Lustre 1.6 Operations Manual

23.2.1 Debug Daemon Option to lctl
The debug_daemon allows users to control the Lustre kernel debug daemon to
dump the debug_kernel buffer to a user-specified file. This functionality uses a
kernel thread on top of debug_kernel. debug_kernel, another sub-command of
lctl, continues to work in parallel with debug_daemon command.
Debug_daemon is highly dependent on file system write speed. File system writes
operation may not be fast enough to flush out all the debug_buffer if Lustre file
system is under heavy system load and continue to CDEBUG to the debug_buffer.
Debug_daemon put ’DEBUG MARKER: Trace buffer full’ into the debug_buffer to
indicate debug_buffer is overlapping itself before debug_daemon flush data to a file.
Users can use lctl control to start or stop Lustre daemon from dumping the
debug_buffer to a file. Users can also temporarily hold daemon from dumping the
file. Use of the debug_daemon sub-command to lctl can provide the same
function.
23.2.1.1 lctl Debug Daemon Commands
This section describes lctl daemon debug commands.
$ lctl debug_daemon start [{file} {megabytes}]
Initiates the debug_daemon to start dumping debug_buffer into a file. The file can
be a system default file,as shown in /proc/sys/lnet/debug_path. The default
patch after Lustre boots is /tmp/lustre-log-$HOSTNAME. Users can specify a new
filename for debug_daemon to output debug_buffer. The new file name shows up
in /proc/sys/lnet/debug_path. Megabytes is the limitation of the file size in
MBs. The daemon wraps around and dumps data to the beginning of the file when
the output file size is over the limit of the user-specified file size. To decode the
dumped file to ASCII and order the log entries by time, run:
lctl debug_file {file} > {newfile}
The output is internally sorted by the lctl command using quicksort.
Chapter 23 Lustre Debugging 23-5