Lustre 1.6 Operations Manual

Or from the device label at any time:
# e2label /dev/sda
lustre-MDT0000
20.1.2 Lustre Timeouts/ Debugging
Lustre uses two types of timeouts.
■ LND timeouts that ensure point-to-point communications complete in finite time
in the presence of failures. These timeouts are logged with the S_LND flag set.
They may not be printed as console messages, so you should check the Lustre log
for D_NETERROR messages, or enable printing of D_NETERROR messages to the
console (echo + neterror > /proc/sys/lnet/printk).
Congested routers can be a source of spurious LND timeouts. To avoid this,
increase the number of LNET router buffers to reduce back-pressure and/or
increase LND timeouts on all nodes on all connected networks. You should also
consider increasing the total number of LNET router nodes in the system so that
the aggregate router bandwidth matches the aggregate server bandwidth.
■ Lustre timeouts that ensure Lustre RPCs complete in finite time in the presence of
failures. These timeouts should always be printed as console messages. If Lustre
timeouts are not accompanied by LNET timeouts, then you need to increase the
lustre timeout on both servers and clients.
Specific Lustre timeouts are described below.
/proc/sys/lustre/timeout
This is the time period that a client waits for a server to complete an RPC (default
100s). Servers wait half of this time for a normal client RPC to complete and a
quarter of this time for a single bulk request (read or write of up to 1 MB) to
complete. The client pings recoverable targets (MDS and OSTs) at one quarter of the
timeout, and the server waits one and a half times the timeout before evicting a
client for being "stale."
Note – Lustre sends periodic ‘PING’ messages to servers with which it had no
communication for a specified period of time. Any network activity on the
filesystem that triggers network traffic toward servers also works as a health check.
/proc/sys/lustre/ldlm_timeout
This is the time period for which a server will wait for a client to reply to an initial
AST (lock cancellation request) where default is 20s for an OST and 6s for an MDS.
If the client replies to the AST, the server will give it a normal timeout (half of the
client timeout) to flush any dirty data and release the lock.
Chapter 20 LustreProc 20-3