Lustre 1.6 Operations Manual

20.1.3 Adaptive Timeouts in Lustre
Lustre 1.6.5 introduces an adaptive mechanism to set RPC timeouts. This feature
causes servers to track actual RPC completion times, and to report estimated
completion times for future RPCs back to clients. The clients use these estimates to
set their future RPC timeout values. If server request processing slows down for any
reason, the RPC completion estimates increase, and the clients allow more time for
RPC completion.
If RPCs queued on the server approach their timeouts, then the server sends an early
reply to the client, telling the client to allow more time. In this manner, clients avoid
RPC timeouts and disconnect/reconnect cycles. Conversely, as a server speeds up,
RPC timeout values decrease, allowing faster detection of non-responsive servers
and faster attempts to reconnect to a server's failover partner.
Caution – In Lustre 1.6.5, adaptive timeouts are disabled by default in order not to
require users applying this maintenance release to use adaptive timeouts. Adaptive
timeouts will be enabled, by default, in Lustre 1.8.
In previous Lustre versions, the static obd_timeout (/proc/sys/lustre/timeout)
value was used as the maximum completion time for all RPCs; this value also
affected the client-server ping interval and initial recovery timer. Now, with adaptive
timeouts, obd_timeout is only used for the ping interval and initial recovery
estimate. When a client reconnects during recovery, the server uses the client's
timeout value to reset the recovery wait period; i.e., the server learns how long the
client had been willing to wait, and takes this into account when adjusting the
recovery period.
Chapter 20 LustreProc 20-5