Lustre 1.6 Operations Manual

Indeed, there will be cases where a user job will die prior to the HA software
triggering a failover. You can certainly shorten timeouts, add monitoring, and take
other steps to decrease this probability. But there is a serious trade-off – shortening
timeouts increases the probability of false-triggering a busy system. Increasing
monitoring takes the system resources, and can likewise cause a false trigger.
Unfortunately, hard failover solutions capable of catching failures in the sub-second
range generally require special hardware. As a result, they are quite expensive.
Tip – Failover of the Lustre client is dependent on the obd_timeout parameter. The
Lustre client does not attempt failover until the request times out. Then, the client
tries resending the request to the original server (if again, an obd_timeout occurs).
After that, the Lustre client refers to the import list for that target and tries to
connect (in a round-robin manner) until one of the nodes replies. The timeouts for
the connection are much lower (obd_timeout / 20, 5).
8-22 Lustre 1.6 Operations Manual • September 2008