Lustre 1.6 Operations Manual

How do I abort recovery? Why would I want to?
If an MDS or OST is not gracefully shut down, for example a crash or power outage
occurs, the next time the service starts it is in "recovery" mode.
This provides a window for any existing clients to re-connect and re-establish any
state which may have been lost in the interruption. By doing so, the Lustre software
can completely hide failure from user applications.
The recovery window ends when either:
■ All clients which were present before the crash have reconnected; or
■ A recovery timeout expires
This timeout must be long enough to for all clients to detect that the node failed and
reconnect. If the window is too short, some critical state may be lost, and any inprogress
applications receive an error. To avoid this, the recovery window of Lustre
1.x is conservatively long.
If a client which was not present before the failure attempts to connect, it receives an
error, and a message about recovery displays on the console of the client and the
server. New clients may only connect after the recovery window ends.
If the administrator knows that recovery will not succeed, because the entire cluster
was rebooted or because there was an unsupported failure of multiple nodes
simultaneously, then the administrator can abort recovery.
With Lustre 1.4.2 and later, you can abort recovery when starting a service by adding
--abort-recovery to the lconf command line. For earlier Lustre versions, or if the
service has already started, follow these steps:
1. Find the correct device. The server console displays a message similar to:
"RECOVERY: service mds1, 10 recoverable clients, last_transno 1664606"
2. Obtain a list of all Lustre devices. On the MDS or OST, run:
lctl device_list
3. Look for the name of the recovering service, in this case "mds1":
3 UP mds mds1 mds1_UUID 2
4. Instruct Lustre to abort recovery, run:
lctl --device 3 abort_recovery
The device number is on the left.
D-4 Lustre 1.6 Operations Manual • September 2008