Lustre 1.6 Operations Manual

What should I do if I suspect device corruption (Example: disk errors)
Keep these points in mind when trying to recover from device-induced corruption.
■
■
■
Stop using the device as soon as possible (if you have a choice).
The longer corruption is present on a device, the greater the risk that it will cause
further corruption. Normally, ext3 marks the filesystem read-only if any
corruption is detected or if there are I/O errors when reading or writing
metadata to the filesystem. This can only be cleared by shutting down Lustre on
the device (use --force or reboot if necessary).
Proceed carefully
If you take incorrect action, you can make an otherwise-recoverable situation
worse. ext3 has very robust metadata formats and can often recover a large
amount of data even when a significant portion of the device is bad.
Keep a log of all actions and output in a safe place.
If you perform multiple filesystem checks and/or actions to repair the filesystem,
save all logs. They may provide valuable insight into problems encountered.
Normally, the first thing to do is a read-only filesystem check, after the Lustre
service (MDS or OST) has been stopped. If it is not possible to stop the service,
you can run a read-only filesystem check when the device is in use. If running a
filesystem check while the device is in use, e2fsck cannot always coordinate data
gathered at the start of the run with data gathered later in the run and will report
incorrect filesystem errors. The number of errors is dependent upon the length of
check (approximately equal to the device size) and the load on the filesystem. In
this situation, you should run e2fsck multiple times on the device and look for
errors that are persistent across runs, and ignore transient errors.
To run a read-only filesystem check, we recommend that you use the latest
e2fsck, available at http://www.sun.com/software/products/lustre/get.jsp.
On the system with the suspected bad device (in the example below, /dev/sda is
used), run:
[root@mds]# script /root/e2fsck-1.sda
Script started, file is /root/e2fsck-1.sda
[root@mds]# e2fsck -fn /dev/sda
e2fsck 1.35-lfck8 (05-Feb-2005)
Warning: skipping journal recovery because doing a read-only
filesystem check
Pass 1: Checking inodes, blocks, and sizes
[root@mds]# exit
Script done, file is /tmp/foo
D-30 Lustre 1.6 Operations Manual • September 2008