Lustre 1.6 Operations Manual

Description
If an MDS or an OST becomes corrupt, you can run a distributed check on the
filesystem to determine what sort of problems exist.
1. Run 'e2fsck -f' on the individual MDS / OST that had problems to fix any local
filesystem damage.
It is a very good idea to run this e2fsck under "script" so you have a log of
whatever changes it made to the filesystem (in case this is needed later). After
this is complete, you can bring the filesystem up if necessary to reduce the outage
window.
2. Run a full e2fsck of the MDS to create a database for lfsck.
The -n option is critical for a mounted filesystem, otherwise you might corrupt
your filesystem. The mdsdb file can grow fairly large, depending on the number
of files in the filesystem (10 GB or more for millions of files, though the actual file
size is larger because the file is sparse). It is fastest if this is written to a local
filesystem because of the seeking and small writes. Depending on the number of
files, this step can take several hours to complete. In the following example,
/tmp/mdsdb is the database file.
e2fsck -n -v --mdsdb /tmp/mdsdb /dev/{mdsdev}
Chapter 28 User Utilities (man1) 28-13