Lustre 1.6 Operations Manual

10.1.1 Reliability
A quick calculation (shown below), makes it clear that without further redundancy,
RAID5 is not acceptable for large clusters and RAID6 is a must.
Calculation
Take a 1 PB filesystem (2000 disks of 500GB capacity). The MTF 1 of a disk is about
1000 days and repair time at 10% of disk bandwidth is close to 1 day (500 GB at 5
MB/sec = 100,000 sec = 1 day). This means that the expected failure rate is
2000/1000 = 2 disks per day.
If we have a RAID5 stripe that is ~10 wide, then during 1 day of rebuilding, the
chance that a second disk in the same array fails is about 9 / 1000 ~= 1/100. This
means that in the expected period of 50 days, a double failure in a RAID5 stripe
leads to data loss.
So RAID6 or another double parity algorithm is necessary for OST storage. For the
MDS, we recommend RAID0+1 storage.
10.1.2 Selecting Storage for the MDS and OSS
The MDS does a large amount of small writes. For this reason, we recommend that
you use RAID1 storage. Building RAID1 Linux MD devices and striping over these
devices with LVM makes it easy to create an MDS filesystem of 1-2 TB (for example,
with 4 or 8 500 GB disks).
It is considered mandatory that you use disk monitoring software, so rebuilds
happen without any delay. We recommend backups of the metadata filesystems.
This can be done with LVM snapshots or using raw partition backups.
We also recommend using a kernel version of 2.6.15 or later with bitmap RAID
rebuild features. This reduces RAID recovery time from a rebuild to quick
resynchronization.
1. Mean Time to Failure
10-2 Lustre 1.6 Operations Manual • September 2008