Lustre 1.6 Operations Manual

33.6 Maximum Size of a Filesystem
For i386 systems with 2.6 kernels, the block devices are limited to 16 TB. Each OST or
MDT can have a filesystem up to 8 TB, regardless of whether 32-bit or 64-bit kernels
are on the server. (For 2.6 kernels, the 8 TB limit is imposed by ext3). Currently,
testing is underway to allow filesystems up to 16 TB.
You can have multiple OST filesystems on a single node. Currently, the largest
production Lustre filesystem has 448 OSTs in a single filesystem. There is a compiletime
limit of 8150 OSTs in a single filesystem, giving a theoretical filesystem limit of
nearly 64 PB.
Several production Lustre filesystems have around 200 OSTs in a single filesystem.
The largest filesystem in production is at least 1.3 PB (184 OSTs). All these facts
indicate that Lustre would scale just fine if more hardware is made available.
33.7 Maximum File Size
Individual files have a hard limit of nearly 16 TB on 32-bit systems imposed by the
kernel memory subsystem. On 64-bit systems this limit does not exist. Hence, files
can be 64-bits in size. Lustre imposes an additional size limit of up to the number of
stripes, where each stripe is 2 TB. A single file can have a maximum of 160 stripes,
which gives an upper single file limit of 320 TB for 64-bit systems. The actual
amount of data that can be stored in a file depends upon the amount of free space in
each OST on which the file is striped.
33.8 Maximum Number of Files or
Subdirectories in a Single Directory
Lustre uses the ext3 hashed directory code, which has a limit of about 25 million
files. On reaching this limit, the directory grows to more than 2 GB depending on the
length of the filenames. The limit on subdirectories is the same as the limit on
regular files in all later versions of Lustre due to a small ext3 format change.
In fact, Lustre is tested with ten million files in a single directory. On a properlyconfigured
dual-CPU MDS with 4 GB RAM, random lookups in such a directory are
possible at a rate of 5,000 files / second.
Chapter 33 System Limits 33-3