Lustre 1.6 Operations Manual

3.4.2 OSS Memory Requirements
When planning the hardware for an OSS node, consider the memory usage of
several components in the Lustre system. Although Lustre versions 1.4 and 1.6 do
not cache file data in memory on the OSS node, there are a number of large memory
consumers that need to be taken into account. Also consider that future Lustre
versions will cache file data on the OSS node, so these calculations should only be
taken as a minimum requirement.
By default, each Lustre ldiskfs filesystem has 400 MB for the journal size. This can
pin up to an equal amount of RAM on the OSS node per filesystem. In addition, the
service threads on the OSS node pre-allocate a 1 MB I/O buffer for each ost_io
service thread, so these buffers do not need to be allocated and freed for each I/O
request. Also, a reasonable amount of RAM needs to be available for filesystem
metadata. While no hard limit can be placed on the amount of filesystem metadata,
if more RAM is available, then the disk I/O is needed less often to retrieve the
metadata. Finally, if you are using TCP or other network transport that uses system
memory for send/receive buffers, this must also be taken into consideration.
Also, if the OSS nodes are to be used for failover from another node, then the RAM
for each journal should be doubled, so the backup server can handle the additional
load if the primary server fails.
OSS Memory Usage for a 2 OST server (major consumers):
■ 400MB journal size * 2 OST devices = 800MB
■ 1.5MB per OST IO thread * 256 threads = 384MB
■ e1000 RX descriptors, RxDescriptors=4096 for 9000 byte MTU = 128MB
This consumes over 1,300 MB just for the pre-allocated buffers, and does not include
memory for the OS or filesystem metadata. For a non-failover configuration, 2 GB of
RAM would be the minimum. For a failover configuration, 3 GB of RAM would be
the minimum.
3-10 Lustre 1.6 Operations Manual • September 2008