Lustre 1.6 Operations Manual

21.5.5 Further Tuning Tips
Here are some tips we have drawn from testing at a large installation:
■
■
■
Use the full device instead of a partition (sda vs sda1). When using the full device,
Lustre writes nicely-aligned 1 MB chunks to disk. Partitioning the disk can
destroy this alignment and will noticeably impact performance.
Separate the EXT3 OST into two LUNs, a small LUN for the EXT3 journal and a
big one for the "data".
Since Lustre 1.0.4, we supply EXT3 mkfs options when we create the OST like
-j, -J and so on in the following manner (where /dev/sdj has been formatted
before as a journal). The journal size should not be larger than 1 GB (262144 4 KB
blocks) as it can consume up to this amount of RAM on the OSS node per OST.
# mke2fs -O journal_dev -b 4096 /dev/sdj [optional size]
Tip – A very important tip—on the S2A DDN 8500 storage array, you need to create
one OST per TIER, especially in write through (see output below). This is of concern
if you have 16 tiers. Create 16 OSTs consisting of one tier each, instead of eight made
of two tiers each.
- Performance is significantly better on the S2A DDN 9500 and 9550 storage arrays
with two tiers per LUN.
- Do NOT partition the DDN LUNs, as this causes all I/O to the LUNs to be
misaligned by 512 bytes. The DDN RAID stripes and cachelines are aligned on 1 MB
boundaries. Having the partition table on the LUN causes all 1 MB writes to do a
read-modify-write on an extra chunk, and ALL 1 MB reads to, instead, read 2 MB
from disk into the cache, causing a noticeable performance loss.
- You are not obliged to lock in cache the small LUNs.
- Configure the MDT on a separate volume that is configured as RAID 1+0. This
reduces the MDT I/O and doubles the seek speed.
Chapter 21 Lustre Tuning 21-9