Lustre 1.6 Operations Manual

22.3.13 Replacing An Existing OST or MDS
The OST filesystem is simply a normal ext3 filesystem. To copy the contents of an
existing OST to a new OST (or an old MDS to a new MDS), use one of these
methods:
■ Connect the old OST disk and new OST disk to a single machine, mount both,
and use rsync to copy all data between the OST filesystems.
For example:
■
■
mount -t ext3 /dev/old /mnt/ost_old
mount -t ext3 /dev/new /mnt/ost_new
rsync -aSv /mnt/ost_old/ /mnt/ost_new
# note trailing slash on ost_old/
If you are unable to connect both sets of disk to the same computer, use rsync to
copy over the network using rsh (or ssh with -e ssh):
rsync -aSvz /mnt/ost_old/ new_ost_node:/mnt/ost_new
Use the same procedure for the MDS, with one additional step:
cd /mnt/mds_old; getfattr -R -e base64 -d . > /tmp/mdsea; \
; cd /mnt/mds_new; setfattr \
--restore=/tmp/mdsea
22.3.14 Handling/Debugging Error "- 28"
Linux error -28 is -ENOSPC and indicates that the filesystem has run out of space.
You need to create larger filesystems for the OSTs. Normally, Lustre reports this to
your application. If the application is checking the return code from its function
calls, then it decodes it into a textual error message like "No space left on device." It
also appears in the system log messages.
During a "write" or "sync" operation, the file in question resides on an OST which is
already full. New files that are created do not use full OSTs, but existing files
continue to use the same OST. You need to expand the specific OST or copy/stripe
the file over to an OST with more space available. You encounter this situation
occasionally when creating files, which may indicate that your MDS has run out of
inodes and needs to be enlarged. To check this, use df -i
Chapter 22 Lustre Troubleshooting Tips 22-13