Lustre 1.6 Operations Manual

How to fix bad LAST_ID on an OST
The filesystem must be stopped on all servers prior to performing this procedure.
For hex decimal translations:
Use GDB:
(gdb) p /x 15028
$2 = 0x3ab4
Or bc:
echo "obase=16; 15028" | bc
1. Determine a reasonable value for LAST_ID. Check on the MDS:
# mount -t ldiskfs /dev/ /mnt/mds
# od -Ax -td8 /mnt/mds/lov_objid
There is one entry for each OST, in OST index order. This is what the MDS thinks the
last in-use object is.
2. Determine the OST index for this OST.
# od -Ax -td4 /mnt/ost/last_rcvd
It will have it at offset 0x8c.
3. Check on the OST. With debugfs, check LAST_ID:
debugfs -c -R 'dump /O/0/LAST_ID /tmp/LAST_ID' /dev/XXX ; od -Ax -td8
/tmp/LAST_ID"
4. Check objects on the OST:
mount -rt ldiskfs /dev/{ostdev} /mnt/ost
# note the ls below is a number one and not a letter L
ls -1s /mnt/ost/O/0/d* | grep -v [a-z] |
sort -k2 -n > /tmp/objects.{diskname}
tail -30 /tmp/objects.{diskname}
This shows you the OST state. There may be some pre-created orphans, check for
zero-length objects. Any zero-length objects with IDs higher than LAST_ID should
be deleted. New objects will be pre-created.
If the OST LAST_ID value matches that for the objects existing on the OST, then it is
possible the lov_objid file on the MDS is incorrect. Delete the lov_objid file on the
MDS and it will be re-created from the LAST_ID on the OSTs.
Appendix D Lustre Knowledge Base D-25