- Page 1 and 2:
Lustre 1.6 Operations Manual Sun M
- Page 3:
Please Recycle
- Page 6 and 7:
2. Understanding Lustre Networking
- Page 8 and 9:
6. Configuring Lustre - Examples 6-
- Page 10 and 11:
9. Configuring Quotas 9-1 9.1 Worki
- Page 12 and 13:
15. Backup and Restore 15-1 15.1 Lu
- Page 14 and 15:
20. LustreProc 20-1 20.1 /proc Entr
- Page 16 and 17:
22. Lustre Troubleshooting Tips 22-
- Page 18 and 19:
Part IV Lustre for Users 24. Free S
- Page 20 and 21:
29. Lustre Programming Interfaces (
- Page 22 and 23:
33. System Limits 33-1 33.1 Maximum
- Page 24 and 25:
Shell Prompts Shell C shell C shell
- Page 26 and 27:
xxiv Lustre 1.6 Operations Manual
- Page 29:
PART I Lustre Architecture Lustre i
- Page 32 and 33:
1.1 Lustre File System Lustre is a
- Page 34 and 35:
1.2.4 OSTs An OSTs provide back-end
- Page 36 and 37:
1.3 Files in the Lustre File System
- Page 38 and 39:
1.3.1 Lustre File System and Stripi
- Page 40 and 41:
1.3.3 Lustre System Capacity Lustre
- Page 42 and 43:
1.5 Lustre Networking In clusters w
- Page 44 and 45:
Note - Lustre does not provide redu
- Page 46 and 47:
Key features of LNET include: ■
- Page 49 and 50:
CHAPTER 3 Prerequisites This chapte
- Page 51 and 52:
3.2 Using a Pre-Packaged Lustre Rel
- Page 53 and 54:
Another option is to: 1. Install db
- Page 55 and 56:
3.3.4 Choosing a Proper Kernel I/O
- Page 57 and 58:
3.4 Memory Requirements This sectio
- Page 59 and 60:
CHAPTER 4 Lustre Installation This
- Page 61 and 62:
4.1.1 MountConf MountConf is shorth
- Page 63 and 64:
Note - For detailed information on
- Page 65 and 66:
Mounting Lustre on a client node $
- Page 67 and 68:
We are mounting by disk label here
- Page 69 and 70:
4.2.2.2 Mount with Inactive OSTs Mo
- Page 71 and 72:
Client mount for foo: mount -t lust
- Page 73 and 74:
$ mount -t lustre -L testfs-MDT0000
- Page 75 and 76:
$ cd /tmp/kernels/linux-2.6.9 $ rm
- Page 77 and 78:
Use standard RPM commands to instal
- Page 79 and 80:
4.3.2.2 Liblustre The Lustre librar
- Page 81 and 82:
4.4 Building a Lustre Source Tarbal
- Page 83 and 84:
CHAPTER 5 Configuring the Lustre Ne
- Page 85 and 86:
5.1.5 Determine Appropriate Mount P
- Page 87 and 88:
Note - Depending on the Linux distr
- Page 89 and 90:
live_router_check_interval, dead_ro
- Page 91 and 92:
5.2.3 Downed Routers There are two
- Page 93 and 94:
5.3.2 Stopping LNET Before the LNET
- Page 95 and 96:
CHAPTER 6 Configuring Lustre - Exam
- Page 97 and 98:
6.1.2 Lustre with Separate MGS and
- Page 99 and 100:
How to Create a CSV File Five diffe
- Page 101 and 102:
Linux LVM LV (Logical Volume) The C
- Page 103 and 104:
The lustre_config.csv file looks li
- Page 105 and 106:
Example 1: Simple Lustre configurat
- Page 107 and 108:
CHAPTER 7 More Complicated Configur
- Page 109 and 110:
Because megan and oscar match the f
- Page 111 and 112:
7.2 Elan to TCP Routing Servers meg
- Page 113 and 114:
7.3.3 Start clients For the TCP cli
- Page 115 and 116:
CHAPTER 8 Failover This chapter des
- Page 117 and 118:
For proper resource fencing, the He
- Page 119 and 120:
8.1.5 Roles of Nodes in a Failover
- Page 121 and 122:
8.4.3 Hardware Requirements for Fai
- Page 123 and 124:
8.5.1.1 Configuring Heartbeat This
- Page 125 and 126:
. Monitor the syslog on both nodes.
- Page 127 and 128:
To get the proper syntax, run: $ st
- Page 129 and 130:
8.6 Using MMP The multiple mount pr
- Page 131 and 132:
8.7.2 Configuring the Hardware Hear
- Page 133 and 134:
Use this procedure: 1. Create the b
- Page 135 and 136:
8.7.3.2 Testing 1. Pull power from
- Page 137 and 138:
CHAPTER 9 Configuring Quotas This c
- Page 139 and 140:
To enable quotas automatically when
- Page 141 and 142:
The lfs command now includes these
- Page 143 and 144:
This sets a much smaller granularit
- Page 145 and 146:
The quota_bunit_sz parameter displa
- Page 147 and 148:
9.1.5.2 Quota Limits Available quot
- Page 149 and 150:
Quota Event nowait_for_pending_blk_
- Page 151 and 152:
CHAPTER 10 RAID This chapter descri
- Page 153 and 154:
10.1.3 Understanding Double Failure
- Page 155 and 156:
10.2 Insights into Disk Performance
- Page 157 and 158:
OST Filesystem: RAID5 with 5 or 9 d
- Page 159 and 160:
10.2.1 Sample Graphs The graphs in
- Page 161 and 162:
10.2.1.2 Graphs for Read Performanc
- Page 163 and 164:
10.3 Creating an External Journal W
- Page 165 and 166:
CHAPTER 11 Kerberos This chapter de
- Page 167 and 168:
Note - The Heimdal implementation o
- Page 169 and 170:
General Installation Notes ■ ■
- Page 171 and 172:
. Install the keytab. Note - There
- Page 173 and 174:
# uppercase the hex nid=$(echo $nid
- Page 175 and 176:
11.2.2 Types of Lustre-Kerberos Fla
- Page 177 and 178:
11.2.2.3 Customized Flavor In most
- Page 179 and 180:
11.2.2.6 Rules, Syntax and Examples
- Page 181 and 182:
CHAPTER 12 Bonding This chapter des
- Page 183 and 184:
# ethtool eth1 Settings for eth1: S
- Page 185 and 186:
12.4 Bonding Module Parameters Bond
- Page 187 and 188:
The examples below are from RedHat
- Page 189 and 190:
12.5.1 Examples This is an example
- Page 191 and 192:
12.6 Configuring Lustre with Bondin
- Page 193 and 194:
CHAPTER 13 Upgrading Lustre The cha
- Page 195 and 196:
13.2.2 Supported Upgrade Paths The
- Page 197 and 198:
[root@mds1]# tunefs.lustre --mgs --
- Page 199 and 200:
13.2.5 Upgrading Multiple File Syst
- Page 201 and 202:
13.3 Upgrading Lustre from 1.6.3 to
- Page 203 and 204:
13.4 Downgrading Lustre from 1.6.4
- Page 205 and 206:
CHAPTER 14 Lustre SNMP Module The L
- Page 207 and 208:
14.3 Using the Lustre SNMP Module O
- Page 209 and 210:
CHAPTER 15 Backup and Restore This
- Page 211 and 212:
15.1.3.1 Backing Up an MDS File To
- Page 213 and 214:
3. Mount the filesystem. ■ For 2.
- Page 215 and 216:
2. Format LVM volumes as Lustre tar
- Page 217 and 218:
15.3.4 Restoring From Old Snapshot
- Page 219 and 220:
CHAPTER 16 POSIX This chapter descr
- Page 221 and 222:
7. When the system displays this pr
- Page 223 and 224:
16.3 Isolating and Debugging Failur
- Page 225 and 226:
If this single test is causing prob
- Page 227 and 228:
CHAPTER 17 Benchmarking The benchma
- Page 229 and 230:
Version 1.03 --Sequential Output--
- Page 231 and 232:
17.3 IOzone Benchmark IOZone is a f
- Page 233 and 234:
CHAPTER 18 Lustre Recovery This cha
- Page 235 and 236:
18.2.2 MDS Failure (and Failover) R
- Page 237:
PART III Lustre Tuning, Monitoring
- Page 240 and 241:
19.1.1 Downloading an I/O Kit You c
- Page 242 and 243:
The sgpdd_survey script must be cus
- Page 244 and 245:
3. Determine the obdfilter instance
- Page 246 and 247:
19.2.2.3 Output of the sbdfilter_su
- Page 248 and 249:
19.3 PIOS Test Tool The PIOS test t
- Page 250 and 251:
19.3.2 PIOS I/O Modes There are sev
- Page 252 and 253:
Offset(o): Distance between two suc
- Page 254 and 255:
19.3.4 PIOS Examples To create a 1
- Page 256 and 257:
19.4.1.2 Utilities LNET self-test h
- Page 258 and 259:
Note - This script can be easily ad
- Page 260 and 261:
19.4.3.2 Group This section lists l
- Page 262 and 263:
del_group NAME Removes a group from
- Page 264 and 265:
There are only two test types: --pi
- Page 266 and 267:
19.4.3.4 Other Commands This sectio
- Page 268 and 269:
show_error [--session] [GROUP]|[IDs
- Page 270 and 271:
20.1 /proc Entries for Lustre This
- Page 272 and 273:
proc/sys/lustre/fail_loc This is th
- Page 274 and 275:
20.1.3.1 Configuring Adaptive Timeo
- Page 276 and 277:
Note - Changing adaptive timeouts s
- Page 278 and 279:
Credits work like a semaphore. At s
- Page 280 and 281:
20.2 Lustre I/O Tunables The sectio
- Page 282 and 283:
20.2.2 Watching the Client RPC Stre
- Page 284 and 285:
20.2.4 Client Read-Write Extents Su
- Page 286 and 287:
20.2.5 Watching the OST Block I/O S
- Page 288 and 289:
20.2.7 mballoc History /proc/fs/ldi
- Page 290 and 291:
Number of groups scanned (grps colu
- Page 292 and 293:
20.2.9 Locking /proc/fs/lustre/ldlm
- Page 294 and 295:
20.3.1 RPC Information for Other OB
- Page 296 and 297: Where: Parameter Cur. Count Cur. Ra
- Page 298 and 299: 20-30 Lustre 1.6 Operations Manual
- Page 300 and 301: 21.1.0.1 OSS Service Thread Count T
- Page 302 and 303: 21.3 Options to Format MDT and OST
- Page 304 and 305: 21.3.3.3 Number of Inodes for OST F
- Page 306 and 307: 21.5.3 Setting Write-Back Cache Per
- Page 308 and 309: For example, one OST per tier LUNLa
- Page 310 and 311: 21.7 Lockless I/O Tunables The lock
- Page 312 and 313: 22.1.2 Lustre Logs The error messag
- Page 314 and 315: 22.3 Lustre Performance Tips This s
- Page 316 and 317: 22.3.4 OSTs Become Read-Only If the
- Page 318 and 319: 22.3.6 Changing Parameters You can
- Page 320 and 321: 22.3.8 Default Striping These are t
- Page 322 and 323: 22.3.12 Handling/Debugging "Bind: A
- Page 324 and 325: You may also receive this error if
- Page 326 and 327: 22.3.17 Handling/Debugging "LustreE
- Page 328 and 329: 22.3.22 Number of OSTs Needed for S
- Page 330 and 331: 23.1 Lustre Debug Messages Each Lus
- Page 332 and 333: 23.2 Tools for Lustre Debugging The
- Page 334 and 335: debug_daemon stop Completely shuts
- Page 336 and 337: 4. If you already have a debug log
- Page 338 and 339: 23.2.8 Adding Debugging to the Lust
- Page 340 and 341: 23.3 Troubleshooting with strace Th
- Page 342 and 343: 23.4.1 Determine the Lustre UUID of
- Page 344 and 345: 23-16 Lustre 1.6 Operations Manual
- Page 348 and 349: 24.1 Querying Filesystem Space The
- Page 350 and 351: 24.2 Using Quotas The lfs quota com
- Page 352 and 353: 25.1.1 Advantages of Striping There
- Page 354 and 355: Generally, a good stripe size for s
- Page 356 and 357: 25.3 lfs setstripe - Setting File L
- Page 358 and 359: 25.3.3 Creating a File on a Specifi
- Page 360 and 361: 25.5 Performing Direct I/O Starting
- Page 362 and 363: 25.7 Striping Using llapi Use llapi
- Page 364 and 365: int get_my_uuids(int fd) { struct o
- Page 366 and 367: c = get_my_uuids(file); rintf("Writ
- Page 368 and 369: Note - ACL support is a system-rang
- Page 370 and 371: 26.2 Root Squash Functionality Lust
- Page 372 and 373: 26-6 Lustre 1.6 Operations Manual
- Page 374 and 375: 27.1 Expanding the Filesystem by Ad
- Page 376 and 377: NEWCHK=$($CKSUM $NEWNAME | awk '{pr
- Page 378 and 379: 27-6 Lustre 1.6 Operations Manual
- Page 381 and 382: CHAPTER 28 User Utilities (man1) Th
- Page 383 and 384: lfs setquota [-u|--user|-g|--group]
- Page 385 and 386: Option Description --print0 Prints
- Page 387 and 388: Option Description quotachown [-i]
- Page 389 and 390: $ lfs find /mnt/lustre Efficiently
- Page 391 and 392: 28.2 lfsck The e2fsprogs package co
- Page 393 and 394: Description If an MDS or an OST bec
- Page 395 and 396: 3. Make this file accessible on all
- Page 397 and 398:
Example: lfsck -n -v --mdsdb /home/
- Page 399 and 400:
28.3 Filefrag The e2fsprogs package
- Page 401 and 402:
28.4 Mount Lustre uses the standard
- Page 403 and 404:
CHAPTER 29 Lustre Programming Inter
- Page 405 and 406:
29.1.3 Parameters ■ ■ Name of t
- Page 407 and 408:
CHAPTER 30 Setting Lustre Propertie
- Page 409 and 410:
Examples System default size is 4MB
- Page 411 and 412:
30.1.3 llapi_file_open The llapi_fi
- Page 413 and 414:
Description The llapi_quotactl() co
- Page 415 and 416:
CHAPTER 31 Configuration Files and
- Page 417 and 418:
31.2.1 LNET Options This section de
- Page 419 and 420:
Here is a more complicated situatio
- Page 421 and 422:
31.2.1.4 forwarding ("") This is a
- Page 423 and 424:
Variable keepalive_idle (30,Wc) kee
- Page 425 and 426:
31.2.4 RapidArray LND The RapidArra
- Page 427 and 428:
Variable rnr_cnt (6,Wc) rnr_nak_tim
- Page 429 and 430:
31.2.7 Portals LND (Linux) The Port
- Page 431 and 432:
Variable credits (128) peercredits
- Page 433 and 434:
The following environment variables
- Page 435 and 436:
To obtain the optimal performance f
- Page 437 and 438:
CHAPTER 32 System Configuration Uti
- Page 439 and 440:
Option Description --failnode=nid,.
- Page 441 and 442:
32.2 tunefs.lustre The tunefs.lustr
- Page 443 and 444:
Examples Changing the MGS’s NID a
- Page 445 and 446:
Network Configuration Option Descri
- Page 447 and 448:
Virtual Block Device Operations Lus
- Page 449 and 450:
Examples lctl $ lctl lctl > dl 0 UP
- Page 451 and 452:
Options Option Description :=[:] Th
- Page 453 and 454:
32.5 New Utilities in Lustre 1.6 Th
- Page 455 and 456:
32.5.4 Application Profiling Utilit
- Page 457 and 458:
32.5.7 Flock Feature Lustre now inc
- Page 459 and 460:
32.5.9 l_getgroups The l_getgroups
- Page 461 and 462:
32.5.11 llstat The llstat utility d
- Page 463 and 464:
32.5.12 lst The lst utility starts
- Page 465 and 466:
32.5.13 plot-llstat The plot-llstat
- Page 467 and 468:
CHAPTER 33 System Limits This chapt
- Page 469 and 470:
33.6 Maximum Size of a Filesystem F
- Page 471 and 472:
33.12 OSS RAM Size for a Single OST
- Page 473 and 474:
APPENDIX A Feature List Networks TC
- Page 475 and 476:
QSW LND tx_maxconfig ntxmsgs nnblk_
- Page 477 and 478:
Portals LND (Linux) ntx concurrent_
- Page 479 and 480:
APPENDIX B Task List Key Concepts s
- Page 481 and 482:
APPENDIX C Version Log Manual Versi
- Page 483 and 484:
Manual Version Date Details of Edit
- Page 485 and 486:
Manual Version Date Details of Edit
- Page 487 and 488:
Manual Version Date Details of Edit
- Page 489 and 490:
APPENDIX D Lustre Knowledge Base Th
- Page 491 and 492:
How can I check if a filesystem is
- Page 493 and 494:
What does "denying connection for n
- Page 495 and 496:
Now for the LOVs: lmc -m test.xml -
- Page 497 and 498:
How do I set striping for a large n
- Page 499 and 500:
How do I configure recoverable / fa
- Page 501 and 502:
How do I backup / restore a Lustre
- Page 503 and 504:
9. Format the new device. The easie
- Page 505 and 506:
What extra resources are required f
- Page 507 and 508:
To see which node is serving that O
- Page 509 and 510:
Currently (v1.4.5) the Voltaire IB
- Page 511 and 512:
1. Shut down all clients and server
- Page 513 and 514:
How to fix bad LAST_ID on an OST Th
- Page 515 and 516:
Why can't I run an OST and a client
- Page 517 and 518:
Information on the Lustre Networkin
- Page 519 and 520:
In many cases, the extent of corrup
- Page 521 and 522:
How do I determine which Lustre ser
- Page 523 and 524:
■ On the journal: ext3 journaling
- Page 525 and 526:
When mounting an MDT filesystm, the
- Page 527 and 528:
Glossary A ACL Administrative OST f
- Page 529 and 530:
F Failback Failout OST Failover FID
- Page 531 and 532:
LDLM lfind lfs lfsck liblustre Llit
- Page 533 and 534:
MGS Management Service. A software
- Page 535 and 536:
Portal A concept used by LNET. LNET
- Page 537 and 538:
W Wide striping Strategy of using m
- Page 539 and 540:
Index Numerics 1.6 utilities, 32-17
- Page 541 and 542:
file formats, quotas, 9-11 file str
- Page 543 and 544:
modprobe.conf, 7-1, 7-5, 7-6 module
- Page 545 and 546:
vib (Voltaire InfiniBand), 2-2 T ta