Lustre 1.6 Operations Manual

More documents

Recommendations

Info

PART I Lustre Architecture Lustre is a storage-architecture for clusters. The central component is the Lustre file system, a shared file system for clusters. The Lustre file system is currently available for Linux and provides a POSIX-compliant UNIX file system interface. The Lustre architecture is used for many different kinds of clusters. It is best known for powering seven of the ten largest high-performance computing (HPC) clusters in the world with tens of thousands of client systems, petabytes (PBs) of storage and hundreds of gigabytes per second (GB/sec) of I/O throughput. Many HPC sites use Lustre as a site-wide global file system, servicing dozens of clusters on an unprecedented scale.
Page 1 and 2: Lustre 1.6 Operations Manual Sun M
Page 3: Please Recycle
Page 6 and 7: 2. Understanding Lustre Networking
Page 8 and 9: 6. Configuring Lustre - Examples 6-
Page 10 and 11: 9. Configuring Quotas 9-1 9.1 Worki
Page 12 and 13: 15. Backup and Restore 15-1 15.1 Lu
Page 14 and 15: 20. LustreProc 20-1 20.1 /proc Entr
Page 16 and 17: 22. Lustre Troubleshooting Tips 22-
Page 18 and 19: Part IV Lustre for Users 24. Free S
Page 20 and 21: 29. Lustre Programming Interfaces (
Page 22 and 23: 33. System Limits 33-1 33.1 Maximum
Page 24 and 25: Shell Prompts Shell C shell C shell
Page 26 and 27: xxiv Lustre 1.6 Operations Manual
Page 31 and 32: CHAPTER 1 Introduction to Lustre Th
Page 33 and 34: 1.2 Lustre Components A Lustre file
Page 35 and 36: FIGURE 1-1 shows the expected inter
Page 37 and 38: FIGURE 1-3 shows how a file open op
Page 39 and 40: 1.3.2 Lustre Storage The storage at
Page 41 and 42: 1.4 Lustre Configurations Lustre fi
Page 43 and 44: 1.6 Lustre Failover and Rolling Upg
Page 45 and 46: CHAPTER 2 Understanding Lustre Netw
Page 47: PART II Lustre Administration Lustr
Page 50 and 51: The Lustre software is released und
Page 52 and 53: 3.2.2 Lustre Tools The lustre-.rpm
Page 54 and 55: 3.3 Environmental Requirements When
Page 56 and 57: 3.3.5 Changing the I/O Scheduler Th
Page 58 and 59: 3.4.2 OSS Memory Requirements When
Page 60 and 61: 4.1 Installing Lustre Use this proc
Page 62 and 63: 4.2 Quick Configuration of Lustre A
Page 64 and 65: Starting Lustre on any OST Node Giv
Page 66 and 67: 4.2.1.3 Filesystem Name The filesys
Page 68 and 69: 4.2.2 More Complex Configurations I
Page 70 and 71: 4.2.2.5 Failout Designate an OST as
Page 72 and 73: 4.2.3.2 Running the Writeconf Comma
Page 74 and 75: For example, the lustre/kernel_patc
Page 76 and 77: 4. You can now use Quilt to apply a
Page 78 and 79:
Example: lustre-1.5.97-\ 2.6.9_42.x
Page 80 and 81:
4.3.3 Building from Source Currentl
Page 82 and 83:
This example is for the RHEL 2.6 ke
Page 84 and 85:
5.1.3 Identify Network Interfaces t
Page 86 and 87:
5.2 Configuring Your Lustre Network
Page 88 and 89:
5.2.1.1 SilverStorm InfiniBand Opti
Page 90 and 91:
5.2.2.1 LNET Routers All LNET route
Page 92 and 93:
5.3 Starting and Stopping LNET Lust
Page 94 and 95:
5-12 Lustre 1.7 Operations Manual
Page 96 and 97:
6.1.1.2 Configuration Generation an
Page 98 and 99:
8. Make a mount point on all the OS
Page 100 and 101:
Linux LVM PV (Physical Volume) The
Page 102 and 103:
Lustre target The CSV line format i
Page 104 and 105:
Using CSV with lustre_config Once y
Page 106 and 107:
# configuring LVM2-PV on oss162.clu
Page 108 and 109:
Networks On the servers: options ln
Page 110 and 111:
7.1.3 Start Clients TCP clients can
Page 112 and 113:
7.3 Load Balancing with Infiniband
Page 114 and 115:
You could create these configuratio
Page 116 and 117:
In Lustre, failover means that a cl
Page 118 and 119:
8.1.3 Heartbeat The Heartbeat packa
Page 120 and 121:
8.3 MDS Failover The MDS has only o
Page 122 and 123:
8.5 Setting Up Failover with Heartb
Page 124 and 125:
# Use manual failback auto_failback
Page 126 and 127:
In this example, ost1 is the shared
Page 128 and 129:
A simple method to halt the system
Page 130 and 131:
8.7 Setting Up Failover with Heartb
Page 132 and 133:
8.7.2.2 Configuring Lustre Configur
Page 134 and 135:
Basic Configuration - Adding STONIT
Page 136 and 137:
Indeed, there will be cases where a
Page 138 and 139:
Caution - Although quotas are avail
Page 140 and 141:
9.1.2 Creating Quota Files and Quot
Page 142 and 143:
9.1.3 Resetting the Quota To reset
Page 144 and 145:
Additional information: Grace perio
Page 146 and 147:
9.1.5 Known Issues with Quotas Usin
Page 148 and 149:
9.1.6 Lustre Quota Statistics Lustr
Page 150 and 151:
Involving Lustre Support in Quotas
Page 152 and 153:
10.1.1 Reliability A quick calculat
Page 154 and 155:
10.1.4 Performance Considerations C
Page 156 and 157:
■ Use the proper block device wit
Page 158 and 159:
■ Consider using writeback cache
Page 160 and 161:
FIGURE 10-2 Write - RAID0, 64K chun
Page 162 and 163:
FIGURE 10-4 Read - RAID5, 64K chunk
Page 164 and 165:
2. On the OSS, perform these steps:
Page 166 and 167:
11.2 Lustre Setup with Kerberos Set
Page 168 and 169:
11.2.1.3 Configuring Lustre for Ker
Page 170 and 171:
11.2.1.4 Configuring Kerberos To co
Page 172 and 173:
11.2.1.5 Setting the Environment Pe
Page 174 and 175:
11.2.1.7 Running GSS Daemons If you
Page 176 and 177:
Basic Flavor Authentication RPC Mes
Page 178 and 179:
11.2.2.4 Specifying Security Flavor
Page 180 and 181:
11.2.2.7 Authenticating Normal User
Page 182 and 183:
12.2 Requirements The most basic re
Page 184 and 185:
12.3 Using Lustre with Multiple NIC
Page 186 and 187:
3. Attach one or more slave interfa
Page 188 and 189:
7. Use ethtool or ifconfig to check
Page 190 and 191:
In the following example, the bond0
Page 192 and 193:
Page 194 and 195:
13.2 Upgrading Lustre from 1.4.12 t
Page 196 and 197:
Note - If your system is upgraded f
Page 198 and 199:
4. Upgrade and start the OSTs for t
Page 200 and 201:
7. Upgrade the old MDT. install new
Page 202 and 203:
6. Install new Lustre version and r
Page 204 and 205:
Page 206 and 207:
14.1 Installing the Lustre SNMP Mod
Page 208 and 209:
Page 210 and 211:
15.1.2 Performing Device-level Back
Page 212 and 213:
15.1.3.2 Backing Up an OST File Fol
Page 214 and 215:
15.3 LVM Snapshots on Lustre Target
Page 216 and 217:
cfs21:~# mount -t lustre /dev/volgr
Page 218 and 219:
Permanent disk data: Target: back-O
Page 220 and 221:
16.1 Installing POSIX To install PO
Page 222 and 223:
16.2 Running POSIX Tests Against Lu
Page 224 and 225:
For example, if the fifth single te
Page 226 and 227:
Page 228 and 229:
17.1 Bonnie++ Benchmark Bonnie++ is
Page 230 and 231:
2. Install the IOR software (per th
Page 232 and 233:
3. Run the IOZone software (per the
Page 234 and 235:
18.2 Types of Failure Different typ
Page 236 and 237:
18.2.4 Network Partition The partit
Page 239 and 240:
CHAPTER 19 Lustre I/O Kit This chap
Page 241 and 242:
19.2.1 sgpdd_survey Use the sgpdd_s
Page 243 and 244:
Network Here, the script drives one
Page 245 and 246:
NETS: set this if you are using a n
Page 247 and 248:
19.2.3 ost_survey The ost_survey to
Page 249 and 250:
19.3.1 Synopsis pios [--chunksize|-
Page 251 and 252:
19.3.3 PIOS Parameters PIOS has fiv
Page 253 and 254:
Parameter --regioncount_low=RL --re
Page 255 and 256:
19.4 LNET Self-Test LNET self-test
Page 257 and 258:
19.4.1.6 Test A test is a configura
Page 259 and 260:
new_session [--timeout SECONDS] [--
Page 261 and 262:
list_group [NAME] [--active] [--bus
Page 263 and 264:
19.4.3.3 Batch and Test This sectio
Page 265 and 266:
un NAME Runs the batch. $ lst run b
Page 267 and 268:
stat [--bw] [--rate] [--read] [--wr
Page 269 and 270:
CHAPTER 20 LustreProc This chapter
Page 271 and 272:
Or from the device label at any tim
Page 273 and 274:
20.1.3 Adaptive Timeouts in Lustre
Page 275 and 276:
Parameter at_early_margin at_extra
Page 277 and 278:
Server statistic files also show th
Page 279 and 280:
20.1.5 Free Space Distribution The
Page 281 and 282:
proc/fs/lustre/osc//max_pages_per_r
Page 283 and 284:
20.2.3 Client Read-Write Offset Sur
Page 285 and 286:
Per-Process Client I/O Statistics T
Page 287 and 288:
20.2.6 Mechanics of Lustre Readahea
Page 289 and 290:
pid inode goal result foundgrps cr
Page 291 and 292:
The following tunables, providing m
Page 293 and 294:
20.3 Debug Support /proc/sys/lnet/d
Page 295 and 296:
The OST .../stats files can be used
Page 297 and 298:
20.3.1.1 llobdstat The llobdstat ut
Page 299 and 300:
CHAPTER 21 Lustre Tuning This chapt
Page 301 and 302:
21.2 LNET Tunables This section des
Page 303 and 304:
21.3.3 Overriding Default Formattin
Page 305 and 306:
■ S2A 8500 ■ S2A 9500 ■ S2A 9
Page 307 and 308:
21.5.5 Further Tuning Tips Here are
Page 309 and 310:
21.6.1 Network Tunables With a larg
Page 311 and 312:
CHAPTER 22 Lustre Troubleshooting T
Page 313 and 314:
22.2 Submitting a Lustre Bug If, af
Page 315 and 316:
22.3.3 OST Object is Missing or Dam
Page 317 and 318:
5. You can delete these files with
Page 319 and 320:
22.3.7 Viewing Parameters To view t
Page 321 and 322:
3. If you have a separate MGS (that
Page 323 and 324:
22.3.13 Replacing An Existing OST o
Page 325 and 326:
Call trace: filter_do_bio+0x3dd/0xb
Page 327 and 328:
22.3.19 Drawbacks in Doing Multi-cl
Page 329 and 330:
CHAPTER 23 Lustre Debugging This ch
Page 331 and 332:
23.1.1 Format of Lustre Debug Messa
Page 333 and 334:
23.2.1 Debug Daemon Option to lctl
Page 335 and 336:
To turn off network logging with ch
Page 337 and 338:
23.2.5 Printing to /var/log/message
Page 339 and 340:
■ OBD_RACE: Similar to OBD_FAIL_C
Page 341 and 342:
23.4 Looking at Disk Content In Lus
Page 343 and 344:
Requests in the history include "li
Page 345:
PART IV Lustre for Users This part
Page 348 and 349:
24.1 Querying Filesystem Space The
Page 350 and 351:
24.2 Using Quotas The lfs quota com
Page 352 and 353:
25.1.1 Advantages of Striping There
Page 354 and 355:
Generally, a good stripe size for s
Page 356 and 357:
25.3 lfs setstripe - Setting File L
Page 358 and 359:
25.3.3 Creating a File on a Specifi
Page 360 and 361:
25.5 Performing Direct I/O Starting
Page 362 and 363:
25.7 Striping Using llapi Use llapi
Page 364 and 365:
int get_my_uuids(int fd) { struct o
Page 366 and 367:
c = get_my_uuids(file); rintf("Writ
Page 368 and 369:
Note - ACL support is a system-rang
Page 370 and 371:
26.2 Root Squash Functionality Lust
Page 372 and 373:
Page 374 and 375:
27.1 Expanding the Filesystem by Ad
Page 376 and 377:
NEWCHK=$($CKSUM $NEWNAME | awk '{pr
Page 378 and 379:
Page 381 and 382:
CHAPTER 28 User Utilities (man1) Th
Page 383 and 384:
lfs setquota [-u|--user|-g|--group]
Page 385 and 386:
Option Description --print0 Prints
Page 387 and 388:
Option Description quotachown [-i]
Page 389 and 390:
$ lfs find /mnt/lustre Efficiently
Page 391 and 392:
28.2 lfsck The e2fsprogs package co
Page 393 and 394:
Description If an MDS or an OST bec
Page 395 and 396:
3. Make this file accessible on all
Page 397 and 398:
Example: lfsck -n -v --mdsdb /home/
Page 399 and 400:
28.3 Filefrag The e2fsprogs package
Page 401 and 402:
28.4 Mount Lustre uses the standard
Page 403 and 404:
CHAPTER 29 Lustre Programming Inter
Page 405 and 406:
29.1.3 Parameters ■ ■ Name of t
Page 407 and 408:
CHAPTER 30 Setting Lustre Propertie
Page 409 and 410:
Examples System default size is 4MB
Page 411 and 412:
30.1.3 llapi_file_open The llapi_fi
Page 413 and 414:
Description The llapi_quotactl() co
Page 415 and 416:
CHAPTER 31 Configuration Files and
Page 417 and 418:
31.2.1 LNET Options This section de
Page 419 and 420:
Here is a more complicated situatio
Page 421 and 422:
31.2.1.4 forwarding ("") This is a
Page 423 and 424:
Variable keepalive_idle (30,Wc) kee
Page 425 and 426:
31.2.4 RapidArray LND The RapidArra
Page 427 and 428:
Variable rnr_cnt (6,Wc) rnr_nak_tim
Page 429 and 430:
31.2.7 Portals LND (Linux) The Port
Page 431 and 432:
Variable credits (128) peercredits
Page 433 and 434:
The following environment variables
Page 435 and 436:
To obtain the optimal performance f
Page 437 and 438:
CHAPTER 32 System Configuration Uti
Page 439 and 440:
Option Description --failnode=nid,.
Page 441 and 442:
32.2 tunefs.lustre The tunefs.lustr
Page 443 and 444:
Examples Changing the MGS’s NID a
Page 445 and 446:
Network Configuration Option Descri
Page 447 and 448:
Virtual Block Device Operations Lus
Page 449 and 450:
Examples lctl $ lctl lctl > dl 0 UP
Page 451 and 452:
Options Option Description :=[:] Th
Page 453 and 454:
32.5 New Utilities in Lustre 1.6 Th
Page 455 and 456:
32.5.4 Application Profiling Utilit
Page 457 and 458:
32.5.7 Flock Feature Lustre now inc
Page 459 and 460:
32.5.9 l_getgroups The l_getgroups
Page 461 and 462:
32.5.11 llstat The llstat utility d
Page 463 and 464:
32.5.12 lst The lst utility starts
Page 465 and 466:
32.5.13 plot-llstat The plot-llstat
Page 467 and 468:
CHAPTER 33 System Limits This chapt
Page 469 and 470:
33.6 Maximum Size of a Filesystem F
Page 471 and 472:
33.12 OSS RAM Size for a Single OST
Page 473 and 474:
APPENDIX A Feature List Networks TC
Page 475 and 476:
QSW LND tx_maxconfig ntxmsgs nnblk_
Page 477 and 478:
Portals LND (Linux) ntx concurrent_
Page 479 and 480:
APPENDIX B Task List Key Concepts s
Page 481 and 482:
APPENDIX C Version Log Manual Versi
Page 483 and 484:
Manual Version Date Details of Edit
Page 485 and 486:
Page 487 and 488:
Page 489 and 490:
APPENDIX D Lustre Knowledge Base Th
Page 491 and 492:
How can I check if a filesystem is
Page 493 and 494:
What does "denying connection for n
Page 495 and 496:
Now for the LOVs: lmc -m test.xml -
Page 497 and 498:
How do I set striping for a large n
Page 499 and 500:
How do I configure recoverable / fa
Page 501 and 502:
How do I backup / restore a Lustre
Page 503 and 504:
9. Format the new device. The easie
Page 505 and 506:
What extra resources are required f
Page 507 and 508:
To see which node is serving that O
Page 509 and 510:
Currently (v1.4.5) the Voltaire IB
Page 511 and 512:
1. Shut down all clients and server
Page 513 and 514:
How to fix bad LAST_ID on an OST Th
Page 515 and 516:
Why can't I run an OST and a client
Page 517 and 518:
Information on the Lustre Networkin
Page 519 and 520:
In many cases, the extent of corrup
Page 521 and 522:
How do I determine which Lustre ser
Page 523 and 524:
■ On the journal: ext3 journaling
Page 525 and 526:
When mounting an MDT filesystm, the
Page 527 and 528:
Glossary A ACL Administrative OST f
Page 529 and 530:
F Failback Failout OST Failover FID
Page 531 and 532:
LDLM lfind lfs lfsck liblustre Llit
Page 533 and 534:
MGS Management Service. A software
Page 535 and 536:
Portal A concept used by LNET. LNET
Page 537 and 538:
W Wide striping Strategy of using m
Page 539 and 540:
Index Numerics 1.6 utilities, 32-17
Page 541 and 542:
file formats, quotas, 9-11 file str
Page 543 and 544:
modprobe.conf, 7-1, 7-5, 7-6 module
Page 545 and 546:
vib (Voltaire InfiniBand), 2-2 T ta
show all

Lustre 1.6 Operations Manual

Create successful ePaper yourself

Delete template?

Save as template?