QLogic OFED+ Host Software User Guide, Rev. B

More documents

Recommendations

Info

QLogic OFED+ Host Software User Guide QLogic OFED+ Version 1.5 Distributed SA Configuration File . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-19 SID . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-19 ScanFrequency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 LogFile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 Dbg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 Other Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 MPI over uDAPL. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 Changing the MTU Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21 Managing the TrueScale Driver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-22 Configure the TrueScale Driver State . . . . . . . . . . . . . . . . . . . . . . . . . 3-23 Start, Stop, or Restart TrueScale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-23 Unload the Driver/Modules Manually. . . . . . . . . . . . . . . . . . . . . . . . . . 3-24 TrueScale Driver Filesystem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-24 More Information on Configuring and Loading Drivers. . . . . . . . . . . . . . . . . 3-25 Performance Settings and Management Tips . . . . . . . . . . . . . . . . . . . . . . . 3-25 Homogeneous Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26 Adapter and Other Settings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26 Remove Unneeded Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-28 Disable Powersaving Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 Hyper-Threading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 Host Environment Setup for MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-29 Configuring for ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-30 Configuring ssh and sshd Using shosts.equiv . . . . . . . . . . 3-30 Configuring for ssh Using ssh-agent . . . . . . . . . . . . . . . . . . . 3-32 Process Limitation with ssh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33 Checking Cluster and Software Status. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34 ipath_control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34 iba_opp_query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-35 ibstatus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36 ibv_devinfo. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36 ipath_checkout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-37 4 Running QLogic MPI on QLogic Adapters Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1 QLogic MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1 PSM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 Other MPIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 Linux File I/O in MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2 MPI-IO with ROMIO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 Getting Started with MPI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 iv D000046-005
QLogic OFED+ Host Software User Guide QLogic OFED+ Version 1.5 Copy Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 Create the mpihosts File. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-3 Compile and Run an Example C Program . . . . . . . . . . . . . . . . . . . . . 4-4 Examples Using Other Programming Languages . . . . . . . . . . . . . . . . 4-5 QLogic MPI Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-6 Use Wrapper Scripts for Compiling and Linking . . . . . . . . . . . . . . . . . 4-7 Configuring MPI Programs for QLogic MPI . . . . . . . . . . . . . . . . . . . . . 4-8 To Use Another Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-9 Compiler and Linker Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10 Process Allocation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-11 TrueScale Hardware Contexts on the DDR and QDR InfiniBand Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-12 Enabling and Disabling Software Context Sharing . . . . . . . . . . . 4-13 Restricting TrueScale Hardware Contexts in a Batch Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-13 Context Sharing Error Messages . . . . . . . . . . . . . . . . . . . . . . . . 4-14 Running in Shared Memory Mode . . . . . . . . . . . . . . . . . . . . . . . 4-14 mpihosts File Details. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-15 Using mpirun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-16 Console I/O in MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-18 Environment for Node Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-19 Environment Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-20 Running Multiple Versions of TrueScale or MPI . . . . . . . . . . . . . . . . . 4-22 Job Blocking in Case of Temporary InfiniBand Link Failures. . . . . . . . 4-23 Performance Tuning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23 CPU Affinity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-23 mpirun Tunable Options . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 MPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 MPD Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-24 Using MPD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-25 QLogic MPI and Hybrid MPI/OpenMP Applications . . . . . . . . . . . . . . . . . . . 4-25 Debugging MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26 MPI Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-26 Using Debuggers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-27 QLogic MPI Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-28 5 Using Other MPIs Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1 Installed Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2 Open MPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3 D000046-005 v
Page 1 and 2: QLogic OFED+ Host Software User Gui
Page 3: Table of Contents 1 Introduction Ho
Page 15 and 16: Preface The QLogic OFED+ Host Softw
Page 17 and 18: Preface Technical Support Technical
Page 19 and 20: 1 Introduction How this Guide is Or
Page 21 and 22: 1-Introduction Interoperability QLo
Page 23 and 24: 2 Step-by-Step Cluster Setup and MP
Page 25 and 26: 3 TrueScale Cluster Setup and Admin
Page 27 and 28: 3-TrueScale Cluster Setup and Admin
Page 55 and 56:
3-TrueScale Cluster Setup and Admin
Page 57 and 58:
Page 59 and 60:
Page 61 and 62:
Page 63 and 64:
4 Running QLogic MPI on QLogic Adap
Page 65 and 66:
4-Running QLogic MPI on QLogic Adap
Page 67 and 68:
Page 69 and 70:
Page 71 and 72:
Page 73 and 74:
Page 75 and 76:
Page 77 and 78:
Page 79 and 80:
Page 81 and 82:
Page 83 and 84:
Page 85 and 86:
Page 87 and 88:
Page 89 and 90:
Page 91 and 92:
5 Using Other MPIs Introduction Thi
Page 93 and 94:
5-Using Other MPIs Open MPI Open MP
Page 95 and 96:
5-Using Other MPIs MVAPICH Further
Page 97 and 98:
5-Using Other MPIs Managing Open MP
Page 99 and 100:
5-Using Other MPIs Platform (Scali)
Page 101 and 102:
5-Using Other MPIs Intel MPI Intel
Page 103 and 104:
5-Using Other MPIs Intel MPI When u
Page 105 and 106:
5-Using Other MPIs Improving Perfor
Page 107 and 108:
6 Performance Scaled Messaging Intr
Page 109 and 110:
6-Performance Scaled Messaging Usin
Page 111 and 112:
7 Dispersive Routing Infiniband use
Page 113 and 114:
7-Dispersive Routing • Static_Des
Page 115 and 116:
8 gPXE gPXE Setup gPXE is an open s
Page 117 and 118:
8-gPXE Preparing the DHCP Server in
Page 119 and 120:
8-gPXE Netbooting Over InfiniBand P
Page 121 and 122:
8-gPXE Netbooting Over InfiniBand W
Page 123 and 124:
8-gPXE Netbooting Over InfiniBand #
Page 125 and 126:
8-gPXE Netbooting Over InfiniBand #
Page 127 and 128:
8-gPXE Netbooting Over InfiniBand T
Page 129 and 130:
8-gPXE HTTP Boot Setup 5. Create an
Page 131 and 132:
A mpirun Options Summary This secti
Page 133 and 134:
A-mpirun Options Summary Quiescence
Page 135 and 136:
A-mpirun Options Summary Tuning Opt
Page 137 and 138:
A-mpirun Options Summary Format Opt
Page 139 and 140:
B Benchmark Programs Several MPI pe
Page 141 and 142:
B-Benchmark Programs Benchmark 2: M
Page 143 and 144:
B-Benchmark Programs Benchmark 4: M
Page 145 and 146:
C VirtualNIC Interface Configuratio
Page 147 and 148:
C-VirtualNIC Interface Configuratio
Page 149 and 150:
Page 151 and 152:
Page 153 and 154:
Page 155 and 156:
Page 157 and 158:
Page 159 and 160:
Page 161 and 162:
D SRP Configuration SRP Configurati
Page 163 and 164:
D-SRP Configuration QLogic SRP Conf
Page 165 and 166:
Page 167 and 168:
Page 169 and 170:
Page 171 and 172:
Page 173 and 174:
Page 175 and 176:
Page 177 and 178:
Page 179 and 180:
Page 181 and 182:
Page 183 and 184:
Page 185 and 186:
D-SRP Configuration OFED SRP Config
Page 187 and 188:
E Integration with a Batch Queuing
Page 189 and 190:
E-Integration with a Batch Queuing
Page 191 and 192:
E-Integration with a Batch Queuing
Page 193 and 194:
F Troubleshooting This appendix des
Page 195 and 196:
F-Troubleshooting Kernel and Initia
Page 197 and 198:
F-Troubleshooting Kernel and Initia
Page 199 and 200:
F-Troubleshooting OpenFabrics and I
Page 201 and 202:
F-Troubleshooting Performance Issue
Page 203 and 204:
F-Troubleshooting QLogic MPI Troubl
Page 205 and 206:
Page 207 and 208:
Page 209 and 210:
Page 211 and 212:
Page 213 and 214:
Page 215 and 216:
Page 217 and 218:
Page 219 and 220:
Page 221 and 222:
Page 223 and 224:
Page 225 and 226:
G ULP Troubleshooting Troubleshooti
Page 227 and 228:
G-ULP Troubleshooting Troubleshooti
Page 229 and 230:
Page 231 and 232:
Page 233 and 234:
Page 235 and 236:
Page 237 and 238:
Page 239 and 240:
Page 241 and 242:
H Write Combining Introduction Writ
Page 243 and 244:
H-Write Combining MTRR Mapping and
Page 245 and 246:
I Useful Programs and Files The mos
Page 247 and 248:
I-Useful Programs and Files Summary
Page 249 and 250:
Page 251 and 252:
Page 253 and 254:
Page 255 and 256:
Page 257 and 258:
Page 259 and 260:
Page 261 and 262:
I-Useful Programs and Files Common
Page 263 and 264:
Page 265 and 266:
Page 267 and 268:
Page 269 and 270:
J Recommended Reading Reference mat
Page 272:
Corporate Headquarters QLogic Corpo
show all

QLogic OFED+ Host Software User Guide, Rev. B

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?