QLogic OFED+ Host Software User Guide, Rev. B

4–Running QLogic MPI on QLogic Adapters
Performance Tuning
Job Blocking in Case of Temporary InfiniBand Link Failures
By default, as controlled by mpirun’s quiescence parameter -q, an MPI job is
killed for quiescence in the event of an InfiniBand link failure (or unplugged cable).
This quiescence timeout occurs under one of the following conditions:
• A remote rank’s process cannot reply to out-of-band process checks.
• MPI is inactive on the InfiniBand link for more than 15 minutes.
To keep remote process checks but disable triggering quiescence for temporary
InfiniBand link failures, use the -disable-mpi-progress-check option with a
nonzero -q option. To disable quiescence triggering altogether, use -q 0. No
matter how these options are used, link failures (temporary or other) are always
logged to syslog.
If the link is down when the job starts and you want the job to continue blocking
until the link comes up, use the -t -1 option.
Performance Tuning
CPU Affinity
These methods may be used at runtime. Performance settings that are typically
set by the system administrator are listed in “Performance Settings and
Management Tips” on page 3-25.
InfiniPath attempts to run each node program with CPU affinity set to a separate
logical processor, up to the number of available logical processors. If CPU affinity
is already set (with sched_setaffinity() or with the taskset utility), then
InfiniPath will not change the setting.
Use the taskset utility with mpirun to specify the mapping of MPI processes to
logical processors. This combination makes the best use of available memory
bandwidth or cache locality when running on dual-core Symmetric
MultiProcessing (SMP) cluster nodes.
The following example uses the NASA Advanced Supercomputing (NAS) Parallel
Benchmark’s Multi-Grid (MG) benchmark and the -c option to taskset.
$ mpirun -np 4 -ppn 2 -m $hosts taskset -c 0,2 bin/mg.B.4
$ mpirun -np 4 -ppn 2 -m $hosts taskset -c 1,3 bin/mg.B.4
The first command forces the programs to run on CPUs (or cores) 0 and 2. The
second command forces the programs to run on CPUs 1 and 3. See the taskset
man page for more information on usage.
To turn off CPU affinity, set the environment variable IPATH_NO_CPUAFFINITY.
This environment variable is propagated to node programs by mpirun.
D000046-005 B 4-23