QLogic OFED+ Host Software User Guide, Rev. B

E–Integration with a Batch Queuing System
Using SLURM for Batch Queuing
The sort | uniq -c component determines the number of times each unique
line was printed. The awk command converts the result into the mpihosts file
format used by mpirun. Each line consists of a node name, a colon, and the
number of processes to start on that node.
NOTE:
This is one of two formats that the file can use. See “Console I/O in MPI
Programs” on page 4-18 for more information.
Simple Process Management
At this point, the script has enough information to be able to run an MPI program.
The next step is to start the program when the batch system is ready, and notify
the batch system when the job completes. This is done in the final part of
batch_mpirun, for example:
mpirun -np $np -m $mpihosts_file "$mpi_prog" $@
exit_code=$
scancel ${SLURM_JOBID}
rm -f $mpihosts_file
exit $exit_code
Clean Termination of MPI Processes
The InfiniPath software normally ensures clean termination of all MPI programs
when a job ends, but in some rare circumstances an MPI process may remain
alive, and potentially interfere with future MPI jobs. To avoid this problem, run a
script before and after each batch job that kills all unwanted processes. QLogic
does not provide such a script, but it is useful to know how to find out which
processes on a node are using the QLogic interconnect. The easiest way to do
this is with the fuser command, which is normally installed in /sbin.
Run these commands as a root user to ensure that all processes are reported.
# /sbin/fuser -v /dev/ipath
/dev/ipath: 22648m 22651m
In this example, processes 22648 and 22651 are using the QLogic interconnect. It
is also possible to use this command (as a root user):
# lsof /dev/ipath
This command displays a list of processes using InfiniPath. Additionally, to get all
processes, including stats programs, ipath_sma, diags, and others, run the
program in this way:
# /sbin/fuser -v /dev/ipath*
lsof can also take the same form:
# lsof /dev/ipath*
E-4 D000046-005 B