QLogic OFED+ Host Software User Guide, Rev. B

E–Integration with a Batch Queuing System
Lock Enough Memory on Nodes when Using SLURM
The following command terminates all processes using the QLogic interconnect:
# /sbin/fuser -k /dev/ipath
For more information, see the man pages for fuser(1) and lsof(8).
Note that hard and explicit program termination, such as kill -9 on the mpirun
Process ID (PID), may result in QLogic MPI being unable to guarantee that the
/dev/shm shared memory file is properly removed. As many stale files
accumulate on each node, an error message can appear at startup:
node023:6.Error creating shared memory object in shm_open(/dev/shm
may have stale shm files that need to be removed):
If this occurs, administrators should clean up all stale files by using this command:
# rm -rf /dev/shm/psm_shm.*
See “Error Creating Shared Memory Object” on page F-23 for more information.
Lock Enough Memory on Nodes when Using
SLURM
This section is identical to information provided in “Lock Enough Memory on
Nodes When Using a Batch Queuing System” on page F-22. It is repeated here
for your convenience.
QLogic MPI requires the ability to lock (pin) memory during data transfers on each
compute node. This is normally done via /etc/initscript, which is created or
modified during the installation of the infinipath RPM (setting a limit of
128 MB, with the command ulimit -l 131072).
Some batch systems, such as SLURM, propagate the user’s environment from
the node where you start the job to all the other nodes. For these batch systems,
you may need to make the same change on the node from which you start your
batch jobs.
If this file is not present or the node has not been rebooted after the infinipath
RPM has been installed, a failure message similar to one of the following will be
generated.
The following message displays during installation:
$ mpirun -np 2 -m ~/tmp/sm mpi_latency 1000 1000000
iqa-19:0.ipath_userinit: mmap of pio buffers at 100000 failed:
Resource temporarily unavailable
iqa-19:0.Driver initialization failure on /dev/ipath
iqa-20:1.ipath_userinit: mmap of pio buffers at 100000 failed:
Resource temporarily unavailable
iqa-20:1.Driver initialization failure on /dev/ipath
D000046-005 B E-5