QLogic OFED+ Host Software User Guide, Rev. B

F–Troubleshooting
QLogic MPI Troubleshooting
See “Compiler Cannot Find Include, Module, or Library Files” on page F-14,
“Compiling on Development Nodes” on page F-15, and “Specifying the Run-time
Library Path” on page F-15 for additional information.
Process Limitation with ssh
MPI jobs that use more than eight processes per node may encounter an ssh
throttling mechanism that limits the amount of concurrent per-node connections
to 10. If you have this problem, a message similar to this appears when using
mpirun:
$ mpirun -m tmp -np 11 ~/mpi/mpiworld/mpiworld
ssh_exchange_identification: Connection closed by remote host
MPIRUN: Node program(s) exitted during connection setup
If you encounter a message like this, you or your system administrator should
increase the value of MaxStartups in your sshd configurations.
NOTE:
This limitation applies only if -distributed=off is specified. By default,
with -distributed=on, you will not normally have this problem.
Number of Processes Exceeds ulimit for Number of Open
Files
When users scale up the number of processes beyond the number of open files
allowed by ulimit, mpirun will print an error message. The ulimit for the
number of open files is typically 1024 on both Red Hat and SLES systems. The
message will look similar to:
MPIRUN.up001: Warning: ulimit for the number of open files is only
1024, but this mpirun request requires at least
open files (sockets). The shell ulimit for open files needs to be
increased.
This is due to limit:
descriptors 1024
The ulimit can be increased; QLogic recommends an increase of
approximately 20 percent over the number of CPUs. For example, in the case of
2048 CPUs, ulimit can be increased to 2500:
ulimit -n 2500
The ulimit needs to be increased only on the host where mpirun was started,
unless the mode of operation allows mpirun from any node.
D000046-005 B F-19