QLogic OFED+ Host Software User Guide, Rev. B

3–TrueScale Cluster Setup and Administration
QLogic Distributed Subnet Administration
If you are using the Installer tool, you can set the OpenSM default behavior at the
time of installation.
OpenSM only needs to be enabled on the node that acts as the subnet manager,
so use the chkconfig command (as a root user) to enable it on the node where it
will be run:
chkconfig opensmd on
The command to disable it on reboot is:
chkconfig opensmd off
You can start opensmd without rebooting your machine by typing:
/etc/init.d/opensmd start
You can stop opensmd again by typing:
/etc/init.d/opensmd stop
If you want to pass any arguments to the OpenSM program, modify the following
file, and add the arguments to the OPTIONS variable:
/etc/init.d/opensmd
For example:
Use the UPDN algorithm instead of the Min Hop algorithm.
OPTIONS="-R updn"
For more information on OpenSM, see the OpenSM man pages, or look on the
OpenFabrics web site.
QLogic Distributed Subnet Administration
As InfiniBand clusters are scaled into the Petaflop range and beyond, a more
efficient method for handling queries to the Fabric Manager is required. One of
these issues is that while the Fabric Manager can configure and operate that
many nodes, under certain conditions it can become overloaded with queries from
those same nodes.
For example, consider an InfiniBand fabric consisting of 1,000 nodes, each with 4
processors. When a large MPI job is started across the entire fabric, each process
needs to collect InfiniBand path records for every other node in the fabric - and
every single process is going to be querying the subnet manager for these path
records at roughly the same time. This amounts to a total of 3.9 million path
queries just to start the job!
In the past, MPI implementations have side-stepped this problem by hand crafting
path records themselves, but this solution cannot be used if advanced fabric
management techniques such as virtual fabrics and mesh/torus configurations are
being used. In such cases, only the subnet manager itself has enough information
to correctly build a path record between two nodes.
D000046-005 B 3-13