QLogic OFED+ Host Software User Guide, Rev. B

7–Dispersive Routing
Internally, PSM utilizes dispersive routing differently for small and large
messages. Large messages are any messages greater-than or equal-to 64K. For
large messages, the message is split into message fragments of 128K by default
(called a window). Each of these message windows is sprayed across a distinct
path between ports. All packets belonging to a window utilize the same path
however the windows themselves can take a different path through the fabric.
PSM assembles the windows that make up an MPI message before delivering it to
the application. This allows limited out of order semantics through the fabrics to be
maintain with little overhead. Small messages on the other hand always utilize a
single path when communicating to a remote node however different processes
executing on a node can utilize different paths for their communication between
the nodes. For example, two nodes A and B each with 8 processors per node.
Assuming the fabric is configured for a LMC of 3, PSM constructs 8 paths through
the fabric as described above and a 16 process MPI application that spans these
nodes (8 process per node). Then:
• Each MPI process is automatically bound to a given CPU core numbered
between 0-7. PSM does this at startup to get improved cache hit rates and
other benefits.
• Small Messages sent from a process on core N will use path N.
NOTE:
Only path N will be used by this process for all communications to any
process on the remote node.
• For a large message, each process will utilize all of the 8 paths and spray
the windowed messages across it.
The above highlights the default path selection policy that is active in PSM when
running on non-zero LMC configured fabrics. There are 3 other path selection
policies that determine how to select the path (or path index from the set of
available paths) used by a process when communicating with a remote node. The
above path policy is called adaptive. The 3 remaining path policies are static
policies that assign a static path on job startup for both small and large message
transfers.
• Static_Src: Only one path per process is used for all remote
communications. The path index is based on the CPU number the process
is running.
NOTE:
Multiple paths are still used in the fabric if multiple processes (each on
a different CPU) are communicating.
7-2 D000046-005 B