QLogic OFED+ Host Software User Guide, Rev. B

7 Dispersive Routing
Infiniband uses deterministic routing that is keyed from the Destination LID (DLID)
of a port. The Fabric Manager programs the forwarding tables in a switch to
determine the egress port a packet takes based on the DLID.
Deterministic routing can create hotspots even in full bisection bandwidth (FBB)
fabrics for certain communication patterns if the communicating node pairs map
onto a common upstream link, based on the forwarding tables. Since routing is
based on DLIDs, the InfiniBand fabric provides the ability to assign multiple LIDs
to a physical port using a feature called Lid Mask Control (LMC). The total number
of DLIDs assigned to a physical port is 2^LMC with the LIDS being assigned in a
sequential manner. The common InfiniBand fabric uses a LMC of 0, meaning
each port has 1 LID assigned to it. With non-zero LMC fabrics, this results in
multiple potential paths through the fabric to reach the same physical port. For
example, multiple DLID entries in the port forwarding table that could map to
different egress ports.
Dispersive routing, as implemented in the PSM, attempts to avoid congestion
hotspots described above by “spraying” messages across these paths. A
congested path will not bottleneck messages flowing down the alternate paths
that are not congested. The current implementation of PSM supports fabrics with
a maximum LMC of 3 (8 LIDs assigned per port). This can result in a maximum of
64 possible paths between a SLID, DLID pair ([SLID, DLID],[SLID, DLID+1],
[SLID,DLID+2]…..[SLID,DLID+8],[SLID+1, DLID],[SLID+1, DLID+1]…..[SLID+7,
DLID+8]). Keeping state associated with these many paths requires large amount
of memory resources, with empirical data showing not much gain in performance
beyond utilizing a small set of multiple paths. Therefore PSM reduces the number
of paths actually used in the above case to 8 where the following paths are the
only ones considered for transmission — [SLID, DLID], [SLID + 1, DLID + 1],
[SLID + 2, DLID + 2] ….. [SLID + N, DLID + N]. This makes the resource
requirements manageable while providing most of the benefits of dispersive
routing (congestion avoidance by utilizing multiple paths).
D000046-005 B 7-1