QLogic OFED+ Host Software User Guide, Rev. B
QLogic OFED+ Host Software User Guide, Rev. B
QLogic OFED+ Host Software User Guide, Rev. B
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
3–TrueScale Cluster Setup and Administration<br />
<strong>QLogic</strong> Distributed Subnet Administration<br />
The Distributed Subnet Administration (SA) solves this problem by allowing each<br />
node to locally replicate the path records needed to reach the other nodes on the<br />
fabric. At boot time, each Distributed SA queries the subnet manager for<br />
information about the relevant parts of the fabric, backing off whenever the subnet<br />
manager indicates that it is busy. Once this information is in the Distributed SA's<br />
database, it is ready to answer local path queries from MPI or other InfiniBand<br />
applications. If the fabric changes (due to a switch failure or a node being added<br />
or removed from the fabric) the Distributed SA updates the affected portions of the<br />
database. The Distributed SA is installed and runs on every node in the fabric,<br />
except the management node running <strong>QLogic</strong> FM.<br />
Applications that use Distributed SA<br />
The <strong>QLogic</strong> PSM Library has been extended to take advantage of Distributed SA.<br />
Therefore, all MPIs that use the <strong>QLogic</strong> PSM library can take advantage of the<br />
Distributed SA. Other applications must be modified specifically to take advantage<br />
of it. For developers writing applications that use the Distributed SA, please refer<br />
to the header file /usr/include/Infiniband/ofedplus_path.h for information on<br />
modifying applications to use the Distributed SA. This file can be found on any<br />
node where the Distributed SA is installed. For further assistance please contact<br />
<strong>QLogic</strong> Support.<br />
Virtual Fabrics and the Distributed SA<br />
The Distributed SA is designed to be aware of Virtual Fabrics, but to only store<br />
records for those Virtual Fabrics that match Service ID records in the Distributed<br />
SA's configuration file. In addition, the Distributed SA recognizes when multiple<br />
SIDs match the same Virtual Fabric and will only store one copy of each path<br />
record within a Virtual Fabric. Next, SIDs that match more than one Virtual Fabric<br />
will be assigned to a single Virtual Fabrics. Finally, Virtual Fabrics that do not<br />
match SIDs in the Distributed SA's database will be ignored.<br />
Configuring the Distributed SA<br />
In order to absolutely minimize the number of queries made by the Distributed SA,<br />
it is important to configure it correctly, both to match the configuration of the Fabric<br />
Manager and to exclude those portions of the fabric that will not be used by<br />
applications using the Distributed SA. The configuration file for the Distributed SA<br />
is named /etc/sysconfig/iba/qlogic_sa.conf.<br />
Default Configuration<br />
As shipped, the <strong>QLogic</strong> Fabric Manager creates a single virtual fabric, called<br />
“Default” and maps all nodes and Service IDs to it, and the Distributed SA ships<br />
with a configuration that lists a set of thirty-one Service IDs (SID<br />
0x1000117500000000 through 0x100011750000000f and 0x1 through 0xf). This<br />
results in an arrangement like the one shown in Figure 3-2<br />
3-14 D000046-005 B
Change language