QLogic OFED+ Host Software User Guide, Rev. B

F–Troubleshooting
Performance Issues
In large cluster configurations, switches may be attached to other switches to
supply the necessary inter-node connectivity. Problems with these inter-switch (or
intermediate) links are sometimes more difficult to diagnose than failure of the
final link between a switch and a node. The failure of an intermediate link may
allow some traffic to pass through the fabric while other traffic is blocked or
degraded.
If you notice this behavior in a multi-layer fabric, check that all switch cable
connections are correct. Statistics for managed switches are available on a
per-port basis, and may help with debugging. See your switch vendor for more
information.
QLogic recommends using FastFabric to help diagnose this problem. If
FastFabric is not installed in the fabric, there are two diagnostic tools, ibhosts
and ibtracert, that may also be helpful. The tool ibhosts lists all the
InfiniBand nodes that the subnet manager recognizes. To check the InfiniBand
path between two nodes, use the ibtracert command.
Performance Issues
The following sections discuss known performance issues.
Large Message Receive Side Bandwidth Varies with
Socket Affinity on Opteron Systems
On Opteron systems, when using the QLE7240 or QLE7280 in DDR mode, there
is a receive side bandwidth bottleneck for CPUs that are not adjacent to the PCI
Express root complex. This may cause performance to vary. The bottleneck is
most obvious when using SendDMA with large messages on the farthest sockets.
The best case for SendDMA is when both sender and receiver are on the closest
sockets. Overall performance for PIO (and smaller messages) is better than with
SendDMA.
Erratic Performance
Sometimes erratic performance is seen on applications that use interrupts. An
example is inconsistent SDP latency when running a program such as netperf.
This may be seen on AMD-based systems using the QLE7240 or QLE7280
adapters. If this happens, check to see if the program irqbalance is running.
This program is a Linux daemon that distributes interrupts across processors.
However, it may interfere with prior interrupt request (IRQ) affinity settings,
introducing timing anomalies. After stopping this process (as a root user), bind
IRQ to a CPU for more consistent performance. First, stop irqbalance:
# /sbin/chkconfig irqbalance off
# /etc/init.d/irqbalance stop
D000046-005 B F-9