R&M Data Center Handbook

www.datacenter.rdm.com
The InfiniBand concept is aware of four network components: The Host Channel Adapter (HCA), the Target
Channel Adapter (TCA), the switch and the router. Every component has one or more ports and can be connected
to another component using a speed class (1x, 4x, 8x, 12x).
IB1X plug for 1xInfiniBand / IB12X plug for 12xInfiniBand
Sierra Technologies
So, for example, an HCA can be connected with one
or more switches, which for their part connect
input/output devices via a TCA with the switch. As a
result, the Host Channel Adapter can communicate
with one or more Target Channel Adapters via the
switch. Multi-point connections can be set up
through the switch. The InfiniBand router’s method
of operation is essentially comparable to that of a
switch; however the router can transfer data from
local subnets to other subnets.
InfiniBand will normally transmit over copper cables, as they are also used for 10 gigabit Ethernet. Transmission
paths of up to 15 meters are possible.
If longer paths need to be bridged over, fiber-optic media converters can be accessed, which convert the
InfiniBand channels on individual fiber pairs. Optical ribbon cables with MPO plugs are used for this purpose, in
other words the same plug type as in 40/100 gigabit Ethernet.
In the early years, the transmission rate was 2.5 Gbit/s, which resulted in a usable transmission rate of 250 MB/s
per link in both directions, in the case of 8B/10B coding. In addition to this standard data rate (1x), InfiniBand
defines bundling of 4, 8 and 12 links for higher transmission rates. The resulting transmission speeds lie at 10
Gbit/s, or a 1 GB/s usable data rate, and 30 Gbit/s or 3 GB/s. The range for a connection with glass fibers (single
mode) can come to 10 kilometers.
Newer InfiniBand developments have a ten time higher data rate of 25 Gbit/s and reach data rates at levels 1x, 4x,
8x and 12x between 25 Gbit/s and 300 Gbit/s or 25 GB/s.
3.8.6. Protocols for Redundant Paths
If Fibre Channel over Ethernet is integrated into the data center or voice and
video into LAN, then chances are good that the network will have to undergo
a redesign. This is because current network designs build on the provision of
substantial overcapacities.
The aggregation/distribution and core layer are often laid out as a tree
structure in a multiple redundant manner. In this case, the redundancy is only
used in case of a fault. As a result, large bandwidth resources are squandered
and available redundant paths are used only insufficiently or inefficiently,
if at all.
However, short paths and thus lower delay times are the top priority for new
real-time applications in networks. So-called "shortest path bridging" will implement
far-reaching changes in this area.
The most important protocols developed for network redundancy are listed
below. Each of these protocols has its own specific characteristics.
Spanning Tree
The best-known and oldest protocol, that prevents “loops” in star-shaped structured Ethernet networks and therefore
allows redundant paths, is the Spanning Tree Protocol (STP). The method was standardized in IEEE 802.1D
and came into use in the world of layer 2 switching.
The STP method works as follows:
• If redundant network paths exist, only one path is active. The others are passive – or actually they are
“turned off” (for loop prevention).
• The switches determine independently which paths are active and which are passive (negotiation).
Manual configuration is not required.
• If an active network path fails, a recalculation is performed for all network paths and the required
redundant connection is built.
Page 106 of 156 © 08/2011 Reichle & De-Massari AG R&M Data Center Handbook V2.0