T-BLADE 2 TECHNICAL GUIDEBOOK - T-Platforms

T-Blade 2 Technical GuideBook 

www.t-platforms.com

Disclaimer: 

This document is provided for informational purposes only 

and may contain inaccuracies. For up-to-date information 

please contact your T-Platforms representative. 

T-Blade Technical Guide Book, rev. A0 .

Table of contents 

1. General overview ..........................................................................................................................................................................4 

2. Compute Module ........................................................................................................................................................................5 

2.1. Compute module board ....................................................................................................................................................5 

2.1.1. Board geometry and dimensions ................................................................................................................................................................5 

2.1.2. Compute node featureset ...............................................................................................................................................................................6 

2.1.3. Compute node special features ....................................................................................................................................................................7 

2.2. Compute module power and connectivity features ........................................................................................7 

2.3. Compute module heat sink..............................................................................................................................................8 

3. InfiniBand Switch Modules ..................................................................................................................................................9 

4. Management and Switch Module (MSM) ...........................................................................................................10 

4.1. Management processor block ....................................................................................................................................10 

4.2. Gigabit Ethernet switches block ...............................................................................................................................11 

4.3. Special networks block (FPGA) ...................................................................................................................................11 

4.4. Global clock distribution block ..................................................................................................................................11 

4.5. Management and Switch Module Connectors.................................................................................................12 

5. Backplane ........................................................................................................................................................................................13 

6. Blade chassis .................................................................................................................................................................................14 

6.3. Cooling subsystem .............................................................................................................................................................15 

7. Other features ...............................................................................................................................................................................16 

7.1. Power-on procedure ........................................................................................................................................................16 

7.2. Emergency Shutdown procedure .............................................................................................................................16 

8. Cluster Management and Monitoring .....................................................................................................................16 

9. Basic infrastructure requirements .................................................................................................................................17 

9.1. Electricity ...................................................................................................................................................................................17 

9.2. Cooling .......................................................................................................................................................................................17 

9.3. Cabinet infrastructure .......................................................................................................................................................17 

9.4. Floors and layout ..................................................................................................................................................................17 

10. Operating and file systems compatibility ...........................................................................................................17 

11. System specification .............................................................................................................................................................18 

T-Blade 2 Enclosure .........................................................................................................................................................................................................18 

T-Blade 2 Compute Node ...........................................................................................................................................................................................18 

T-Blade 2 external ports and networks ...............................................................................................................................................................18 

Appendix 1. Compute Module diagram ......................................................................................................................19 

Appendix 2. Intra-chassis QDR IB links diagram .....................................................................................................20 

Appendix 3. Management and Switch Module diagram ..............................................................................21 


1. General overview 

Welcome to T-Blade , the leading compute density solution from T-Platforms engineered 

specifically for the world’s largest x86 HPC installations. 

Introduced in 009, T-Blade creates an unmatched solution for customers with the most 

demanding HPC needs, and perfectly complements the other elements of T-Platforms’ HPC 

portfolio, including the Cell BE-based PowerXCell 8 1RU compute nodes and the 5RU 10-node 

T-Blade I system. 

The driving force behind the T-Blade 2 design is the use of industry-standard compute, 

interconnect, and management architectures packaged in an HPC-optimized manner to deliver 

just the right mix of performance, density, redundancy, and manageability. 

T-Blade 2: designed for density, performance, and reliability: 

■ 

■ 

■ 

■ 

■ 

■ 

7U-high chassis for mounting into standard 19” rack 

16 hot-pluggable compute modules with 64 Intel Xeon 56xx series CPUs; each compute 

module contains 2 dual-processor compute nodes on a single board 

2 integrated 36-port QDR InfiniBand switches 

Integrated management module with GbE switches and special HPC features 

Air cooling using hot-swappable high performance fans 

11KW, N+1 redundant power supplies 

Although T-Blade 2 features standard technologies, T-Platforms has focused its engineering and design expertise on overcoming the scalability 

bottlenecks that so often plague commodity clusters based built with off-the-shelf components. This bottleneck is largely caused by poor collective 

communications efficiency, which needs special support at the hardware level to maintain excellent applications performance with high nodecount 

systems. T-Blade 2 features dedicated global barrier and global interrupt networks, ensuring effective application scalability even on 

systems containing many thousands of nodes. 

The global barrier network supports fast synchronization between the tasks of a large-scale application, while the global interrupt network 

significantly reduces the negative impact of OS jitter by synchronizing process scheduling over the entire system. As a result, processors 

communicate much more efficiently with each other, enabling high scalability even for the most demanding parallel applications. 

The two networks are managed by an FPGA chip integrated into a dedicated management module (see Appendix) that monitors all subsystems 

and components, enabling remote system management, emergency shutdowns, etc. 

To ensure smooth and fast data transfer, and reduce network congestion in high node-count systems, T-Blade 2 incorporates extra external 

InfiniBand ports, providing an impressive 1.6Tb/sec of bandwidth. 

T-Platforms has engineered the T-Blade 2 platform for reliability as well as performance. The system features neither hard discs nor cables inside 

the enclosure, dramatically decreasing outages caused by mechanical failure within a node. Reliability is further enhanced within each enclosure 

with hot-swappable, N+1 redundant power supplies and cooling fans. 

Petascale computing demands deep changes in system software to efficiently support application scalability. T-Blade 2 is optionally supplied 

with a comprehensive ‘all-in-one’ system software stack that includes an optimized Linux OS core and system libraries, as well as all necessary 

management and monitoring software components. This integrated software stack provides an out-of-the-box experience, reducing system 

installation time and administration costs. 

The T-Blade Linux core provides specialized support to the global barrier and global interrupt networks, an optimization that greatly speeds up 

inter-node communications over more traditional designs. And our new system management software, part of Clustrx OS, ensures seamless 

scalability up to 12,000 compute nodes with near-real time monitoring capability. T-Blade 2 also features aggressive power saving technology, and 

support of topology-driven resource allocation which improves memory usage and accelerates real applications performance. 

Best of all, the T-Blade 2 HPC platform is field-proven: it is the primary building block of the 420TF Lomonosov cluster deployed at Moscow State 

University. This system was recognized at number 12 on the November 2009 TOP500 list of the world’s fastest computers and is considered to 

be the largest supercomputer in Eastern Europe. 


2. Compute Module 

A T-Blade 2 compute module consists of a single printed circuit board (PCB) with two separate dual-processor compute nodes that are mounted 

on a specially-engineered heat sink that provides the necessary cooling for the entire module. Each compute module board comes with memory 

modules preinstalled (Figure 1), in 16-board bulk packages. 

2.1. Compute module board 

Guides 

Figure 1. 

System 

board tray 

Backplane connector 

DDR3 memory modules 

2.1.1. Board geometry and dimensions 

Intel Xeon 5500 Series 

QDR IB controllers 

The geometry of a compute module PCB is shown in Figure 1. This airflow-optimized board is attached for reliability both to the heat sink and to 

a board tray using mounting holes. Each board attaches to the system backplane using a power and signal connector mounted on the PCB, and 

is supported by mechanical guides located on each tray. 

T-Blade Technical Guide Book, rev. A0 . 

Heatsink 

DC-DC 

converters 

5

2.1.2. Compute node featureset 

Each of the two compute nodes on a PCB features: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Two Intel Westmere (56xx series) CPUs, each with a maximum TDP of 95W and clock speeds up to 2.93GHz 

12GB memory per compute node in a 3-channel design that uses unbuffered ECC DDR3-1333 RAM 

6GB or 12GB of memory per each CPU socket 

4 memory modules (Figure 2) in a proprietary mezzanine card design with 27 memory chips (9 chips in each of the 3 

channels) 

The Intel Tylersburg 24D+ ICH10 chipset, with QPI links providing 6.4 Gtransfers/sec (3.2GHz) between CPUs on a board 

A single-port Mellanox ConnectX QDR InfiniBand chip connected to the Tylersburg chipset via PCIe 2.0 x8 link 

A single-port GbE controller connected to ICH10 

A USB controller for miniSD or microSD flash, and a connector for the flash card 

Baseboard management controller (BMC) and a separate single-port GbE controller for the BMC connection (refer to the section 

“Compute node management and monitoring” for a discussion of the BMC and the management and monitoring feature set) 

One serial port, connected to the BMC 

Five GPIO signals from ICH10 chip going through the backplane to the FPGA chip on the blade management module for HPCspecific 

purposes (described in the section “Compute node special features”) 

A centralized storage design with no separate SAS/SATA storage controller; the links from the integrated SATA controller are not 

routed, and the appropriate BIOS portion is disabled 

Two microbuttons for POWER and RESET, placed on the rear edge of the compute module 

Figure 2. DDR3 Memory Module 

6 T-Blade Technical Guide Book, rev. A0 . 

See Compute node Diagram in Appendix 1

2.1.3. Compute node special features 

■ 

■ 

■ 

It is possible to use either an internal or external clock for the system. Clock selection is implemented using a special IDT5T9GL02 

model (or similar) chip, and the clock select pin on that chip goes to the DIP switch placed on the rear bottom part of the PCB 

to provide manual switching of the clocking mode. There is also an LED placed on the rear edge of each compute module to 

indicate whether the system is using the internal or external clock. 

Two interrupt pins on the ICH10 chip are routed through the backplane to the FPGA chip on the management module; the FPGA 

chip is discussed in greater detail in a separate document. 

5 GPIO pins on the ICH10 chip on each compute node are routed through the backplane to the FPGA chip on the management 

module. 

Compute node management and monitoring is based on an industry standard baseboard management controller, or BMC. The BMC is accessed 

through the management module using a dedicated GbE interface to eliminate the possibility of interference with application performance. The 

BMC provides the following hardware monitoring features: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Temperature for each CPU core 

Memory temperature – one temperature sensor per memory block dedicated to each CPU 

Two additional temperature sensors on CPU voltage regulator modules 

The voltage at each CPU core 

3.3V, 5V, 12V and 5Vsb voltages 

Memory and PCI error conditions; for memory errors the exact memory channel is reported to aid in error resolution 

InfiniBand and GbE counters (optional) 

The following management capabilities are supported: 

■ 

■ 

■ 

■ 

■ 

Remote power on/power off/reset 

Remote KVM-over-IP and full SOL console access starting from the boot process, including the POST procedure and BIOS 

setup 

Compute node BIOS update and NVRAM save/restore without involving any OS-dependent utilities 

Boot select priority 

Remote BMC firmware upgrade 

All BMC features can be accessed remotely via Telnet and SSH using key authentication for added security. 

2.2. Compute module power and connectivity features 

■ 

All power and connectivity for a compute module is provided via backplane power and signal connectors. The power connector 

provides -48V DC input with a minimum of 18A current for every -48V channel. 

The following signals are routed to the backplane via the signal connector: 

■ 

■ 

■ 

■ 

■ 

2 QDR InfiniBand 4x links (1 per compute node) 

4 GbE links (1 GbE + 1 BMC GbE connection per compute node) 

2 external clock links (1 per compute node); it is possible to use just one external clock link by using a clock extender on the 

compute module 

4 remote interrupt links (2 per compute node) 

10 global barrier network links (5 GPIO links per compute node) 


2.3. Compute module heat sink 

The compute board is attached to a tray and a specially-designed heat sink that covers the entire board. The heat sink also serves as a rail for 

the compute module boards that helps them slide reliably into compute module bays of the enclosure. The geometry of the heat sink is shown in 

figure 3. 

The heat sink base plate has variable thickness according to the component height on the compute module board. A rubber material with low 

thermal resistance provides good thermal contact between the components and the heat sink. 

The heat sink design and choice of material are based on extensive thermal analysis performed by T-Services to achieve an optimal compute 

node design. This engineering resulted in a final design that is not only more efficient but also lighter than standard heat sinks, reducing the overall 

T-Blade 2 system weight below 160kg. 

There is also a retention mechanism mounted on the rear edge of the heat sink for plugging/unplugging the compute module (Figure 3). 

Figure 3. 

Figure 4. 

Air current velocity and distribution between the board 

and heat sink for 60CFM Fan design variant. 

Heatsink Backplane 

connector 

PCB 


Retention 

Mechanism 

Figure 5. 

Temperature distribution at the base of aluminum-based 

heat sink design variant.

3. InfiniBand Switch Modules 

There are two non-redundant integrated InfiniBand switch modules provided with the T-Blade 2 system, each located at the back of the blade 

chassis. Each InfiniBand switch module is based on Mellanox’s InfiniScale IV QDR InfiniBand switch silicon. A total of 36 QDR InfiniBand 4x ports 

is provided: 16 QDR ports are routed through the backplane to the compute modules, and 20 QDR ports go to the rear panel to provide additional 

flexibility when connecting centralized storage or heterogeneous nodes (see Appendix 2). Each blade module port is connected sequentially, with 

ports 1-16 connected to one switch, and ports 17-36 to the other. The intra-chassis switch topology is two independent switches, while the interchassis 

topology can be any option supported by the InfiniBand subnet manager. 

Figure 6. QDR InfiniBand Modules 

■ 

■ 

■ 

The external panel ports are implemented using QSFP connectors. 

Remote control and monitoring of the InfiniBand switch module is implemented via an I C link going through the backplane to 

the management module. 

The InfiniBand switch module is powered via backplane power connector using -48V DC input. 

The rear panel of the IB Switch module features: 

■ 

■ 

■ 

■ 

20 QSFP connectors for InfiniBand ports 

“Link initialized” and “Link active” LEDs for each port 

Two mounting handles (retention mechanism, Figure 1) 

Ventilation holes 


9

4. Management and Switch Module (MSM) 

The MSM provides the following functions: 

■ 

■ 

■ 

■ 

Blades and chassis management and monitoring 

Gigabit Ethernet switching 

Support for the global barrier and global interrupt networks 

Global clock distribution 

The MSM is a 1U-high device located at the rear side of the blade enclosure. It consists of four 

functional blocks: 

■ 

■ 

■ 

■ 

Management processor block (4.1) 

Gigabit Ethernet switches block (4.2) 

Special networks block (4.3) 

Global clock distribution block (4.4) 

For large-scale installations, the management module is connected to the specialized external global clock distribution switch to reduce OS 

jitter, improving application performance and predictability. The management module continuously collects software and hardware events, and 

consolidates them for reporting to the cluster management node and a ruggedized Black Box appliance. Combined with Clustrx management 

services, this two-stage process gives near real-time monitoring capability of 12,000 events per second using a single 2-way management node 

(see Section 8 for more information). 

4.1. Management processor block 

The management processor block features the following components: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Low voltage Intel Yonah-class CPU 

Intel 3100 MICH 

Single DDR2 memory slot populated with 2GB ECC module 

Dual-port GbE controller on a PCI-E bus (Intel 82576) 

BMC (AST2050) 

SIO chip 

40GB SSD drive 

The following interfaces are provided: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

1 GbE interface from the Intel 82576 to the auxiliary network switch 

1 GbE interface from the Intel 82576 to the management network switch 

1 10/100 Ethernet interface from the BMC sharing one of the RJ45 connectors on the rear panel. This makes the BMC and 

management module CPU available remotely in case of issues with GbE switches 

1 PCI-E interface to the special networks block (FPGA) 

1 video interface from the BMC to the D-SUB connector on the rear panel 

2 USB interfaces to the connectors on the rear panel 

1 USB interface to the LCD module (via backplane connector) 

1 RS-232 interface to the RJ45 connector on the rear panel 

1 RS-232 interface to the GbE switches management processor (Marvel 88F5181) 

I C bus interface going to the backplane (for fans, PSUs, and InfiniBand switch management). This I C bus is shared by both the 

BMC and the management module CPU, and enables the BMC to power individual PSUs up and down 

See Appendix for more information 


Figure 7. 

Management and Switch Module

4.2. Gigabit Ethernet switches block 

The Gigabit Ethernet switches block consists of two separate switches: 

■ 

■ 

Auxiliary network switch, used for generic node access (SSH, job management, etc.) 

Management network switch, used for access to BMCs 

Each of the switches contains two Marvell DX270 chips stacked together. Both switches are managed via a Marvell 88F5181 processor running 

Linux and custom firmware. 

The auxiliary network switch has the following connections: 

■ 

■ 

■ 

■ 

32 GbE links going through the backplane to the compute nodes 

1 GbE link going to the network controller of the management CPU (Intel 82576) 

1 GbE link going to the Marvel 88F5181 

2 10G uplinks going to the XFP connectors on the rear panel of the management module 

The management network switch has the following connections: 

■ 

■ 

■ 

■ 

32 GbE links routed through the backplane to the BMCs of the compute nodes 

1 GbE link routed to the network controller of the management CPU (Intel 82576) 

1 GbE link routed to the Marvel 88F5181 

2 GbE uplinks routed to the RJ45 connectors on the rear panel of the management module 

One of the RJ45 connectors on the rear panel can also be used to provide remote access to the BMC of the management module. The Marvel 

88F5181 processor provides two RS-232 interfaces to the CPU of the management module. 

See Appendix 3 for more information. 

4.3. Special networks block (FPGA) 

The special networks block is used to implement the global barrier and global interrupt networks. It is based on a Xilinx XC5VLX50T FPGA 

(XC5VLX50T-3FFG665C), and also contains an FPGA boot flash device. 

The following interfaces are provided: 

■ 

■ 

■ 

■ 

■ 

■ 

PCI-E link from the FPGA to the Intel 3100 MICH chip 

32x5 (160 total) single-wire links from the FPGA to the GPIO pins of the ICH10 chips on the compute nodes 

2 single-wire links from the FPGA to the IRQ pins of the ICH10 chips on the compute nodes (split into 32x2 links on the 

backplane) 

5 differential links from the FPGA to the RJ45 connectors on the rear panel for the global barrier network 

2 differential links from the FPGA to the RJ45 connectors on the rear panel for the global interrupt network 

4 differential links from MGTs on the FPGA to the RJ45 connector on the rear panel (reserved for future use). 

See Appendix for more information. 

4.4. Global clock distribution block 

Global clock distribution is used to adjust the frequency of CPUs and memory on all compute nodes. Clock distribution may be provided via a clock 

source on the management module, or by an external clock signal. For better signal quality, differential signaling should be used for the external 

clock; the differential clock input is converted to a single-wire on the management module. The choice of using an external or embedded clock 

source is made by the clock selector chip, controlled using the switch on the rear panel of the management module. The resulting clock signal 

goes to the backplane and then is split to provide the clock source for all compute nodes. 

See Appendix 3 for more information. 


11

4.5. Management and Switch Module Connectors 

The following connectors are present on the management module: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

The backplane signal connector provides the following signals: 

■ 32 GbE links from compute nodes to the auxiliary network switch 

■ 32 GbE links from the BMCs of the compute nodes to the management network switch 

■ I C link to the Intel 3100 MICH chip and the management node BMC 

■ 32x5 GPIO links from the FPGA 

■ interrupts links from the FPGA 

■ Сlock signal to the compute nodes 

■ USB link to the LCD module 

Backplane power connector that provides main 48V power input and 5Vsb power input to the BMC 

Rear panel connectors: 

■ 2 XFP 10GbE connectors for auxiliary network switch uplinks (ETH1 & ETH2) 

■ 2 RJ45 connectors for management network switch uplinks (ETH3 & ETH4) 

■ One port is GbE 

■ Another one is 100Mb and shared among few devices 

1 RJ45 100Mb connector for global barrier network links 1-4 (FPGA block) (SPN1) 

1 RJ45 100Mb connector for global barrier network link 5, global interrupt network links 1 and 2, and an external clock input 

(FPGA block) (SPN2) 

1 RJ45 100Mb connector for 4 differential signals, reserved for future use (FPGA block) (RIO) 

1 D-SUB connector for video output (VGA) 

2 USB connectors (USB) 

1 RJ45 connector for RS-22 port (serial console) (Serial). 

ETH1 

ETH2 

ETH3 

ETH4 

Figure 8. 

Rear panel connectors of management and switch module. 

VGA RIO 

SPN2 SPN1 


SERIAL 

USB

5. Backplane 

The backplane is used to provide power to the compute modules, the InfiniBand switch modules, and the management and switch module (MSM), 

and to provide connectivity between these system components. It also provides hot plug functionality to the compute modules. The backplane 

delivers electrical signals rated at up to 13kW of power with standard system signaling using high-frequency Infiniband QDR links (32 links @ 

40Gigatransfers). 

The backplane has very few active components; those that are included are responsible only for global clock distribution logic. It is a low-profile 

board, optimized for efficient airflow, and based on a 24-layer PCB design. The following connectors are on the rear side of the backplane: 

■ 

■ 

■ 

16 combined power/signal connectors to the compute nodes 

2 combined power/signal connectors to the InfiniBand switch modules 

2 combined power/signal connectors to the MSM 

The front side of the backplane is used for power input. The following components are placed on the front side of the backplane: 

■ 

■ 

■ 

I C connectors to the PSU’s 

I C connectors to the fan modules 

USB connector to the LCD panel 

The overall backplane geometry is shown in Figure 9. 

Figure 9. System backplane, rear view. 

Management Module Interface 

2 Infiniband Switch interfaces 


Blade module interfaces 

13

6. Blade chassis 

The T-Blade 2 chassis accommodates the following field-replaceable unit (FRU) components: 

■ 

■ 

■ 

■ 

■ 

16 hot-plug compute modules 

2 InfiniBand switch modules 

1 management and switch module 

10+2 hot-swappable 80mm fan modules 

6 hot-swappable power supplies. 

There is also a button-controlled, 4-line LCD panel connected through the backplane to the USB port on the blade management module. There is 

no main power switch on the blade enclosure front panel, nodes are powered up remotely or using microbuttons to power on/off/reset individual 

blades. 

The dimensions of the chassis: 

■ 

■ 

■ 

Height: 310mm (7U) 

Width (not including the rails and mounting brackets): 430mm, fits a standard 19” rack 

Depth: 860 mm 

The chassis can be mounted into a standard 19” rack using fixed (non-sliding) rails, and there are mounting brackets in the front side of the 

chassis that can be screwed to vertical rails in the rack. T-Platforms recommends that customers install an empty chassis into the rack cabinet, 

and then populate it with the modules. A fully populated T-Blade 2 chassis has an approximate weight of 153kg, and caution is advised when 

handling it. 

10 HS fans 

LCD panel 

2 HS fans 

6 Power supplies 

Figure 10. Front section view Figure 11. Rear section view 

Figure 12. Extraction of Compute Module 

AC power 


16 twin blade bays 

IB QDR Switches 

Figure 13. Extraction of IB Switch 

Management module

6.2. Power subsystem 

The T-Blade 2 power subsystem is comprised of six power supplies. They are connected through six PSU Interface boards to the T-Shaped 

Power Distribution board which connects the hot-pluggable power supplies (and system fans) to the system backplane. The power supplies and 

fans are the only off-the-shelf components in a T-Blade 2 system; for more information on the T-Blade 2 system assembly, please refer to the 

document “T-Blade 2 system layout and components.” 

The 2725W Lineage Power-made rectifier (model CP2725AC54Z) 

is a compact 1RU design (Figure 14). As the integral power 

distribution system of T-Blade 2, it features a front-to-back airflow 

rectifier, an RS485 interface, redundant I C links, PFC, and both 

under/over voltage and thermal protection. 

For more information about the PSU, please follow this link: 

http://www.lineagepower.com/BinaryGet.aspx?ID=f75abb6f-abcf- 

4835-b811-f3b6dabd98b5 

6.3. Cooling subsystem 

T-Blade 2 uses front-to-back airflow cooling, zoned into upper and lower airflow sections within the blade enclosure. Extensive engineering and 

design work was invested in the cooling design of T-Blade 2, with both thermal simulations and physical tests conducted to ensure sufficient 

system cooling at constant peak loads. 

Compute module cooling is provided via ten hot-swappable 80mm 

axial fans (Figure 15). The fans have connectors for DC power, speed 

monitoring, and PWM-based speed control. Each fan is placed into a 

separate hot-swappable module, and there are louvers in the chassis 

to prevent backward airflow in the case of a fan failure or removal. Fan 

modules come equipped with a small board that holds the fan controller. 

This controller is connected through the backplane via the I C interface to 

the blade management module. 

There are two additional 80mm fans below the LCD panel to provide 

cooling for the InfiniBand switch modules and the blade management 

module. Each of the two fan clusters is N+1 redundant, and uses identical 

high performance 14,000 RPM fans. To increase their operational lifetime 

they are housed at the front of the enclosure which lets them operate in a 

close-to-ambient air temperature range. 


Figure 14. System Power supplies 

Figure 15. System fans 

15

7. Other features 

7.1. Power-on procedure 

After connecting the blade system to the power source, the power supplies enter stand by mode, providing only 5Vsb power to the blade 

management module. The power supplies are then switched on using a remote connection to the BMC, or by pressing the power button. At 

this point the Management and Switch module, the InfiniBand switch modules, and the BMCs on the compute modules begin their initialization 

routines. After initialization is complete, the compute modules are turned on using a remote connection to their BMCs. 

7.2. Emergency Shutdown procedure 

The T-Blade 2 system has Emergency Shutdown functionality implemented in the management module (MSM). Depending upon the firmware 

version, either an individual blade module or entire enclosure can automatically be powered down in case of a critical management event. 

Emergency shutdown helps to avoid physical damage to system silicon. 

As a reminder, each blade can also be manually switched on, off, and reset using two microbuttons located on the rear edge of blade module PCB. 

8. Cluster Management and Monitoring 

The advanced monitoring and management capabilities of the T-Blade 2 are implemented at a clusterwide 

level. The optional Clustrx OS TP Edition and its Clustrx Watch monitoring suite are tightly integrated 

with the T-Blade system and its management module, along with standalone command line utilities for 

advanced scripting. Clustrx is Petascale-ready high performance computing OS, developed by T-Massive 

Computing, part of T-Platforms group. The Clustrx OS management subsystem requires a console system 

and single management node, and provides the following functionality: 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

Automated cluster deployment, using one console with single installation DVD media 

CLI- and GUI-based access 

Optimized such that cluster installation and tuning require only run-of-the-mill system administration 

skills 

The installation GUI reduces the time of basic installation and tuning of the cluster from installation to Linpack test readiness 

down to 2 hours (for systems with a standard topology, not including storage system installation time) 

The monitoring system tracks both the compute and infrastructure cluster subsystems (except for Ethernet and InfiniBand 

switches at the time of writing) 

The monitoring system can be configured to deliver notifications to systems administrators 

The management subsystem reacts automatically with precise remedies to correct abnormal operation, including automated 

equipment power down. 

Monitoring latency is minimized to ensure on-time delivery of corrective measures, with up to 150 events per second gathered 

from individual compute nodes (300-500 events per second targeted in the near future) 

When combined with Clustrx CNL (Compute Node Linux), the Clustrx monitoring and management suite is an exaFLOPS-ready 

operating environment covering L3 clusters with up to 12,000 compute nodes (support for L4 clusters of up to 210,000 compute 

nodes is planned) 

Supports a variety of compute node operating systems; currently validated with Clustrx OS (future support for RHEL, SUSE, 

Windows, and others is planned) 

A variety of parallel file systems are supported, including Lustre and Panasas 

It is possible to use some other popular management packages with the T-Blade 2 systems, as they are IPMI-compliant and device connectors 

can be developed upon request. 

16 T-Blade Technical Guide Book, rev. A0 .

9. Basic infrastructure requirements 

The T-Blade 2 system requires professional installation to ensure proper system operation and uptime. Installation is typically supplied as a part 

of a complete High Performance Computer system deployed by T-Platforms installation specialists. The design of the compute room and support 

infrastructure is an integral part of each T-Blade 2-based HPC system, and it is common for T-Platforms to work closely with customers on the 

design and deployment of the entire HPC system installation in a turn-key fashion. 

The following subsections cover the basic prerequisites for T-Blade 2-based HPC cluster deployment. 

9.1. Electricity 

■ 

■ 

■ 

9.2. Cooling 

■ 

■ 

■ 

■ 

80-415VAC, 5-wire, 3 Phase electrical distribution system 

Each T-Blade 2 system must be supplied with at least 11kW of uninterruptable power 

Each T-Blade 2 system must be fed by a dedicated power switch or circuit breaker rated at 32A 

Each T-Blade 2 enclosure requires 600 cubic feet per minute (CFM) of front-to-back airflow (1,019 cubic meters per hour) 

The site cooling system must be able to operate at an ambient temperature of up to 55°C, mixing in a sufficient amount of cold 

air to support the nominal operational mode of conditioning systems. 

Required cooling performance of 11KW for each T-Blade 2 system 

The inflow air temperature range should be in the range of 10°C to 30°C; an energy-efficient design rule is to keep inflow air at 

not less than 20°C, with optimal air temperature at 25°C. 

9.3. Cabinet infrastructure 

■ 

■ 

■ 

EIA 310-D (or later) compliant cabinet 

Cabinet rack depth of at least 900mm; newer cabinet designs with 1000m rack depth are recommended 

Depending upon the interconnect cabling, the total enclosure weight with attached cabling infrastructure can vary from 130kg 

to 00kg 

9.4. Floors and layout 

■ 

■ 

■ 

■ 

Antistatic raised floors rated for the fully loaded rack cabinet weight are recommended 

Recommended front isle width is not less than 1.0 meter 

Recommended back isle width is not less than 0.9 meter 

Final isle design width will vary with individual cooling design requirements 

10. Operating and file systems compatibility 

As a standards-based HPC system, T-Blade 2 generally supports many Linux distributions, including RHEL and SUSE. It is also possible to use 

the Clustrx Watch with the major Linux Distributions to enable fine-grain monitoring capabilities. For customers interested in a Windows-based 

installation, Windows HPC Server 2008 does run on a T-Blade 2 system. 

Note: The Clustrx T-Platforms Edition OS is the only distribution that currently supports the global barrier and interrupt network 

functionality of the T-Blade 2 platform. 

Currently supported parallel file systems include, but are not limited to, Lustre and Panasas. 


17

11. System specification 

T-Blade 2 Enclosure 

Form factor ■ 16 hot-plug computing modules (32 dual processor compute 

nodes) in a 7U enclosure 

■ 2 modules with 36-port QDR InfiniBand switches 

■ Dedicated management module 

Peak performance per enclosure 4.5 TFlops 

Density 384 quad-core processors (2,304 cores) per standard 19’’ 42U rack 

Peak performance per rack TFlops 

RAM ■ Up to 384GB (1.5GB per core) 

■ Up to 768GB (3GB per core) 

Power consumption per enclosure (max. configuration) 11 KW 

Performance/power consumption ratio 0.4GFLOPS/W 

Cooling design 12 hot-swap redundant cooling fans in front of the chassis 

Operating temperature 10-30°C 

Dimensions (HxWxD), mm 310х430х860mm 

System weight (fully configured) 152.6 Kg 

T-Blade 2 Compute Node 

Processor capacity/type 2 six-core Intel Xeon E5600 processors, up to 2.93GHz 

Chipset Intel 5520+ICH10 

RAM up to 24GB of DDR3-1333/1066/800 

Internal Storage Integrated MicroSD slot 

Expansion slots no 

Ethernet interface 1 GbE port 

Interconnect support integrated QDR InfiniBand 

Network interface capacity 40GB/s 

LED power, system ID 

Management integrated service processor with KVM over IP support 

Dimensions (HxWxD), mm 26х225х612 

T-Blade 2 external ports and networks 

System network QDR InfiniBand 40Gb/s 

40 external ports per enclosure 

Management (auxiliary) network 10G Ethernet, 2 external ports per enclosure 

Service network One GbE external port and one 100Mb external port per enclosure 

Global barrier network 1 uplink port to support large system topologies 

Global interrupt network 1 uplink port to support large system topologies 

18 T-Blade Technical Guide Book, rev. A0 .

Appendix 1. Compute Module diagram 

UDDR3 

CHA 

CHB 

CPU0 

QPI 

CPU1 

CHA 

CHB 

UDDR3 

UDDR3 

CHA 

CHB 

CPU2 

QPI 

CPU3 

CHA 

CHB 

UDDR3 

UDDR3 

UDDR3 

UDDR3 

UDDR3 

6.4GT/s 

CHC 

6.4GT/s 

UDDR3 

CHC 

Nehalem-EP 

Nehalem-EP 

CHC 

UDDR3 

UDDR3 

Nehalem-EP 

Nehalem-EP 

CHC 

UDDR3 

QPI 6.4GT/s 

QPI 6.4GT/s 

QPI 6.4GT/s 

QPI 6.4GT/s 

Intel 

82576 

PCIe 

PCIe 

Intel 

82576 

PCIe 

PCIe 

Tylersburg 24D 

Mellanox 

ConnectX 

QDR 

Tylersburg 24D 

Mellanox 

ConnectX 

QDR 

Infiniband 

Serdes 

ESI 

ESI 

MicroSD 

MicroSD 

USB 

AST2050 

USB 

AST2050 

ICH10 

ICH10 


Serdes 

Infiniband 

High speed connector 

to middle plane 

19

Appendix 2. Intra-chassis QDR IB links diagram 

Compute modules 

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 


Backplane signal connector Backplane signal connector 

-48V 

-48V 

16 x QDR IB 4x links Power connector 16 x QDR IB 4x links 

Power connector 

I 2 C link 

QDR IB switch 

(Mellanox InfiniScale IV) 

I 2 C link 

QDR IB switch 

(Mellanox InfiniScale IV) 

20 x QDR IB 4x links 

20 x QDR IB 4x links 

20 x QSPF connectors 

20 x QSPF connectors

Appendix 3. Management and Switch Module diagram 

RJ45 

MDI 

PORT 1 

AST2050 

BMC 

MII 

PORT 5 

88E6161 

U122 

PORT 4 

MDI 

PORT 0 

88E1116 

U39 

U98 

XFP XFP 

RGMII 

SYSTEM 

RJ45 

MDI 

CPU 

479 

Yonah 

SGMII 

88E2011 

U77 

88E2011 

U76 

IMVP-6 

For CPU 

CPU 

88F5181 

88E1112 

U103 

U40 

PORT 0 

Intel 

82576 

PORT 0 

SGMII 

P1 

PORT 1 

SMI 

PORT 25 PORT 26 

P0 

PORT 1 

PCI BUS 

FSB 400 MHZ 

DDRII MEMORY *1 

PCIE x4 

DDR2 400 MHZ 

Intel 3100 

PCIE x4 

USB*2 

PCIE x4 

SATA 

PCI 

LPC I2C 


USB 2.0 

HGX 10G 

PORT 20 

PORT 24 

PORT 26 

PORT 24 

DX270 DX270 PORT 8 DX270 HGX 10G DX270 

X24 

PORT 

X8 

PORT 

X16 

PORT 

X16 

PORT 

U18 U17 U16 U15 

PORT 25 

HGX 10G 

MICH 

PORT 0 PORT 1 PORT 0 PORT 1 PORT 4 PORT 5 PORT 0 PORT 1 

SATA*1 

SERDES 

SERDES 

BP 

BP 

FBGA 

HM 

SIO 

FWH 

MII 

(DME1737) (W83793G) 

BMC 

AST2050 

(FIRMWARE HUB) 

PCI-E 

FBGA 

LX50T 

RJ45 

Serial port *2 

D-SUB 

one to RJ45 

one to 88F8181 

RJ45 RJ45 

21

Disclaimer: 

This document is provided for informational purposes only 

and may contain inaccuracies. For up-to-date information 

please contact your T-Platforms representative. 



About T-Platforms 

Established in 2002, T-Platforms provides comprehensive HPC systems, software and services 

with customer installations consistently included on the TOP500 worldwide list of most powerful 

supercomputers. Lomonosov, a T-Platforms system installed at Moscow State University, has 

been widely recognized as the #1 ranked supercomputer in Eastern Europe and the #12 ranked 

supercomputer worldwide. 

T-Platforms is a one-stop source for companies looking for the competitive advantage of HPC 

technology, but lacking the resources necessary to fully adopt and embrace a supercomputing 

environment. The portfolio of solutions offered by T-Platforms starts with early stage analysis and 

documentation of user requirements, and progresses all the way to turnkey supercomputer center 

design. The company’s highly scalable T-Blade family of HPC systems utilize Clustrx, a robust 

operating system built specifically for HPC, that ensures next-generation scalability and fidelity to 

support the path from petascale to exascale. 

T-Platforms also delivers a unique added value with its ability to provide end-to-end modeling, 

simulation and analysis services, and deep technical talent with particular expertise in areas such as 

CFD, structural analysis, and other extreme computational disciplines, a level of support not available 

from most HPC platform suppliers. 

T-Platforms is part of T-Platforms Group which consists of T-Platforms, T-Services, T-Massive 

Computing, and T-Design, with locations in Hannover, Moscow, Kiev and Taipei. 

For more information, please visit www.t-platforms.com. 

The T-Blade 1 computational row at 60TFlops MSU Chebyshev installation 

T-Platforms 

Moscow, Russia 

Leninsky Prospect 113/1 Suite E-520 

Tel.: +7 (495) 956 54 90 

Fax: +7 (495) 956 54 15 

info@t-platforms.com 

http://www.t-platforms.com 

© T-Platforms 2010 

tPlatforms GmbH 

Woehlerstrasse 42, D-30163, 

Hannover, Germany 

Tel.: +49 (511) 203 885 40 

Tax.: +49 (511) 203 885 41 

T-Platforms, T-Platforms logo, T-Blade, Clustrx TP edition are trademarks or registered trademarks of T-Platforms, JSC. Other brand names and 

trademarks are property of their respective owners. 

This document is for informational purposes only. T-Platforms reserves the right to make changes without further notice to any products herein. The 

content provided is as is and without express or implied warranties of any kind.

T-BLADE 2 TECHNICAL GUIDEBOOK - T-Platforms

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?