Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
important design goal. The testbed was architected for flexibility and to support research, and the hardware
was chosen to be similar to the high-end hardware in HPC clusters, thus catering to scientific applications.
Argonne. The Magellan testbed at Argonne includes computational, storage, and networking infrastructure.
There is a total of 504 iDataplex nodes as described above. In addition to the core compute cloud, Argonne’s
Magellan has three types of hardware that one might expect to see within a typical HPC cluster: Active
Storage servers, Big Memory servers, and GPU servers, all connected with QDR InfiniBand provided by two
large, 648-port Mellanox switches.
There are 200 Active Storage servers, each with dual Intel Nehalem quad-core processors, 24 GB of
memory, 8x500 GB SATA drives, 4x50 GB SSD, and a QDR InfiniBand adapter. The SSD was added to
facilitate exploration of performance improvements for both multi-tiered storage architectures and Hadoop.
These servers are also being used to support the ANI research projects.
Accelerators and GPUs are becoming increasingly important in the HPC space. Virtualization of heterogeneous
resources such as these is still a significant challenge. To support research in this area, Argonne’s
testbed was outfitted with 133 GPU servers, each with dual 6 GB NVIDIA Fermi GPU, dual 8-core AMD
Opteron processors, 24 GB memory, 2x500 GB local disks, and a QDR InfiniBand adapter. Virtualization
work on this hardware was outside of the scope of the Magellan project proper; see [48, 86] for details.
Several projects were interested in exploring the use of machines with large amounts of memory. To
support them, 15 Big Memory servers were added to the Argonne testbed. Each Big Memory server was
configured with 1 TB of memory, along with 4 Intel Nehalem quad-core processors, 2x500 GB local disks,
and a QDR InfiniBand adapter. These were heavily used by the genome sequencing projects, allowing them
to load their full databases into memory.
Argonne also expanded the global archival and backup tape storage system shared between a number of
divisions at Argonne. Tape movers, drives, and tapes were added to the system to increase the tape storage
to support the Magellan projects.
Finally, there is 160 terabytes (TB) of global storage. In total, the system has over 150 TF of peak floating
point performance with 8,240 cores, 42 TB of memory, 1.4 PB of storage space, and a single 10-gigabit (Gb)
external network connection. A summary of the configuration for each node is shown in Table 5.1.
Table 5.1: Node configuration on the Argonne Magellan cluster. There are 852 nodes in the cluster.
Feature
Description
Processor
Dual 2.66 GHz Intel Quad-core Nehalem (704 nodes),
Quad Intel Nehalem Quad-core Nehalem (15 nodes),
Dual 2 GHz 8-core AMD Opteron (133 GPU nodes)
GPUs
NVidia Fermi 2070 6 GB cards (133 nodes, 2 per node)
Memory
Local Disk
High Performance Network
Ethernet Network
Management
24 GB (837 nodes), 1 TB (15 nodes)
1 TB SATA (504 nodes),
8x500 GB SATA and 4x50 GB SSD (200 nodes),
2x500 GB SATA (148 nodes)
40 Gb InfiniBand (4X QDR)
On-board 1 Gb
IPMI
NERSC. The NERSC Magellan testbed also provided a combination of computing, storage, and networking
resources. There are 720 iDataplex nodes as described earlier; 360 of them have local 1 TB drives. In addition,
20x400GB SSDs from Virident were acquired to explore the impact and performance of the SSD technology.
The total system has over 60 TF of peak floating point performance. A summary of the configuration for
each node is shown in Table 5.2.
The InfiniBand fabric was built using InfiniBand switches from Voltaire. Since the system is too large
to fit within a single switch chassis, multiple switches are used, and they are connected together via 12x
24