Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
9.1.2 Machines
All results were obtained during normal, multi-user, production periods on all machines unless specified
otherwise.
Magellan. All the experiments described in this section were performed on the Magellan system at NERSC.
Chapter 5 described the testbed in detail; here we summarize the key characteristics relevant to understanding
the benchmarking results. Magellan is a 720-node IBM iDataPlex cluster. Each node has two quad-core
Intel Nehalem processors running at 2.67 Ghz, 24 GB of RAM, and two network connections: a single Quad
Data Rate (QDR) InfiniBand network connection and a GiB Ethernet connector. The InfiniBand network
is locally a fat tree with a global 2-D mesh. Our 10G network is based on the Juniper Qfabric Switching
System with four Q/F nodes, two Q/F Interconnects, and Junos 11.3.
Some early experiments were performed on Carver, a machine identical in hardware to the Magellan
cloud testbed at NERSC. Carver is a 400-node IBM iDataPlex cluster also located at NERSC, and the
nodes have an identical configuration to Magellan. All of the codes compiled on Carver used version 10.0 of
the Portland Group suite and version 1.4.1 of OpenMPI.
The virtual machine environment on Magellan at NERSC is based on Eucalyptus 1.6 (studies in Section
9.1.4) and 2.0 (studies in Section 9.1.5), an open source software platform that allows organizations
to leverage existing Linux-based infrastructure to create private clouds. Our Eucalyptus installation uses
Kernel-based Virtual Machines(KVM) as a hypervisor. We modified Eucalyptus to use Virtio for disk access.
We use the KVM option for the emulated e1000 nic for the network. All codes were compiled with the Intel
compiler version 11.1 and used version 1.4.2 of OpenMPI.
All tests were performed on an instance type configured with 8 CPUs/20 G memory/20 G disk. The guest
OS is CentOS release 5.5 (Linux kernel 2.6.28-11). We set up a virtual cluster where the head node mounts
a block store volume that has all the application binaries and data. The block store volume is mounted on
the other virtual machines via NFS. All the virtual machine communication traffic goes over the Ethernet
network.
Amazon EC2. Amazon EC2 is a virtual computing environment that provides a web services API for
launching and managing virtual machine instances. Amazon provides a number of different instance types
that have varying performance characteristics. CPU capacity is defined in terms of an abstract Amazon EC2
Compute Unit. One EC2 Compute Unit is approximately equivalent to a 1.0–1.2 GHz 2007 Opteron or 2007
Xeon processor. For our tests we used the m1.large instance type, which has four EC2 Compute Units, two
virtual cores with two EC2 Compute Units each, and 7.5 GB of memory. The nodes are connected with
Gigabit Ethernet. To ensure consistency, the binaries compiled on Lawrencium were used on EC2. All of
the nodes are located in the US East region in the same availability zone. Amazon offers no guarantees on
proximity of nodes allocated together, and there is significant variability in latency between nodes.
In addition to the variability in network latency, we also observed variability in the underlying hardware
on which the virtual machines are running. By examining /proc/cpuinfo, we are able to identify the actual
CPU type of the non-virtualized hardware. In our test runs, we identified three different CPUs: the Intel
Xeon E5430 2.66 GHz quad-core processor, the AMD Opteron 270 2.0 GHz dual-core processor, and the
AMD Opteron 2218 HE 2.6 GHz dual-core processor. Amazon does not provide a mechanism to control
which underlying hardware virtual machines are instantiated on. Thus, we almost always end up with a
virtual cluster running on a heterogeneous set of processors. This heterogeneity also meant we were unable
to use any of the processor-specific compiler options.
During the course of the project Amazon started providing access to Cluster Compute (CC) instances
in addition to the other types of instances previously available. These new CC instances are significantly
different from the other types in terms of performance characteristics. This is the first instance type which
guarantees the hardware architecture of the nodes and reduces the chance for any performance variation
due to varying hardware types. Amazon has defined the specification on these CC instances to have dual
socket Intel Nehelam CPUs at 2.97 GHz and 23 GB of memory per node. The nodes are connected by
48