Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
9.2 OpenStack Benchmarking
All of the tests in this section were run on the ALCF OpenStack virtual machines or virtual clusters, and
a corresponding number of IBM x3650 servers were used for the bare-metal (or ’raw’) comparison. This
allowed us to determine the precise overhead cost of running in an open-stack virtualized environment. All
software between the virtual and raw hardware clusters was identical, and MPICH2 v1.4 was used by the
applications.
9.2.1 SPEC CPU
SPEC-CPU2006 is an intensive benchmarking suite made available by the Standard Performance Evaluation
Corporation [78]. It is designed to stress the CPU, memory, and compiler of a given system for the purpose
of acquiring general performance numbers. A total of 29 benchmarks were run in two passes on both the
virtual machine and on raw hardware; the first pass used integer operations and the second pass used floating
point operations. Only the time-to-solution is reported below.
(a) Integer
(b) Floating point
Figure 9.10: SPEC CPU benchmark results.
At first glance, the results in Figure 9.10 seem to show greater performance for the virtualized machine; the
most notable exception being MCF. This particular benchmark is noted as having been modified for increased
memory cache performance. Two other benchmarks that show better performance on the raw hardware are
MILC and CactusADM; these are also known as particularly memory-cache friendly applications. This leads
us to believe that KVM hides the performance penalty associated with cache misses. As such, CPU cycles
spent idle while an application replenishes its L2 cache are only added to the total runtime on hardware.
Further investigation of the impact of the memory cache was out of the scope of this project. Other studies
have shown the effects of memory performance in VMs[43]
9.2.2 MILC
As an example of a primarily compute-bound scientific application, the MIMD Lattice Computation[61] application
was chosen as the second benchmark. MIMD Lattice Computation is a commonly used application
for parallel simulation of four-dimensional lattice gauge theory. Two passes were run on each cluster, using
one core per node and 8 cores per node, respectively.
The results in Figure 9.11 show an almost six-fold increase in runtime for the one-core-per-node run, which
is almost certainly a result of poor MPI performance due to high latency (quantified below in the Phloem
SQMR discussion). As for the 256-core run, while the time-to-solution improves for the raw hardware cluster,
it increases by a factor of five for the virtual cluster. Again, this is a result of poor MPI performance due to
high latency, an exponential increase in the number of core-to-core messages being passed, and contention
among processes for network resources. This is further exacerbated by the OpenStack network model which
requires a single network service to mediate all internal traffic.
60