Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
100
12
Percentage
Performance
with
Respect
to
Na4ve
90
80
70
60
50
40
30
20
10
TCP
o
IB
TCP
o
Eth
VM
Percentage
Performance
Rela/ve
to
Na/ve
10
8
6
4
2
TCP
o
IB
TCP
o
Eth
IB/VM
0
DGEMM
 STREAM
 Ping
Pong
 RandRing
Bandwidth
 Bandwidth
HPL
 FFTE
 PTRANS
 RandAccess
0
Ping
Pong
Latency
RandRing
Latency
Figure 9.3: Performance of the HPCC challenge benchmarks relative to the native hardware configuration
for each of the cases examined. (a) Shows the performance for all of the components of the benchmark
except the latency measurements which are shown in (b) for clarity. Note the very different vertical scale on
(b).
performance of each of the benchmarks relative to that obtained on the native host using the InfiniBand
interconnect. First we consider Figure 9.3a. The first two measurements, DGEMM and STREAM, are
measures of the floating point performance and memory bandwidth, respectively. As they do not depend on
the network, they are in this case a probe to allow us to determine the impact of virtualization on compute
performance. In both cases, the performance inside the virtual machine is about 30% slower than running on
the native hardware, which is in line with previous measurements. [46] There seems to be two reasons: the
overhead in translating the memory address instructions, and the KVM virtual environment not supporting
all the instructions on the CPU such as SSE, etc. Subsequent to this study, a later version of KVM was
installed on the Magellan testbed that supports the advanced CPU instructions in the virtual machine. The
CPU overhead was observed to be negligible after this upgrade.
Next we consider the network bandwidth measurements, of which we made two kinds, ping-pong and
random ring. Ping-pong is a measure of point-to-point bandwidth, whereas random ring consists of each
task simultaneously sending to a randomly selected partner, with the intent to induce network contention.
The ping-pong results show a decrease in performance relative to the native measurement for each of the
cases examined. The biggest drop relative to the native performance comes from using TCP over IB, with
additional decreases in performance observed for TCP over Ethernet and virtualization. The random ring
bandwidth clearly shows the lack of capability of the Ethernet connection to cope with significant amounts of
network contention. In this case, in contrast to the ping-pong case, the virtualization does not significantly
further decrease the performance. The network latency measurements, both ping-pong and random ring,
(Figure 9.3b) show qualitatively different trends to the bandwidths, as well as a significantly larger overall
performance decrease, even at the TCP over InfiniBand level. The principal increase in latency occurs when
switching to TCP over IB, then a further increase occurs from within the virtual machine. The TCP over
IB and TCP over Ethernet latencies are approximately the same, which indicates the latency is primarily
a function of the transport mechanism, as one would expect. The performance of the HPL benchmark is
primarily a function of the DGEMM performance and the ping-pong bandwidth. The exact ratio depends
upon the balance point of the machine, thus as the network capabilities become relatively worse, the HPL
performance decreases. Both FFT and PTRANS are sensitive to random ring bandwidth and mirror the
trends observed there. Random access is sensitive to network latency primarily and in a similar manner
mimics the trends observed previously in the network latency measurements.
The performance of our application benchmarks relative to the native using the various machine config-
53