Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Figure 9.25 shows some preliminary analysis of the data collected on the NERSC Magellan system over
a period of seven months, for over 220K runs of several classes of HPC applications. The Magellan batch
queue was available to NERSC users; however, it is not clear how the workload on Magellan compares with
the general workload on NERSC systems. Figure 9.25a shows the scatter plot of number of nodes used by
a job vs duration of the job. Most of the jobs in the period that we profiled used fewer than 256 cores. The
total wall clock time used by the job on all processors did tend to vary from a few hundred to a few thousand
hours. Figure 9.25b shows a similar trend, but the X-axis shows the number of processes instead of tasks.
Figure 9.25c shows the scatter plot of the jobs percentage of I/O vs percentage of communication. The
jobs closer to the origin of the graph (i.e., less communication and less I/O) are likely to do well in cloud
environments. The graph also seems to indicate that there are clear clusters of I/O vs communication.
Figure 9.25d shows the percentage of CPUs vs percentage of I/O. A number of these applications tend to be
CPU intensive, thus clustering in the > 50% part of the X-axis. Applications that are closer to the X-axis
are likely to do better in virtual environments. Additional analysis will need to classify jobs by user and
application to understand these patterns better and also understand the volume of I/O and communication.
The IPM data is a rich source of workload data that helps us understand the types of applications that
run well on systems such as Magellan.
9.7 Discussion
Virtualized cloud computing environments promise to be useful for scientific applications that need customized
software environments. Virtualization is known to have a significant performance impact for scientific
applications. We analyzed the performance of a number of different interconnect technologies and I/O
performance to understand the performance trade-offs in this space and gain an understanding of what a
private cloud configured for scientific computing should look like.
9.7.1 Interconnect
Our benchmarking approach enabled us to understand the impact of the layers in the hierarchy of protocols.
Although performance is highly dependent on the type of workload, we see that the performance decrease
from virtualization is largely due to overheads in the communication, i.e., the reliance of the virtual machines
on TCP over Ethernet as the communication mechanism.
Our results show that while the difference between the interconnects is small at lower core counts, the
impact is significant even at the mid-range size of problems of 256 cores to 1024 cores. While the bandwidth
is slightly impacted at higher concurrency, the latency takes a much higher hit. Scientific applications tend to
run at higher concurrencies to achieve the best time to solution. Thus to serve the needs of these applications,
we need good interconnects.
In addition to the network fabric and the protocol, the virtualization software stack imposes an additional
overhead to the performance of the applications. This overhead also is impacted by the communication
pattern of the application and the concurrency of the run.
The higher bound on the performance on virtual machines today is what is achievable with TCP over
Ethernet clusters. Our experiments show that the availability of InfiniBand interconnects on virtual machines
would boost the performance of scientific applications by reducing the communication overheads.
Scientific applications would be able to leverage the benefits of virtualization and get a significant increase
in performance if future environments had InfiniBand available for the communication network.
Our performance evaluation illustrates the importance of a high-bandwidth, low-latency network. However,
combining virtual machines and an InfiniBand (IB) network presents several challenges. Possible
methods for extending the InfiniBand network inside a virtual machine include routing IP over InfiniBand
from the Hypervisor to the virtual machine; bridging Ethernet over InfiniBand through the Linux bridge to
the virtual machine; and full virtualization of the NIC. We will briefly discuss these options, including the
benefits and trade-offs, and the current issues.
80