Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
• The Argonne Laboratory Computing Resource Center (LCRC), working with Magellan project staff,
was able to develop a secure method to extend the production HPC compute cluster into the ALCF
Magellan testbed, allowing jobs to utilize the additional resources easily, while accessing the LCRC
storage and running within the same computing environment.
• The STAR experiment at the Relativistic Heavy Ion Collider (RHIC) used Magellan for real-time data
processing. The goal of this particular analysis is to sift through collisions searching for the “missing
spin.”
• Biologists used the ALCF Magellan cloud to quickly analyze strains suspected in the E. coli outbreak
in Europe in summer 2011.
• Magellan was honored with the HPCwire Readers’ Choice Award for “Best Use of HPC in the Cloud”
in 2010 and 2011.
• Magellan benchmarking work won the “Best Paper Award” at both CloudCom 2010 and DataCloud
2011.
• Magellan played a critical role in demonstrating the capabilities of the 100 Gbps network deployed
by the Advanced Networking Initiative (ANI). Magellan provided storage and compute resources that
were used by many of the demonstrations performed during SC11. The demonstrations were typically
able to achieve rates of 80-95 Gbps.
1.4 Approach
Our approach has been to evaluate various dimensions of cloud computing and associated technologies. We
adopted an application-driven approach where we assessed the specific needs of scientific communities while
making key decisions in the project.
A distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility (ALCF)
and the National Energy Research Scientific Computing Center (NERSC). The testbed consists of IBM
iDataPlex servers and a mix of special servers, including Active Storage, Big Memory, and GPU servers
connected through an Infiniband fabric. The testbed also has a mix of storage options, including both
distributed and global disk storage, archival storage, and two classes of flash storage. The system provides
both a high-bandwidth, low-latency quad-data rate InfiniBand network as well as a commodity Gigabit
Ethernet network. This configuration is different from a typical cloud infrastructure but is more suitable for
the needs of scientific applications. For example, InfiniBand may be unusual in a typical commercial cloud
offering; however, it allowed Magellan staff to investigate a range of performance points and measure the
impact on application performance.
The software stack on the Magellan testbed was diverse and flexible to allow Magellan staff and users to
explore a variety of models with the testbed. Software on the testbed included multiple private cloud software
solutions, a traditional batch queue with workload monitoring, and Hadoop. The software stacks were
evaluated for usability, ease of administration, ability to enforce DOE security policies, stability, performance,
and reliability.
We conducted a thorough benchmarking and workload analysis evaluating various aspects of cloud computing,
including the communication protocols and I/O using micro as well as application benchmarks. We
also provided a cost analysis to understand the cost effectiveness of clouds. The main goal of the project was
to advance the understanding of how private cloud software operates and identify its gaps and limitations.
However, for the sake of completeness, we performed a comparison of the performance and cost against
commercial services as well.
We evaluated the use of MapReduce and Hadoop for various workloads in order to understand the
applicability of the model for scientific pipelines as well as to understand the various performance tradeoffs.
We worked closely with scientific groups to identify the intricacies of application design and associated
3