Magellan Final Report - Office of Science - U.S. Department of Energy

Executive Summary
The goal of Magellan, a project funded through the U.S. Department of Energy (DOE) Office of Advanced
Scientific Computing Research (ASCR), was to investigate the potential role of cloud computing in addressing
the computing needs for the DOE Office of Science (SC), particularly related to serving the needs of midrange
computing and future data-intensive computing workloads. A set of research questions was formed to
probe various aspects of cloud computing from performance, usability, and cost. To address these questions,
a distributed testbed infrastructure was deployed at the Argonne Leadership Computing Facility (ALCF)
and the National Energy Research Scientific Computing Center (NERSC). The testbed was designed to be
flexible and capable enough to explore a variety of computing models and hardware design points in order
to understand the impact for various scientific applications. During the project, the testbed also served as
a valuable resource to application scientists. Applications from a diverse set of projects such as MG-RAST
(a metagenomics analysis server), the Joint Genome Institute, the STAR experiment at the Relativistic
Heavy Ion Collider, and the Laser Interferometer Gravitational Wave Observatory (LIGO), were used by
the Magellan project for benchmarking within the cloud, but the project teams were also able to accomplish
important production science utilizing the Magellan cloud resources.
Cloud computing has garnered significant attention from both industry and research scientists as it has
emerged as a potential model to address a broad array of computing needs and requirements such as custom
software environments and increased utilization among others. Cloud services, both private and public, have
demonstrated the ability to provide a scalable set of services that can be easily and cost-effectively utilized
to tackle various enterprise and web workloads. These benefits are a direct result of the definition of cloud
computing: on-demand self-service resources that are pooled, can be accessed via a network, and can be
elastically adjusted by the user. The pooling of resources across a large user base enables economies of scale,
while the ability to easily provision and elastically expand the resources provides flexible capabilities.
Following the Executive Summary we summarize the key findings and recommendations of the project.
Greater detail is provided in the body of the report. Here we briefly summarize some of the high-level
findings from the study.
• Cloud approaches provide many advantages, including customized environments that enable users to
bring their own software stack and try out new computing environments without significant administration
overhead, the ability to quickly surge resources to address larger problems, and the advantages
that come from increased economies of scale. Virtualization is the primary strategy of providing these
capabilities. Our experience working with application scientists using the cloud demonstrated the
power of virtualization to enable fully customized environments and flexible resource management,
and their potential value to scientists.
• Cloud computing can require significant initial effort and skills in order to port applications to these
new models. This is also true for some of the emerging programming models used in cloud computing.
Scientists should consider this upfront investment in any economic analysis when deciding whether to
move to the cloud.
• Significant gaps and challenges exist in the areas of managing virtual environments, workflows, data,
cyber-security, and others. Further research and development is needed to ensure that scientists can
i