Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
as Eucalyptus and Hadoop, require us to study the impact of cloud computing on scientific environments.
The Magellan project has focused on understanding the unique requirements of DOE science applications
and the role cloud computing can play in scientific communities. However the identified gaps and challenges
apply more broadly to scientific applications using cloud environments.
1.1 Magellan Goals
The goal of the Magellan project is to investigate how the cloud computing business model can be used to
serve the needs of DOE Office of Science applications. Specifically, Magellan was charged with answering
the following research questions:
• Are the open source cloud software stacks ready for DOE HPC science
• Can DOE cyber security requirements be met within a cloud
• Are the new cloud programming models useful for scientific computing
• Can DOE HPC applications run efficiently in the cloud What applications are suitable for clouds
• How usable are cloud environments for scientific applications
• When is it cost effective to run DOE HPC science in a cloud
In this report, we summarize our findings and recommendations based on the experiences over the course
of the project in addressing the above research questions.
1.2 NIST Cloud Definition
The term “cloud computing” has been used to refer to different concepts, models, and services over the
last few years. For the rest of this report we use the definition for cloud computing provided by the
National Institute of Standards and Technology (NIST), which defines cloud computing as a model for
enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g.,
networks, servers, storage, applications, and services) that can be rapidly provisioned and released with
minimal management effort or service provider interaction [63].
High performance computing (HPC) centers such as those funded by the Department of Energy provide
a number of these key features, including resource pooling, broad network access, and measured services
based on user allocations. However, there is limited support for rapid elasticity or on-demand self-service in
today’s HPC centers.
1.3 Impact
The Magellan project has made an impact in several different areas. Magellan resources were available to end
users in several different configurations, including virtual machines, Hadoop, and traditional batch queues.
Users from all the DOE SC offices have taken advantage of the systems. Some key areas where Magellan
has made an impact in the last two years:
• The Magellan project is the first to conduct an exhaustive evaluation of the use of cloud computing
for science. This has resulted in a number of publications in leading computer science conferences and
workshops.
• A facility problem at the Joint Genome Institute led to a pressing need for backup computing hardware
to maintain its production sequencing operations. NERSC partnered with ESnet and was able to
provision Magellan hardware in a Hardware as a Service (HaaS) model to help the Institute meet its
demands.
2