Magellan Final Report - Office of Science - U.S. Department of Energy

Chapter 1
Overview
Cloud computing has served the needs of enterprise web applications for the last few years. The term “cloud
computing” has been used to refer to a number of different concepts (e.g., MapReduce, public clouds, private
clouds, etc.), technologies (e.g., virtualization, Apache Hadoop), and service models (e.g., Infrastructureas-a-Service
[IaaS], Platform-as-a-Service [PaaS], Software-as-a-Service[SaaS]). Clouds have been shown to
provide a number of key benefits including cost savings, rapid elasticity, ease of use, and reliability. Cloud
computing has been particularly successful with customers lacking significant IT infrastructure or customers
who have quickly outgrown their existing capacity.
The open-ended nature of scientific exploration and the increasing role of computing in performing science
has resulted in a growing need for computing resources. There has been an increasing interest over the last few
years in evaluating the use of cloud computing to address these demands. In addition, there are a number of
key features of cloud environments that are attractive to some scientific applications. For example, a number
of scientific applications have specific software requirements including OS version dependencies, compilers and
libraries, and the users require the flexibility associated with custom software environments that virtualized
environments can provide. An example of this is the Supernova Factory, which relies on large data volumes
for the supernova search and has a code base which consists of a large number of custom modules. The
complexity of the pipeline necessitates having specific library and OS versions. Virtualized environments also
promise to provide a portable container that will enable scientists to share an environment with collaborators.
For example, the ATLAS experiment, a particle physics experiment at the Large Hadron Collider at CERN,
is investigating the use of virtual machine images for distribution of all required software [10]. Similarly, the
MapReduce model holds promise for data-intensive applications. Thus, cloud computing models promise to
be an avenue to address new categories of scientific applications, including data-intensive science applications,
on-demand/surge computing, and applications that require customized software environments. A number of
groups in the scientific community have investigated and tracked how the cloud software and business model
might impact the services offered to the scientific community. However, there is a limited understanding
of how to operate and use clouds, how to port scientific workflows, and how to determine the cost/benefit
trade-offs of clouds, etc. for scientific applications.
The Magellan project was funded by the American Recovery and Reinvestment Act to investigate the
applicability of cloud computing for the Department of Energy’s Office of Science (DOE SC) applications.
Magellan is a joint project at the Argonne Leadership Computing Facility (ALCF) and the National Energy
Research Scientific Computing Center (NERSC). Over the last two years we have evaluated various dimensions
of clouds—cloud models such as Infrastructure as a Service (IaaS) and Platform as a Service(PaaS),
virtual software stacks, MapReduce and its open source implementation (Hadoop). We evaluated these on
various criteria including stability, manageability, and security from a resource provider perspective, and
performance and usability from an end-user perspective.
Cloud computing has similarities with other distributed computing models such as grid and utility computing.
However, the use of virtualization technology, the MapReduce programming model, and tools such
1