Magellan Final Report - Office of Science - U.S. Department of Energy

Chapter 11
Application Experiences
A diverse set of scientific applications have used the Magellan cloud testbed, resulting in significant scientific
discoveries while helping to evaluate the use of cloud computing for science. Early adopters of cloud computing
have typically been scientific applications that are largely data parallel, since they are good candidates
for cloud computing. These applications are primarily throughput-oriented (i.e., there is no tight coupling
between tasks); the data requirements can be large but are well constrained; and some of these applications
have complex software pipelines, and thus can benefit from customized environments. Cloud computing
systems typically provide greater flexibility to customize the environment when compared with traditional
supercomputers and shared clusters, so these applications are particularly well suited to clouds.
We outline select scientific case studies that have used Magellan successfully over the course of the project.
The diversity of Magellan testbed setups enabled scientific users to explore a variety of cloud computing technologies
and services, including bare-metal provisioning, virtual machines (VMs), and Hadoop. The Magellan
staff worked closely with the users to reduce the learning curve associated with these new technologies. In
chapter 9 we outlined some of the performance studies from our applications. In this chapter, we focus on
the advantages of using cloud environments, and we identify gaps and challenges in current cloud solutions.
We discuss case studies in three areas of cloud usage: (a) bare metal provisioning (Section 11.1), (b)
virtualized resources (Section 11.2), and c) Hadoop (Section 11.3). We discuss the suitability of cloud models
for various applications and how design decisions were implemented in response to cloud characteristics.
In section 11.4 we discuss gaps and challenges in using current cloud solutions for scientific applications,
including feedback that we collected from the user community through a final survey.
11.1 Bare-Metal Provisioning Case Studies
Many scientific groups need dedicated access to computing resources for a specific period of time, often
requiring custom software environments. Virtual environments have been popularized by cloud computing
technologies as a way to address these needs. However, virtual environments do not always work for scientific
applications due to performance considerations (Chapter 9) or the difficulties with creating and maintaining
images (Section 11.4). In this section, we highlight a set of diverse use cases that illustrate alternate
provisioning models for applications that need on-demand access to resources.
11.1.1 JGI Hardware Provisioning
Hardware as a Service (HaaS) offers one potential model for supporting the DOE’s scientific computing
needs. This model was explored early with Magellan when a facility issue at the Joint Genome Institute
(JGI) led to their need for rapid access to additional resources. NERSC responded to this by allocating 120
nodes of Magellan to JGI.
102