Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
cloud environments. Similarly, I/O intensive applications take a substantial hit when run inside virtual
environments.
• Cloud programming models such as MapReduce and the resulting ecosystem show promise for addressing
the needs of many data-intensive and high-throughput scientific applications. However, current
tools have gaps for scientific applications. The MapReduce model emphasizes the data locality
and fault tolerance that are important in large systems. Thus there is a need for tools that provide
MapReduce implementations which are tuned for scientific applications.
• Current cloud tools do not provide an out-of-box solution to address application needs. There is
significant design and programming required to manage the data and workflows in these environments.
Virtual machine environments require users to configure and create their software images with all
necessary packages. Scientific groups will also need to maintain these images with security patches and
application updates. There exist a number of performance, reliability, and portability challenges with
cloud images that users must consider carefully. There are limited user-side tools available today to
manage cloud environments.
• One way clouds can achieve cost efficiency is though consolidation of resources and higher average
utilization. DOE Centers already consolidate workloads from different scientific domains and have
high average utilization, typically greater than 90%. Even with conservative cost analysis, we show
how two DOE Centers are more cost-efficient than private clouds.
As noted in the final point, a key benefit of clouds is the consolidation of resources. This typically leads to
higher utilization, improved operational efficiency, and lower acquisition cost through increased purchasing
power. If one looks across the scientific computing landscape within DOE there are variety of models for how
scientists access computing resources. These cover the full range of consolidation and utilization scales. At
one end of the spectrum is the small group or departmental cluster. These systems are often under-utilized
and represent the best opportunity to achieve better efficiency. Many of the DOE National Laboratories
have already taken efforts to consolidate these resources into institutional clusters operating under a variety
of business models (institutionally funded, buy-in/condo, etc.). In many ways, these systems act as private
clouds tuned for scientific applications and effectively achieve many of the benefits of cloud computing. DOE
HPC centers provide the next level consolidation, since these facilities serve users from many institutions
and scientific domains. This level of consolidation is one reason why many of the DOE HPC centers operate
at high levels of utilization.
Clouds have certain features that are attractive for scientific groups needing support for on-demand access
to resources, sudden surges in resource needs, customized environments, periodic predictable resource needs
(e.g., monthly processing of genome data, nightly processing of telescope data), or unpredictable events such
as computing for disaster recovery. Cloud services essentially provide a differentiated service model that
can cater to these diverse needs, allowing users to get a virtual private cluster with a certain guaranteed
level of service. Clouds are also attractive to high-throughput and data-intensive workloads that do not fit
in current-day scheduling and allocation policies at supercomputing centers. DOE labs and centers should
consider adopting and integrating features of cloud computing into their operations in order to support
more diverse workloads and further enable scientific discovery. This includes mechanisms to support more
customized environments, but also methods of providing more on-demand access to cycles. This could be
achieved by: a) maintaining idle hardware at additional costs to satisfy potential future requests, b) sharing
cores/nodes typically at a performance cost to the user, and c) utilizing different scheduling policies such
as preemption. Providing these capabilities would address many of the motivations that lead scientists
to consider cloud computing while still preserving the benefits of typical HPC systems which are already
optimized for scientific applications.
Cloud computing is essentially a business model that emphasizes on-demand access to resources and
cost-savings through consolidation of resources. Overall, whether cloud computing is suitable for a certain
application, science group, or community depends on a number of factors. This was noted in NIST’s draft
document on Cloud Computing, “Cloud Computing Synopsis and Recommendations”, which stated,
130