Magellan Final Report - Office of Science - U.S. Department of Energy

Recommendations
The Magellan project has evaluated cloud computing models and various associated technologies and explored
their potential role in addressing the computing needs of scientists funded by the Department of Energy’s
Office of Science. Our findings highlight both the potential value and the challenges of exploiting cloud
computing for DOE SC applications. Here we summarize our recommendations for DOE SC, DOE resource
providers, application scientists, and tool developers. A number of these recommendations do not fit within
current scope and budgets, and additional funding beyond currently funded DOE projects might be required.
DOE Office of Science. The findings of the Magellan project demonstrate some potential benefits of
cloud computing for meeting the computing needs of the DOE SC scientific community. Cloud features such
as customized environments and the MapReduce programming model help in addressing the needs of some
current users of DOE resources as well as the needs of some new scientific applications that need to scale
up from current environments. The findings also show that existing DOE HPC centers are cost effective
compared with commercial providers. Consequently, DOE should work with the DOE HPC centers and
DOE resource providers to ensure that the expanding needs of the scientific community are being met. If
cases emerge where cloud models are deemed necessary in order to fully address these needs, we recommend
that DOE should first consider a private cloud computing strategy. This approach can provide many of the
benefits of cloud environments while being more cost effective, allowing for more optimized offerings, and
better addressing the security, performance, and data management challenges.
DOE Resource Providers. There is a rising class of applications that do not fit the current traditional
model of large-scale tightly coupled applications that run in supercomputing centers, yet have increasing
needs for computation and storage resources. In addition, various scientific communities need custom
software environments or shared virtual clusters to meet specific collaboration or workload requirements.
Additionally, clouds provide mechanisms and tools for high-throughput and data-intensive applications that
have large-scale resource requirements. The cloud model provides solutions to address some of these requirements,
but these can be met by DOE resource providers as well. We make some specific recommendations
to DOE resource providers for providing these features. Resource providers include the large ASCR facilities
such as the Leadership Computing Facilities and NERSC; institutional resources operated by many of the
labs; and the computing resources deployed in conjunction with many large user facilities funded by other
science program offices.
1. DOE resource providers should investigate mechanisms to support more diverse workloads such as highthroughput
and data-intensive workloads that increasingly require more compute and storage resources.
Specifically, resource providers should consider more flexible queueing policies for these loosely coupled
computational workloads.
2. A number of user communities require a different resource provisioning model where they need exclusive
access to a pre-determined amount of resources for a fixed period of time due to increased computational
load. Our experience with the Material Genomes project, the Joint Genome Institute, and the analysis
of the German E. coli strains demonstrates that on-demand resource provisioning can play a valuable
vi