Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
3.1 Collaborations and Synergistic Activities
Magellan research was conducted as a collaboration across both ALCF and NERSC, as well as leveraging
the expertise of other collaborators and projects. At a high-level, Magellan collaboration activities can be
classified into three categories:
• Magellan Core Research. Magellan staff actively performed the research necessary to answer the
research questions outlined in the report with respect to understanding the applicability of cloud
computing for scientific applications.
• Synergistic Research Activities. Magellan staff participated in a number of key collaborations in
research related to cloud computing.
• Magellan Resource Enabling User Research. Magellan resources were used extensively by scientific
users in their simulations and data analysis. Magellan staff often provided key user support in
facilitating this research.
We outline the key Magellan collaborations in this section. Additional collaborations are highlighted
throughout the report as well. Specifically the application case studies are highlighted in Chapter 11.
Cloud Benchmarking. The benchmarking of commercial cloud platforms was performed in collaboration
with the IT department at Lawrence Berkeley National Laboratory (LBNL), which manages some of
the mid- range computing clusters for scientific users; the Advanced Technology Group at NERSC, which
studies the requirements of current and emerging NERSC applications to find hardware design choices and
programming models; and the Advanced Computing for Science (ACS) Department in the Computational
Research Division (CRD) at LBNL, which seeks to create software and tools to enable science on diverse
resource platforms. Access to commercial cloud platforms was also possible through collaboration with the
IT division and the University of California Center for Information Technology Research in the Interest of
Society (CITRIS), which had existing contracts with different providers. This early benchmarking effort
resulted in a publication at CloudCom 2010 [46] that was awarded Best Paper.
MapReduce/Hadoop Evaluation. Scientists are struggling with a tsunami of data across many domains.
Emerging sensor networks, more capable instruments, and ever increasing simulation scales are generating
data at a rate that exceeds our ability to effectively manage, curate, analyze, and share it. This is exacerbated
by the limited understanding and expertise on the hardware resources and software infrastructure required
for handling these diverse data volumes. A project funded through the Laboratory Directed Research and
Development (LDRD) program at Lawrence Berkeley Laboratory is looking at the role that many new,
potentially disruptive, technologies can play in accelerating discovery. Magellan resources were used for this
evaluation. Additionally, staff worked closely with a summer student on the project evaluating the specific
application patterns that might benefit from Hadoop. Magellan staff also worked closely with the Grid Computing
Research Laboratory at SUNY, Binghamton in a comparative benchmarking study of MapReduce
implementations and an alternate implementation of MapReduce that can work in HPC environments. This
collaboration resulted in two publications in Grid 2011 [25, 24].
Collaboration with Joint Genome Institute. The Magellan project leveraged the partnership between
the Joint Genome Institute (JGI) and NERSC to benchmark the IMG and MGM pipelines on a variety of
platforms. Project personnel were also involved in pilot projects for the Systems Biology Knowledge Base.
They provided expertise with technologies such as HBASE which were useful in guiding the deployments
on Magellan. Magellan resources were also deployed for use by JGI as a proof-of-concept of Hardware-as-a-
Service. JGI also made extensive use of the Hadoop cluster.
10