Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
of the pipeline makes it necessary to have specific library and OS versions and ends up being a barrier to
making use of other large resources. The Supernova Factory project finds cloud computing attractive due
to the ability to control software environments and the ability to manage and control user accounts and
groups for access to the software. Initial experiments conducted by the group in collaboration with Magellan
project personnel on Amazon EC2 show that the cloud is a feasible platform for this application. There is
also interest in using Hadoop to coordinate and manage the loosely coupled jobs.
4.4.4 ATLAS
The ATLAS project is investigating the use of cloud platforms to support analysis jobs. The ATLAS project
has hundreds of jobs that operate on terabytes of data and can greatly benefit from timely access to cloud
resources. The cloud environment also promises to be an effective platform for transitioning scientific codes
from testing on the desktop to large-scale cloud resources. The group is investigating the use of virtual
machine images for distribution of all required software [10]. This would enable sites to boot the virtual
machines at different sites with minimal or no work involved with software management.
4.4.5 Integrated Microbial Genomes (IMG) Pipeline
The Integrated Microbial Genomes (IMG) pipeline at the DOE Joint Genome Institute (JGI) provides analysis
of microbial community metagenomes in the integrated context of all public reference isolate microbial
genomes. IMG has workloads that need to run periodically every few weeks to months for content maintenance
[8]. Timeliness of completion of workloads is critical for the community, and the tremendous growth
of these data sets makes access to large number of resources critical. The computational stage consists of
functional annotation of individual genes, identification of pair-wise genes, and identification of chromosomal
clusters. The most computationally intensive step is performing a BLAST analysis against a reference
database. Subsequent steps characterize the genes based on the alignments reported by BLAST. The BLAST
output alone is typically over a terabyte. Consequently the analysis of the output to find the top matches and
identify taxons can be time consuming and must be done in parallel. There is interest in using technologies
such as Hadoop to ease management of these loosely coupled application runs.
Currently the output of “all vs. all” pairwise gene sequence comparisons is stored in compressed files.
However, modifying individual entries and querying the data is not easy in this format. The group is
interested in exploring the use of HBase for managing data, which will allow the users to update individual
rows and perform simple queries.
4.5 Summary
The results of the detailed survey have helped us in understanding the science requirements for cloud environments
and have influenced the direction of research in the project. We summarize the requirements
gathered from the user survey and corresponding project activities:
• The user requirements for cloud computing are diverse, ranging from access to custom environments
to the MapReduce programming model. These diverse requirements guided our flexible software stack.
Users of Magellan had access to (a) traditional batch queue access with the additional capability of
custom software environments through xCAT; (b) customized virtual machines through Eucalyptus or
OpenStack front ends, enabling users to port between commercial providers and the private cloud; (c)
a Hadoop installation that allowed users to access the MapReduce programming model, the Hadoop
Distributed File System and other job management features such as fault tolerance. More details of
the software stack are presented in Chapter 5, our Hadoop evaluation is presented in Chapter 10, and
our user experiences are summarized in Chapter 11.
• It is important to understand whether commercial cloud platforms such as Amazon EC2 and private
cloud software such as Eucalyptus and Hadoop met the needs of the science. We identified the existing
21