Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
11.2.5 Integrated Metagenome Pipeline
The Integrated Microbial Genomes (IMG) system hosted at the DOE Joint Genome Institute (JGI) supports
analysis of microbial community metagenomes in the integrated context of all public reference isolate
microbial genomes. The content maintenance cycle for data involves running BLAST for identifying pairwise
gene similarities between new metagenome and reference genomes, where the reference genome baseline
is updated with new (approximately 500) genomes every four months. This processing takes about three
weeks on a Linux cluster with 256 cores. Since the size of the databases is growing, it is important that
the processing can still be accomplished in a timely manner. The primary computation in the IMG pipeline
is BLAST, a data parallel application that does not require communication between tasks and thus has
similarities with traditional cloud applications. The need for on-demand access to resources makes clouds
an attractive platform for this workload.
The pipeline is primarily written in Perl, but it includes components written in Java, as well as compiled
components written in C and C++. The pipeline also uses several reference collections (typically called
databases), including one for RNA alignment and a periodically updated reference database for BLAST.
The pipeline and databases are currently around 16 GB in size. This does not include Bio-perl, BLAST, and
other utilities. The pipeline was run across both Magellan sites through Eucalyptus. A simple task farmer
framework was used to distribute the workload across both sites. As virtual machines came up, a client
would query the main server for work and run the computation.
11.2.6 Fusion
Fusion is a traditional 320-node high-performance compute cluster operated by the Laboratory Computing
Resource Center at Argonne. In order to expand the capabilities of the cluster, a project was launched to
prototype an application that could offload traditional HPC style jobs to Magellan resources at the ALCF
in a transparent fashion. HPC batch jobs are often pre-coupled with a lot of expectations regarding file
system locations, amount and type of resources, software availability, etc. To seamlessly offload these types
of jobs on the ALCF Magellan cloud, a compute node image was created that establishes a VPN connection
back to the Fusion HPC cluster on launch, and appears to the users as one of the default batch computation
systems. Upon submission of a job, the application requests a number of instances from the cloud on the
fly, passes the user’s job to them for execution, and terminates them when the work is completed. The end
result was a success; a number of test serial jobs were offloaded to the Magellan cloud at ALCF. This case
study demonstrates a scenario where serial jobs could be redirected to cloud resources, enabling the HPC
resources to cater to the larger parallel jobs, therefore making better use of specialized hardware.
11.2.7 RAST
SEED is an open-source comparative-analysis genomics effort. The compute-intensive protein-base searches
were offloaded from their traditional cluster onto the Magellan resources at ALCF during periods of heavy
demand. Incoming raw genome sequences were processed with Glimmer to identify individual genes. These
were subsequently annotated with RAST. The subsequent protein-base sequences were then processed via
BLAST on a persistent 32-node Magellan virtual cluster against the NCBI and Swiss-Prot protein-sequence
databases, which were shared among the two clusters via NFS. The results were then fed into the SEED
database, enabling users to query similarities among genomes. This was a heavily compute-bound application,
and performance was deemed to be very good on virtual resources.
RAST had a number of similar challenges in managing virtual machine images and tying to existing
infrastructure such as NFS and databases to make this load balancing transparent.
11.2.8 QIIME
QIIME (Quantitative Insights Into Microbial Ecology) is a toolset for analyzing high throughput 16S amplicon
datasets. 16S rRNA is a highly conserved structure in microbial genomes that can be sequenced
111