Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
data collected at Brookhaven National Laboratory’s Relativistic Heavy Ion Collider. Previously, STAR has
demonstrated the use of cloud resources both on Amazon and at other local sites [28, 56]. STAR used
Magellan resources to process near-real-time data from Brookhaven for the 2011 run data. The need for
on-demand access to resources to process real-time data with a complex software stack makes it useful to
consider clouds as a platform for this application.
The STAR pipeline transferred the input files from Brookhaven to a NERSC scratch directory. The
files were then transferred using scp to the VMs; and after the processing was done, the output files were
moved back to the NERSC scratch directory. The output files were finally moved to Brookhaven, where
the produced files were catalogued and stored in mass storage. STAR calibration database snapshots were
generated every two hours. Those snapshots were used to update database information in each of the running
VMs once per day. Update times were randomized between VMs to optimize the snapshot availability.
The STAR software stack was initially deployed on NERSC/Magellan. After two months of successful
running, the software stack was expanded to a coherent cluster of over 100 VMs from three resource pools
including NERSC Eucalyptus (up to 60 eight-core VMs), ALCF Nimbus cloud (up to 60 eight-core VMs),
and ALCF OpenStack cloud (up to 30 eight-core VMs). Figure 11.3 shows the virtual machines used and
corresponding aggregated load for a single run over three days.
In summary, over a period of four months, about 18,000 files were processed, 70 TB of input data was
moved to NERSC from Brookhaven, and about 40 TB of output data was moved back to Brookhaven. The
number of simultaneous jobs over the period of time varied between 160 and 750. A total of about 25K CPU
days or 70 CPU years was used for the processing.
The STAR data processing is a good example of applications that can leverage a cloud computing testbed
such as Magellan. The software can be easily packaged as a virtual machine image, and the jobs within the
STAR analysis also require little communication, thus having minimal impact from virtualization overheads.
Using cloud resources enabled the data processing to proceed during the five months of experiments and to
finish at nearly the same time.
There were a number of lessons learned. The goal was to build a VM image that mirrored the STAR
reconstruction and analysis workflow from an existing system at NERSC. Building such an image from
scratch required extensive system administration skills and took weeks of effort. Additionally, several days
effort was required when the image was ported to the ALCF Nimbus and OpenStack clouds. The image
creator needs to be careful not to compromise the security of the VMs by leaving personal information
like passwords or usernames in the images. Additionally, the image must be as complete as possible while
limiting its size, since it resides in memory. Using cloud resources was not a turnkey operation and required
significant design and development efforts. The team took advantage of on-demand resources with custom
scripts to automate and prioritize the workflow.
11.2.2 Genome Sequencing of Soil Samples
Magellan resources at both ALCF and NERSC were used to perform genome sequencing of soil samples
pulled from two plots at the Rothamsted Research Center in the UK. Rothamsted is one of the oldest
soil research centers and has a controlled environment, off limits to farming and any other disturbances by
humans, reaching back 350 years. The specific aim of this project was to understand the impact of long-term
plant influence (rhizosphere) on microbial community composition and function. Two distinct fields were
selected to understand the differences in microbial populations associated with different land management
practices.
The demonstration used Argonne’s MG-RAST metagenomics analysis software to gain a preliminary
overview of the microbial populations of these two soil types (grassland and bare-fallow). MG-RAST is a
large-scale metagenomics data analysis service. Metagenomics use shotgun sequencing methods to gather
a random sampling of short sequences of DNA from community members in the source environment. MG-
RAST provides a series of tools for analyzing the community structure (which organisms are present in
an environment) and the proteins present in an environment, which describe discrete functional potentials.
MG-RAST also provides unique statistical tools including novel sequence quality assessment and sample
107