Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
expand its compute capacity. Around the world, scientists were submitting additional E. coli genomes to
the RAST servers to be annotated as word got out that DNA sequences were available. Additional backend
computing instances were made available by the ALCF Magellan staff to keep up with demand for increased
annotation. Such on-demand changes to handle the load and provide the fast turnaround were made possible
by the Magellan cloud tools. This project was unique in that it utilized both bare-metal provisioning to
leverage native hardware performance (provided by the Argonne bcfg2 and Heckle tools), as well as ALCF’s
OpenStack cloud for populating the databases.
Once the genomes were annotated, a comprehensive genome content analysis was done that required
building molecular phylogenies (evolutionary trees) for each of the proteins in these new strains. These
datasets enabled a detailed comparison to the nearly two hundred strains of E. coli that are in the public
databases. This comparative analysis was quickly published on the internet and made available to the
community. Overall, work that normally would have required many months was completed in less than
three days by leveraging the on-demand computing capability of the ALCF Magellan cloud. This effort
demonstrated the potential value in using cloud computing resources to quickly expand the capacity of
computational servers to respond to urgent increases in demand.
11.2 Virtual Machine Case Studies
In our second set of case studies, we outline some of applications that used the Eucalyptus and OpenStack
clouds at both sites to harness a set of virtual machines (VMs) for their application needs. A number of
applications from diverse domains have used Magellan’s cloud computing infrastructure. A number of these
applications tend to be data-parallel, since these are the most suitable to run in cloud environments. We
outline the suitability, gaps, and challenges of virtual cloud environments for these select applications.
11.2.1 STAR
Figure 11.3: A plot of the status of processing near-real-time data from the STAR experiment using Magellan
at NERSC over a three-day period. The yellow circles show the number of instances running; the green
triangles reflect the total load average across the instances; and the red squares plot the number of running
processing tasks. (Image courtesy of Jan Balewski, STAR Collaboration).
STAR is a nuclear physics experiment that studies fundamental properties of nuclear matter from the
106