Magellan Final Report - Office of Science - U.S. Department of Energy

Magellan Final Report
Figure 11.2: An AMR volume rendering of FLASH Astrophysics simulation using vl3.
VASP errors, and updating the database.
The workflow manager was run for three weeks within an allocated sub-queue on Magellan consisting
of 80 nodes. During this time the group ran approximately 20,000 VASP calculations and successfully
determined the relaxed ground state energy for 9000 inorganic crystal structures. The number of cores for
each VASP MPI calculations was chosen to be consistent with the number of cores on a Magellan node. The
project found 8 cores to be a good match for addressing the wide distribution of computational demands
(e.g., number of electrons, k-point mesh) inherent to high-throughput computing.
There were several challenges running high-throughput quantum mechanical MPI calculations on Magellan.
For example, the workflow manager running on the login node would often get killed due to resource
limits. Another problem was the robustness of the workflow manager. This was related to updating the
state in the MasterQueue database that would corrupt the database. The virtual cluster on Magellan was an
essential component to compute in an atomic manner and as a result, dramatically simplified the workflow.
This allowed us to avoid bundling several VASP calculations into a single Portable Batch System (PBS)
script. This bundling process generates another layer of complexity whose sole purpose is to work within
the constraints of the queue policies of supercomputing centers, which limit the number of jobs per user.
Magellan was an excellent solution for high-throughput computing for the Materials Project. The virtual
cluster that NERSC staff provided allowed the users to leverage the computing resources in a flexible way, as
per the needs of the application, enabling the group to focus on the challenging materials science problems
of data storage and data validation.
The Materials Project workflow is an example of a class of next-generation projects that require access
to a large number of resources for a specific period of time for high-throughput computing. There is limited
or no support available for such workloads at DOE centers today: job queue policies limit the number of
jobs allowed in the queue; there are no tools to enable such high-throughput jobs to seamlessly access a large
number of resources; and machine firewall policies often restrict or constrain processes such as workflow
managers and connections to external databases. Cloud technologies provide a way to enable customized
environments for specific time periods while maintaining isolation to eliminate any impact to other services
at the center.
11.1.5 E. coli
During the weekend of June 3–5, 2011, hundreds of scientists worldwide were involved in a spontaneous
and decentralized effort to analyze two strains of E. coli implicated in an outbreak of food poisoning in
Germany. Both strains had been sequenced only hours before and released over the internet. An Argonne
and Virginia tech team worked throughout the night and through the weekend to annotate the genomes
and to compare them to known E. coli strains. During the roughly 48-hour period, many E. coli strains
were annotated using Argonne’s RAST annotation system, which used the Magellan testbed at ALCF to
105