National Energy Research Scientific Computing Center

64 NERSC ANNUAL REPORT 2015
‘Superfacilities’ Concept
Showcases Interfacility
Collaborations
A GISAXS pattern from a
printed organic photovoltaic
sample as it dried in the
beamline, part of a
“superfacility” experiment
based at Berkeley Lab.
Image: Craig Tull, Lawrence
Berkeley National Laboratory
Throughout 2015 NERSC saw increased interest
in the coupled operation of experimental and
computational facilities throughout the DOE Office
of Science complex—the so-called “superfacility.”
A superfacility is two or more facilities interconnected
by means of a high performance network
specifically engineered for science (such as ESnet)
and linked together using workflow and data
management software such that the scientific output
of the connected facilities is greater than it
otherwise could be. This represents a new
experimental paradigm that allows HPC resources to be brought to bear on experimental work being
done at, for example, a light source beamline within the context of the experimental run itself.
Adding HPC to facility workflows is a growing trend, and NERSC is leveraging its experience with
the Joint Genome Institute, the Advanced Light Source, the Large Hadron Collider, the Palomar
Transient Factory, AmeriFlux and other experimental facilities to shape the interfaces, schedules and
resourcing models that best fit what our facility users need. Treating facilities as users of HPC
requires long-term data planning, real-time resource scheduling and pipelining of workflows.
NERSC’s superfacility strategy focuses on scalable data analysis services that can be prototyped and
reused as a new interfacility collaborations emerge. For example, in 2015 a collaborative effort linking
the Advanced Light Source (ALS) at Berkeley Lab with supercomputing resources at NERSC and
Oak Ridge National Laboratory via ESnet yielded exciting results in organic photovoltaics research.
In a series of experiments, the operators of a specialized printer of organic photovoltaics used HPC to
observe the structure of the material as it dried. Organic photovoltaics show promise as less expensive,
more flexible materials for converting sunlight to electricity.
During the experiments, scientists at the ALS simultaneously fabricated organic photovoltaics using
a miniature slot-die printer and acquired beamline data of the samples as they were made. This data
was transferred to NERSC via ESnet, where it was put into the SPOT Suite data portal and
underwent analysis to translate the image pixels into the correct reciprocal space. Using the Globus
data management tool developed by the University of Chicago and Argonne National Laboratory, the
output from those jobs was then sent to the Oak Ridge Leadership Computing Facility for analysis,
where the analysis code ran on the Titan supercomputer for six hours nonstop on 8,000 GPUs.
A unique feature of this experiment was co-scheduling beamline time at the ALS with time and
resources at NERSC and Oak Ridge, plus utilizing ESnet to link it all together. This was the
first time a beamline was able to simultaneously take data and perform large-scale analysis on
a supercomputer.
NERSC and the 2015 Nobel Prize in Physics
In 2015 NERSC was once again fortunate to be part of the story behind a Nobel Prize. This time it
was the 2015 Nobel Prize in Physics, awarded to the leaders of two large neutrino experiments:
Takaaki Kajita of Tokyo University, who led the Super-Kamiokande experiment, and Arthur B.
McDonald of Queen’s University in Kingston, Ontario, Canada, head of the Sudbury Neutrino
Observatory (SNO). According to the Nobel Prize committee, Kajita and McDonald were honored
“for the discovery of neutrino oscillations, which shows that neutrinos have mass.”