TGQR 2010Q4 Report.pdf - Teragridforum.org

2.2.18
Physics: The Geophysical High Order Suite for Turbulence (GHOST) (PI: Annick
Pouquet, NCAR)
The Geophysical High Order Suite for Turbulence (GHOST) is a framework for studying
fundamental turbulence via direct numerical simulation (DNS) and modeling. Running on
TeraGrid computers in 2010, GHOST became the first known successful attempt to combine
OpenMP and MPI in a large-scale hybrid scheme for pseudo-spectral fluid dynamics. This
hybridization enables highly efficient use of computing resources both within each node and
between nodes across a network. Using a TeraGrid allocation at NICS, NCAR researchers were
able to do a 3,072 3 simulation of a rotating flow—the largest such simulation in the world—on
nearly 20,000 processor cores. The efficiency achieved with
GHOST allowed them to study flows at a higher resolution and
with greater fidelity by using all the cores on each node. Such
efficiency gains are allowing researchers to solve problems they
could not do before by significantly reducing the processing time
on large core-count computer systems. Pablo Mininni, Duane
Rosenberg, Raghu Reddy, and Annick Pouquet presented their
paper on the topic, Investigation of Performance of a Hybrid
MPI-OpenMP Turbulence Code, at TG'10, and it received the
conference award for best paper in the technology track.
2.2.19
Student Training: Use of TeraGrid Systems in the
Classroom and Research Lab (PI: Andrew Ruether,
Swarthmore College)
Swarthmore College uses the TeraGrid for both student
coursework and faculty research. When Ruether taught
Swarthmore’s Distributed and Parallel Computing course, all
students were given TeraGrid accounts, and they used TeraGrid
systems for class projects. Two of computer science Professor Tia
Newhall’s students extended the work done in the class to a
summer project and presented their results at the TeraGrid 2010
conference. The students developed and tested a novel
parallelization technique for solving the K-Nearest Neighbor
problem. The algorithm is used to classify objects from a large
set. A few examples of its use include discovering medical
images that contain tumors from a large medical database,
recognizing fingerprints from a national fingerprint database and
finding certain types of astronomical objects in a large
Figure 2.18. These four turbulence
visualizations show Taylor-Green flow at
increasing resolutions using 64 3 , 256 3 ,
1,024 3 , and 2,048 3 grid points. This
modeling advancement using a hybrid
MPI-OpenMP code is only possible on
systems with significantly more processor
cores than NCAR’s most powerful
supercomputer. GHOST code has been run
successfully on Kraken and Ranger.
GHOST code scales well on these and
other platforms for hydrodynamic runs
with up to 3,072 3 grid points. These results
demonstrate that pseudo-spectral codes
such as GHOST – using FFT algorithms
that are prevalent in many numerical
applications – can scale linearly to a large
number of processors.
astronomical database. The TeraGrid allowed the students to run large-scale experiments
necessary to demonstrate that their solution worked well for real-world problems.
Swarthmore also has three faculty members—in computer science, physics and chemistry—doing
publishable work using TeraGrid resources. Because Swarthmore does not have a highperformance
computing system, the TeraGrid has been a key resource for faculty researchers who
have reached the limits of desktop computing. In one case, this reduced the run time of a
simulation from one week on a desktop computer to one hour using Purdue's Condor pool, which
offers nearly 37,000 processors and is available as a high-throughput TeraGrid resource. This
allowed
the
research
group to
run many
Figure 2.19. The gray lines are magnetic field lines in plasma and the colored dotted lines are charged
particle orbits through the magnetic field. TeraGrid resources were used to model many particle orbits to get a
sense of the confinement in this configuration. Knowledge gained from the modeling could be useful in
designing fusion reactors based on magnetic confinement.
25