Annual Meeting - SCEC.org

Group 1 – CME/PetaSHA | Poster Abstracts
Community Modeling Environment (CME);
Petascale Cyberfacility for Physics-Based Seismic Hazard
Analysis (PetaSHA)
1-019
SUPPORTING SCEC PETASHAKE SIMULATIONS Cui Y, Zhu J, Chourasia A, Olsen KB,
Dalguer LA, Lee K, Davis S, Day SM, Juve G, Maechling PJ, and Jordan TH
We have performed a set of very-large scale earthquake simulations on the San Andreas Fault on
the TeraGrid computing resources, which used more than ten millions of NSF TeraGrid allocations
over the past year. The simulations are a collaborative, inter-disciplinary research program
coordinated by the Southern California Earthquake Center (SCEC) that makes extensive use of
large-scale, physics-based, numerical modeling of earthquake phenomena. This is part of the
USGS-led public earthquake preparedness exercise called ShakeOut exercise.
The SCEC AWP-Olsen based ShakeOut simulations were scientifically and computationally
challenging for several reasons including: (1) the simulations were performed for a large (600km x
300km x 80km) geographical volume, (2) the simulations were run at high resolution (100m spacing
on a regular grid resulting in 14.4 billion mesh points), and (3) in order to simulate the earthquake
rupture more realistically, source descriptions based on dynamic rupture simulations (ShakeOut-
D) using about ? million subfaults was used as a primary input to the wave propagation
simulations. In addition, ShakeOut simulations with kinematic source descriptions (ShakeOut-K)
were also carried out for code verification purposes and to compare ground motions computed for
various level of complexity.
The simulations were coordinately executed on multiple TeraGrid systems, including the 504
Teraflops Ranger system at Texas Advanced Computing Center (TACC), and Datastar p655 at San
Diego Supercomputer Center (SDSC). Additional SDSC Datastar fat nodes p690 and Blue Gene/L
were used for pre- and post-processing of these capability simulations. A large part of a hundred
terabytes of simulation outputs were registered directly from TACC disks to the SCEC digital
library on SAM-QFS at SDSC, managed through SDSC’s latest iRODs (integrated Rule-Oriented
Data System). We aggressively optimized the iRODs transfer rate up to 135 MB/s, which is many
times faster than previously recorded.
Visualization for SCEC ShakeOut-K and ShakeOut-D simulations were performed on site at TACC.
Visualizations were created by color mapping the data then overlaying contextual information like
freeways, fault lines and topography. Comparative side by side visualization for these two sets of
simulations were also created. The visualization movies are available at
http://visservices.sdsc.edu/projects/scec/shakeout.
1-020
SCEC CME AWP-OLSEN APPLICATION ENHANCEMENTS FOR PETASCALE
COMPUTING Cui Y, Kaiser TH, Zhu J, Lee K, Maechling PJ, Olsen KB, and Jordan TH
The SCEC CME AWP-Olsen code has been through many optimizations over last few years to meet
SCEC research requirements such as running earthquake simulations at 0.5Hz at the TeraShake
scale. The recent SCEC ShakeOut exercise is performed at 1Hz, the computational requirement is
16x larger than previous TeraShake simulations. The increased input size, often exceeding one
terabyte in size, adds additional challenge to the scalability, memory and fault tolerance of the
2008 SCEC Annual Meeting | 77