PNNL-13501 - Pacific Northwest National Laboratory

Study Control Number: PN00021/1428
Code Development for NWGrid/NWPhys
Harold E. Trease
The NWGrid/NWPhys system represents enabling technology for solving dynamic, multiscale, coupled computational
physics problems on hybrid grids using massively parallel computers in the following areas: computational biology and
bioengineering, atmospheric dynamics and transport, subsurface environmental remediation modeling, engineering,
computational and molecular chemistry, and semiconductor design. The NWGrid/NWPhys system provides one of the
components for building a new Problem Solving Environment modeled after EECE/NWChem. NWGrid/NWPhys brings
advanced grid and computational simulation technologies to the Laboratory and to the DOE research community.
Project Description
The purpose of this project is to develop NWGrid/
NWPhys into production codes that integrate automated
grid generation, time-dependent adaptivity, applied
mathematics, and numerical analysis for hybrid grids on
distributed parallel computing systems. This system
transforms complex geometries into computable hybrid
grids upon which computational physics problems can
then be solved. To run the necessary, large memory
problems, NWGrid has been ported to other platforms
including those in the EMSL Molecular Science
Computing Facility (MSCF) and at the National Energy
Research Scientific Computing Center (NERSC), with
more memory and central processing unit power.
Additional Global Array sparse matrix operations that are
needed to parallelize NWPhys have been defined and
tested. A first draft of the NWGrid Users Manual is being
reviewed and a Graphical User Interface is being
designed.
Introduction
In the past, NWGrid was run in parallel on the ASCI
Silicon Graphics, Inc., System cluster (a predominantly
global memory, 64-bit machine) at Los Alamos National
Laboratory with ASCI parallelization software. To be
able to run the necessary, large memory problems,
NWGrid needed to be ported to other platforms, with
more memory and central processing unit power.
In the past, NWPhys was run in parallel on CRAYs and
Connection Machines at Los Alamos National
Laboratory. It is not currently running in a full-assembled
configuration in parallel anywhere. Pieces are running in
serial on a Silicon Graphics, Inc., system at our Laboratory.
Portions of the code are being rewritten on an as-
needed-basis for ongoing work. To be able to parallelize
it, functions for sparse matrix operations were needed.
The NWGrid/NWPhys command suite was not well
documented and there were few examples. Most
experienced users knew only small portions that applied
to their specific applications. The code currently runs
with a command line driven interface. A users manual
and graphical user interface are needed.
Results and Accomplishments
NWGrid Code Development. To run the necessary,
large memory problems, the scalar version of NWGrid
has been ported to other platforms including those in the
EMSL/MSCF and at NERSC, with more memory and
central processing unit power. This involved converting
it from a 64-bit Silicon Graphics, Inc., code to 32-bit
architectures, such as IBM (MPP1), LINUX INTEL PC,
LINUX ALPHA PC, and SUNs and porting it to each
machine’s parallel libraries. NWGrid is evolving from
FORTRAN 77 and C to FORTRAN 90.
NWPhys Code Development. To be able to parallelize
NWPhys, a number of sparse matrix operations are being
added to the EMSL internal software package, Global
Arrays, to support unstructured meshes. These new
functions are compatible with the directives provided by
the Connection Machine, which are already used by
NWPhys. They have been defined and tested and are in
an initial parallel implementation of NWPhys. NWPhys
is being rewritten to FORTRAN 90 during this conversion
process.
Documentation. The first draft of the NWGrid Users
Manual has a full description with format, parameters,
examples, and special notes of the over 120 NWGrid
Computational Science and Engineering 103