PNNL-13501 - Pacific Northwest National Laboratory

tetra-scale computing, envisioned as being in the near
future for parallel computing. Multi-block grids
comprising orthogonal sub-grids will allow incorporation
of subsurface complexities into the computational domain
without significantly increasing the total number of
required grid cells.
Approach
This project has focused on implementing capabilities for
orthogonal curvilinear coordinate grids and multi-block
grids into a series of sequential and parallel multifluid
subsurface flow and transport simulators. Curvilinear
coordinate grid capabilities were implemented into the
software using the conventional approach of transforming
the governing differential equations that describe
subsurface flow and transport, using grid metrics (i.e.,
geometric transformations between the physical domain
and computational domain). For static grids the grid
metrics are computed once, resulting in minimal
additional computational effort. For dynamic grids, such
as adaptive grids used to simulate the migration of dense
brines from leaking storage tanks into the vadose zone,
grid metrics are computed for each grid level. The
parallel implementation of the subsurface flow and
transport simulator uses a FORTRAN preprocessor,
which interprets directives imbedded in the source coding,
to convert the source coding from sequential Fortran to
parallel FORTRAN with message passing protocols.
Multi-block grids require different data structures and
distributions than the conventional single-block structured
grid. The principal focus for implementing multi-block
capabilities in parallel has been in the development of
directives and preprocessor translators to handle multiblock
data forms.
Results and Accomplishments
The principal outcomes of the project have been the
development of capabilities for solving multifluid
subsurface flow and transport problems on curvilinear
coordinates and the development of a FORTRAN
preprocessor directive, data structure, and interpreter for
multi-block grids. To demonstrate these computational
capabilities, an experiment involving multifluid flow in a
bedded porous media with a sloped interface is being
simulated using a structured Cartesian grid, tilted
Cartesian grid, and boundary-fitted orthogonal grid.
Simulation results for the different grids will be compared
against laboratory data, where the key issues are the
behavior of a light-nonaqueous-phase-liquid near a sloped
textural interface. The ability of the simulator to predict
260 FY 2000 Laboratory Directed Research and Development Annual Report
the observed behavior of the light-nonaqueous-phaseliquid
migration strongly depends on the soil-moisture
characteristic function and the geometric representation of
the computational grid. For this study, a boundary-fitted
curvilinear grid has a significant advantage over Cartesian
and tilted Cartesian grids in that it accurately honors both
the geometry of the sloped interface and the container
boundaries.
Curvilinear coordinate system capabilities have been
incorporated into eleven operational modes of the
multifluid subsurface flow and transport simulators
(sequential and parallel implementations): 1) water
(aqueous system with passive gas), 2) water-air (aqueousgas
system), 3) water-salt (aqueous-brine system with
passive gas), 4) water-salt-air (aqueous-brine-gas system),
5) water-air-energy (nonisothermal aqueous-gas system),
6) water-salt-air-energy (nonisothermal aqueous-brine-gas
system), 7) water-oil (aqueous-nonaqueous liquid system
with passive gas), 8) water-oil-air (aqueous-nonaqueous
liquid-gas system), 9) water-oil-dissolved oil (aqueousnonaqueous
liquid system with passive gas and kinetic
dissolution), 10) water-oil-surfactant (aqueousnonaqueous
liquid-surfactant system with passive gas),
and 11) water-oil-alcohol (aqueous-nonaqueous liquidternary
mixture with passive gas).
Writing long-lasting computer codes that distribute data
and computations across an array of parallel processors is
difficult in the advancing technology of computer
hardware and system software. The concept chosen to
develop enduring scientific software for this project was
that of using a programmable parallel FORTRAN
preprocessor, which translates sequential FORTRAN
coding with embedded directives into parallel coding.
The embedded parallel coding distributes arrays, supports
task and data parallelism, implements parallel
input/output and other processes required to use
distributed memory, shared memory symmetric
multiprocessor, and clustered parallel computer systems.
This approach allows the code developer to focus on the
science and applied mathematics of multifluid subsurface
flow and transport without having to deal with the details
and protocols of parallel processing. Implementing this
approach for multi-block grids required the development
of preprocessor directives and programming the
preprocessor to interpret these directives. The
preprocessor has been programmed to handle block
structures using the derived types and structures
capabilities of FORTRAN 90. Preprocessor routines were
written to handle defining blocks, allocating variables,
creating variables, and distributing blocks across the
processors.