CERFACS CERFACS Scientific Activity Report Jan. 2010 – Dec. 2011

THE OASIS COUPLER
2.3 OASIS4
Since the beginning of the IS-ENES project in March 2009, CERFACS, CNRS and DKRZ (Deutsches
Klimarechenzentrum GmbH, in Hamburg, Germany) have been collaborating on the development and
evaluation of a new fully parallel version of the coupler, OASIS4. In particular, OASIS4 includes a library
performing a fully parallel calculation of the source neighbour weights and addresses needed for the
regridding of the coupling fields between the source and target grids (hereafter called the “neighbourhood
search library”) originally developed by NEC Laboratories Europe - IT Research Division (NLE-IT). This
work resulted in June 2011 in the release of the latest OASIS4 version, OASIS4 1beta ([CC29], [CC10]).
OASIS4 was used by Météo-France, KNMI (Netherlands), and MPI-M (Germany) in the framework of
the EU GEMS project (lead by ECMWF) for 3D coupling between atmospheric dynamic and atmospheric
chemistry models and is currently used by SMHI (Sweden), the Alfred Wegener Institute for Polar and
Marine Research (AWI in Germany), the BoM in Australia for ocean-atmosphere 2D regional or global
coupling. In the framework of the METAFOR project, OASIS4 was adapted so to allow the use of the
Common Information Model standard to configure the coupling exchanges ([CC28]).
However, this work also lead us to conclude that OASIS4 parallel neighbourhood search library, originally
developed by NEC, presents some fundamental weaknesses in its design. In particular, the support of
unstructured grids was not originally included and it would be very difficult to add it in the current code.
Also, it is now very clear that it was developed with efficiency as the prime criteria, leaving aside readability
and ease of development.
Therefore we decided to retrofit OASIS4 for pre-computed regridding weights, thereby circumventing
its parallel neighbourhood search library, and use what we call the OASIS4 “user-defined” interpolation
([CC23]). To use this functionality, the user has to describe the links associating specific source grid
points with specific target grid points in a separate NetCDF file. For each link, the index of the source
point, the index of the target point and the weight associated to that link have to be provided. The
OASIS4 communication library reads these indices and weights and automatically defines on each side
a non-geographical grid with one point for each link. The multiplication of the source field values by the
appropriate weights is done in parallel on the source side and a parallel redistribution of the results is
done directly between the source and the target processes by the OASIS4 communication library. The userdefined
regridding has been validated with simple toy models. It now has to be validated for real grids and
in real coupled ESMs.
2.4 OASIS3-MCT
The second solution is based on the modification of OASIS3 to use the Model Coupling Toolkit 5 (MCT)
developed by the Argonne National Laboratory in the USA. MCT requires that the regridding weights are
pre-computed offline, but then implements fully parallel regridding and exchanges of the coupling fields
and has proven parallel performance. MCT embodies a generic approach for creating coupled applications.
Its design philosophy, based on flexibility and minimal invasiveness, is close to the OASIS approach. MCT
uses distributed objects to store the coupling data and pre-computed regridding weights as well as a domain
decomposition descriptor (DDD) to describe the parallel decomposition of the component models. Parallel
communication patterns are computed automatically from the source and target DDDs. To use MCT,
one has to load the coupling data into MCT data types and call MCT parallel matrix multiplication (for
regridding) and communication methods. Parallel data transfer is then accomplished by pairs of send/receive
methods with coupling data and communication pattern as inputs. MCT is, most notably, the underlying
coupling software used in National Center for Atmospheric Research Community Earth System Model 1
(NCAR CESM1) [1]. Interfacing MCT in OASIS3 will provide a valid and efficient solution to remove
current OASIS3 bottleneck ; in this case, the coupling fields will be regridded in parallel in the source
5 http ://www.mcs.anl.gov/research/projects/mct/
102 Jan. 2010 – Dec. 2011