CERFACS CERFACS Scientific Activity Report Jan. 2010 – Dec. 2011

COUPLING TOOLS AND HPC CLIMATE MODELING
Scientifique (CNRS), also devoting one full time engineer for OASIS since 5 years, is in that respect
particularly valuable.
During the last two years, active user support was provided to the OASIS users. Within IS-ENES, OASIS
Dedicated User Support was offered to IPSL (1 person-month), SMHI (2 persons-months) and ETH Zurich
(2 persons-mponths) (see [CC16], [CC24]). In addition, a new web site 3 is now open to the community
through the IS-ENES portal and offers different services such as on-line documentation, user guides,
tutorial, FAQs, user forum and tips for best practices. The central position of CERFACS in the coupled
climate modelling community was also confirmed by the success of the workshop “Coupling Technologies
for Earth System Modelling : Today and Tomorrow” 4 organised in collaboration with the Georgia Institute
of Technology in December 2010. Forty-five participants from around the world (20 from France, 12 from
other EU countries, 11 from the US, and 2 from China) explored the trade-offs involved in the different
approaches to coupling in use throughout the climate modeling community and laid out a vision for coupling
Earth System Models (ESMs) in the year 2020 (see also [CC1]).
2.2 OASIS3
Although it is foreseen that it will become a bottleneck for high resolution ESMs running on massively
parallel platforms, OASIS3 is still being used successfully in different high resolution ESMs with a
reasonnable overhead, thanks to its field-by-field “pseudo-parallelism”.
For example, the EC-Earth model, which underlying atmospheric model IFS, was increased to T799 ( 25km,
or 843,000 points) and 62 levels, along with a 0.25-degree (1.5M points), 75 depth-level configuration of the
NEMO ocean model, was run on the Ekman cluster at the PDC Centre for High-Performance Computing
on Sweden (1268 nodes of dual-sock quad-core AMD Opteron processors, i.e. a total of 10144 cores)
with different numbers of cores for each component and OASIS3. Different combinations were tested. A
coupling overhead of about 3% was observed with the 512-128-10 configuration (i.e. when 512, 128 and
10 cores are respectively used for IFS, NEMO, and OASIS3) and an overhead of about 1.3% was observed
for the 800-256-10 configuration, which in both cases are still reasonnable overheads.
It has to be noted however that in these configurations there is some imbalance between the components
elapse times, which allows OASIS3 to interpolate the fields when the fastest component waits for the
slowest. For other coupled models, this may not be possible. A high-resolution version of CNRM-CM5 (50
km-atmosphere, 0.25 degree-ocean) has been compiled and run on more than 1000 cores on the PRACE
tier-0 “curie” Bullx supercomputer to study regional scale / large scale interactions. The ocean model is
used in a mixed layer mode (a configuration called NEMIX) to simplify and better understand coupled
processes. In this configuration, the component models run sequentially and OASIS3 cost is directly added
to the component elapse times. On 1024 cores (500 for the atmosphere, 512 for the ocean, 12 for the
coupler), it was observed that OASIS3 takes up to 20% of total elapsed time to perform interpolations and
communications between coupled components, which becomes an overhead barely acceptable.
Two approaches are currently followed to introduce true parallelism into OASIS and remove the observed
bottleneck. The first one relies on the OASIS4 version of the coupler using its “user-defined” interpolation,
and the second one is based on the modification of OASIS3 to use the Model Coupling Toolkit (MCT).
3 https ://verc.enes.org/models/software-tools/oasis
4 https ://verc.enes.org/models/software-tools/oasis/general-information/events
CERFACS ACTIVITY REPORT 101