TGQR 2010Q4 Report.pdf - Teragridforum.org

Machine # Processors Domains Grids per domain # e- per proc. Wall Clock(sec)
Kraken 1024 128 8x512x512 262144 31441
Kraken 4096 512 4x1024x1024 524288 67115
PI. Axelrod (Univ. of Arizona, Physics). Research Allocation for the Large Synoptic Survey
Telescope Data Challenge 3b. Continuing through 08/11. Darren Adams (NCSA), Raghu Reddy
(PSC), and Chris Jordan (TACC) are providing ASTA support for this PI. There is no update this
quarter for this project.
PI.Radko (Naval Postgraduate School, Physical Oceanography) Numerical Modeling of Doublediffusive
Convection. Continuing through 09/11. ASTA staff for this project is Gabriele Jost.
The goal of the AUS is to increase the performance and scalability of Prof. Radko's 2D and 3D
double diffusion convection (DDC) implementations TILT2D and TILT3D. The codes are
currently parallelized base on OpenMP, which limits their scalability on SMP clusters to 1 node
(eg 16 threads on Ranger).
The ASTA staff and PI’s team are pursuing 3 possibilities:
1. Develop an MPI implementation of TILT3D
2. Investigate the use of GPGPUs (eg on Longhorn at TACC) for TILT2D
3. Explore the use of very large scale SMP nodes (eg. SGI Altix UV)
During the first quarter they focused the development of an MPI based implementation of
TILT3D. The work items below were performed.
Item 1: Planning and discussing a strategy with Prof. Radko
=============================================================
The application consists of 2 main programs. One program is used to generate initial input data
for numerical experiments, the other program performs the DDC calculations. It was decided to
use a 1D domain decomposition of the 3D arrays in the 3rd dimension.
The code to generate the input data currently generates a single binary file with input data. For the
distributed version they decided to keep this approach. Each MPI process will read its part of the
input data from the binary file.
The code DDC code uses FFTW to perform 3D real-to-complex FFTs. In order to perform the
FFT on distributed data the structure of the code will be changed to
1. perform 2D FFT, using FFTW real-to-complex in the xy-plane
2. use MPI_All_to_all to redistribute the data
3. perform a 1D complex-to-complex FFT in z dimension
4. use MPI_All_to_all to obtain the original distribution in z-dimension
Item 2: Start implementation
============================
Change the input generation code from sequential into an MPI code. Input generation was
changed to allow direct access to the binary file. This is to support the reading of multiple MPI
processes from different locations of the files.
Test the code for running on 1 MPI process.
Implement several test cases for performing a 3D FFTW in 2 steps: 2D real-to-complex (R2C) in
xy, 1D complex-to-complex (C2C) in z dimension.
60