FIAS Scientific Report 2011 - Frankfurt Institute for Advanced Studies ...

Lattice QCD on GPUs
Collaborators: M. Bach 1 , V. Lindenstruth 1 , O. Philipsen 2 , C. Pinke 2 , C. Schäfer 2 , L. Zeidlewicz 2
1 Frankfurt Institute for Advanced Studies, 2 Institut für Theoretische Physik, Goethe-Universität Frankfurt am Main
Quantum Chromodynamics (QCD) is the known theory of the strong force and part of the Standard Model of
particle physics. Its phase diagram is a problem of particular interest and is investigated in current, state-of-theart
collider experiments at CERN and, in the near future, at FAIR. Lattice QCD provides a first principle access
to this problem.
Lattice simulations require an enormous amount of computing power. They sample the phase space using
Hybrid Monte Carlo techniques, requiring the fermion matrix to be inverted many times. The inversion of this
sparse quadratic matrix, typically of size 10 8 × 10 8 , is the most time consuming part of the algorithm. To get
physical results simulations need to be carried out at different lattice spacings and extrapolated to the continuum
limit.
Sparse matrix inversion is completely dominated by the memory bandwidth available in the system. Today
Graphics Processing Units (GPUs) provide significantly more bandwidth than CPUs, making them a promising
platform for QCD codes. However, so far GPU-enabled Lattice QCD codes have always been focused on the
proprietary NVIDIA CUDA programming interface, locking the usability of the code to hardware of this one
vendor.
We aim at building a versatile Lattice QCD solution that can achieve optimum performance on a wide variety of
modern hardware architectures. Therefore we base our solution on OpenCL, an open programming standard for
parallel programming supported by all major hardware vendors. We focus on remaining with a single source
source code, ensuring maintainability and easier verification of code correctness, while achieving maximum
performance over a wide range of system configurations.
Bandwidth GB/s
140
120
100
80
60
40
20
16^3x4
16^3x8
24^3x4
16^3x16
16^3x20
16^3x24
16^3x28
32^3x4
24^3x12
Dslash Performance
24^3x16
Lattice Size
32^3x8
24^3x20
24^3x24
32^3x12
GPU Bandwidth
140
GPU Gflops
CPU Bandwidth 120
CPU Gflops 100
Dslash Performance on one AMD HD5870 and two Operon 6172 in LOEWE-CSC for a variety of lattice sizes
Our first focus for performance optimization was the LOEWE-CSC supercomputer, implementing AMD GPUs.
We have carefully analyzed the characteristics of the memory controller of AMD GPUs. Using that knowledge
we were able to achieve more than 70 double precision Gflops in the Dslash calculation, the main part of the
fermion matrix, while still being able to run the same code parallelized on CPU only system. This is competitive
with NVIDIA-only solutions, however, in addition, the methods for memory controller analysis and the
OpenCL based code that can be parametrized to match different memory controllers enable high performance
on a wider range of hardware.
Related publications in 2011:
M. Bach, O. Philipsen, C. Pinke, C. Schäfer, L. Zeidlewicz, LatticeQCD using OpenCL, Proceedings of the
XXIX International Symposium on Lattice Field Theory - Lattice 2011.
120
24^3x32
80
60
40
20
Gflops