FIAS Scientific Report 2011 - Frankfurt Institute for Advanced Studies ...

High Performance GPU-based DGEMM and Linpack
Collaborators: D. Rohr 1 , M. Bach 1 , M. Kretz 1 , V. Lindenstruth 1
1 Frankfurt Institute for Advanced Studies
Linpack is a popular benchmark for supercomputers and builds the basis of the half-yearly Top500 list. It
iteratively solves a dense system of linear equations. Each iteration consists of panel factorization, panel
broadcast, LASWP, U-matrix broadcast and trailing matrix update [1]. The trailing matrix update is performed
via a matrix multiplication (DGEMM) and is the most compute intensive step. It is a well known fact that
GPUs excel at matrix multiplication. For these reasons, a heterogeneous Linpack has been implemented for
the LOEWE-CSC cluster, which performs the update step on the GPU and all other tasks on the processor.
Since only one process per node can use the GPU efficiently, instead of one MPI process per CPU core only
one process pre node is used, which uses the GPU and is multi-threaded itself. Therefore, all other tasks during
Linpack have been parallelized. The employed multi-threaded BLAS libraries were to support reservation of
CPU cores for GPU pre- and postprocessing. A binary patch to the AMD driver reduces the page fault rate.
The DGEMM of the update step is split in many multiplications of submatrices called tiles, which are small
enough to fit in GPU memory. Processing of these tiles is arranged in a pipeline which ensures continuous
GPU utilization and allows for concurrent GPU and CPU processing as well as DMA transfer. In addition, a
dynamic buffering ensures that no tile is ever retransferred. For not wasting GPU cycles during non-DGEMM
Linpack-tasks, the implementation allows for executing factorization and broadcast for the next iteration in
parallel to the DGEMM of the current iteration and pipelines the LASWP task. Linpack performance has been
demonstrated to scale linearly up to several hundreds of nodes [1].
Usually, the per-node performance of multi-node Linpack runs is limited by the slowest nodes. For clusters
with nodes of different performance levels, e.g. the LOEWE-CSC has CPU-only and GPU-enabled compute
notes, the new Linpack allows for an unequal distribution of the workload among the systems. Measurements
show that the implementation is actually able to achieve almost the accumulated performance of all
participating nodes with less than 3% granularity loss.
Multiple GPUs can be used in parallel with e.g. two GPUs
showing a speedup of about factor 1.98. Up to four GPUs in
one system have been tested reaching 2 TFlop/s of Linpack
performance. With three GPUs an efficiency of about 1200
GFlop/J was demonstrated, corresponding to a second place in
the Green500 list at the time the experiment was conducted [2].
The DGEMM kernel on the GPU achieves about 90% of the
theoretical peak performance on both the Cypress and Cayman
series of AMD GPUs. 75% of the accumulated theoretical
CPU and GPU performance are available in Linpack. The
LOEWE-CSC ranked place 22 in the November 2010 Top500
list demonstrating the highest efficiency of all listed GPU
clusters with respect to theoretical peak performance.
GPU CPU
Core 23
...
Core 10
Core 9
...
Core 3
Core 2
Core 14
Core 13
Core 12
Core 1
Core 0
GPU 0
GPU 1
GPU 2
U
BCAST
LASWP + DTRSM: 0-5120
LASWP + DTRSM: 5120-n
GotoBLAS, Columns 0-1024
Factorization
GotoBLAS
Columns 1024-l
Rows k-n
PANEL
BROADCAST
Iteration N Iteration N+1
GotoBLAS CPU DGEMM
MergeBuffer GPU 2
MergeBuffer GPU 1
DivideBuffer GPU 1 & 2
MergeBuffer GPU 0
Scheduling, DivideBuffer GPU 0
Time
Columns l-n, Rows k-n
GPU DGEMM KERNELS
Columns 1024-n, Rows 0-k
U
BCAST
Figure 1: Concurrent execution of all HPL tasks
on CPU cores and GPUs
The current implementation for the CAL-framework is
momentarily being improved to support OpenCL and CUDA and the new AMD Graphics Core Next GPU.
Related publications in 2011:
1) M. Bach, M. Kretz, V. Lindenstruth, D. Rohr Optimized HPL for AMD GPU and multi-core CPU usage,
Computer Science - Research and Development 26, 153 (2011)
2) D. Rohr, M. Bach, M. Kretz, V. Lindenstruth Multi-GPU DGEMM and HPL on highly energy efficient
clusters, IEEE Micro, Special Issue: CPU, GPU, and Hybrid Computing
119