Lecture Notes in Computer Science 4917

184 H. Servat et al.
Figures 3.a and 4.a show the total execution time divided on communication
and non communication time of the 3D FFT. We have measured the communication
time done in the executions with no computation into the SPEs. Non
communication time in the Figures is the total execution time minus the communication
time. That is, non communication time is the computation that is not
overlapped with communication. Figures 3.b and 4.b show the total execution
time divided on the five mandatory steps of the 3D FFT, but the 128-A and
128 3 grid size case (Figure 3.b). For 128-A and 128 3 grid size case, the 128 × 128
plane fits in the LS of the SPE and Steps 1, 2, and 3 are joined together in one
Step called FFT2D in the Figure, and Steps 4, 5 are joined into another Step
called transf+fft in the Figure.
First, independently of using one or two SPEs to swap submatrices on the
transpositions, communication time increases when reducing the blocksize because
this reduction affects the memory bandwidth that we obtain. In particular,
one row DMA transfers, which happens on the second matrix transpostion (for
Nx × Nz planes), misuse the EIB memory bandwidth, specially for 32 blocksize.
For rows of 32 elements, there are only 256 bytes to transfer, which is less
than the minimum number of bytes (1024 bytes) to achieve near peak memory
bandwidth [18]. Besides, DMA list transfers can help to improve memory bandwidth
on second matrix transposition. A DMA list transfer allows gather/scatter
operations, which increases the size of those 256-byte DMA transfers.
Figures 3 and 4 show performance differences between A 3D FFT and B
3D FFT transposition strategies on the communication part. That is because,
in the A 3D FFT strategy, one SPE uses double buffering when swapping and
transposing the two B × B submatrices, while, in the B 3D FFT strategy no
double buffering is done; in this case one SPE performs the matrix transposition
of only one submatrix. Therefore, the execution time of the A 3D-FFT strategy,
for the same blocksize, is less than for B 3D-FFT strategy. However, for 256 3 grid
size, B 3D FFT strategy shows better performance when using 128 blocksizes
(Figure 4). That is because we increase the usage of the memory bandwidth of
the EIB and the computation to be done in a SPE. The minimum DMA transfer
size is 1024 bytes, that is a row of 128 elements, two 4-byte floats per element.
That is the minimum size to achieve peak memory bandwidth [18].
Finally, for 128-A and 128 3 grid size case, we have reduced the DMA transfers
to do, and increased the re-use of the data per DMA transfer (Figure 3).
Figure 5 shows the speed-up based on the execution time with 1 SPE, using
the best blocksize for each transposition strategy and grid size. The speedup
achieved is not linear. That is mostly due to the contention accessing to main
memory and the conflicts on the EIB. However, one SPE strategy scales very
well for grid size 128 3 thanks to the reduction of the DMA transfers. For two
SPE strategy, we estimate the execution time considering that we have the same
speedup for one and two SPE strategies for 2 SPEs.
Finally, synchronization, necessary on accessing to the work scheduler list,
does not seem to be a performance bottleneck. However, we have to interleave
wait loops to read an atomic counter to reduce the contention.