Intel® Trace Analyzer User's Reference Guide

Intel® Trace Analyzer User's Reference Guide
Note that the time spent in MPI_Sendrecv grows with the process number while
the time for MPI_Allreduce decreases. The Message Profile (the section called
“The Message Profile”) in the bottom right corner of Figure 1.9, “Many Charts”
shows that messages traveling from a higher rank to a lower rank need more and
more time with increasing rank while the messages traveling from lower rank to
higher rank do reveal a weak even-odd kind of pattern.
Figure 1.9. Many Charts
As poisson_sendrecv.single.stf is such a striking example of serialization,
next to all Charts provided by the Intel Trace Analyzer reveal this interesting
pattern. But in real world cases it might be necessary to formulate a hypothesis
regarding how the program should behave and to check this hypothesis using
the most adequate Chart.
A possible way to improve the performance of the program is to use nonblocking
communication to replace the usage of MPI_Sendrecv and to avoid the
serialization in this way. One iteration of the resulting program looks like the one
shown in Figure 1.10, “One iteration of the improved version”. Note that a single
iteration now takes about 0.9 milliseconds, while it took about 1.4 milliseconds
before the change.
Figure 1.10. One iteration of the improved version
To compare two trace files the Intel Trace Analyzer offers the so called
Comparison View (see Chapter 7, Comparison of two Trace Files). Just
use the menu entry View Menu → View → Compare in the View showing
poisson_sendrecv.single.stf. In the now appearing dialog choose another
Document number: 318120-002 8