Compile-time Loop Splitting for Distributed Memory ... - Stanford AI Lab

on a uniprocessor. In removing the uniprocessor, scalable multiprocessors also remove the
single memory address space. Array referencing must not only consider where in memory
an array is located, but also in which processor’s memory, thus requiring more calculation.
Virtual memory can eliminate the additional addressing calculations by providing the
abstraction of a uniform, linear address space. However, this abstraction results in an
efficiency loss because an unnecessarily large number of memory references may need to be
satisfied by a remote node.
The remainder of this section describes in more detail what has been outlined above.
First, Section 2.2 presents task and data partitioning, while introducing terms relevant to
loop splitting. Then, Section 2.3 describes the procedure for determining the optimal task
and data partitions. Section 2.4 describes the problem with array referencing on a
distributed memory multiprocessor. Finally, Section 2.5 examines a method of
multiprocessor array referencing, the complex expressions involved, and the possibility of
reducing these expressions.
2.2 Partitioning
While a uniprocessor executes an entire program, a processing element (PE) in a
distributed memory multiprocessor executes only a portion of the program. Dividing a
program into portions for the PEs is called partitioning, of which there are two types –
task partitioning and data partitioning.
2.2.1 Task Partitioning
Performed for both distributed and shared memory multiprocessors, task partitioning
assigns a portion of work to each processor such that the sum of all processor tasks is
equivalent to the original single task. Two types of task partitioning exist: dynamic and
static. Dynamic task partitioning is determined at the time of execution, where processes
spawn other processes until all processors are busy. This is achieved by fork operations,
which are later joined to merge the results of two processors onto one processor. The fork
and join operations are well-suited to recursive algorithms.
14