COPYRIGHT 2008, PRINCETON UNIVERSITY PRESS

368 chapter 1414.11 Parallelization StrategyA typical organization of a program for both serial and parallel tasks isMain task programMain routineSerial subroutine aParallel sub 1 Parallel sub 2 Parallel sub 3Summation taskThe user organizes the work into units called tasks, with each task assigningwork (threads) to a processor. The main task controls the overall execution as wellas the subtasks that run independent parts of the program (called parallel subroutines,slaves, guests, orsubtasks). These parallel subroutines can be distinctivesubprograms, multiple copies of the same subprogram, or even for loops.It is the programmer’s responsibility to ensure that the breakup of a code intoparallel subroutines is mathematically and scientifically valid and is an equivalentformulation of the original program.As a case in point, if the most intensive part of aprogram is the evaluation of a large Hamiltonian matrix, you may want to evaluateeach row on a different processor. Consequently, the key to parallel programming isto identify the parts of the program that may benefit from parallel execution. To dothat the programmer should understand the program’s data structures (discussedbelow), know in what order the steps in the computation must be performed, andknow how to coordinate the results generated by different processors.The programmer helps speed up the execution by keeping many processorssimultaneously busy and by avoiding storage conflicts among different parallelsubprograms. You do this load balancing by dividing your program into subtasks ofapproximately equal numerical intensity that will run simultaneously on differentprocessors. The rule of thumb is to make the task with the largest granularity−101COPYRIGHT 2008, PRINCET O N UNIVE R S I T Y P R E S SEVALUATION COPY ONLY. NOT FOR USE IN COURSES.ALLpup_06.04 — 2008/2/15 — Page 368