Is Parallel Programming Hard, And, If So, What Can You Do About It?

Appendix FAnswers to Quick QuizzesF.1 Chapter 1: IntroductionQuick Quiz 1.1:Come on now!!! Parallel programming has beenknown to be exceedingly hard for many decades.You seem to be hinting that it is not so hard. Whatsort of game are you playing?Answer:If you really believe that parallel programming isexceedingly hard, then you should have a readyanswer to the question “Why is parallel programminghard?” One could list any number of reasons,ranging from deadlocks to race conditions to testingcoverage, but the real answer is that it is not reallyall that hard. After all, if parallel programming wasreally so horribly difficult, how could a large numberof open-source projects, ranging from Apache toMySQL to the Linux kernel, have managed tomaster it?A better question might be: ”Why is parallelprogramming perceived to be so difficult?” To seethe answer, let’s go back to the year 1991. PaulMcKenney was walking across the parking lot to Sequent’sbenchmarkingcentercarryingsixdual-80486Sequent Symmetry CPU boards, when he suddenlyrealized that he was carrying several times the priceof the house he had just purchased. 1 This high costof parallel systems meant that parallel programmingwas restricted to a privileged few who worked for anemployer who either manufactured or could affordto purchase machines costing upwards of $100,000— in 1991 dollars US.1 Yes, this sudden realization did cause him to walk quitea bit more carefully. Why do you ask?In contrast, in 2006, Paul finds himself typingthese words on a dual-core x86 laptop. Unlike thedual-80486 CPU boards, this laptop also contains2GB of main memory, a 60GB disk drive, a display,Ethernet, USB ports, wireless, and Bluetooth.And the laptop is more than an order of magnitudecheaperthanevenoneofthosedual-80486CPUboards, even before taking inflation into account.Parallel systems have truly arrived. They are nolonger the sole domain of a privileged few, but somethingavailable to almost everyone.The earlier restricted availability of parallel hardwareis the real reason that parallel programmingis considered so difficult. After all, it is quite difficultto learn to program even the simplest machineif you have no access to it. Since the age of rare andexpensive parallel machines is for the most part behindus, the age during which parallel programmingis perceived to be mind-crushingly difficult is comingto a close. 2Quick Quiz 1.2:How could parallel programming ever be as easy assequential programming???Answer:It depends on the programming environment.SQL [Int92] is an underappreciated success story,as it permits programmers who know nothingabout parallelism to keep a large parallel systemproductively busy. We can expect more variationson this theme as parallel computers continue tobecome cheaper and more readily available. Forexample, one possible contender in the scientific andtechnical computing arena is MATLAB*P, whichis an attempt to automatically parallelize comonmatrix operations.2 Parallel programming is in some ways more difficult thansequential programming, for example, parallel validation ismore difficult. But no longer mind-crushingly difficult.