Is Parallel Programming Hard, And, If So, What Can You Do About It?

2 CHAPTER 1. INTRODUCTIONIn fact, in 2008, it is becoming difficult to find asingle-CPU desktop system, with single-core CPUsbeing relegated to netbooks and embedded devices.Second, the advent of low-cost and readily availablemulticore system means that the once-rare experienceof parallel programming is now availableto almost all researchers and practitioners. In fact,parallel systems are now well within the budget ofstudents and hobbyists. We can therefore expectgreatly increased levels of invention and innovationsurrounding parallel systems, and that increased familiaritywill over time make once-forbidding field ofparallel programming much more friendly and commonplace.Third, where in the 20 th century, large systems ofhighly parallel software were almost always closelyguarded proprietary secrets, the 21 st century hasseen numerous open-source (and thus publicly available)parallel software projects, including the Linuxkernel [Tor03], database systems [Pos08, MS08],and message-passing systems [The08, UoC08]. Thisbook will draw primarily from the Linux kernel, butwill provide much material suitable for user-level applications.Fourth, even though the large-scale parallelprogrammingprojects of the 1980s and 1990s werealmost all proprietary projects, these projects haveseeded the community with a cadre of developerswho understand the engineering discipline requiredto develop production-quality parallel code. A majorpurposeofthisbookistopresentthisengineeringdiscipline.Unfortunately, the fifth difficulty, the high costof communication relative to that of processing, remainslargely in force. Although this difficulty hasbeen receiving increasing attention during the newmillenium, according to Stephen Hawkings, the finitespeed of light and the atomic nature of matterislikelytolimitprogressinthisarea[Gar07,Moo03].Fortunately, thisdifficultyhasbeeninforcesincethelate 1980s, so that the aforementioned engineeringdiscipline has evolved practical and effective strategiesfor handling it. In addition, hardware designersare increasingly aware of these issues, so perhaps futurehardware will be more friendly to parallel softwareas discussed in Section 2.3.Quick Quiz 1.1: Come on now!!! Parallel programminghas been known to be exceedingly hardfor many decades. You seem to be hinting that it isnot so hard. What sort of game are you playing?However, even though parallel programmingmight not be as hard as is commonly advertised, itis often more work than is sequential programming.Quick Quiz 1.2: How could parallel programmingever be as easy as sequential programming???Itthereforemakessensetoconsideralternativestoparallel programming. However, it is not possible toreasonably consider parallel-programming alternativeswithout understanding parallel-programminggoals. This topic is addressed in the next section.1.2 Parallel ProgrammingGoalsThe three major goals of parallel programming (overand above those of sequential programming) are asfollows:1. Performance.2. Productivity.3. Generality.Quick Quiz 1.3: What about correctness, maintainability,robustness, and so on???Quick Quiz 1.4: And if correctness, maintainability,and robustness don’t make the list, why doproductivity and generality???Quick Quiz 1.5: Given that parallel programsare much harder to prove correct than are sequentialprograms, again, shouldn’t correctness really be onthe list?Quick Quiz 1.6: What about just having fun???Each of these goals is elaborated upon in the followingsections.1.2.1 PerformancePerformance is the primary goal behind mostparallel-programming effort. After all, if performanceis not a concern, why not do yourself a favor,just write sequential code, and be happy? It willvery likely be easier, and you will probably get donemuch more quickly.Quick Quiz 1.7: Are there no cases where parallelprogramming is about something other than performance?Note that “performance” is interpreted quitebroadly here, including scalability (performance perCPU) and efficiency (for example, performance perwatt).That said, the focus of performance has shiftedfrom hardware to parallel software. This changein focus is due to the fact that Moore’s Law has