Is Parallel Programming Hard, And, If So, What Can You Do About It?

Chapter 1IntroductionParallel programming has earned a reputation asone of the most difficult areas a hacker can tackle.Papers and textbooks warn of the perils of deadlock,livelock, race conditions, non-determinism,Amdahl’s-Law limits to scaling, and excessive realtimelatencies. And these perils are quite real; weauthors have accumulated uncounted years of experiencedealing with them, and all of the emotionalscars, grey hairs, and hair loss that go with such anexperience.However, new technologies have always been difficultto use at introduction, but have invariably becomeeasier over time. For example, there was atime when the ability to drive a car was a rare skill,but in many developed countries, this skill is nowcommonplace. This dramatic change came aboutfor two basic reasons: (1) cars became cheaper andmore readily available, so that more people had theopportunity to learn to drive, and (2) cars becamesimpler to operate, due to automatic transmissions,automatic chokes, automatic starters, greatly improvedreliability, and a host of other technologicalimprovements.The same is true of a host of other technologies,including computers. It is no longer necessary tooperate a keypunch in order to program. Spreadsheetsallow most non-programmers to get resultsfrom their computers that would have required ateam of specialists a few decades ago. Perhaps themost compelling example is web-surfing and contentcreation, which since the early 2000s has beeneasily done by untrained, uneducated people usingvarious now-commonplace social-networking tools.As recently as 1968, such content creation was afar-out research project [Eng68], described at thetime as “like a UFO landing on the White Houselawn”[Gri00].Therefore, if you wish to argue that parallel programmingwill remain as difficult as it is currentlyperceivedbymanytobe, itisyouwhobearstheburdenof proof, keeping in mind the many centuries ofcounter-examples in a variety of fields of endeavor.1.1 Historic Parallel ProgrammingDifficultiesAs indicated by its title, this book takes a differentapproach. Ratherthancomplainaboutthedifficultyofparallelprogramming, itinsteadexaminesthereasonswhy parallel programming is difficult, and thenworks to help the reader to overcome these difficulties.As will be seen, these difficulties have falleninto several categories, including:1. The historic high cost and relative rarity of parallelsystems.2. The typical researcher’s and practitioner’s lackof experience with parallel systems.3. The paucity of publicly accessible parallel code.4. Thelackofawidelyunderstoodengineeringdisciplineof parallel programming.5. The high cost of communication relative to thatof processing, even in tightly coupled sharedmemorycomputers.Many of these historic difficulties are well on theway to being overcome. First, over the past fewdecades, the cost of parallel systems has decreasedfrom many multiples of that of a house to a fractionof that of a used car, thanks to the advent ofmulticore systems. Papers calling out the advantagesof multicore CPUs were published as earlyas 1996 [ONH + 96], IBM introduced simultaneousmulti-threading into its high-end POWER familyin 2000, and multicore in 2001. Intel introducedhyperthreading into its commodity Pentium line inNovember 2000, and both AMD and Intel introduceddual-core CPUs in 2005. Sun followed withthe multicore/multi-threaded Niagara in late 2005.