Is Parallel Programming Hard, And, If So, What Can You Do About It?

F.2. CHAPTER 2: HARDWARE AND ITS HABITS 275well.2. If you prefer a more academic and rigoroustreatment of parallel programming, you mightlike Herlihy’s and Shavit’s textbook [HS08].This book starts with an interesting combinationof low-level primitives at high levels of abstractionfrom the hardware, and works its waythrough locking and simple data structures includinglists, queues, hash tables, and counters,culminating with transactional memory.3. If you would like an academic treatment ofparallel programming that keeps to a morepragmatic viewpoint, you might be interestedin the concurrency chapter from Scott’s textbook[Sco06] on programming languages.4. If you are interested in an object-oriented patternisttreatment of parallel programming focussingon C++, you might try Volumes 2 and4 of Schmidt’s POSA series [SSRB00, BHS07].Volume 4 in particular has some interestingchapters applying this work to a warehouse application.The realism of this example is attestedto by the section entitled “Partitioningthe Big Ball of Mud”, wherein the problemsinherent in parallelism often take a back seatto the problems inherent in getting one’s headaround a real-world application.5. If your primary focus is scientific and technicalcomputing, and you prefer a patternist approach,you might try Mattson et al.’s textbook[MSM05]. It covers Java, C/C++,OpenMP, and MPI. Its patterns are admirablyfocusedfirstondesign, thenonimplementation.6. If you are interested in POSIX Threads, youmight take a look at David R. Butenhof’sbook [But97].7. If you are interested in C++, but in a Windowsenvironment, you might try Herb Sutter’s“Effective Concurrency” series in Dr. DobbsJournal [Sut08]. This series does a reasonablejob of presenting a commonsense approach toparallelism.8. If you want to try out Intel Threading BuildingBlocks, then perhaps James Reinders’sbook [Rei07] is what you are looking for.9. Finally, those preferring to work in Java mightbe well-served by Doug Lea’s textbooks [Lea97,GPB + 07].In contrast, this book meshes real-world machineswith real-world algorithms. If your sole goal is tofind an optimal parallel queue, you might be betterserved by one of the above books. However, if youare interested in principles of parallel design thatallow multiple such queues to operate in parallel,read on!F.2 Chapter 2: Hardware andits HabitsQuick Quiz 2.1:Why should parallel programmers bother learninglow-level properties of the hardware? Wouldn’t itbe easier, better, and more general to remain at ahigher level of abstraction?Answer:It might well be easier to ignore the detailed propertiesof the hardware, but in most cases it wouldbe quite foolish to do so. If you accept that the onlypurpose of parallelism is to increase performance,and if you further accept that performance dependson detailed properties of the hardware, then itlogically follows that parallel programmers aregoing to need to know at least a few hardwareproperties.This is the case in most engineering disciplines.Would you want to use a bridge designed by an engineerwho did not understand the properties of theconcrete and steel making up that bridge? If not,why would you expect a parallel programmer to beable to develop competent parallel software withoutat least some understanding of the underlying hardware?Quick Quiz 2.2:What types of machines would allow atomic operationson multiple data elements?Answer:One answer to this question is that it is often possibleto pack multiple elements of data into a singlemachine word, which can then be manipulatedatomically.A more trendy answer would be machines supportingtransactional memory [Lom77]. However,such machines are still (as of 2008) research curiosities.The jury is still out on the applicability oftransactional memory [MMW07, PW07, RHP + 07].