Is Parallel Programming Hard, And, If So, What Can You Do About It?

F.3. CHAPTER 3: TOOLS OF THE TRADE 277Answer:Get a roll of toilet paper. In the USA, each roll willnormally have somewhere around 350-500 sheets.Tear off one sheet to represent a single clock cycle,setting it aside. Now unroll the rest of the roll.The resulting pile of toilet paper will likely representa single CAS cache miss.For the more-expensive inter-system communicationslatencies, use several rolls (or multiple cases)of toilet paper to represent the communications latency.Important safety tip: make sure to account forthe needs of those you live with when appropriatingtoilet paper!Quick Quiz 2.7:Given that distributed-systems communication isso horribly expensive, why does anyone bother withthem?Answer:There are a number of reasons:1. Shared memory multiprocessor have strict sizelimits, If you need more than a few thousandCPUs, you have no choice but to use a distributedsystem.2. Extremely large shared-memory systems tendto be quite expensive and to have even longercache-miss latencies than does the small four-CPU system shown in Table 2.1.3. The distributed-systems communications latenciesdo not necessarily consume the CPU, whichcan often allow computation to proceed in parallelwith message transfer.4. Many important problems are “embarrassinglyparallel”, so that extremely large quantities ofprocessingmaybeenabledbyaverysmallnumberof messages. SETI@HOME [aCB08] is butone example of such an application. These sortsof applications can make good use of networksof computers despite extremely long communicationslatencies.It is likely that continued work on parallel applicationswill increase the number of embarrassinglyparallel applications that can run well on machinesand/or clusters having long communications latencies.That said, greatly reduced hardware latencieswould be an extremely welcome development.F.3 Chapter 3: Tools of theTradeQuick Quiz 3.1:But this silly shell script isn’t a real parallelprogram!!! Why bother with such trivia???Answer:Because you should never forget the simple stuff!!!Please keep in mind that the title of this book is“Is Parallel Programming Hard, And, If So, WhatCan You Do About It?”. One of the most effectivethings you can do about it is to avoid forgetting thesimple stuff! After all, if you choose to do parallelprogramming the hard way, you have no one butyourself to blame for it being hard.Quick Quiz 3.2:Is there a simpler way to create a parallel shellscript? If so, how? If not, why not?Answer:One straightforward approach is the shell pipeline:grep $pattern1 | sed -e ’s/a/b/’ | sortFor a sufficiently large input file, grep will patternmatchin parallel with sed editing and with theinput processing of sort. See the file parallel.shfor a demonstration of shell-script parallelism andpipelining.Quick Quiz 3.3:But if script-based parallel programming is so easy,why bother with anything else?Answer:In fact, it is quite likely that a very large fractionof parallel programs in use today are script-based.However, script-based parallelism does have itslimitations:1. Creation of new processes is usually quiteheavyweight, involving the expensive fork()and exec() system calls.2. Sharing of data, including pipelining, typicallyinvolves expensive file I/O.3. The reliable synchronization primitives availableto scripts also typically involve expensivefile I/O.