Is Parallel Programming Hard, And, If So, What Can You Do About It?

F.14. CHAPTER C: WHY MEMORY BARRIERS? 311would be a parametric study using a single independentvariable. If the program run study tooka single argument, then we could use the followingbash script to run two instances in parallel, asmight be appropriate on a two-CPU system:run study 1 > 1.out& run study 2 > 2.out; waitOne could of course argue that the bash ampersandoperator and the “wait” primitive are in factsynchronization primitives. If so, then consider thatthis script could be run manually in two separatecommand windows, so that the only synchronizationwould be supplied by the user himself or herself.Quick Quiz B.2:What problems could occur if the variable counterwere incremented without the protection of mutex?Answer:On CPUs with load-store architectures, incrementingcounter might compile into something like thefollowing:LOAD counter,r0INC r0STORE r0,counterOn such machines, two threads might simultaneouslyload the value of counter, each increment it,and each store the result. The new value of counterwillthenonlybeonegreaterthanbefore,despitetwothreads each incrementing it.Quick Quiz B.3:How could you work around the lack of a perthread-variableAPI on systems that do not provideit?Answer:One approach would be to create an array indexedby smp thread id(), and another would be to usea hash table to map from smp thread id() to anarray index — which is in fact what this set of APIsdoes in pthread environments.Another approach would be for the parent to allocatea structure containing fields for each desiredper-thread variable, then pass this to the child duringthread creation. However, this approach can imposelarge software-engineering costs in large systems.To see this, imagine if all global variables ina large system had to be declared in a single file,regardless of whether or not they were C static variables!F.14 Chapter C: Why MemoryBarriers?Quick Quiz C.1:What happens if two CPUs attempt to invalidatethe same cache line concurrently?Answer:One of the CPUs gains access to the shared busfirst, and that CPU “wins”. The other CPU mustinvalidate its copy of the cache line and transmitan “invalidate acknowledge” message to the otherCPU.Of course, the losing CPU can be expected to immediatelyissue a “read invalidate” transaction, sothe winning CPU’s victory will be quite ephemeral.Quick Quiz C.2:When an “invalidate” message appears in a largemultiprocessor, every CPU must give an “invalidateacknowledge” response. Wouldn’t the resulting“storm” of “invalidate acknowledge” responsestotally saturate the system bus?Answer:It might, if large-scale multiprocessors were in factimplemented that way. Larger multiprocessors,particularly NUMA machines, tend to use so-called“directory-based” cache-coherence protocols toavoid this and other problems.Quick Quiz C.3:If SMP machines are really using message passinganyway, why bother with SMP at all?Answer:There has been quite a bit of controversy on thistopic over the past few decades. One answer is thatthe cache-coherence protocols are quite simple, andtherefore can be implemented directly in hardware,gaining bandwidths and latencies unattainable bysoftware message passing. Another answer is thatthe real truth is to be found in economics due to therelative prices of large SMP machines and that ofclusters of smaller SMP machines. A third answeris that the SMP programming model is easier touse than that of distributed systems, but a rebuttalmight note the appearance of HPC clusters andMPI. And so the argument continues.