Is Parallel Programming Hard, And, If So, What Can You Do About It?

C.6. EXAMPLE MEMORY-BARRIER SEQUENCES 1715. CPU 0 executes b=1. It already owns this cacheline (in other words, the cache line is already ineither the “modified” or the “exclusive” state),so it stores the new value of “b” in its cacheline.6. CPU 0 receives the “read” message, and transmitsthe cache line containing the now-updatedvalue of “b” to CPU 1, also marking the line as“shared” in its own cache.7. CPU 1 receives the cache line containing “b”and installs it in its cache.8. CPU 1 can now finish executing while(b==0)continue, and since it finds that the value of“b” is 1, it proceeds to the next statement,which is now a memory barrier.9. CPU 1 must now stall until it processes all preexistingmessages in its invalidation queue.10. CPU 1 now processes the queued “invalidate”message, and invalidates the cache line containing“a” from its own cache.11. CPU 1 executes the assert(a==1), and, sincethe cache line containing “a” is no longer inCPU 1’s cache, it transmits a “read” message.12. CPU 0 responds to this “read” message withthe cache line containing the new value of “a”.13. CPU 1 receives this cache line, which containsa value of 1 for “a”, so that the assertion doesnot trigger.With much passing of MESI messages, the CPUsarrive at the correct answer. This section illustrateswhy CPU designers must be extremely careful withtheir cache-coherence optimizations.C.5 Read and Write MemoryBarriersIn the previous section, memory barriers were usedto mark entries in both the store buffer and the invalidatequeue.Butinourcodefragment,foo()hadno reason to do anything with the invalidate queue,and bar() simlarly had no reason to do anythingwith the store queue.Many CPU architectures therefore provide weakermemory-barrier instructions that do only one or theother of these two. Roughly speaking, a “read memorybarrier” marks only the invalidate queue and a“write memory barrier” marks only the store buffer,while a full-fledged memory barrier does both.The effect of this is that a read memory barrierorders only loads on the CPU that executes it, sothatallloadsprecedingthereadmemorybarrierwillappear to have completed before any load followingthe read memory barrier. Similarly, a write memorybarrier orders only stores, again on the CPU thatexecutesit, andagainsothatallstoresprecedingthewrite memory barrier will appear to have completedbefore any store following the write memory barrier.A full-fledged memory barrier orders both loads andstores, but again only on the CPU executing thememory barrier.If we update foo and bar to use read and writememory barriers, they appear as follows:1 void foo(void)2 {3 a = 1;4 smp_wmb();5 b = 1;6 }78 void bar(void)9 {10 while (b == 0) continue;11 smp_rmb();12 assert(a == 1);13 }Some computers have even more flavors of memorybarriers, but understanding these three variantswill provide a good introduction to memory barriersin general.C.6 Example Memory-BarrierSequencesThis section presents some seductive but subtly brokenuses of memory barriers. Although many ofthem will work most of the time, and some will workall the time on some specific CPUs, these uses mustbe avoided if the goal is to produce code that worksreliably on all CPUs. To help us better see the subtlebreakage, we first need to focus on an orderinghostilearchitecture.C.6.1 Ordering-Hostile ArchitecturePaul has come across a number of ordering-hostilecomputersystems, butthenatureofthehostilityhasalways been extremely subtle, and understanding ithas required detailed knowledge of the specific hardware.Rather than picking on a specific hardware