Is Parallel Programming Hard, And, If So, What Can You Do About It?

174 APPENDIX C. WHY MEMORY BARRIERS?Loads Reordered After Loads?Loads Reordered After Stores?Stores Reordered After Stores?Stores Reordered After Loads?Atomic Instructions Reordered With Loads?Atomic Instructions Reordered With Stores?Dependent Loads Reordered?Incoherent Instruction Cache/Pipeline?Alpha Y Y Y Y Y Y Y YAMD64YARMv7-A/R Y Y Y Y Y Y YIA64 Y Y Y Y Y Y Y(PA-RISC) Y Y Y YPA-RISC CPUsPOWER TM Y Y Y Y Y Y Y(SPARC RMO) Y Y Y Y Y Y Y(SPARC PSO) Y Y Y YSPARC TSO Y Yx86 Y Y(x86 OOStore) Y Y Y Y YzSeries R○ Y YTable C.5: Summary of Memory Orderingtice.The common ”just say no” approach to memorybarriers can be eminently reasonable where itapplies, but there are environments, such as theLinux kernel, where direct use of memory barriersis required. Therefore, Linux provides a carefullychosen least-common-denominator set of memorybarrierprimitives, which are as follows:• smp mb(): “memory barrier” that orders bothloads and stores. This means that loads andstores preceding the memory barrier will becommitted to memory before any loads andstores following the memory barrier.• smp rmb(): “read memory barrier” that ordersonly loads.• smp wmb(): “write memory barrier” that ordersonly stores.• smp read barrier depends() that forces subsequentoperations that depend on prior operationsto be ordered. This primitive is a no-opon all platforms except Alpha.• mmiowb() that forces ordering on MMIO writesthat are guarded by global spinlocks. Thisprimitive is a no-op on all platforms on whichthe memory barriers in spinlocks already enforceMMIO ordering. The platforms witha non-no-op mmiowb() definition include some(but not all) IA64, FRV, MIPS, and SH systems.This primitive is relatively new, so relativelyfew drivers take advantage of it.The smp mb(), smp rmb(), and smp wmb() primitivesalso force the compiler to eschew any optimizationsthat would have the effect of reorderingmemory optimizations across the barriers. Thesmp read barrier depends() primitive has a similareffect, but only on Alpha CPUs. See Section12.2 for more information on use of these primitives.Theseprimitives generate code only in SMPkernels, however, each also has a UP version (mb(),rmb(), wmb(), and read barrier depends(), respectively)that generate a memory barrier even inUP kernels. The smp versions should be used inmost cases. However, these latter primitives are usefulwhen writing drivers, because MMIO accessesmust remain ordered even in UP kernels. In absenceof memory-barrier instructions, both CPUsand compilers would happily rearrange these accesses,which at best would make the device actstrangely, and could crash your kernel or, in somecases, even damage your hardware.Somostkernelprogrammersneednotworryaboutthe memory-barrier peculiarities of each and everyCPU, as long as they stick to these interfaces. Ifyou are working deep in a given CPU’s architecturespecificcode, of course, all bets are off.Furthermore, all of Linux’s locking primitives(spinlocks, reader-writer locks, semaphores, RCU,...) include any needed barrier primitives. So ifyou are working with code that uses these primitives,you don’t even need to worry about Linux’smemory-ordering primitives.That said, deep knowledge of each CPU’smemory-consistency model can be very helpfulwhen debugging, to say nothing of when writingarchitecture-specific code or synchronization primitives.Besides, they say that a little knowledge is a verydangerousthing. Justimaginethedamageyoucoulddo with a lot of knowledge! For those who wish tounderstand more about individual CPUs’ memoryconsistencymodels, thenextsectionsdescribesthoseof the most popular and prominent CPUs. Although