Is Parallel Programming Hard, And, If So, What Can You Do About It?

234 APPENDIX D. READ-COPY UPDATE IMPLEMENTATIONSpoint), then rcu_read_unlock() must decrementrcu_flipctr[0], but if the matching rcu_read_lock() executed during grace-period stage 1 (thenew stage), then rcu_read_unlock() must insteaddecrement rcu_flipctr[1].The critical point is that all rcu_flipctr elementstracking the old RCU read-side critical sectionsmust strictly decrease. Therefore, once thesum of these old counters reaches zero, it cannotchange.The rcu_read_lock() primitive uses the bottombit of the current grace-period counter(rcu_ctrlblk.completed&0x1) to index the rcu_flipctr array, and records this index in the taskstructure. The matching rcu_read_unlock() usesthis recorded value to ensure that it decrements acounter corresponding to the one that the matchingrcu_read_lock() incremented. Of course, ifthe RCU read-side critical section has been preempted,rcu_read_lock() might be decrementingthe counter belonging to a different CPU than theone whose counter was incremented by the matchingrcu_read_lock().Each CPU also maintains rcu_flip_flag andand rcu_mb_flag per-CPU variables. The rcu_flip_flag variable is used to synchronize the startof each grace-period stage: once a given CPUhas responded to its rcu_flip_flag, it must refrainfrom incrementing the rcu_flip array elementthat now corresponds to the old grace-period stage.The CPU that advances the counter (rcu_ctrlblk.completed) changes the value of each CPU’s rcu_mb_flag to rcu_flipped, but a given rcu_mb_flagmay be changed back to rcu_flip_seen only by thecorresponding CPU.The rcu_mb_flag variable is used to force eachCPU to execute a memory barrier at the end of eachgrace-period stage. These memory barriers are requiredto ensure that memory accesses from RCUread-side critical sections ending in a given graceperiodstageareorderedbeforetheendofthatstage.This approach gains the benefits memory barriersat the beginning and end of each RCU read-sidecritical section without having to actually executeall those costly barriers. The rcu_mb_flag is setto rcu_mb_needed by the CPU that detects thatthe sum of the old counters is zero, but a givenrcu_mb_flag is changed back to rcu_mb_done onlyby the corresponding CPU, and even then only afterexecuting a memory barrier.D.4.2.2 Data StructuresThis section describes preemptible RCU’s majordata structures, including rcu_ctrlblk, rcu_data, rcu_flipctr, rcu_try_flip_state, rcu_try_flip_flag, and rcu_mb_flag.rcu ctrlblk Thercu_ctrlblkstructureisglobal,and holds the lock that protects grace-period processing(fliplock) as well as holding the globalgrace-period counter (completed). The leastsignificantbit of completed is used by rcu_read_lock() to select which set of counters to increment.rcu data The rcu_data structure is a per-CPUstructure, and contains the following fields:• lock guards the remaining fields in this structure.• completedis usedtosynchronizeCPU-local activitywith the global counter in rcu_ctrlblk.• waitlistcount is used to maintain a count ofthe number of non-empty wait-lists. This fieldis used by rcu_pending() to help determine ifthis CPU has any RCU-related work left to bedone.• nextlist, nextail, waitlist, waittail,donelist, and donetail form lists containingRCU callbacks that are waiting for invocationat the end of a grace period. Each list has a tailpointer, allowing O(1) appends. The RCU callbacksflow through these lists as shown below.• rcupreempt_trace accumulates statistics.Figure D.62 shows how RCU callbacks flowthrough a given rcu_data structure’s lists, from creationby call_rcu() through invocation by rcu_process_callbacks(). Each blue arrow representsone pass by the grace-period state machine, whichis described in a later section.rcu flipctr As noted earlier, the rcu_flipctrper-CPUarrayofcounterscontainsthecounterpairsthat track outstanding RCU read-side critical sections.Any given counter in this array can go negative,forexample, whenataskismigratedtoadifferentCPU in the middle of an RCU read-side criticalsection. However, the sum of the counters will stillremain positive throughout the corresponding graceperiod, and will furthermore go to zero at the endof that grace period.