Is Parallel Programming Hard, And, If So, What Can You Do About It?

266 APPENDIX E. FORMAL VERIFICATION1 struct rcu_dynticks {2 int dynticks_nesting;3 int dynticks;4 int dynticks_nmi;5 };67 struct rcu_data {8 ...9 int dynticks_snap;10 int dynticks_nmi_snap;11 ...12 };Figure E.18: Variables for Simple Dynticks InterfaceE.8 Simplicity Avoids FormalVerificationThe complexity of the dynticks interface for preemptableRCU is primarily due to the fact that bothirqs and NMIs use the same code path and the samestate variables. This leads to the notion of providingseparate code paths and variables for irqs and NMIs,as has been done for hierarchical RCU [McK08a] asindirectly suggested by Manfred Spraul [Spr08b].E.8.1 State Variables for SimplifiedDynticks InterfaceFigure E.18 shows the new per-CPU state variables.These variables are grouped into structs toallow multiple independent RCU implementations(e.g.,rcuandrcu_bh)toconvenientlyandefficientlyshare dynticks state. In what follows, they can bethought of as independent per-CPU variables.The dynticks_nesting, dynticks, anddynticks_snap variables are for the irq code paths,and the dynticks_nmi and dynticks_nmi_snapvariables are for the NMI code paths, although theNMI code path will also reference (but not modify)the dynticks_nesting variable. These variablesare used as follows:dynticks_nesting: This counts the number of reasonsthat the corresponding CPU should bemonitored for RCU read-side critical sections.If the CPU is in dynticks-idle mode, then thiscounts the irq nesting level, otherwise it is onegreater than the irq nesting level.dynticks: This counter’s value is even if the correspondingCPU is in dynticks-idle mode andthere are no irq handlers currently running onthat CPU, otherwise the counter’s value is odd.In other words, if this counter’s value is odd,then the corresponding CPU might be in anRCU read-side critical section.1 void rcu_enter_nohz(void)2 {3 unsigned long flags;4 struct rcu_dynticks *rdtp;56 smp_mb();7 local_irq_save(flags);8 rdtp = &__get_cpu_var(rcu_dynticks);9 rdtp->dynticks++;10 rdtp->dynticks_nesting--;11 WARN_ON_RATELIMIT(rdtp->dynticks & 0x1, &rcu_rs);12 local_irq_restore(flags);13 }1415 void rcu_exit_nohz(void)16 {17 unsigned long flags;18 struct rcu_dynticks *rdtp;1920 local_irq_save(flags);21 rdtp = &__get_cpu_var(rcu_dynticks);22 rdtp->dynticks++;23 rdtp->dynticks_nesting++;24 WARN_ON_RATELIMIT(!(rdtp->dynticks & 0x1), &rcu_rs);25 local_irq_restore(flags);26 smp_mb();27 }Figure E.19: Entering and Exiting Dynticks-IdleModedynticks_nmi: This counter’s value is odd if thecorresponding CPU is in an NMI handler, butonly if the NMI arrived while this CPU was indyntick-idle mode with no irq handlers running.Otherwise, the counter’s value will be even.dynticks_snap: This will be a snapshot of thedynticks counter, but only if the current RCUgrace period has extended for too long a duration.dynticks_nmi_snap: This will be a snapshot of thedynticks_nmi counter, but again only if thecurrent RCU grace period has extended for toolong a duration.If both dynticks and dynticks_nmi have takenon an even value during a given time interval, thenthe corresponding CPU has passed through a quiescentstate during that interval.Quick Quiz E.19: But what happens if an NMIhandler starts running before an irq handler completes,and if that NMI handler continues runninguntil a second irq handler starts?E.8.2 Entering and LeavingDynticks-Idle ModeFigure E.19 shows the rcu_enter_nohz() and rcu_exit_nohz(), which enter and exit dynticks-idlemode, also known as “nohz” mode. These two functionsare invoked from process context.