Is Parallel Programming Hard, And, If So, What Can You Do About It?

42 CHAPTER 4. COUNTING1 #define THEFT_IDLE 02 #define THEFT_REQ 13 #define THEFT_ACK 24 #define THEFT_READY 356 int __thread theft = THEFT_IDLE;7 int __thread counting = 0;8 unsigned long __thread counter = 0;9 unsigned long __thread countermax = 0;10 unsigned long globalcountmax = 10000;11 unsigned long globalcount = 0;12 unsigned long globalreserve = 0;13 unsigned long *counterp[NR_THREADS] = { NULL };14 unsigned long *countermaxp[NR_THREADS] = { NULL };15 int *theftp[NR_THREADS] = { NULL };16 DEFINE_SPINLOCK(gblcnt_mutex);17 #define MAX_COUNTERMAX 100Figure 4.20: Signal-Theft Limit Counter Datasets the theft state to READY.Once the slowpath sees a thread’s theft stateis READY, the slowpath is permitted to steal thatthread’scount. Theslowpaththensetsthatthread’stheft state to IDLE.Quick Quiz 4.37: In Figure 4.19, why is theREQ theft state colored blue?Quick Quiz 4.38: In Figure 4.19, what is thepoint of having separate REQ and ACK theftstates? Why not simplify the state machine by collapsingthem into a single state? Then whicheverof the signal handler or the fastpath gets there firstcould set the state to READY.4.4.4 Signal-Theft Limit Counter ImplementationFigure 4.20 (count_lim_sig.c) shows the datastructures used by the signal-theft based counterimplementation. Lines 1-7 define the states andvalues for the per-thread theft state machine describedin the preceding section. Lines 8-17 are similarto earlier implementations, with the addition oflines 14 and 15 to allow remote access to a thread’scountermax and theft variables, respectively.Figure 4.21 shows the functions responsible formigrating counts between per-thread variables andthe global variables. Lines 1-7 shows global_count(), which is identical to earlier implementations.Lines 9-19 showsflush_local_count_sig(),which is the signal handler used in the theft process.Lines 11 and 12 check to see if the theft state isREQ,and,ifnotreturnswithoutchange. Line13executesa memory barrier to ensure that the samplingof the theft variable happens before any change tothat variable. Line 14 sets the theft state to ACK,and, if line 15 sees that this thread’s fastpaths arenot running, line 16 sets thetheft state to READY.1 static void globalize_count(void)2 {3 globalcount += counter;4 counter = 0;5 globalreserve -= countermax;6 countermax = 0;7 }89 static void flush_local_count_sig(int unused)10 {11 if (ACCESS_ONCE(theft) != THEFT_REQ)12 return;13 smp_mb();14 ACCESS_ONCE(theft) = THEFT_ACK;15 if (!counting) {16 ACCESS_ONCE(theft) = THEFT_READY;17 }18 smp_mb();19 }2021 static void flush_local_count(void)22 {23 int t;24 thread_id_t tid;2526 for_each_tid(t, tid)27 if (theftp[t] != NULL) {28 if (*countermaxp[t] == 0) {29 ACCESS_ONCE(*theftp[t]) = THEFT_READY;30 continue;31 }32 ACCESS_ONCE(*theftp[t]) = THEFT_REQ;33 pthread_kill(tid, SIGUSR1);34 }35 for_each_tid(t, tid) {36 if (theftp[t] == NULL)37 continue;38 while (ACCESS_ONCE(*theftp[t]) != THEFT_READY) {39 poll(NULL, 0, 1);40 if (ACCESS_ONCE(*theftp[t]) == THEFT_REQ)41 pthread_kill(tid, SIGUSR1);42 }43 globalcount += *counterp[t];44 *counterp[t] = 0;45 globalreserve -= *countermaxp[t];46 *countermaxp[t] = 0;47 ACCESS_ONCE(*theftp[t]) = THEFT_IDLE;48 }49 }5051 static void balance_count(void)52 {53 countermax = globalcountmax -54 globalcount - globalreserve;55 countermax /= num_online_threads();56 if (countermax > MAX_COUNTERMAX)57 countermax = MAX_COUNTERMAX;58 globalreserve += countermax;59 counter = countermax / 2;60 if (counter > globalcount)61 counter = globalcount;62 globalcount -= counter;63 }Figure 4.21: Signal-Theft Limit Counter Value-Migration Functions