Actas JP2011 - Universidad de La Laguna

Actas XXII Jornadas de Paralelismo (JP2011) , La Laguna, Tenerife, 7-9 septiembre 2011MemoryoperationP0 keeperP1 initiatorhome MCHitOperationsuccessfulMiss resolvedFig. 4.HitAccessto cacheYesNon−coherentrequestAccessto TLBMissPrivatepage in TLBMark pageas sharedMissNoMiss resolvedCoherentrequestCoherencerecoveryFaultAdd pagetable entry(private)NoAccess topage tableYesIs this thepage keeper?HitStore page tableentry in TLBPrivate pageBlock diagram of the general working scheme.YesNotrigger coherence recoveryrecovery requestpage P keeper P0lock page P in TLBevict cachedevictionsblocks of Pwait for pendingoperationsset TLB entry to sharedunlock page P in TLBrecovery donepage Pend coherence recoveryset page table entry to sharedOSwrite datato memoryTIMEP is used when a memory reference causes a cachemiss. Hence, if the cache miss is for a block belongingto a private page, a non-coherent requestis issued. Otherwise, a coherent request is issued.Non-coherent requests override the coherence protocoland they are always served by main memory. Inaddition, directory caches do not track them. Thisbehaviour has two primary advantages. First, neithera lookup nor an insertion in the directory cacheis required, which helps to reduce the latency of cachemisses, the contention at memory controllers, andthe energy consumption. Second, directory cachesare not so contended and, therefore, their capacitycan be better exploited to track blocks that reallyneed coherence.B. Detection of Shared PagesAs shown in Figure 3, page tables require threeadditional fields. Private (P) indicates whether thepage is private or shared. If P is set, keeper indicatesthe identity of the only processor that hasaccessed the page blocks. The cached-in-TLB bit(C) indicates whether the keeper field is valid ornot. These extra fields only require extra OS storagespace, which is very small. Particularly, their size is2 + log 2 (N) bits, where N is the number of processorsin the system. Thus, assuming an 8-processorsystem, only 5 extra bits per entry are required.On a page table fault, the OS allocates a new pagetable entry with the virtual to physical address translation.In addition, since every newly loaded page isconsidered as private, P is set and C is cleared indicatingthat the entry is not cached in any TLByet. When a TLB miss takes place in a processor,it retrieves the information from the page table andstores it in its TLB. The extra fields in the page tableare updated inside a critical section during the resolutionof TLB misses according to the following fourpremises: (1) if C is clear (i.e., the page blocks havenot been cached yet), C is set and the identity of theprocessor requesting the page table entry is kept inkeeper; (2) if both C and P are set and keeper matchesthe requester (the keeper processor suffered a TLBeviction and it requires such information again), nochanges are necessary; (3) if both C and P are setand keeper does not match the requester (two differentprocessors are trying to access blocks within thesame private page), the coherence recovery mechanismis triggered and, when it finishes, P is cleared;and (4) if C is set and P is clear (the page is shared),no changes are necessary.Fig. 5. Flushing-based recovery mechanism. P0 and P1 areprocessors and MC is the home node. Solid arrows aremessages due to the recovery mechanism, whereas dashedarrows are messages due to the coherence protocol.Figure 4 depicts all the actions that take place onmemory operations.C. Coherence Recovery MechanismBefore turning a private page into shared, the recoverymechanism has to ensure that the directorycache keeps proper track of every block within thepage. This can be done by two different strategies:(1) not modifying the directory cache and evictingall the page blocks from the keeper’s cache (flushingbasedrecovery) or (2) updating the directory cachewith proper track of every cached block within thepage (updating-based recovery). Next two sectionsexplain these mechanisms in detail.C.1 Flushing-based Recovery MechanismThe simplest way to restore the coherence statusof the blocks belonging to a page that has to becomeshared is by evicting all the page blocks so that thenext time any of them is accessed, the directory cachecan begin to keep proper track of them. This mechanismworks as follows.First, the initiator (node that triggers the coherencerecovery) issues a recovery request for the involvedpage to its keeper, which is obtained from thepage table during the page table fault resolution.Second, on the recovery request arrival, the keeperperforms four operations: (1) it locks the correspondingTLB entry by setting the L bit, which preventsthe keeper from issuing new requests for any of thepage blocks; (2) the keeper performs a cache lookup,flushing (and, if required, writing-back) every cachedblock within the involved page; (3) the keeper checksits MSHR (Miss Status Holding Registers) structureto wait for the completion of the pending misses orevictions for any of the page blocks; and (4) thekeeper marks the TLB entry corresponding to theinvolved page as shared, unlocks it, and sends to theinitiator a recovery done message.Third, when the initiator receives the recoverydone message, the recovery mechanism finalizes andthe page is set as shared in both the page table andthe local TLB. Notice that, during this process, theOS has exclusive access to the involved page table entryand no other processor can access it so that raceconditions cannot take place. Figure 5 illustrates anexample of how this mechanism works.JP2011-199