Unit OS5: Windows Memory Management

Windows Memory Management (I)Memory Manager Concepts and Services

2Roadmap for This LectureMemory Manager FundamentalsMemory Management ServicesVirtual Address Space LayoutWorking SetsKernel Mode HeapsHeap ManagerPhysical Memory LimitsMemory management APIs

3IntroductionMain Tasks:Translating (mapping) a process’s virtual address space intophysical memory, so running threads correctly referencephysical addressPaging in and out contents between memory and disk whenthreads try to use more memory than currently availableAdditional Services:Memory-mapped files (section object)Copy-on-write memoryLarge, sparse address spaceUse of larger physical memory than max address space for aprocess (AWE)

4Key FeaturesFully reentrant and provides synchronization toPage frame number (PFN) databaseSection objects and working setsPage file creationLazy evaluationSharing – usage of prototype PTEs (page table entries)Extensive usage of copy-on-write...whenever possibleShares memoryCopy-on-writeMapped files

5Main ComponentsSystem services for allocating, deallocating, and managing virtualmemoryAn access fault trap handler for resolving hardware-detected memorymanagement exceptions and making virtual pages resident on behalf of aprocessSix system threadsWorking set manager (priority 16) – drives overall memory managementpolicies, such as working set trimming, aging, and modified page writingProcess/stack swapper (priority 23) -- performs both process and kernelthread stack inswapping and outswappingModified page writer (priority 17) – writes dirty pages on the modified listback to the appropriate paging filesMapped page writer (priority 17) – writes dirty pages from mapped filesto diskDereference segment thread (priority 18) is responsible for cache andpage file growth and shrinkageZero page thread (priority 0) – zeros out pages on the free list

6ServicesCaller can manipulate own/remote memoryParent process can allocate/deallocate, read/write memory ofchild processSubsystems manage memory of their client processes thiswayMost services are exposed through Windows APIPage granularity virtual memory functions (Virtualxxx...)Memory-mapped file functions (CreateFileMapping,MapViewofFile)Heap functions (Heapxxx, Localxxx (old), Globalxxx (old))Services for device drivers/kernel code (Mm...)

7Large PagesArchitecture Small Page Size Large Page SizeX86 4 KB 4 MB (2MB on PAE)X64 4 KB 2 MBIA64 8 KB 16 MBVirtual memory divided into pages – smallest unit ofprotectionTwo page sizes: small and large.Advantage of large page:Faster address translation of references to other data in the same pageSingle Translation Lookaside Buffer (TLB) entry used to map larger areaLarge pages are used to map NTOSKRNL, HAL, nonpagedpool, and the PFN database of a “large memorysystem”

8Large PagesCan specify other drivers to map with large pages:HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\MemoryManagement\LargePageDrivers (multi-string)Applications can use large pages for processmemoryVirtualAlloc with MEM_LARGE_PAGE flagCan query if system supports large pages withGetLargePageMinimum

9Large PagesDisadvantage of large pages:disables kernel write protectionWith small pages, OS/driver code pages are mapped as readonly; with large pages, entire area must be mapped read/writeDrivers can then modify/corrupt system & driver code withoutimmediately crashing systemDriver Verifier turns large pages offCan also override by changingHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\MemoryManagement\LargePageMinimum to FFFFFFFF

10Protecting MemoryAttributePAGE_NOACCESSPAGE_READONLYPAGE_READWRITEPAGE_EXECUTEPAGE_EXECUTE_READPAGE_EXECUTE_READWRITEPAGE_WRITECOPYPAGE_EXECUTE_WRITECOPYPAGE_GUARDDescriptionRead/write/execute causes access violationWrite/execute causes access violation; read permittedRead/write accesses permittedAny read/write causes access violation; execution of code ispermitted (relies on special processor support)Read/execute access permitted (relies on special processorsupport)All accesses permitted (relies on special processor support)Write access causes the system to give process a private copyof this page; attempts to execute code cause access violationWrite access causes creation of private copy of page.Any read/write attempt raises EXCEPTION_GUARD_PAGEand turns off guard page status

11Reserving & Committing MemoryOptional 2-phase approach to memory allocation:1. Reserve address space (in multiples of page size)2. Commit storage in that address spaceCan be combined in one call (VirtualAlloc, VirtualAllocEx)Reserved memory:Range of virtual addresses reserved for future use (contiguous buffer)Accessing reserved memory results in access violationFast, inexpensiveCommitted memory:Has backing store (pagefile.sys, memory-mapped file)Either private or mapped into a view of a sectionDecommit via VirtualFree, VirtualFreeExA thread‘s user-mode stack is constructed usingthis 2-phase approach: initial reserved size is 1MB,only 2 pages are committed: stack & guard page

12Allocation GranularityEach region of reserved process address spacebegin on multiple of allocation granularityGetSystemInfo or GetNativeSystemInfo functionValue is 64 KBSize and base of each reserved memory regionis multiple of the system page size (e.g. 4 KB onx86 systems)

13Shared MemoryLike most modern OS’s, Windows provides away for processes to share memoryHigh speed IPC (used by ALPC, which isused by RPC)Threads share address space, butapplications may be divided into multipleprocesses for stability reasonsIt does this automatically for shareable pagesE.g. code pages in an EXE or DLLProcesses can also create shared memorysectionsUnderlying primitives for sharing is “sectionobjects”, or “file mapping objects” in WindowsAPISection object connected to open file mapped file (or file backed section)Section object connected to committedmemory page file backed sectionProcess 1 virtual memoryProcess 2 virtual memoryPhysical memorycompilerimage

14Mapped FilesA way to take part of a file and map it to a range of virtual addresses(address space is 2 GB, but files can be much larger)Bytes in the file then correspond one-for-one with bytes in the region ofvirtual address spaceRead from the “memory” fetches data from the filePages are kept in physical memory as neededChanges to the memory are eventually written back to the file (can requestexplicit flush)Initial mapped files in a process include:The executable image (EXE)One or more Dynamically Linked Libraries (DLLs)Processes can map additional files as desired (data files or additionalDLLs)

15Viewing DLLs & Memory MappedFilesProcess Explorer lists memory mapped files

16Data Execution PreventionPurposesPrevents code from executing in a memory page not specifically marked asexecutableStops exploits that rely on getting code executed in data areasRelies on hardware ability to mark pages as non executableSetting BCD nxAttempts to execute code in a page marked no execute result in:User mode: access violation exceptionKernel mode: ATTEMPTED_EXECUTE_OF_NOEXECUTE_MEMORYbugcheck (blue screen)Memory that needs to be executable must be marked as such using pageprotection bits on VirtualAlloc and VirtualProtect APIs:PAGE_EXECUTE, PAGE_EXECUTE_READ,PAGE_EXECUTE_READWRITE, PAGE_EXECUTE_WRITECOPY

17Controlling DEPBCD nx values:OptIn: Turn on DEP for necessary Windows programs and servicesonlyOptOut: Turn on DEP for all programs and services except the onesselectedAlwaysOn: Enables DEP for all components with no ability to excludecertain applicationsAlwaysOff: Disables DEP (not recommended)

18DEP on 64-bit WindowsAlways applied to all64-bit processes anddevice driversProtects user andkernel stacks, pagedpool, session pool32-bit processesdepend on configurationsettings

19DEP on 32-bit WindowsHardware DEP used when running 32-bit Windows on systems that supportitWhen enabled, system boots PAEkernel (Ntkrnlpa.exe)Kernel mode: applied to kernel stacks,but not paged/session poolUser mode: depends on systemconfigurationEven on processors without hardwareDEP, some limited protectionimplemented for exception handlers

20Software DEPOlder processors which don’t support hardware “noexecution” protectionException handling mechanismimage built with safe structure exception handling verify thatexception is registered in the function tableNot built with safe structure exception handling exceptionhandler is located in the memory region marked as executableStack cookiesSpecial code at the beginning and end of functionCompare cookie valuesPointer encodingEncoding with cookie valueCorrupt decoded pointer crashes program

21Copy-On-Write PagesUsed for sharing between process address spacesPages are originally set up as shared, read-only, faultedfrom the common fileAccess violation on write attempt alerts pagerpager makes a copy of the page and allocates it privately to theprocess doing the write, backed to the paging fileSo, only need unique copies for the pages in the shared regionthat are actually written (example of “lazy evaluation”)Original values of data are still sharede.g. writeable data initialized with C initializers

22How Copy-On-Write WorksBeforeOrig. DataPage 1Page 2Orig. DataPage 3ProcessAddressSpacePhysicalmemoryProcessAddressSpace

23How Copy-On-Write WorksAfterOrig. DataPage 1Page 2Mod’d. DataPage 3ProcessAddressSpaceCopy of page 2PhysicalmemoryProcessAddressSpace

24Shared Memory = File Mapped byMultiple Processes00000000Process AProcess BUseraccessiblev.a.s.Useraccessiblev.a.s.7FFFFFFFNote, the shared regionmay be mapped atdifferent addresses in thedifferent processesPhysicalMemory

25Virtual Address Space (V.A.S.)Process space contains:The applicationyou’re running(.EXE and .DLLs)A user-mode stack for eachthread (automatic storage)All static storage defined bythe applicationUseraccessibleKernel-modeaccessible}Unique perprocess000000007FFFFFFF}System-80000000wideFFFFFFFF

26Virtual Address Space (V.A.S.)System space contains:Executive, kernel, and HALStatically-allocated systemwidedata cellsPage tables (remapped foreach process)Executive heaps (pools)Kernel-mode device drivers(in nonpaged pool)File system cacheA kernel-mode stack forevery thread in everyprocessUseraccessibleKernel-modeaccessible}Unique perprocess000000007FFFFFFF}Systemwide80000000FFFFFFFF

27Windows User ProcessAddress Space LayoutRange Size Function0x0 – 0xFFFF 64 KB No-access region to catch incorrect pointer ref.0x10000 -07FFEFFFF2 GB minus atleast 192kbThe private process address space0x7FFDE000 -0x7FFDEFFF0x7FFDF000 -0x7FFDFFFF0x7FFE0000 -0x7FFE0FFF0x7FFE1000 –0x7FFEFFFF0x7FFF0000 –0x7FFFFFFF4 KB Thread Environment Block (TEB) for first thread, more TEBsare created at the page prior to that page4 KB Process Environment Block (PEB)4 KB Shared user data page – read-only, mapped to system space,contains system time, clock tick count, version number(avoid kernel-mode transition)60 KB No-access region64 KB No-access region to prevent threads from passing buffersthat straddle user/system space boundary

3GB Process Space Option00000000Unique perprocess,accessible inuser or kernelmodePer process,accessible onlyin kernelmodeBFFFFFFFC0000000System wide,accessibleonly in kernelmodeFFFFFFFFUnique perprocess .EXE code(= per Globals appl.),user Per-thread mode usermode stacks.DLL codeProcess heapsProcess page tables,hyperspaceExec, kernel,HAL,drivers, etc.Only available on:Windows 2003 Server, Enterprise Edition& Windows 2000 Advanced Server, XPSP2Limits phys memory to 16 GB/3GB option in BOOT.INIWindows Server 2003 and XP SP2supports variations from 2GB to 3GB(/USERVA=)Provides 3 GB per-process addressspaceCommonly used by database servers (forfile mapping).EXE must have “large address spaceaware” flag in image header, or they’relimited to 2 GB (specify at link time or withimagecfg.exe from ResKit)Chief “loser” in system space is file systemcacheBetter solution: address windowingextensionsEven better: 64-bit Windows28

29Large Address Space AwareImagesImages marked as “large address space aware”:Lsass.exe – Security ServerInetinfo.exe—Internet Information ServerChkdsk.exe – Check Disk utilityDllhst3g.exe – special version of Dllhost.exe (for COM+ applications)Esentutl.exe - jet database repair toolTo see this, type:Imagecfg \windows\system32\*.exe > large_images.txtThen search for “large” in large_images.txt

30Large Address Space Aware on64-bitsImages marked large address space aware geta full 4 GB process virtual address spaceOS isn’t mapped there, so space is available forprocess

31Address Windowing Extensions32-bit user process by default has2 GB virtual address spaceAWE functions allow Windowsprocesses to allocate largeamounts of physical memory andthen map “windows” into thatmemoryApplications: database servers cancache large databasesUp to programmer to controlProcess virtual memoryPhysical memoryAWE memoryAWE memoryAWE memoryAWE memory never paged out a security featureSteps:1. Allocate physical mem to be used2. Create a region of v.a.s. to act as a window to map views of physicalmem3. Map views of the physical mem into window

32Working SetsWorking Set:The set of pages in memory at any time for a given process, orAll the pages the process can reference without incurring a pagefaultPer process, private address spaceWS limit: maximum amount of pages a process can ownImplemented as array of working set list entries (WSLE)Soft vs. Hard Page Faults:Soft page faults resolved from memory (standby/modified page lists)Hard page faults require disk accessWorking Set Dynamics:Page replacement when physical memory limit reached and pagefault happensReplacement policies:First-in-first (FIFO)Least Recently Used (LRU)

33Working Set ManagementDefault working set size: min 50 pages, max 345 pagesMemory manager examines working set limits and amount of free memorywhen page fault occursWorking set manager initiates automatic working set trimming whenphysical memory runs lowBalance Set Manager threadCreated at system initializationWakes up every secondExecutes MmWorkingSetManagerTrimming process WS when required: from current down to minimal WS forprocesses with lowest page fault rateAware of the system cache working setProcess can be out-swapped if all threads have pageable kernel stack

34Kernel-Mode Heaps(System Memory Pools)Nonpaged poolReside in physical memory at all timesNo page faultsPage faults can’t be satisfied at DPC or abovePaged poolCan page in and out of the physical memoryDevice drivers that don’t need to access memory from DPC andabovePool type Max on 32-bit Max on 64-bitNon-paged75% of physical mem or 2 GB(whichever smaller)Paged 2 GB 128 GB75% of physical memory or128 GB (whichever smaller)

35Look-Aside ListsAnother fast memory allocation mechanismContains only fixed-size blocksFaster no spinlocksExecutive components and device drivers useExInitializeNPagedLookasideList to create lookasidelists that match frequently allocated datastructuresSize adjusted by balance set manager once asecond

36Heap ManagerMost apps allocates smaller blocks than 64KBHeap manager manages allocations inside the largermemory areas reserved by VirtualAlloc functionAllocation granularity:32-bit: 8 bytes64-bit: 16 bytesDesigned to optimize memory usage and performancewith these smaller allocationsExists in Ntdll.dll and Ntoskrnl.exeEach process has a default process heap:1 MB but expandable

Heap Manager StructureTwo layers:Core heapCore heapOptional front-end layerCommon across user-mode and kernelmodeManages blocks inside segments,segments, policies for extending heap,committing and decommitting memoryand large blocksFront-end (for user mode only)Low fragmentation heap (LFH)Avoids fragmentation by allocatingblocks according to bucket sizesMore popular allocation classesperformed by LFHCan be disabled by HeapSetInformationAPIApplicationWindows heapAPIsFront-end heaplayerCore heap layerMemoryManagerHeapmanager37

38Physical MemoryAmount of physical memory affects performanceLimit other resources e.g. nonpaged pools, OS buffers backed byphysical memorySystem virtual memory limit sum of roughly the size of physicalmemory and configured size of paged files32-bit limit is 64GB:Data structure that keeps track of physical memory (e.g. PFN database)grows with the physical memoryLarge system would consume too much of virtual addressMapping pieces of PFN database into the system address as needed toomuch complexity and overhead with mapping, unmapping and locking2 TB limit on 64-bit windows not hardware/implementation limit:largest test Itanium system was 2TB

Physical Memory Limits32-Bit 64-Bit Limiting FactorWindows Server 2008 Datacenter and enterprise64 GB 2 TB on IA641 TB on x64PFN database size on 32-bit,hardware platform on 64-bitWindows Server 2008Standard and Web serverWindows Server 20084 GB 32 GB LicensingN/A 128 GB LicensingHPC EditionWindows VistaUltimate, Enterprise, BusinessWindows VistaHome PremiumWindows VistaHome Basic4 GB 128 GB Licensing on 64-bit,hardware support and drivercompatibility on 32-bit4 GB 16 GB Licensing on 64-bit,hardware support and drivercompatibility on 32-bit4 GB 8 GB Licensing on 64-bit,hardware support and drivercompatibility on 32-bit39

40I/O SupportI/O Support operations:Locking/Unlocking pages in memoryMapping/Unmapping Locked Pages into current addressspaceMapping/Unmapping I/O spaceGet physical address of a locked pageProbe page for accessMemory Descriptor ListStarting VADSize in BytesArray of elements to be filled with physical page numbersPhysically contiguous vs. Virtually contiguous

41Cache SupportSystem wide cache memoryRegion of system paged area reserved at initialization timeInitial default: 512 MB (min. 64MB if /3GB, max 960 MB)Managed as system wide working setA valid cache page is valid in all address spacesLock the page in the cache to prevent WS removalWS Manager trimming thread is aware of this special WSNot accessible from user modeOnly views of mapped files may reside in the cacheFile Systems and Server interaction supportMap/Unmap view of section in system cacheLock/Unlock pages in system cacheRead section file in system cachePurge section

42Windows Memory Allocation APIsHeapCreate, Alloc, etc. (process heap APIs)Windows equivalent of malloc(), free(), etc.VirtualAlloc( MEM_RESERVE )VirtualAlloc( MEM_COMMIT )VirtualFreeVirtualQuery

43Windows API MemoryManagement ArchitectureWindows ProgramC library: malloc, freeHeap API:• HeapCreate,HeapDestroy,• HeapAlloc, HeapFreeVirtual Memory APIMemory-Mapped Files API:• CreateFileMapping,• CreateViewOfFileWindows Kernel withVirtual Memory ManagerPhysical MemoryDisc &File System

44Further ReadingMark E. Russinovich et al. Windows Internals, 5th Edition, MicrosoftPress, 2009.Chapter 9 - Memory ManagementIntroduction to memory manager (from pp.699)Working sets (from pp. 822)Services the Memory Manager Provides (from pp. 704)Kernel-mode Heaps (from pp. 721)Heap manager (from pp. 729)Physical memory limits (from pp. 818)

45Source Code ReferencesWindows Research Kernel sources\base\ntos\mm – Memory manager\base\ntos\inc\mm.h – additional structuredefinitions\base\ntos\cache – Cache manager

Lab Demo: View & Create & UseDLL2013-10-12

ContentView DLLs using “listdll.exe” from sysinternalsCreate your own DLLUsing your own DLL