Advanced Mac OS X Rootkits [PDF - Black Hat

4 DINO A. DAI ZOVI DDZ@THETA44.ORG4. Kernel RPC Subsystem InjectionA number of Mach RPC servers exist within the kernel. The MachIPC abstractions allow RPC servers to run in the kernel or user-modeMach tasks, often only requiring a recompile with different options toMiG. In Mac OS X Leopard, the Mach clock, clock priv, host priv,host security, ledger, lock set, mach host, mach port, mach vm, processor,processor set, security, task, thread act, and vm map subsystemsare all implemented as kernel servers. Mac OS X adds a fewnew in-kernel Mach servers for the IOKit and UserNotification systems.Running these servers in the kernel improves performance as thedelivery of Mach IPC messages requires less mode switches betweenuser and kernel mode.In the xnu kernel, IPC messages received by the kernel are routed toin-kernel RPC servers through the mig_buckets hash table. This datastructure and the functions that operate on it are defined in the xnukernel source file osfmk/kern/ ipc_kobject.c. Each routine in the MiGsubsystem is inserted in the hash table by storing its routine identifier,function pointer, and maximum message size. Incoming messageshave their msgh_id header field set to the routine identifier and the kernelfunction ipc_kobject_server uses this field to find the RPC serverroutine to handle the request.While the mig_buckets hash table is statically initialized by the kernel,a rootkit can easily dynamically inject new subsystems into it,as described in [5] and shown in Figure 1. The injected or existingsubsystems can also be removed as shown in Figure 2.Just as easily as new subsystems are added or removed, existingsubsystems can be modified. The subsystems in this hash table performmany of the critical functions of the Mach kernel, including task,thread, and memory management. A kernel-based rootkit could overwritethe routine pointers in this table in order to intercept messagesto these servers. The request messages and their replies could be modifiedin transit in order to modify the behavior of the in- kernel MachRPC server. In this respect, this technique is similar to how moretraditional rootkits intercept system calls by patching the system calltable on Unix-like operating systems or the System Service Table onWindows-based operating systems.5. MachiavelliHistorical Mach-based operating systems often included the NetMessageServer [2]. The NetMessage Server transparently extended MachIPC across the network to other hosts. Each node ran a NetName