Operating System Protection And Rings - Giac

Operating System Protection And Rings
Jim Hurst

Operating System Protection And Rings
Introduction
Hacker break-ins are so common that they are no longer news. This paper explores a
system compromise from the other side: How does an operating system protect itself
from attack? The short answer is that the OS applies layers of trust, or protection rings, to
executing programs—with the kernel residing in the most privileged ring. User programs
have restricted access to the resources of the privileged ring.
The discussion briefly explains kernels and memory segregation into rings. It then
explores how rings function in program execution, in addition to the attack mechanisms
used for privilege escalation.
Kernels: Micro, Monolithic, and Hybrid
The operating system is the software that runs your computer, and the heart of the
operating system is the kernel. The kernel functions as a liaison between the hardware
and software. While hiding the implementation details from the other software that uses
these devices, it manages access to memory, processor, and I/O devices. User programs
use system calls and inter-process communications to request and receive resources from
the kernel. The operating system normally segregates the available memory into a portion
reserved for the kernel alone (the kernel space), and user space, where less privileged
programs run.
Kernels are categorized based on how they process in the same address space. A typical
solution, done for efficiency, is for the kernel to run all code in a single large address
space. This is referred to as a monolithic kernel, and it is the solution that Linux, Unix,
and Mac OS use. In contrast, QNX and other systems provide only basic services in the
kernel and make system calls to user space programs to provide other functions. The
Windows NT kernel (used in Windows 2000, XP, 2003, and Vista) is known as a hybrid
kernel, because it combines aspects of both monolithic and micro kernels.
The security implications of hybrid kernel are huge. The power to execute code in the
kernel space implies complete ownership of the system. Therefore, system architects and
security engineers must guarantee the integrity of all software running in the kernel. One
way an intruder can completely subvert a system is to run hostile code in kernel space.
All common modern operating systems are multitasking, which simply means that the
operating system runs many different programs at once. It does so by assigning each of
the different programs their own section of memory (their address space) and sharing the
processor cycles among the programs (time-sharing). This is a powerful technique, which
allows one to listen to music while working on a document, with both a chat window and
a browser open. It also presents a security risk: How do you prevent hostile programs
from accessing unauthorized data?
OS Rings
In practice, this means that formal mechanisms must be in place to segregate the trusted
operating system from untrusted user programs. The most reliable way to accomplish this
is in hardware. If the segregation occurs in software, a software failure (such as a buffer

overflow) can be used to compromise the system. The first system to support rings in
hardware was the MULTICS time-sharing system in the 1960s, which included eight
rings. This approach of hardware-enforced rings has been almost universally adopted by
later architectures.
The most common CPU architecture in use today is the x86 compatible architecture.
Beginning with the 80286 chipset, the x86 family has provided two main methods of
addressing memory: real mode and protected mode. Real mode, limited to a single
megabyte of memory, quickly became obsolete. Protected mode provided numerous new
features to support multitasking. These included segmenting processes, so that they could
no longer write outside their address space, along with hardware support for virtual
memory and task switching.
In the x86 family, protected mode uses four priority levels, numbered 0 to 3. System
memory is divided into segments, and each segment is assigned a priority level. The
processor uses the priority level to determine what can and cannot be done with code or
data within a segment. The term rings comes from the MULTICS system, where
privilege levels were visualized as a set of concentric rings. Ring 0 is considered to be the
innermost ring, with total control of the processor. Ring 3, the outermost ring, is provided
only with restricted access.
Windows, Linux, and most Unix variants all use rings, although they have generally
dropped the four-ring structure and instead adopted a two-layer approach that uses only
rings 0 and 3. Security mechanisms in the hardware enforce restrictions on ring 3 by
limiting code access to segments, paging, and input/output. If a user program running in
ring 3 tries to address memory outside of its segments, a hardware interrupt stops code
execution. Some assembly language instructions are not even available for execution
outside of Ring 0.
Rings as Gated Communities
The discussion above shows how the operating system is separated from user
applications, as the kernel uses a different address space that is hardware-locked against
access from users. This poses the question of how user applications communicate with
the OS. Gates between rings provide user applications tightly controlled access to ring 0
resources. An example of this occurs when a user attempts to connect to the Internet with
a browser. The browser is requesting access to the network interface, which is a hardware
resource controlled by ring 0. The operating system policies in place determine whether
the privilege is granted. Privileged programs, such as a firewall client, on your computer
an intercept the request and allow or deny it based on which application or user is making
the request. Similar uses of policy occur in file protections: Policy can specify that only
the owner of a file can access it. Therefore, when a user application attempts to read the
file on the disk, the operating system in ring 0 verifies that the user is authorized for
access before permitting the file read.
The Gatekeepers: Device Drivers and Libraries
Device drivers (or drivers) are the programs the operating system uses to read from and
write to hardware devices. These are specialized programs written to provide an interface
between a specific piece of hardware and a specific operating system. This makes device

drivers a target of hacker attacks. If a hacker can replace a device driver with his own
version, the device driver runs in kernel space, and then it can be used to access to any
resource on the system.
That is why the Windows operating system is particularly finicky about replacing device
drivers. It prefers to use drivers that Microsoft has digitally signed.
By using strong public key cryptography to verify the authenticity of a piece of software,
the operating system makes it harder for malicious drivers to be installed (unless, of
course, the user overrides the warning, which happens routinely).
When a user program needs to access privileged resources, it makes a system call, which
causes execution to transfer from the user program to a kernel library function. The
calling program is interrupted, information to restart it later is saved in the call stack, and
the library code begins to execute in privileged mode. When the library function
completes its execution, the call stack is unwound to restore the processor state, and
control is returned to the user program. If the library function takes some time to execute,
the user program can be blocked for the duration. One common set of library routines is
the C library (libc or MSLibC). These libraries handle the details of passing control from
user to kernel programs and switching to supervisor mode.
Privilege Escalation
There are several ways that attackers attempt to beat this system. Because user programs
are effectively in a memory jail and cannot access the kernel address space, the intruder
goal is usually to overwrite some piece of code on the file system with a corrupted
version that gives the hacker privileges. If a rogue program can replace a system
executable with a hacked version, full ownership of the system can be achieved.
There are two classic ways to hack a system in this manner
• Create a buffer overflow in an existing program and use this to execute hacker
code.
• Trick the user into executing a hostile program through a web browser, mail
program, or other application.
After the hostile code is executing in user space, it has all the rights of the current user.
The hackers want full ownership of the system. In many cases, executing any code
provides full control, because users often have administration rights. There is no need for
a typical corporate desktop user to have full privilege, so these users are over-privileged
(this is the default profile in Microsoft Windows, for example). In the enterprise,
establishing profiles with minimal user privilege is a good policy that helps protect the
operating system from compromise.
Fully compromising a system where users do not have administration privileges takes
more work. The hacker has established a beachhead by executing the hostile code, but
she must find some way to increase her access (privilege escalation). There are numerous
techniques to accomplish this, but in essence, they must trick the kernel into executing a
piece of hacker code.

Not all programs are created equally. Hackers often exploit bugs in programs running in
privileged mode. From compromised user account, the hacker can run a buffer overflow
exploit in a system executable that runs with full privileges. In Windows, this often
means an exploit against a Windows Service, because it operates in privileged mode. In
Unix, programs with suid root and world execute permissions are common targets for
privilege escalation.
Another approach is to trick the system into installing a compromised device driver,
perhaps by convincing the user into downloading from a hostile server. The new device
driver then opens up the system to the hacker, which can open network ports that can be
used for remote access.
Summary
Maintaining information security requires absolute confidence in your operating system
software. System designers have approached this by separating the operating system
address space from that the user processes. Most CPU designs support this in hardware.
Ring 0 is memory space where the operating system executes, and ring 3 is where user
applications reside. To achieve full control of a system, intruders need a way to run
programs in ring 0. They can do this by exploiting bugs in privileged programs or by
replacing privileged programs, such as device drivers, with compromised versions.
A well configured system is resistant to privilege escalation and unauthorized access.
This requires the integrity of the operating system software, which is protected by layers
of privilege within the hardware. All hail Ring 0!
References
Websites:
Rings in Computer Security
http://en.wikipedia.org/wiki/Ring_%28computer_security%29
Rootkits on a PCI Card?
http://securitywatch.eweek.com/rootkits/rootkits_on_a_pci_card.html
The Quest for Ring 0
http://www.securityfocus.com/columnists/402/3
Papers
Jerome H. Saltzer, Michael D. Schroeder, The Protection of Information in Computer
Systems
http://cap-lore.com/CapTheory/ProtInf/
Intel Corporation Intel Architecture Software Developer's Manual Volume 3: System
Programming
http://download.intel.com/design/PentiumII/manuals/24319202.pdf

William J. Caelli: Relearning "Trusted Systems" in an Age of NIIP: Lessons from the
Past for the Future.
http://cisse.info/history/CISSE%20J/2002/cael.pdf