A Measurement Study of the Linux TCP/IP Stack Performance and ...

for the Linux Operating System. The Linux OS
has been a popular choice for server class systems
due to its stability and security features, and it is
now used even by large-scale system operators such
as Amazon and Google [11], [13]. In this paper,
we have focused on the network stack performance
of Linux kernel 2.4 and kernel 2.6. Until recently,
kernel 2.4 was the most stable Linux kernel and
was used extensively. Kernel 2.6 is the latest stable
Linux kernel and is fast replacing kernel 2.4.
Although several performance studies of the TCP
protocol stack [1], [6], [9], [10] have been done, this
is the first time a thorough comparison of Linux 2.4
and Linux 2.6 TCP/IP stack performance has been
carried out. We have compared the performance
of these two Linux versions along various metrics:
bulk data throughput, connection throughput
and scalability across multiple processors. We also
present a fine-grained profiling of resource usage
by the TCP/IP stack functions, thereby identifying
the bottlenecks.
In most of the experiments, kernel 2.6 performed
better than kernel 2.4. Although this is to be expected,
we have identified specific changes in Linux
2.6 which contribute to the improved performance.
We also discuss some unexpected results such as
the degraded performance of kernel 2.6 on SMP
architecture when processing a single connection
on an SMP system. We present fine grained kernel
profiling results which explain the performance
characteristics observed in the experiments.
The rest of the paper is organised as follows.
In Section II, we review previous work in TCP/IP
profiling, and discuss some approaches for protocol
processing improvement. Section III discusses the
improvements made in Linux kernel 2.6 which
affect the network performance of the system. Section
IV presents results of performance measurement
on uniprocessor systems, while Section V
discusses results of performance measurement on
multiprocessor systems. In Section VI we discuss
the kernel profiling results and inferences drawn
from it. In Section VII we conclude with our main
observations.
II. BACKGROUND
Several studies have been done earlier on the
performance of the TCP/IP stack processing [1],
[5], [6], [9], [10]. Copying and checksumming,
among others, have usually been identified as expensive
operations. Thus, zero copy networking,
integrated checksum and copying, header prediction
[6], jumbo frame size [1] etc are improvements that
have been explored earlier. Another approach has
been to offload the TCP/IP stack processing to a
NIC with dedicated hardware [24].
Efforts have also been made to exploit the parallelism
available in general purpose machines itself,
by modifying the protocol stack implementation
appropriately. Parallelizing approaches usually deal
with trade-offs between balancing load between
multiple processors and the overhead due to maintenance
of shared data among these processors [3],
[4], [19]. Some of the approaches that are known
to work well include "processor per message" and
"processor per connection" [23]. In the processorper-message
paradigm each processor executes the
whole protocol stack for one message (i.e. packet).
With this approach, heavily used connections can
be efficiently served, however the connection state
has to be shared between the processors. In the
processor-per-connection paradigm, one processor
handles all the messages belonging to a particular
connection. This eliminates the connection state
sharing problem, but can suffer from uneven distribution
of load. Other approaches include "processor
per protocol" ( each layer of the protocol stack is
processed by a particular processor) and "processor
per task" (each processor performs a specific task or
function within a protocol). Both these approaches
suffer from poor caching efficiency.
III. IMPROVEMENTS IN LINUX KERNEL 2.6
The Linux kernel 2.6 was a major upgrade from
the earlier default kernel 2.4 with many performance
improvements. In this section we discuss
some of the changes made in kernel 2.6, which
can have an impact on the performance of the
networking subsystem.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY BOMBAY. Downloaded on April 29, 2009 at 00:47 from IEEE Xplore. Restrictions apply.
2