10-Gigabit Ethernet Switch Performance Testing - Ixia

Enabling a Converged World
10-Gigabit Ethernet Switch
Performance Testing

White Paper
10-Gigabit Ethernet Switch
Performance Testing
sample test plans included

Copyright © 1998-2004 Ixia. All rights reserved.
The information in this document is furnished for
informational use only, is subject to change
without notice, and should not be construed as a
commitment by Ixia. Ixia assumes no
responsibility or liability for any errors or
inaccuracies that may appear in this document.
Ixia and the Ixia logo are trademarks of Ixia. All
other companies, product names, and logos are
trademarks or registered trademarks of their
respective holders.
Ixia
26601 W. Agoura Road
Calabasas, CA 91302
Phone: (818) 871-1800
Fax: (818) 871-1805
Email: info@ixiacom.com
Internet: www.ixiacom.com
2 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Contents
Abstract .....................................................................................................................................5
Introduction ..............................................................................................................................5
10-Gigabit Ethernet LAN ...................................................................................................6
10-Gigabit Ethernet Metro ................................................................................................6
10-Gigabit Ethernet WAN ..................................................................................................6
What Is Inside a Multi-10GE Port Switch? ..............................................................................7
Packet buffer .....................................................................................................................7
Packet processing function ..............................................................................................7
Classification tables ..........................................................................................................7
Context memory ................................................................................................................7
Traffic management function ...........................................................................................7
Switch fabric ......................................................................................................................7
What Happens to a Packet When It Enters a 10GE Port? .....................................................8
Storing the packet .............................................................................................................8
Processing the packet .......................................................................................................8
Metering and statistics recording .................................................................................. 10
Packet modification ....................................................................................................... 10
Traffic management ....................................................................................................... 10
Switch fabric ................................................................................................................... 10
What Are the Challenges in Processing Packets at 10G Rates? ....................................... 11
Stress Point 1: Ingress packet buffer ............................................................................ 11
Stress Point 2: Packet classification ............................................................................. 11
Stress Point 3: Traffic management ............................................................................. 13
Stress Point 4: Crossbar switch and backplane interconnect .................................... 14
Stress Point 5: Multicast replication and queues ........................................................ 14
Stress Point 6: Control plane ......................................................................................... 15
Developing the Right Test Methodology .............................................................................. 16
Test methodology example ............................................................................................ 16
Module-level testing ....................................................................................................... 16
System-level testing ....................................................................................................... 16
Test methodologies ........................................................................................................ 16
Anticipating the behavior of the 10GE switch .............................................................. 18
Ixia’s Test Solution ............................................................................................................... 20
IxExplorer ........................................................................................................................ 20
IxScriptMate .................................................................................................................... 20
Appendix: 10GE Testing Examples ...................................................................................... 22
1. Throughput Testing: RFC 2544 Throughput Test .................................................... 22
2. Testing for Latency: RFC 2544 Latency Tests ......................................................... 24
3. Testing how 10GE ports handle prioritized streams based on QoS parameters .. 25
4. 10GE OSPF Convergence Test .................................................................................. 27
Glossary ................................................................................................................................. 29
Acknowledgements ...............................................................................................................30
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 3

4 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

10-Gigabit Ethernet Switch Performance Testing
Abstract The first generation of 10-Gigabit Ethernet
(10GE) switches, introduced in early 2002
after IEEE 802.3ae standardization,
proved to be subpar. Some switches barely
reached 50 percent of line rate
throughput, even for large packet sizes,
and exhibited poor latency, limited feature
sets, and high per-port costs.
The second generation of 10GE products
has arrived, with substantial packet
processing capabilities that enable
additional services. Key functions in this
Introduction Ethernet networking technology is now
virtually ubiquitous; it has evolved from
10Base-T (IEEE 802.3) in 1983, Fast
Ethernet (IEEE 802.3u) in 1995, 1-Gigabit
Ethernet (802.3z) in 1998 to 10-Gigabit
Ethernet (802.3ae) in 2002. The 10-
Gigabit Ethernet standard provides a
significant increase in bandwidth while
maintaining compatibility with the installed
base of 802.3-standard interfaces, to
protect existing investments in Ethernet
technology. The IEEE 802.3ae 10-Gigabit
Ethernet standard defines both LAN and
WAN physical layers, the latter of which is
compatible with existing SONET
infrastructure.
Ethernet not only dominates the LAN
market, but is also taking hold in the Metro
Area Network (MAN) market. It has
extended into the WAN arena as both its
distance and capacity has increased.
Ethernet at 1-Gigabit and 10-Gigabit
speeds has attracted increasing attention
from carriers operating in the MAN and
WAN market segments.
technology include performing the
necessary packet classification, header
modification, policing of flows, and
queuing/scheduling — all at wire-speed
rates.
This white paper reviews the common
building blocks of a multiple line card
10GE switch, identifies various stress
points within the switch, and offers various
test methodologies to test these stress
points.
After nearly two years of slow growth
during 2001 and 2002, worldwide Layer 2
and Layer 3 Ethernet switch revenue
began to accelerate in mid-2003, and is
expected to grow from $11.8 billion in
2004 to $15 billion in 2006, according to
Infonetics Research's quarterly market
share and forecast service, L2–L3
Ethernet Switches. Worldwide port
shipments are predicted to grow 75
percent — from 161 million in 2003 to 279
million in 2006.
The strongest growth in the Ethernet
switch market has come from 10GE
chassis ports and revenue, driven by 10GE
deployment in service provider metro
networks, and by the proliferation of 1GE
to desktops/laptops as a standard
interface.
10-Gigabit Ethernet has been developed to
support a wide range of applications —
from the enterprise network, through the
edge and metro network, and into the wide
area network.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 5

10-Gigabit Ethernet LAN
The biggest market for 10-Gigabit Ethernet
is likely to be in the corporate backbone.
The cost of 1-Gigabit and 10-Gigabit
Ethernet has dropped significantly, which
will trigger the demand for 10GE services
and products.
In the enterprise network, 10-Gigabit
Ethernet is used for:
• Switch-to-switch links for very highspeed
connections within the same
wiring closet or data center or between
different buildings.
• Aggregation of multiple-fiber
1000BASE-X or copper 1000BASE-T
segments into 10GE uplinks. Nearly all
new desktops and laptops are now
shipped with 10/100/1000BASE-T
ports.
•System Area Networking (SAN)
applications providing server
interconnects for clusters of servers.
10-Gigabit Ethernet plays a key role in
interconnecting the new generation of
high-performance servers with multigigahertz
2/4/8-way processors
required for moving large amount of
data between server farms.
10-Gigabit Ethernet Metro
At the Metro Area Network (MAN) level,
service providers face tremendous
pressure to expand capacity for broadband
local access, high-speed WANs, storage,
and campus /enterprise grids. Standardsbased
10-Gigabit Ethernet enables service
providers to extend their 10GE networks
into the metro edge using their dark fiber,
especially when 10GE is combined with
metro area optical fiber networks based on
Wave Division Multiplexing (WDM).
10GE MANs can be deployed in either a
star or ring topology. Unlike Resilient
Packet Ring (RPR) networks, 10GE
networks use the standard Ethernet MAC
protocol. The latest 10GE metro switches
can provide network reliability similar to
that of networks based on SONET/SDH
rings.
10-Gigabit Ethernet WAN
10GE WAN applications are similar to MAN
applications, but leverage existing SONET
networks rather than new infrastructure.
SONET is the dominant transport protocol
in the WAN backbone today, and most
MAN public services are offered over
SONET networks.
The 10-Gigabit Ethernet interface (WAN
PHY) is compatible with SONET-based Time
Division Multiplexing (TDM) and with the
payload rate of OC-192c/SDH VC-4-64c.
The 10-Gigabit WAN interface facilitates
the transport of native Ethernet packets
across the WAN transport network, with no
need for protocol conversion. This not only
improves the performance of the network,
but makes it simpler and less costly to
manage.
Ethernet’s flexibility and universal
availability will fuel the demand for 10-
Gigabit Ethernet deployment in multiple
network segments, from corporate
networks, to data centers, and throughout
service provider networks. This ongoing
demand will require system vendors to
deliver high-performing 10-Gigabit systems
that meet service providers’ stringent
requirements — requirements that ensure
high availability to corporate networks and
performance that meets service level
agreements (SLAs).
6 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

What Is Inside a
Multi-10GE Port
Switch?
Testing the new generation of 10GE
switches requires a clear understanding of
the various building blocks that make up a
switch, and their interaction, to determine
the various stress points within a switch
and identify the weakest link in the chain.
Figure 1 shows the major components of a
10GE switch, including the line card, the
switch fabric card, and the controller card.
The following subsections provide an
overview of the 10GE switch/router
components crucial to performance
testing.
Packet buffer
The packet buffer is a temporary repository
for arriving packets while they wait to be
processed.
Packet processing function
The packet processor is an optimized ASIC
or programmable device (Network
Processing Unit, or NPU) for processing
and forwarding packets in the data plane
or fast path. It performs specific key tasks
such as parsing the header, pattern
matching or classification, table look-ups,
packet modification, and packet
forwarding, ideally at wire speed.
Classification tables
The classification table is a special
memory used by the packet processor. It
may contain the following databases:
•The routing table determines where to
route incoming packets.
• Access Control Lists (ACLs) contain
information to grant or deny
permission to specific users or groups
•The flow classification table contains
information about a particular user or
group of users, protocols, and
applications. Flow identification
information is used to determine QoS
treatment, packet policing, and perflow
queuing and billing.
•The label table contains information
about VLANs, stacked VLANs, MPLS
labels, etc.
Context memory
Context memory contains instructions
about whether to deny or forward packets,
where to forward them, all the internal
system headers needed to get a packet
from an ingress to an egress port, and the
external packet header (new MAC, IP,
MPLS stacks, etc.). It also holds
information relevant to metering the
packet and other miscellaneous
information about how to process a
packet.
Traffic management function
The traffic management function is used
to regulate the flow of traffic. It forwards
traffic according to a user-defined set of
rules pertaining to priority levels, latency
and bandwidth guarantees, and
congestion levels. It also provides the
buffering required to work with any
queuing mechanisms it uses to manage
traffic flow across the switch fabric. It may
also include the high-speed SERDES
(serializer/deserializer) function used to
connect to the switch fabric.
Switch fabric
The switch fabric provides data plane
interconnection among all line cards in the
system. The switch fabric typically employs
a crossbar to move packets between its
ingress and egress ports. An NxN crossbar
switch fabric allows N line cards in a
system to be interconnected in a chassis.
This allows each slot to simultaneously
send and receive traffic over the switch
fabric.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 7

What Happens to a
Packet When It
Enters a 10GE Port?
Figure 1. 10GE switch building blocks.
This section provides a high-level
walkthrough of the “life of a packet” as it
goes from a port on an ingress line card
through the switch fabric card and exits on
an egress line card. Figure 2 shows the
logical operation of various line cards in a
multi-10GE port switch, and identifies the
stress points (numbered red circles).
Storing the packet
When a packet arrives on a 10GE port, it is
stored in ingress buffer memory while
waiting to be processed by the packet
processor. When the packet processor is
ready, the packet header is copied into the
packet processor memory for processing.
Processing the packet
The packet processor (ASIC or NPU)
examines the packet to determine whether
a packet should be filtered or forwarded. It
makes this determination by parsing the
packet header and then performing packet
and flow classification.
The classification process maps
information extracted from the packet
header to information stored in local tables
maintained by the control plane processor.
The information in the classification tables
typically includes forwarding tables,
routing tables, and profiles or rules for a
given packet or flow, such as policies for
ACLs, Quality of Service (QoS), and Class of
Service (CoS).
The classification typically points to many
fragments of information about a packet.
This information is usually saved in context
memory and may contain instructions
about whether to deny or forward the
packet, where to forward it, all additional
header information to get a packet from
ingress to egress port, the new header
used when it leaves the egress port (new
MAC, IP, MPLS stacks, etc.), information
relevant to policing the packet (see
"Metering and statistics recording" below),
and other information about how to
process the packet.
8 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Figure 2. Logical operation of line cards in a multi-10GE port switch with stress points.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 9

Metering and statistics recording
Packet metering checks the conformance
of a flow to a subscribed traffic policy or
contract. The metered information is then
used to decide whether to discard or
forward the packet. Bandwidth allocations
may be established in contracts that
specify the per-flow, steady state, and
short term peak (burst) rates. Typically, a
Token Bucket Algorithm is used at the
ingress port to check the compliance of
each flow to its contract. (A detailed
discussion of the Token Bucket Algorithm
is outside the scope of this document). The
metered packets that violate the assigned
contract can be marked for a range of
treatments: unconditional drop, drop if no
internal resources available, or pass on
the packet dropping decision to the
downstream device. The marking of
packets can also be used by the egress
line card to decide whether to delay
packets within a packet stream in order to
conform to some defined traffic profile,
referred to as traffic shaping. Traffic
shaping is used to smooth out packet flows
and regulate the rate and volume of traffic
admitted to the network.
Packet modification
Based on the outcome of the classification
process, the next step is to modify the
packet header and determine the packet’s
egress port. Context memory contains
instructions about whether to deny or
forward packets, where to forward them,
and all the internal system headers
needed to get a packet from an ingress to
an egress port. On the egress line card, a
new external packet header (new MAC, IP,
MPLS stacks, etc.) will be inserted by the
egress packet processing function.
Traffic management
The packet processor typically appends an
internal forwarding header to each packet
that it forwards to the traffic management
device. This header contains information
that tells the ingress/egress traffic
managers where to forward the packet
(which port and line card) and the
characteristics of the flow. The traffic
manager then passes through or discards
flagged packets, depending on its policies
for the ingress direction.
Depending on the architecture, forwarded
packets are organized into FIFO (First-In,
First-Out) queues, input queues, or Virtual
Output Queues (VOQs). Queued traffic is
then scheduled for transmission to the
switch fabric based on a scheduling
algorithm — for example, Weighted Round
Robin (WRR), Weighted Fair Queuing
(WFQ), etc.
Depending on whether or not the switch
fabric supports fixed or variable length
packets, packet segmentation into fixed
cells may occur before packets are
transmitted onto the backplane. The
ingress traffic manager typically interacts
with the switch fabric and/or egress traffic
manager to meet switch fabric scheduling
requirements and avoid congestion.
On the egress side, the traffic manager
first performs a reassembly operation if
segmentation was performed on the
ingress side. Received packets are then
placed into an output queue based on
information from the header that was
appended at ingress. If per-flow queuing is
supported, the egress traffic manager
maintains an additional output queue for
each flow. Each output queue provides
traffic shaping to maintain conformance to
QoS policy.
Switch fabric
The switch fabric functions as a highperformance
data plane interconnect
between its ingress and egress ports. Most
new crossbar switch architectures have
embedded high speed SERDES (serializer/
deserializer) interconnecting line cards
through centralized switch devices. A
packet entering a switch is transferred to
the switching engine. The switching engine
10 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

What Are the
Challenges in
Processing Packets
at 10G Rates?
then examines the packet header and
decides where to forward the packet.
When multiple ingress ports contend for
an egress port, congestion becomes the
primary issue at the egress switch port
and, depending on the architecture, may
cause Head-of-Line blocking (HOL). HOL
blocking occurs with input queuing, when a
cell (a segmented packet to be transmitted
across the backplane) has to be held up,
thereby blocking the progress of any cells
This section examines the most common
stress points within a switch, and
discusses conditions that might push them
toward a strained state. Refer to Figure 2.
Stress Point 1: Ingress packet buffer
The ingress packet buffer is a temporary
repository for arriving packets waiting to be
processed by the packet processor.
Depending on the architecture and
efficiency of the packet processor, data in
the packet buffer could build up, resulting
in intermittent (poor) latency, jitter, packet
loss, or even service outage.
In most architectures, when packet buffers
begin to fill beyond a preset threshold, the
packet processor initiates flow control to
the upstream MAC device, requesting it to
stop passing packets. The MAC device
then transmits a special packet requesting
remote ports to delay sending packets for
a period of time. This special packet is
called a pause frame. This helps prevent
buffer overflow, but it does not solve the
packet loss problem completely. If the
packet flow continues and the flow control
signal is not removed by the packet
processor before the MAC device’s buffer
fills up, the MAC device will start dropping
packets.
Generally, the buffer in any part of the
system can build up for two reasons:
• Local or downstream devices are
exceeding their allocated processing
budget; or
behind it, even if they could otherwise be
switched.
Some switch architectures also provide
multicast packet replication. To get around
packet replication issues, most modern
switch fabrics now allow for an incoming
packet to be broadcast to any number of
egress ports. Typically, on the ingress and
egress line cards, separate queues and
scheduling disciplines are maintained for
unicast and multicast traffic.
• There is contention for resources — for
instance, multiple ingress ports on a
switch/router contending for an egress
port.
Buffer buildup can create a chain reaction,
leading to unpredictable behavior in a
switch or router.
Another challenge in packet buffering for
10GE switches is dealing with back-to-back
small packets. For example, at 10 Gbps
speeds, arriving 64-byte packets must be
deposited into buffer memory every 67
nanoseconds (ns), and departing packets
must be retrieved from buffer memory
every 67ns. Thus, to process a stream of
back-to-back 64-byte packets, the packet
buffer memory subsystem must support a
write and a read every 67ns.
Stress Point 2: Packet classification
Packet classification is one of the most
susceptible stress points. Classification
maps information from the packet header
to information stored in local tables
maintained by a control plane processor
(see "Stress Point 6: Control plane"). The
packet processor parses various fields in
the packet header to construct search
keys. These keys are then used to address
various tables. In most high-end
architectures, Ternary Content
Addressable Memory (TCAM) devices or
equivalent technologies are used to map
these keys to addresses, and are capable
of holding millions of entries and
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 11

performing key searches in a matter of few
internal clock cycles. However, complex
applications or routing protocols may
require multiple key searches to drive a
look-up result. Furthermore, separate
classification sequences may be required
to determine what to do with a packet. For
example, the packet processor may
perform an ACL look-up first to decide
whether to forward or deny a packet, and
then do a route look-up to decide where to
forward it. It might also perform a flow
control look-up to provide enhanced
services.
The required degree of packet parsing and
processing is one of the main criteria that
identifies the class of a switch/router. A
simple Layer 2-3 switch only inspects the
L2 header (i.e., MAC header, VLAN), and
the L3 header (i.e., IPv4, IPv6), and, in
some cases, performs limited flow
classification.
Complex packets might require the
classification of multiple L2-L3 headers for
a given packet. Packets of a given protocol
may be encapsulated within one or more
tunnels of varying protocols, as shown in
Figure 3. For example, a system supporting
IPv6 over GRE requires two Layer 3
headers (IPv4 and IPv6) in addition to the
Layer 2 MAC addresses. A more complex
example is a Layer 2 VPN Martini Draft
IPv6 over GRE packet
Dest
MAC
Dest
MAC
Src
MAC Type
Src
MAC
IPv4
header
Ethernet over MPLS packet
Type MPLS
label
GRE
IPv6
header
EXP S TTL Dest
MAC
packet with frames arriving over Ethernet
in a wide range of dispositions, including
IPv4 routing, MPLS switching or Ethernet
bridging.
Flow classification. In addition to complex
packet classification, flow classification
might be required to provide enhanced
services and policies. Flow classification
provides a level of granularity that allows
policies to be established based on the
applications. Any number of combinations
of Layer 3 and Layer 4 information could
be employed to define the QoS or security
policies that are then enforced.
A flow is a collection of one or more packet
streams. In some classes of switches/
routers, in addition to packet classification,
flow classifiers perform stateful analysis of
packets within the packet streams. Flow
classifiers track the protocol state of each
flow as the connection develops. This
makes it possible to track control
connections on well-known ports that
spawn data connections on ephemeral
ports. This is important, since many
protocols establish connections and
negotiate services on well-known
Transmission Control Protocol (TCP) ports
and then establish another ephemeral
port to transfer the data for the network
session.
TCP/UDP
datagram
Figure 3. Packets may be encapsulated in tunnels of varying protocols.
12 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing
Src
MAC
Type
IPv4
header
TCP/UDP
datagram

Table 1. Maximum packet arrival rates over 10-Gigabit Ethernet.
Look-ups and performance. Classification
table information is typically stored in a
look-up table, usually held in a large TCAM
or equivalent technology. When processing
encapsulated packets or packets with
multiple L2/L3 headers (i.e., IPv6 over
IPv4, or MPLS stacks with Ethernet header,
IP header, and TCP) the classification
process might require multiple accesses to
the look-up table for each packet.
Table 1 shows the worst-case performance
packet arrival rate for small packets.
Depending on the packet arrival rate and
number of required look-ups per packet,
the packet processor or the classification
device could become a resource
bottleneck. For example, a back-to-back
Ethernet packet with an IPv6 datagram
requiring ACL and flow look-up might
require 7–8 look-ups per packet. With 12
million packets arriving per second, this
will require handling ~96 million look-ups/
sec.
Stress Point 3: Traffic management
The traffic management function provides
advanced queuing, congestion
management, and hierarchical scheduling
of network traffic for large numbers of
flows. It forwards traffic according to a
user-defined set of rules pertaining to
priority levels, latency and bandwidth
guarantees, and varying congestion levels.
It also provides the packet buffering
required to work with any queuing
mechanisms used to manage traffic flow
Wire rate LAN throughput for minimum-size packet
Ethernet IPv4 14.88 million packets/sec
Ethernet IPv6 12.02 million packets/sec
Ethernet Over MPLS IPv4 12.25 million packets/sec
Ethernet Over MPLS IPv6 10.25 million packets/sec
across the switch fabric. It may also
include the SERDES function.
Communication from line card to switch
fabric requires additional flow control
information, creating overhead and
increased bandwidth. This additional
bandwidth is called speedup. See "Speedup."
on page 14.
The architecture of a switch/router can
affect how the network behaves in times of
heavy demand. An important concern is
how packets are queued as they enter the
switching fabric, that is, how traffic
prioritization is handled and how different
traffic flows are merged through the fabric.
Some products forward all high-priority
packets before any lower-priority packets;
this is called strict priority queuing. Other
products use mechanisms such as
Weighted Fair Queuing (WFQ) to
statistically multiplex packets into the
fabric. On the ingress line card, WFQ allows
packets from lower priority queues to be
interleaved with higher priority traffic into
the switch fabric. This prevents the higher
priority traffic from completely blocking the
lower priority traffic, since the queues are
guaranteed access to the switch fabric for
a predefined proportion of the time.
Packet discarding is also part of traffic
management. During times of congestion,
the traffic manager may need to make
discard decisions based on the availability
of queue space, priority, or destination
port, using a packet discard algorithm like
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 13

Random Early Detection (RED) or Weighted
RED (WRED) for IP traffic.
Some routers and switches are
fundamentally limited by the small number
of queues they have for QoS. Small
numbers of queues are common for classbased
queuing, which limits prioritization
and fairness. Class-based queuing typically
prioritizes traffic based on the Layer 2
header (virtual LAN and source or
destination MAC address, for example),
rather than on higher level information
such as application or protocol type.
Systems with large numbers of queues
facilitate more granular prioritization and
greater fairness. Queues established on a
per-flow basis provide the possibility for
each user session to get its own queue.
However, a large number of CoS and QoS
policies requires large amounts of memory
and hierarchical schedulers capable of
handling hundreds of thousands of flows.
Stress Point 4: Crossbar switch and backplane
interconnect
Although most switch architectures for
modern systems are non-blocking, three
types of blocking can limit performance
when multiple ingress ports are
contending for an egress port: Head-of-
Line (HOL) blocking, input blocking, and
output blocking. HOL blocking can waste
nearly half a crossbar switch's bandwidth if
the cells waiting at each input are stored in
a single First-In, First-Out (FIFO) queue.
Modern switch architectures employ
Virtual Output Queuing (VOQ). VOQ, in
conjunction with a scheduling algorithm,
eliminates most blocking issues. These
scheduling algorithms require the traffic
manager device and the switch fabric to
exchange information, including requests
for permission to transmit, granting of
permissions to transmit, and other
information.
Speed-up. The additional bandwidth
required to support VOQ and related
scheduling algorithms is called speed-up.
A 10GE line card that supports 15 Gbps to
the switch fabric offers 50 percent
speedup. Speed-up is a common way to
reduce input and output blocking by
running the crossbar switch faster than the
external line rate. For example, if the
crossbar switch runs twice as fast as the
external line, the traffic manager can
transfer two cells from each input port, and
two cells to each output port during each
cell time.
The advantage of speed-up is obvious — it
offers more predictable delay and jitter
across the switch ports by delivering more
cells per cell time, and thus reducing the
delay of each cell through the switch. In
fact, sufficient speed-up can guarantee
that every cell is immediately transferred
to the output port, where its departure
time can be precisely scheduled.
Stress Point 5: Multicast replication and
queues
The biggest challenge in handling
multicast packets is packet replication.
Packet replication is generally
accomplished in two stages. The first stage
handles the branch replications from one
ingress line card to multiple egress line
cards. The second stage handles the leaf
replications, and is typically accomplished
on the egress line card. Depending on the
switch architecture, the packet replication
function could cause resource starvation
in memory and CPU processing, as well as
contention with unicast packets, as it
accesses the data plane to forward the
replicated packets.
New generation switches take advantage
of the natural multicast properties of a
crossbar switch and perform cell
replication within the fabric by closing
multiple cross points simultaneously. This
method relieves the ingress line card from
performing packet replication. However,
the second-stage replication on the egress
line card can still cause resource
starvation and congestion.
14 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Stress Point 6: Control plane
The control plane processor handles the
routing and switching control plane, as well
as many system management operations,
such as user configuration, background
diagnostics, statistics and alarm collection
and reporting, network management, etc.
This document focuses only on how the
control and data plane interact for the
purpose of packet processing.
The control plane processor runs the
switch/router’s operating system and is
responsible for the operation of network
routing protocols (OSPF, BGP, IS-IS, IGRP),
network management (SNMP), console
port, diagnostics, etc. In a distributed
architecture, where each line card has its
own control plane processor, a master
control plane processor typically
generates, synchronizes, and distributes
routing tables and other information
among line cards for local forwarding
decisions.
The control plane path interconnects the
management processor(s) with various
data plane blocks, to initialize, configure,
perform diagnostics, and most importantly,
to set up or update routing tables, Layer 2
tables, policies, and QoS/CoS tables.
The control processor can read/write to
any location in the forwarding table,
context memory, and other memories to
support route removal and addition, table
flushing in route flaps, and policy for a
given flow.
The look-up and table management
operation is asynchronous. The route table
may be updated by the control plane
processor while the packet processor is
performing a look-up.
During ordinary system operation and
moderately stressed conditions, the
control plane would not be called upon to
modify more than a few thousand routes
per second in response to a routing
protocol update, while the system is at the
same time forwarding data plane packets.
However, during error conditions, or a
topology alteration known as route flapping
or route convergence, hundreds of
thousands of routes may be modified each
second while packets are still arriving on
each interface at up to line rate.
Therefore, determining how fast a switch/
router can update its routing table requires
test beds that can create/modify hundreds
of thousands of route updates per second
while performing normal data plane
operation, that is, forwarding packets at
line rate.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 15

Developing the
Right Test
Methodology
Understanding the stress points in a
switch allows the development of a test
methodology that can focus on testing
these areas. The test methodology should
address multiple layers, and could include
measuring for:
• Wire speed unicast data throughput
and latency for Layer 2/3 traffic.
• The ability to filter packets at wirespeed
based on MAC addresses, IP
addresses, TCP or UDP ports, or a
combination of these (N-tuple).
• The ability to perform prioritization
based on QoS marking.
• The ability to police traffic based on
user-defined rate limits.
• The ability to handle Head-of-Line
blocking (HOL).
• Wire speed multicast performance.
Test methodology example
Following is an example of a test
methodology for a multi-10GE port switch
that supports the following features:
1. Wire-rate Layer 2/3 (for IPv4)
switching with a minimum packet
size of 64 bytes.
2. Switching capacity per slot that
supports total port capacity on line
card.
3. Filtering based on ACLs.
4. QoS handling based on 802.1p or IP
Type of Service (TOS) bits.
5. Priority scheduling based on
weighted round robin (WRR).
6. Traffic shaping.
7. Routing protocols, including
multicast.
The test methodology is broken out into
module- and system-level testing. In the
real world, local switching on the module
occurs; this is the best-case scenario for
switch performance, because there is no
contention for the fabric. The worst-case
scenario is when all traffic entering the
switch must traverse the fabric,
contending for backplane capacity and
causing over-subscription.
Module-level testing
In this scenario, traffic will stay local to the
line card. This means that switching will
occur locally between ports on the same
line card, and minimal traffic will traverse
the backplane.
System-level testing
In this scenario, all of the ingress traffic is
switched to ports on different line cards.
This means that traffic will be contending
for backplane capacity and will determine
how the system handles oversubscription.
It will be set up with partially meshed
traffic patterns, with unique ingress ports
mapped to unique egress ports, and
multiple ingress ports mapped to a single
egress port.
Test methodologies
1. Layer 2 bidirectional throughput and latency
test. This test determines the Device Under
Test’s (DUT’s) maximum Layer 2
forwarding rate without traffic loss as well
as average latency for different packet
sizes. This test is performed full duplex
with traffic transmitting in both directions.
The DUT must perform packet parsing and
Layer 2 address look-ups on the ingress
port and then modify the header before
forwarding the packet on the egress port.
2. Layer 2 throughput, QoS, and latency test.
This test determines the DUT’s maximum
Layer 2 forwarding rate with packet loss
and latency for different packet sizes. The
DUT must perform a Layer 2 address lookup,
check the 802.1p priority bit value on
the ingress port, send it to the designated
queue, and then modify the header before
forwarding the packet on the egress port.
3. Layer 3 (IPv4) performance test with ACL and
latency. This test determines the DUT’s
maximum IPv4 Layer 3 forwarding rate
with packet loss and latency for different
packet sizes. The DUT must perform
16 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

packet parsing and route look-ups for both
Layer 2 and Layer 3 packets on the ingress
port and then modify the header before
forwarding the packet on the egress port.
The ACL test involves blocking or allowing
traffic through, based on user-defined
classifiers such as IP addresses or Layer 4
port numbers. Ixia routing emulation
software is used to populate the DUT’s
routing table,. For example, OSPF
emulation is used to generate OSPF LSAs
to construct topological databases.
4. Layer 3 (IPv4) performance test with ACL,
QoS, and latency. In addition to test 3 above,
QoS values in each header will force the
classification of the traffic based on IP
Type of Service (TOS) field settings. On the
ingress side, this QoS policy could also be
used for assigning a packet to a specific
queue, packet metering, and policing; on
the egress side, it could be used for packet
shaping.
5. Layer 3 (IPv6) performance test with ACL and
latency. This test methodology is the same
as the previous Layer 3 IPv4 with ACL
performance test, except that it runs IPv6
traffic with a minimum-size packet of 84
bytes instead of 64 bytes. Due to the larger
IPv6 header, the classification and table
look-up functions will require more
bandwidth and processing.
6. Layer 3 (IPv6) performance test with ACL,
QoS, and latency. In addition to test 5 above,
QoS values in each header force the
classification of the traffic, based on the
TOS field setting. On the ingress side, this
QoS policy could also be used for assigning
a packet to a specific queue, packet
metering, and policing; on the egress side,
it could be used for packet shaping.
7. Multicast test. This test uses a multicast
protocol such as IGMP (IPv4) or MLD (IPv6)
to set up multicast sessions between a
multicast transmitter and groups of
receivers. A multicast protocol emulation
can be used to simulate one or more hosts
while the DUTs function as IGMP/MLD
routers. The simulation calls for groups of
simulated hosts to respond to IGMP/MLD
router-generated queries and to generate
reports automatically at regular intervals. A
number of IGMP groups are randomly
shared across a group of hosts.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 17

Anticipating the behavior of the 10GE switch
Each row in the following table represents
a test methodology and its expected
impact on the different stress points
mentioned earlier in this document, based
on the switch’s design criteria.
As an example, in Table 2, row 2 (modulelevel,
Full duplex Layer 2 performance and
latency, with prioritization): all stress
points show low or no stress, except for
Stress Point 3, which points to the line
Table 2. Module-level test methodologies and stress points.
Test Methodology
1. Full duplex Layer 2
performance & latency test
2. Layer 2 QoS
throughput & latency test
3. Layer 3 (IPv4) with ACL
performance & latency test
4. Layer 3 (IPv4)
with QoS & ACL
performance & latency test
5. Layer 3 (IPv6) with ACL
performance & latency test
6. Layer 3 (IPv6)
with QoS & ACL
performance & latency test
7. Full duplex IGMP
multicast test
card function that services the different
priority queues.
However, in row 6 (Layer 3 IPv6
performance, ACL and QoS), Stress Points
1 and 2 show that a high level of strain is
to be expected. This is because we know
that the switch is designed to handle wirespeed
Layer 3 IPv4 packets, but not IPv6.
This may mean that packet classification
for IPv6 addresses may take more clock
cycles, which may require more buffering
on the ingress.
Stress Points
1 2 3 4 5 6
Ingress
Packet Buffer
Packet
Classification
low/no stress
moderate stress
high stress
low/no stress
Traffic
Management
moderate
stress
X-Bar Switch
& Backplane
Interconnect
moderate stress
Multicast
Replication
& Queues
low/no stress
high stress
Control
Plane
18 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

With system-level testing, additional strain
will occur with the traffic management and
switching functions (Stress Points 3 and 4)
because of a high level of traffic
contending for backplane switching
capacity. For example, row 4 (Mesh Layer 3
IPv6, with route flapping), shows additional
Table 3. System-level test methodologies and stress points.
Test Methodology
Mesh L3 IPv4 with
ACL performance & latency
Mesh L3 IPv4 with
QoS and ACL
performance & latency
Mesh L3 IPv6 with
ACL performance & latency
Mesh L3 IPv6 with
QoS & ACL performance
& route flapping
Mesh IGMP multicast
strain not only in Stress Points 3 and 4, but
also in Stress Point 6 (the control
processor). Because of route flapping, it
needs to modify the routing table as
packets continue to arrive on each
interface.
1 2 3
Stress Points
4 5 6
Ingress
Packet Buffer
low/no
stress
high stress
Packet
Classification
low/no stress
Traffic
Management
low/no
stress
moderate stress
low/no
stress
X-Bar Switch
& Backplane
Interconnect
moderate stress
Multicast
Replication
& Queues
high stress
Control
Plane
low/no stress
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 19

Ixia’s Test Solution Ixia provides testing solutions that
measure the performance and
conformance of complex IP networks,
devices, and applications. It was the first to
offer support for a comprehensive 10GE
wire rate test solution — one that can
generate, capture, characterize, and
emulate network and application traffic,
establishing definitive performance and
conformance metrics for a 10GE network
device or system under test. It is also the
first to offer support for XAUI, XENPAK, XFP,
XPAK, and X2 interfaces in the test
industry. For more information on Ixia’s
10GE products, refer to: http://
www.ixiacom.com/products/interfaces/
index.php
Ixia has developed two primary
applications for Ethernet performance
testing, IxExplorer and IxScriptMate, each
with a distinct testing focus.
IxExplorer
IxExplorer provides a high level of flexibility
and functionality in protocol emulation,
traffic generation, and analysis. IxExplorer
is the primary controlling application for
Ixia’s purpose-built hardware test platform,
allowing detailed configuration of protocols
and analysis of test results.
Within IxExplorer, a comprehensive set of
IP signaling and routing protocols is
supported, including OSPF, IS-IS, RIP, BGP,
LDP, and RSVP-TE, as well as multicast
protocols. IxExplorer controls the protocol’s
operation on Ixia’s test hardware
architecture, which supports a CPU
running Linux on each test port. This
dedicated emulation environment allows
hundreds of 10GE routers to be emulated
on each network interface. Up to hundreds
of thousands of LSPs can be signaled from
each interface. Line rate traffic can be
generated over the established
connections.
IxScriptMate
IxScriptMate provides a framework for
running automated test scenarios.
Numerous test suites have been
developed within the IxScriptMate
environment for testing the session and
circuit scalability, traffic throughput
performance, latency, and failover recovery
capabilities of switch/routers. IxScriptMate
simplifies the configuration process by
defining a configuration for the test and
displaying the relevant parameters for user
input. Tests then run automatically, and
the results are presented to the user. This
is especially useful where the test
configuration can involve multiple
protocols and complex traffic generation
requirements.
20 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Table 4. Test methodologies and Ixia test solutions.
Test Methodology Ixia's Solution
1. Full duplex Layer 2
performance & Latency
2. Layer 2 QoS throughput and
latency test
3. Layer 3 (IPv4) with ACL
performance and latency test
4. Layer 3 (IPv4) with QoS and
ACL performance and latency test
5. Layer 3 (IPv6) with ACL
performance and latency test
6. Layer 3 (IPv6) with QoS and
ACL performance and latency test
Table 4 maps the specific Ixia solution
available to test each of the seven test
methodologies previously outlined for
10GE switches.
IxExplorer,
IxScriptMate using RFC 2544 and RFC 2889 tests
IxExplorer,
IxScriptMate QoS Test
IxExplorer,
IxScriptMate RFC 2544, RFC 2889, ATSS tests
IxExplorer,
IxScriptMate QoS Test, BGP, OSPF tests
IxExplorer,
IxScriptMate RFC 2544, RFC 2889, ATSS tests
IxExplorer,
IxScriptMate RFC 2544, RFC 2889, ATSS tests
7. Full duplex IGMP multicast IxExplorer,
IxScriptMate IP Multicast test
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 21

Appendix: 10GE
Testing Examples
The test plans presented in this section
include several examples demonstrating
how Ixia's solution addresses the
challenges of 10GE switch testing.
1. Throughput Testing: RFC 2544 Throughput
Test
Objective. To characterize the 10GE switch
data plane performance in forwarding
traffic. IETF RFC 2544 defines a test
methodology for this characterization,
divided into the following three tests:
• The back-to-back test determines how
the DUT responds to different
quantities of frames with the minimum
Figure 4. RFC 2544: Throughput Test — setup.
gap allowed by the protocol
specification.
• The frame loss test determines how
the DUT responds to streams with
different loading.
• The throughput test finds the highest
rate at which the DUT can forward
frames.
Setup. A minimum of two 10GE test ports is
required for this test. Ixia's IxScriptmate
application can be used to run RFC 2544
tests.
Parameters. Select the test port pairs,
frame size mode and frame sizes, and
number of iterations.
22 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Methodology. This test should involve
several test ports to match the DUT's port
density. Ideally, the test should flood traffic
to every input port of the DUT. A number of
Ixia load modules will be connected to the
DUT. Ixia's IxScriptMate is used to perform
the RFC 2544 benchmark test.
Figure 5. RFC 2544: Throughput Test — results.
Results. The results of the test show the
throughput rates, in frames per second,
obtained for each frame size. The results
also show the average throughput rates for
all the trials.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 23

2. Testing for Latency: RFC 2544 Latency Tests
Objective. To determine the latency of the
DUT, and how much it varies with different
frame sizes.
In this test, frames are transmitted for a
fixed duration. Once per second, the test
tags a frame and transmits it halfway
through the duration time. The test
compares the tagged frame's timestamp
when it was transmitted with the
timestamp when it was received. The
difference between the two timestamps is
the latency.
Setup. A minimum of two 10GE test ports
will be used for this test, in conjunction
with the IxScriptmate RFC2544 test.
Latency will be measured in both
directions.
Parameters. Select the test port pairs,
frame size mode and frame sizes, and
number of iterations.
Figure 6. RFC 2544: Latency Test — results.
Methodology
1. Packets will be sent to the DUT from
each 10GE load module starting with
the selected minimum frame size for
an interval that lasts for selected test
duration.
2. The average latency is then
measured.
3. These two steps will continuously
repeat, using selected frame sizes
every interval until it reaches the
maximum frame size selected.
Results. The results of the test show the
latency in nanoseconds (ns) obtained for
each frame size and test port pair. The
results also show the average latencies
across the pairs.
24 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

3. Testing how 10GE ports handle prioritized
streams based on QoS parameters
Objective. To determine the maximum rate
at which the DUT can forward frames
correctly according to their priority
settings.
The test sets up a configuration in which
many ports, each with a different priority,
sends traffic to one single port. The
receive port may be overloaded in order to
test the receipt of the highest priority
frames.
This test supports both MAC and IP layer
frames; priority bits may be specified
either in the IP precedence bits or in the
10 GE
1 GE
1 GE
802.1p header. Latency may optionally be
calculated per priority level. Test results
are: the transmit and receive rates per
priority, percent loss, and optionally,
latency, for each priority per port.
Setup. A minimum of two test ports will be
used for this test, in conjunction with the
IxScriptmate QoS Many-to-One test. In this
example, one Ixia 10GE test port and two
Ixia 1 GE test ports connected to the DUT –
all traffic directed to a single 10GE port on
the DUT.
Parameters. Select the three test port pairs,
frame sizes, QoS parameters for each port,
and number of iterations.
Figure 7. Testing how 10GE ports handle prioritized streams based on QoS parameters — setup.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 25
10 GE

Methodology
1. Packets will be sent from all ports.
Multiple streams with varying QoS
parameters will be sent to the one
egress port.
2. Total frames received, latency, and
frame forwarding rate are measured
per priority.
3. Steps 1 and 2 continuously repeat,
using selected frame sizes every
interval until the test reaches the
maximum frame size selected.
Results. The results of the test show, on a
per priority stream, the total frames
received, the receive rate, and percentage
loss for each frame size.
Figure 8. Testing how 10GE ports handle prioritized streams based on QoS parameters — results.
26 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

4. 10GE OSPF Convergence Test
Objective. Verify the ability of a router to
switch between preferred and lesspreferred
routes on its 10GE ports when
the preferred routes are withdrawn and readvertised.
This will test the control plane
function on the DUT’s 10GE line card.
In this test, two 10GE ports simulate OSPF
routers, Router 1 and Router 2. Both
routers advertise the same route prefixes
to the simulated network. However, the
routes advertised by Router 1 will have
smaller metrics (lower costs), which should
cause the DUT to forward traffic over them
instead of Router 2’s routes. After
advertising the routes, the transmit port
begins transmitting a stream of packets to
an address in each of the advertised route
prefixes. The DUT should forward all the
packets over the route with the lower
metric (Router 1). After a time interval has
Figure 9. 10GE OSPF Convergence Test — setup.
elapsed, the receive port simulating
Router 1 withdraws its routes. The DUT
should detect that the preferred route has
gone down and switch traffic to Router 2’s
routes. Router 1 then re-advertises its
routes. The DUT should again detect a
change re-route traffic to the receive port
simulating Router 1.
Setup. This test uses three test ports — one
to transmit and two to receive (see Figure
9 below). One Ixia test port (acting as the
transmit port) and two Ixia 10GE (test
ports 1 and 2 acting as receive ports)
connect to the DUT. Receive ports will
emulate OSPF routers. The traffic is
unidirectional. The DUT must have three
ports utilized with two enabled for OSPF. All
three ports should be configured for IP and
have unique subnets in which to
communicate with the tester ports.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 27

Results. The test results provide an average
convergence time for all routes. Figure 10
below displays example results for the
automated OSPF convergence test in
Figure 10. 10GE OSPF Convergence Test — results.
IxScriptMate. In addition to convergence
time, this test also indicates the amount of
lost packets caused by the convergence.
28 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing

Glossary
Access Control List (ACL) A list of the services available on a server, each with a
list of the hosts permitted to use the service.
Context Memory Context memory contains instructions about whether
to deny or forward a packet, where to forward and all
internal system headers needed to get a packet from
an ingress to egress port and the external packet
header (new MAC, IP, MPLS stacks, etc.).
Head-of-Line (HOL) Blocking HOL blocking occurs when the packet at the head of a
queue cannot be transmitted to an output due to a
contending packet from another input. At the same
time, a packet further back in the queue is blocked
although its destination port is free to receive the
packet.
Packet Buffer A temporary repository for arriving packets while they
wait to be processed.
Packet Processor Optimized application-specific integrated circuit (ASIC)
or programmable device (NPU) for processing and
forwarding packets in the data plane or fast path. It
performs specific key tasks such as parsing the
header, pattern matching or classification, table lookups,
packet modification, and packet forwarding,
ideally at wire speed.
Random Early Detection (RED) Random Early Detection (RED) is a mechanism for
avoiding congestion in packet switched networks. RED
controls the average queue size by indicating to the
end hosts when they should temporarily stop sending
packets.
Synchronized Digital Hierarchy
(SDH)
Synchronized Optical Network
(SONET)
SERDES Serializer/Deserializer.
Switch Fabric Provides data plane interconnection between each line
card in the system. The switch fabric typically employs
a crossbar to move packets between its ingress and
egress ports.
An international digital telecommunications network
hierarchy that standardizes transmission around the
bit rate of 51.84 megabits per second, which is also
called STS-1. The SDH specifies how payload data is
framed and transported synchronously across optical
fiber transmission links, without requiring all the links
and nodes to have the same synchronized clock for
data transmission and recovery.
SONET is a standard for optical transport. It allows
different types of formats to be transmitted on one
line.
10-Gigabit Ethernet Switch Performance Testing Copyright © 2004, Ixia 29

Ternary Content Addressable
Memory (TCAM)
Acknowledgements Authors: Ted Fornoles, Alireza Safari
Editor, Illustrator: Elliott Stewart
Content addressable memory refers to a kind of
storage device that includes comparison logic with
each bit of storage.
Time Division Multiplexing (TDM) A method of putting multiple data streams in a single
signal by separating the signal into many segments,
each having a very short duration. Each individual data
stream is reassembled at the receiving end.
Traffic Management Function The traffic management function is used to regulate
the flow of traffic. It will forward traffic according to a
user-defined set of rules pertaining to priority levels,
latency and bandwidth guarantees, and varying
congestion levels. It also provides the necessary
buffering required to work in conjunction with any
queuing mechanisms it uses to manage traffic flow
across the switch fabric.
Wave Division Multiplexing
(WDM)
A type of multiplexing developed for use on optical
fiber. WDM modulates each of several data streams
onto a different part of the light spectrum.
Weighted Fair Queuing (WFQ) A scheduling algorithm that identifies conversations (in
the form of traffic streams), separates packets that
belong to each conversation, and ensures that
capacity is shared fairly between these individual
conversations. WFQ is an automatic way of stabilizing
network behavior during congestion and results in
increased performance and reduced retransmission.
Weighted Random Early
Detection (WRED)
Cisco’s implementation of RED, called Weighted
Random Early Detection (WRED), combines the
capabilities of the RED algorithm with IP Precedence to
provide preferential traffic handling for higher priority
packets.
Weighted Round Robin (WRR) A scheduling algorithm that assigns a weight to each
client (it can be a labeled stream or a port within a
switch/hub, or a group of streams belonging to the
same service level); the assigned weight corresponds
to the pre-assigned bandwidth allocation for the client.
30 Copyright © 2004, Ixia 10-Gigabit Ethernet Switch Performance Testing