PCIe switches with integrated instrumentation speed product ...

UNIT
UNDERTEST
INNOVATION IN ELECTRONICS TEST TECHNOLOGY
Anyone who has been
in the electronic systems
business for an
appreciable amount of
time has seen several
paradigm shifts in the
way systems are designed,
interconnected,
and debugged. One notable
change is in parallel
buses disappearing
and being replaced
by serialized, virtualized,
and switched
connections.
Decades ago, almost
every node in a circuit
could easily be probed
with an oscilloscope.
Only five years ago,
one could count on the
availability of a parallel
bus that could be
monitored with a logic
or protocol analyzer.
Today, systems larger
than a single Systemon-Chip
(SoC) consist of multiple SoC-like components interconnected
by high-speed serial buses, perhaps through a packet
switch. The few visible interconnect traces not hidden under
surface-mount technology components or buried in internal PCB
layers can’t be touched for fear of Heisenberg.
The case for integrated instrumentation
If complex system problems are to be debugged at all or root
cause for simpler problems reached in a matter of minutes to
hours (instead of days to weeks), system interconnect switches
must now also function as logic and protocol analyzers. Switches
also should be capable of evaluating signal integrity at the
physical layer, exercising external links at full wire speed, and
PCIe switches with
integrated instrumentation
speed product development
By Jack Regula
As part of our featured section on the evolution of instrumentation, here’s a perspective on
chip instrumentation integrated directly into circuitry. Determining if a PCI Express (PCIe) link
comes up correctly and quickly diagnosing system problems are now much easier tasks with
integrated instrumentation in PCIe switch silicon and supporting software.
injecting errors of all kinds to evaluate the system’s ability to
detect and recover from such errors. These measures are necessitated
by the nearly insurmountable challenges of hooking up
external test equipment for these purposes, given the high degree
of integration and confines of today’s system packaging.
The 5 Gbps speed of PCIe Gen 2 (see Figure 1) presents additional
challenges to component developers and their customers,
challenges that will be intensified by the Gen 3 specification under
development. The higher frequencies reduce timing and signaling
margins overall, making the physical layer more susceptible to
the nonidealities of real-world transmission line environments and
working against even minimally invasive probing.

To overcome physical layer impairments, SERDES designers utilize
the concept of receive equalization. Using equalization, the
signal integrity eye opening is meaningfully observable only at the
output of the equalizer, buried inside the component. This can only
be done with the aid of integrated instrumentation.
Accelerating product bring-up
Experienced and successful design teams have long implemented
features that allow them to debug problems that arise during the
course of product development. With transistor densities available
today, designers can integrate logic analysis capabilities alongside
critical mission functions, often without affecting cost. The ability
to look at even a tiny subset of the logic nodes in a multimillion gate
ASIC can be of inestimable value in quickly reaching the root cause
of a component- or system-level bug.
While once reserved for factory use only, these advanced capabilities
are now being opened up to customers through software that
provides a high-level interface. This interface frees the customer
from the need to have detailed knowledge about circuit implementation
in order to make sense of the results, while protecting the
manufacturer’s intellectual property. At the same time, the scope
of integrated analysis tools is expanding from a single component
focus to system-wide issues.
While integrated instrumentation is useful throughout the development
cycle, its value can be appreciated most readily when considering
the issues involved in initial board bring-up. When a system
or subsystem consists of several SoCs surrounding a central PCIe
packet switch, the switch is the focus of attention after that first
power-on. Have the links come up? Is signal integrity as predicted
and required? Did enumeration and subsequent configuration
succeed? Are the packet flow patterns nominal? The sooner these
questions are answered, the sooner the team can zero in on what
Murphy has thrown at them and move toward a successful product
release. With integrated instrumentation backed up by vendor-provided
support software, these answers are readily available.
Figure 1
Using the internal logic analyzer
The first order of business when powering on a new board is to
determine if the PCIe links have come up. PCIe switches commonly
provide a link status output for each port that can control an
LED for visual indication. If a link doesn’t come up, it is extremely
helpful to trace the transitions of the Link Training and Status State
Machine (LTSSM) defined in the PCIe specification.
A switch with even a simple integrated logic analyzer can be configured
to capture and time-stamp each transition of a port’s LTSSM
into internal trace memory during a link training cycle. Afterwards,
the trace can be read out via I2C or PCIe and analyzed with the aid
of supplied software. Such a trace can often quickly discriminate
between an interoperability issue, manufacturing problem, and signal
integrity issue.
If tracing the LTSSM doesn’t identify the answer, an internal logic
analyzer can be pointed first at the output of the SERDES’ serialto-parallel
converter, elastic buffer, descrambler, and so forth in an
attempt to pinpoint the malfunction.
While an integrated logic analyzer may provide the ability to probe
deep into the guts of an ASIC, only those registers or state machines
either close to an external interface or architected as called
for in an industry standard will be meaningful to an end user.
Optimizing link performance
Once each link is up, it’s necessary to estimate the signal integrity
eye opening and optimize it. Advanced SERDES allow the width of
the eye opening to be measured with the help of Built-In Self-Test
(BIST) features. This is done using sometimes patented techniques
that offset the sampling point from the center of the eye and then determine
if this results in bit errors. Links are put in loopback mode
and bit error rates tested using internal bit error rate test logic. The
debug processor can step through settings of the SERDES equalizers
and drive strength options, reporting the ones that work best
and even burning them into an optional serial Electrically Erased
Programmable Read-Only Memory (EEPROM) for automatic loading
at the next power-on cycle.
Software that provides a degree of automation
to this measurement and parameter optimization
process can be provided (see Figure 2).
With a debug processor connected to the
switch’s I2C bus, an engineer has sideband access
to all the switch’s internal registers. The
developer can peek and poke manually while
the application is running to check its progress
or cause a script to be executed that compares
the actual switch configuration state to the expected
one.
The debug processor can configure available
performance monitors and then display the
collected statistics in real time while an application
is running to give a real-time indication

UNIT
UNDERTEST
INNOVATION IN ELECTRONICS TEST TECHNOLOGY
of its health. Purely as a matter of
convenience, similar access can be
provided through the debug port
to downstream devices’ control
and status registers. When used
in this manner, the debug processor
acts like the management or
service processor used in servers
and, indeed, their functions may
be combined.
Evolution of instrumentation
After functional debugging, stress
testing and error injection are two
necessary parts of product development.
A switch’s normal data
path is easily modified to function
as an exerciser. As a first step, the
switch sets aside a portion of its
buffer memory, allows software
to build packets therein, and then
transmits the packets on a selected
link. More advanced implementations
include counting, looping,
and branching mechanisms to support
more complex packet streams
and add a pseudorandom element to the error injection.
When operating some portion of a switch as an exerciser, the
ports involved no longer act as switches. Therefore, this capability
is not as unobtrusive as other development acceleration
features found in PCIe switches. Nevertheless, the simple availability
of a component that can perform error injection and generate
a full wire-speed packet stream on up to an x16 Gen 2 link
is a huge boon to the product validation community.
Implementing embedded instrumentation
At PLX Technology, integrating these capabilities into PCIe silicon
evolved out of a selfish desire to reduce manufacturing costs
and facilitate internal debug and validation efforts. It quickly
became evident that these capabilities were perhaps even more
valuable to customers for the accelerative
effect on their product development process.
Because technology is advancing
to ever-higher interconnect speeds, PLX
believes that integrated instrumentation
is fast becoming essential, especially for
PCIe switches.
Implementing the SERDES eye measurement
capability requires close cooperation
with both SERDES and SERDES BIST
intellectual property vendors, then diligent
work by design teams to exploit the
capabilities thus provided.
Implementing on-chip logic analysis is enabled
by a design methodology that requires
Figure 2
Register Transfer Level (RTL) engineers to provide selectable probe
output of critical states from each major module in much the
same way that they are required to insert verification assertions.
This internal standardization allows script development to partially
automate the addition of the probe buses that feed into the
trigger logic and trace memory into the RTL hierarchy. Standardization
also reduces the software effort required to harness
this data and shape at least some of it into a form that is useful
to an end user.
Integrated instrumentation and supporting software tools are
showing excellent potential and becoming more user-friendly as
the technology develops. This represents a new and promising
direction for PCIe switch vendors and customers – one that can
remove major sources of pain and delay from the system development
process.
Jack Regula is chief scientist at PLX Technology,
based in Sunnyvale, California. An inventor of early
switch fabric technology, he has delivered several
technical presentations at industry conferences and
authored multiple technical articles in leading trade
publications. Jack received a BSEE (cum laude) and
an MEE from Rensselaer Polytechnic University, an
education funded by National Merit and New York
State Regents scholarships, as well as the National
Science Foundation.
PLX Technology
408-774-9060
jregula@plxtech.com
www.plxtech.com