IOS XR Platform Hardware Architectures
1oxAq6l
1oxAq6l
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>IOS</strong> <strong>XR</strong> <strong>Platform</strong> <strong>Hardware</strong><br />
<strong>Architectures</strong><br />
LJ Wobker<br />
Principal Engineer<br />
Lane Wigley<br />
Technical Marketing
Agenda<br />
• Introduction – building a forwarding path<br />
• <strong>Platform</strong> Design & Building Blocks<br />
• <strong>IOS</strong> <strong>XR</strong> <strong>Platform</strong>s<br />
• <strong>Hardware</strong><br />
• Virtualized<br />
Cloud, mobility, and media are driving tremendous traffic growths over provider and web-scale networks. To address this<br />
growth, Cisco offers a diverse portfolio of products running the industry leading <strong>IOS</strong>-<strong>XR</strong>. The <strong>XR</strong> portfolio covers<br />
deployments from web-scale cloud, mobile backhaul, carrier ethernet, data center, and CDN infrastructures. This session<br />
discusses the various hardware architectures for <strong>IOS</strong>-<strong>XR</strong> routing portfolio. The session covers the NCS router portfolio,<br />
the ASR 9000 and <strong>XR</strong> Virtual Router portfolio.
Should I be here?<br />
Today’s topics and not-topics.<br />
Yes!<br />
• <strong>Hardware</strong> architectures<br />
• System-level design<br />
• Data planes<br />
• Packet forwarding<br />
• High-speed, complex,<br />
(expensive!) systems<br />
No!<br />
(or at least not much)<br />
• Routing protocols / design<br />
• Network level designs<br />
• Control plane / OS infrastructure<br />
• Selling a specific product<br />
4
You can’t always get what you want<br />
But if you try sometimes well you just might find you get what you need<br />
• Nothing is free<br />
• Some things are closer to free<br />
• The further from the middle,<br />
the more things “cost”<br />
5
The <strong>IOS</strong> <strong>XR</strong> Router Family<br />
CRS<br />
NCS 5000 NCS 5500 NCS 6000 ASR 9000<br />
6
What’s needed to build a forwarding path?<br />
1. Optical to electrical<br />
2. Transport a signal from optics to NPU<br />
3. Ingress forwarding operations<br />
4. Transport a signal from ingress NPU to egress NPU (fabric)<br />
5. Egress forwarding operations<br />
6. NPU to optics<br />
7. Electrical to optical<br />
2/6<br />
3/5<br />
4<br />
1/7<br />
7
Logical view of forwarding path components<br />
LASERS / RECEIVERS<br />
FAST MEMORY<br />
SILICON<br />
Optics<br />
Tables<br />
SERDES<br />
TCAM MEMORY<br />
TCAM<br />
FAST MEMORY<br />
NPU<br />
TM<br />
Fabric<br />
Interface<br />
ASIC<br />
Optics<br />
Packet<br />
Buffers<br />
SILICON<br />
Fabric<br />
nPower X1<br />
Optics<br />
nPower X1<br />
FIA<br />
Optics<br />
nPower X1<br />
FIA<br />
Optics<br />
nPower X1<br />
FIA<br />
Optics<br />
nPower X1<br />
FIA<br />
SERDES<br />
DRAM<br />
CPU<br />
Ethernet<br />
Switch<br />
Control<br />
Ethernet<br />
8
Liquid cooling<br />
Acoustics<br />
NEBS<br />
Heat Sinks<br />
Die size<br />
On-chip<br />
Serial<br />
DDR3<br />
LLDRAM<br />
Materials<br />
Signal Integrity<br />
Junction temp<br />
Cooling<br />
Airflow<br />
Chassis<br />
Backplane<br />
Filters<br />
Cable Management<br />
Fabric<br />
Connectors<br />
Routing Slot pitch<br />
SERDES<br />
Busbar<br />
Power<br />
PCB<br />
Interposer<br />
Optics<br />
Process<br />
Silicon<br />
CPAK<br />
Fast Convergence<br />
Run to completion<br />
N:N<br />
QSFP28<br />
Pipeline<br />
PPS<br />
MPO<br />
LR4<br />
Features<br />
Stats<br />
Programmability<br />
Router<br />
<strong>Hardware</strong><br />
Building<br />
Blocks<br />
IPoDWDM<br />
Ops/sec<br />
Capacity<br />
Bandwidth<br />
FIB size<br />
Memory<br />
Buffering<br />
ACL scale<br />
TCAM<br />
GDDR5<br />
HMC<br />
SRAM<br />
HBM
Challenge: Scale routers faster than components<br />
Note: exponential scale<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
16x<br />
4x<br />
1x<br />
Moore’s Law<br />
?<br />
?<br />
?<br />
?<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends – POS/Ethernet Standards<br />
Fastest Interface Bandwidth<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
16x<br />
Interface Speeds<br />
10G<br />
40G<br />
100G<br />
Moore’s Law<br />
400G<br />
4x<br />
1x<br />
OC-48<br />
GE<br />
OC-12<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends – POS/Ethernet Standards<br />
Fastest Interface & Router Bandwidth<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
16x<br />
640G<br />
Interface Speeds<br />
40G<br />
10G<br />
Buffered Router Bandwidth<br />
8T<br />
2.4T<br />
100G<br />
Moore’s Law<br />
24T<br />
16T<br />
58T<br />
400G<br />
4x<br />
1x<br />
OC-48<br />
GE 28G<br />
8G OC-12<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends - Optics<br />
Bandwidth/Volume<br />
4096x<br />
Driven by physics, not Moore’s Law, costs shifting<br />
SerDes dependency (NPU interface)<br />
Cooling challenge – 30C lower max temp vs. ASICs<br />
1024x<br />
256x<br />
64x<br />
16x<br />
4x<br />
1x<br />
GBIC<br />
10G XFP<br />
10G SFP+<br />
100G CFP<br />
100G CPAK<br />
40G QSFP+<br />
100G QSFP28<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Silicon & Modular Router PPS – Per NPU/LC<br />
*Mpps for Standalone and Modular/Buffered<br />
4096x<br />
*represents a combination of memory, Silicon & SerDes<br />
1024x<br />
256x<br />
64x<br />
16x<br />
4x<br />
1x<br />
CPU<br />
4<br />
12000<br />
16<br />
75<br />
CRS-1<br />
14x4<br />
ASR 9000<br />
CRS-3<br />
125<br />
45x4<br />
ASR 9000<br />
3200<br />
1200<br />
800 700x6<br />
280x5<br />
150x4<br />
NCS 5500<br />
NCS 6000<br />
ASR 9000<br />
SoC – no buffer<br />
Modular – buffered<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends – Memory<br />
Commodity & Custom Bandwidth<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
16x<br />
4x<br />
1x<br />
0.8<br />
SDR<br />
1.6<br />
DDR<br />
Scaling FIB above ~256K IPv4<br />
usec vs. msec buffering<br />
Operations / second critical for FIB<br />
DDR2<br />
5<br />
DDR3<br />
10<br />
ASR 9000<br />
2 nd & 3 rd gen<br />
NCS 6000<br />
55<br />
GDDR5<br />
HMC<br />
28<br />
NCS 6000<br />
160<br />
100<br />
25<br />
HBM<br />
DDR4<br />
325<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends – SerDes<br />
High-performance Electrical Link (Speed in GHz)<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
Optics to NPU<br />
NPU to fabric<br />
NPU to TCAM<br />
NPU to serial memory<br />
16x<br />
4x<br />
1x<br />
1.25<br />
1.25<br />
2.5<br />
5<br />
11.5<br />
15<br />
25<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Technology Trends Compared<br />
4096x<br />
1024x<br />
256x<br />
64x<br />
Interfaces<br />
PPS<br />
16x<br />
4x<br />
Optics<br />
Memory<br />
SerDes<br />
1x<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016
Modular Buffered Router Bandwidth<br />
Systems growing much faster than components – exponential scale<br />
4096x<br />
1024x<br />
8T<br />
24T<br />
16T<br />
58T<br />
256x<br />
2.4T<br />
64x<br />
640G<br />
Interfaces<br />
PPS<br />
16x<br />
4x<br />
1x<br />
8G<br />
28G<br />
Optics<br />
Memory<br />
SerDes<br />
1998 2000 2002 2004 2006 2008 2010 2012 2014 2016<br />
*similar trends for conductive materials, connectors, fans, power supplies, CPUs, …
<strong>Platform</strong> Design
Why not one platform?<br />
Requirements have a butterfly effect<br />
• An individual requirement may dramatically impact architecture<br />
• Every component “ahead of the curve” adds complexity<br />
• Key Drivers<br />
• Time / Cost<br />
• Buffering<br />
• FIB scale<br />
• System Scale<br />
• Features/Flexibility<br />
20
Time – Comparing 1T LCs in 2013 vs. 2016<br />
• NCS 6000 1T – 200G NPU w/ full FIB, deep buffers, programmable<br />
• Significant investment in nPower X1 silicon development<br />
• Custom memory for FIB and buffering – extremely high bandwidth and ops/sec<br />
• $271M acquisition of Lightwire for small, low-power 100G optics<br />
• ASR 9000 1.2T – similar speed, flexibility, and power – 3 years later<br />
• Commercial NPU, customized for Cisco<br />
• Commodity memories<br />
• Cisco fabric, backward compatible with earlier generations<br />
• QSFP28 optics<br />
• Full range of chassis sizes<br />
21
Buffering<br />
Ports<br />
• On-chip limited to microseconds<br />
• But… Off-chip requires<br />
• ASIC I/O pins – fewer interfaces<br />
• Board space and power<br />
• Commodity bandwidth is limited<br />
• Many devices may be needed<br />
• Better suited for buffering than FIB (1 read/write per packet optimized for 128B read/write)<br />
• Stalled – High-end graphics and networking moving to custom<br />
• Custom memories<br />
• Fewer devices are needed to reach bandwidth, fewer pins<br />
• Development and per-unit costs – Extremely expensive<br />
Ports<br />
Ports<br />
NPU<br />
Ports<br />
Ports<br />
NPU<br />
FIB<br />
Ports<br />
Ports<br />
FIB<br />
Buffers<br />
22
FIB Scale<br />
FIB<br />
• On-chip FIB limited to ~256K entries today<br />
Ports<br />
NPU<br />
Ports<br />
• Large FIB tables require external memory:<br />
Buffers<br />
• ASIC I/O pins redirected from interfaces to memory<br />
• 4-8 reads for every packet – high operations per second<br />
• Board space (density) and power<br />
23
System Scale<br />
Four common approaches<br />
FIB<br />
• Fixed – single NPU / Forwarding ASIC<br />
• All resources dedicated to network interfaces<br />
Ports<br />
NPU<br />
Fabric<br />
• May or may not have external memories<br />
• Fixed – multiple NPUs<br />
Buffers<br />
• Connected via mesh or fabric chips<br />
• Modular – Expand with line cards<br />
• Resources for fabric and usually external memories<br />
• Multi-chassis<br />
• Adds fiber connections to fabric cards (more power & board space)<br />
• Increases software complexity<br />
24
Features and Flexibility<br />
Network Interface<br />
• Packet Processing Engine (PPE)<br />
• C programmable<br />
• Run to completion<br />
• Anything is possible, no pure optimization<br />
• Wide range of pipeline programmability<br />
• NCS 5000 < NCS 5500 < ASR 9000<br />
CRS PPES<br />
• Packet rate is strongly correlated to<br />
cost, power, and flexibility<br />
Ingress<br />
Port<br />
Link<br />
Term Parser Layer<br />
Tunnel<br />
VLAN<br />
Service<br />
Trans<br />
Term<br />
Fwd<br />
PMF<br />
Egress<br />
FEC<br />
Res<br />
FEC<br />
Res<br />
NCS 5500 PIPELINE<br />
25
<strong>IOS</strong> <strong>XR</strong> <strong>Platform</strong>s
The <strong>IOS</strong> <strong>XR</strong> Router Family<br />
CRS<br />
NCS 5000 NCS 5500 NCS 6000 ASR 9000<br />
27
NCS 6000 – First Dense 100G<br />
• Highest router bandwidth capacity<br />
• 128 Tbps first generation multi-chassis (2013)<br />
• Only Tbps card on the market in 2013-2015<br />
• With buffers and full FIB<br />
• Dramatic improvement in power efficiency<br />
• $6000/100G/month power savings over CRS-3 64x100<br />
• Merchant fabric<br />
• Highly programmable forwarding<br />
• Custom memories for FIB and buffering<br />
• Custom CPAK optics<br />
28
NCS 6000 Line Card Architecture<br />
• Slice architecture<br />
• Optics, NPU & FIA per slice<br />
• Run to Completion NPU w/ PPEs<br />
• 1 Generation (so far)<br />
• 1T line cards w/ 200G NPUs (40 nm)<br />
• 400G NPUs in lab<br />
• TCAM for ACL/QoS scale<br />
Optics<br />
Optics<br />
Optics<br />
Optics<br />
Optics<br />
Optics<br />
T<br />
C<br />
A<br />
M<br />
Tables<br />
PPEs<br />
Packet<br />
Buffers<br />
nPower X1<br />
nPower X1<br />
nPower X1<br />
nPower X1<br />
nPower X1<br />
T<br />
M<br />
Fabric<br />
Interface<br />
ASIC<br />
FIA<br />
FIA<br />
FIA<br />
FIA<br />
• Deep buffers (50+ msec)<br />
DRAM<br />
CPU<br />
10X 100GE<br />
29
ASR 9000 Series<br />
• Full range of chassis and interfaces<br />
• 40G, 200G, and 1T generations<br />
• Highly programmable forwarding<br />
• Partnered for a customized NPU – cisco software<br />
• Highest scale, feature, and QoS capabilities<br />
• Cisco fabric<br />
• Commodity memories for FIB and buffering<br />
• Mostly commodity optics<br />
30
ASR 9000 System Design<br />
• Backplane or midplane<br />
• Flexible switch fabric options<br />
• RP and fabric may be integrated into RSP<br />
• Variable # of fabrics for increased capacity and redundancy<br />
• First stage of fabric on line card<br />
• Multiple airflow designs<br />
• Front-to-back – 9922, 9912, 9010<br />
• Side-to-back – 9006<br />
• Side-to-side – 9001, 9004<br />
• Modular options for lower-speed and legacy interfaces<br />
31
ASR 9000 Line Card Architecture<br />
• Slice architecture<br />
• Optics, NPU & FIA per slice<br />
• Flexible pipeline NPU<br />
• 1 st stage of fabric on line card<br />
• 3 Generations (so far)<br />
• 40G line cards w/ 15G NPUs (90 nm)<br />
• 200-360G line cards w/ 60G NPUs (55 nm)<br />
• 800G-1.2T line cards with 240G NPUs (28 nm)<br />
• TCAM for ACL/QoS scale<br />
CPAK<br />
CPAK<br />
CPAK<br />
CPAK<br />
CPAK<br />
CPAK<br />
CPAK<br />
CPAK<br />
FIB TCAM<br />
NP-5c<br />
NP-5c<br />
NP-5c<br />
NP-5c<br />
Packet Buffers<br />
FIA<br />
FIA<br />
FIA<br />
FIA<br />
DRAM<br />
Fabric<br />
CPU<br />
• Huge FIB (5M+) & buffers (200 msec)<br />
8X 100GE<br />
32
ASR 9000 Modular Line Card Architecture<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
PHY<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
PHY<br />
2/4/8X 10GE<br />
10G SFP+<br />
10G SFP+<br />
PHY<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
10G SFP+<br />
PHY<br />
20X 10G<br />
Bay<br />
0<br />
FIB TCAM<br />
NP-5c<br />
FIB TCAM<br />
Packet Buffers<br />
FIA<br />
Packet Buffers<br />
Fabric<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
1G SFP<br />
PHY<br />
40G<br />
QSFP+<br />
40G<br />
QSFP+<br />
PHY<br />
40G<br />
QSFP+<br />
PHY<br />
100G<br />
CPAK<br />
PHY<br />
100G<br />
CPAK<br />
100G<br />
CPAK<br />
PHY<br />
Bay<br />
1<br />
NP-5c<br />
FIA<br />
DRAM<br />
CPU<br />
1G SFP<br />
20X 1G<br />
1/2X 40G<br />
1/2X100G<br />
MODULAR 3 RD GENERATION LC<br />
33
NCS 5500 – 3 rd generation 100G<br />
• Dense 100GE with deep buffers<br />
• 8 & 16-slot modular, 1 & 2 RU fixed<br />
• Up to 57.6 Tbps in ½ rack in 2016<br />
• 2.4T - 3.6T line cards<br />
• Options for route and ACL scale via TCAM<br />
• Highly integrated Silicon<br />
• Single ASIC for forwarding & fabric interface<br />
• Dramatic power reduction to 0.24 W/Gbps<br />
• QSFP28 optics<br />
• GDDR5 commodity buffers<br />
34
NCS 5500 System Design<br />
AIR INLET<br />
• New chassis design for <strong>XR</strong><br />
• Orthogonal direct connect<br />
• Horizontal line cards and vertical fabric<br />
• Direct connection between line cards and fabric cards<br />
• No midplane<br />
• Distributed air intake between cards<br />
• Fans cover fabric cards<br />
FAN<br />
REMOVED<br />
• Provides cool air equally to all optics<br />
• Optics require ~30C cooler operation than silicon<br />
• Avoid preheating air from module to module<br />
REAR VIEW<br />
35
NCS 5500 Line Card Architecture<br />
• Slice architecture<br />
• Optics & Pipelined Forwarding ASIC per slice – integrated FIA<br />
• Optional TCAM for FIB/ACL scale<br />
• 1st Generation<br />
• 3.6T line cards FA at 600G<br />
• 2.9T line cards FA at 720G + TCAM<br />
• Deep VoQ buffers (50+ msec)<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
Optics x 6<br />
Optics x 6<br />
Optics x 6<br />
Optics x 6<br />
Optics x 6<br />
DRAM<br />
36X 100GE<br />
Forwarding<br />
ASIC<br />
FA<br />
FA<br />
FA<br />
FA<br />
FA<br />
CPU<br />
Buffers<br />
36
Buffers<br />
TCAM<br />
Buffers<br />
TCAM<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
SFP+<br />
SFP+<br />
SFP+<br />
SFP+<br />
SFP+<br />
SFP+<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
QSFP28<br />
CPU<br />
DRAM<br />
CPU<br />
DRAM<br />
NCS 5501/5502 Architecture<br />
Forwarding<br />
ASIC<br />
• NCS 5501<br />
• 1RU – Single 800 Gbps Forwarding ASIC<br />
• 4x 100G + 40x 10G<br />
• NCS 5502<br />
• 2RU – 8 600Gbps Forwarding ASICs<br />
• Integrated switch fabric<br />
• 48x 100G<br />
• Optional TCAM for scale<br />
• Deep VoQ buffers (50+ msec)<br />
40x10G<br />
NCS 5501 800G<br />
Switch<br />
Forwarding<br />
ASIC<br />
QSFP x 6 FA<br />
Fabric<br />
QSFP x 6 FA<br />
QSFP x 6 FA<br />
QSFP x 6 FA<br />
4x100G<br />
NCS 5502 4.8T<br />
Switch<br />
QSFP x 6 FA<br />
QSFP x 6 FA<br />
QSFP x 6 FA<br />
48x100G
NCS 5000 <strong>Platform</strong>s<br />
Extending <strong>IOS</strong> <strong>XR</strong> for satellite, ToR, and beyond<br />
• Maximizing Silicon capabilities<br />
• No external memories – small FIB, usec buffers<br />
• All bandwidth to ports, no fabric<br />
• Low cost and power<br />
• Full <strong>IOS</strong> <strong>XR</strong> Routing<br />
NCS 5001<br />
NCS 5002<br />
Forwarding ASIC<br />
32x100<br />
FIB<br />
Buffers<br />
NCS 5011<br />
38
CRS-X<br />
• Scaling CRS into 100G<br />
• 4x 100G<br />
• 40x 100G<br />
• 2x 100G + 5x 40G<br />
• 3x 100G + 1x 100G IPoDWDM<br />
• Up to 51.2T via multi-chassis<br />
• Fully compatible with CRS-1 & CRS-3<br />
39
Virtualized <strong>IOS</strong> <strong>XR</strong><br />
• <strong>IOS</strong> <strong>XR</strong> on x86 hardware with Linux VM & containers<br />
• Router creation in seconds – rapid service deployment<br />
• Runs NCS 6000 software (recompiled) in emulated NPU<br />
• Hosted & small PE with L2 & L3 VPNs<br />
• Ideal route reflector<br />
• 64 bit<br />
• 10+ M routes<br />
40
Virtualized <strong>IOS</strong> <strong>XR</strong> – Performance & QoS<br />
• Targeted for 5-50 Gbps forwarding<br />
• 20+ Gbps with features per socket<br />
• 3-layer Hierarchical QoS<br />
41
Call to Action<br />
• Visit the World of Solutions for<br />
• Walk in Labs – NCS 6000 Zero Packet Loss demo, NCS 5000, NCS 5500<br />
• Technical Solution Clinics<br />
• Lunch and Learn Topics<br />
• DevNet zone related sessions
Complete Your Online Session Evaluation<br />
• Please complete your online session<br />
evaluations after each session.<br />
Complete 4 session evaluations<br />
& the Overall Conference Evaluation<br />
(available from Thursday)<br />
to receive your Cisco Live T-shirt.<br />
• All surveys can be completed via<br />
the Cisco Live Mobile App or the<br />
Communication Stations
You can’t always get what you want<br />
But if you try sometimes well you just might find you get what you need<br />
• Summary & Questions<br />
44
Thank you
Change language