Design Review Template - NETS

Cisco Systems Advanced Services 

National Center for Atmospheric Research 

Campus Design Review 

And Best Practices Recommendation 

Version 1.2 

Corporate Headquarters 

Cisco Systems, Inc. 

170 West Tasman Drive 

San Jose, CA 95134-1706 

USA 

http://www.cisco.com 

Tel: 408 526-4000 

800 553-NETS (6387) 

Fax: 408 526-4100

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, 

INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, 

EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. 

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED 

WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED 

WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. 

The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 

of the FCC rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment 

generates, uses, and can radiate radio-frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. 

Operation of this equipment in a residential area is likely to cause harmful interference, in which case users will be required to correct the interference at their own expense. 

The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in 

accordance with Cisco’s installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a 

Class B digital device in accordance with the specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a 

residential installation. However, there is no guarantee that interference will not occur in a particular installation. 

You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral 

devices. If the equipment causes interference to radio or television reception, try to correct the interference by using one or more of the following measures: 

Turn the television or radio antenna until the interference stops. 

Move the equipment to one side or the other of the television or radio. 

Move the equipment farther away from the television or radio. 

Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by 

different circuit breakers or fuses.) 

Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product. 

The following third-party software may be included with your product and will be subject to the software license agreement: 

CiscoWorks software and documentation are based in part on HP OpenView under license from the Hewlett-Packard Company. HP OpenView is a trademark of the Hewlett-Packard 

Company. Copyright © 1992, 1993 Hewlett-Packard Company. 

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version 

of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California. 

Network Time Protocol (NTP). Copyright © 1992, David L. Mills. The University of Delaware makes no representations about the suitability of this software for any purpose. 

Point-to-Point Protocol. Copyright © 1989, Carnegie-Mellon University. All rights reserved. The name of the University may not be used to endorse or promote products derived from 

this software without specific prior written permission. 

The Cisco implementation of TN3270 is an adaptation of the TN3270, curses, and termcap programs developed by the University of California, Berkeley (UCB) as part of the UCB’s 

public domain version of the UNIX operating system. All rights reserved. Copyright © 1981-1988, Regents of the University of California. 

Cisco incorporates Fastmac and TrueView software and the RingRunner chip in some Token Ring products. Fastmac software is licensed to Cisco by Madge Networks Limited, and the 

RingRunner chip is licensed to Cisco by Madge NV. Fastmac, RingRunner, and TrueView are trademarks and in some jurisdictions registered trademarks of Madge Networks Limited. 

Copyright © 1995, Madge Networks Limited. All rights reserved. 

Xremote is a trademark of Network Computing Devices, Inc. Copyright © 1989, Network Computing Devices, Inc., Mountain View, California. NCD makes no representations about the 

suitability of this software for any purpose. 

The X Window System is a trademark of the X Consortium, Cambridge, Massachusetts. All rights reserved. 

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL 

FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF 

MERCHANTABILITY, FITNESS FOR A PRACTICAL PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE 

PRACTICE. 

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT 

LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS 

SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 

AccessPath, AtmDirector, Browse with Me, CCDE, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco NetWorks logo, the Cisco Powered Network logo, Cisco Systems Networking Academy, 

Fast Step, Follow Me Browsing, FormShare, FrameShare, GigaStack, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness 

Scorecard, MGX, the Networkers logo, Packet, RateMUX, ScriptBuilder, ScriptShare, SlideCast, SMARTnet, TransPath, Unity, Voice LAN, Wavelength Router, and WebViewer are 

trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, Discover All That’s Possible, and Empowering the Internet Generation, are service marks of 

Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert Logo, Cisco IOS, the Cisco IOS logo, 

Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch, IOS, IP/TV, LightStream, MICA, Network Registrar, 

PIX, Post-Routing, Pre-Routing, Registrar, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. 

and certain other countries. 

All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between 

Cisco and any other company. (0105R) 

INTELLECTUAL PROPERTY RIGHTS: 

THIS DOCUMENT CONTAINS VALUABLE TRADE SECRETS AND CONFIDENTIAL INFORMATION OF CISCO SYSTEMS, INC. AND IT’S SUPPLIERS, AND SHALL NOT 

BE DISCLOSED TO ANY PERSON, ORGANIZATION, OR ENTITY UNLESS SUCH DISCLOSURE IS SUBJECT TO THE PROVISIONS OF A WRITTEN NON-DISCLOSURE 

AND PROPRIETARY RIGHTS AGREEMENT OR INTELLECTUAL PROPERTY LICENSE AGREEMENT APPROVED BY CISCO SYSTEMS, INC. THE DISTRIBUTION OF 

THIS DOCUMENT DOES NOT GRANT ANY LICENSE IN OR RIGHTS, IN WHOLE OR IN PART, TO THE CONTENT, THE PRODUCT(S), TECHNOLOGY OF 

INTELLECTUAL PROPERTY DESCRIBED HEREIN. 

Design Review Template 

Copyright © 2001-2, Cisco Systems, Inc. 

All rights reserved. 

COMMERCIAL IN CONFIDENCE. 

A PRINTED COPY OF THIS DOCUMENT IS CONSIDERED UNCONTROLLED.

Contents 

Contents 3 

Document Control 4 

History 4 

Review 4 

About The Document 5 

Document Purpose 5 

Business Profile 5 

Current Topology 6 

Current Design Overview 6 

Overview of Recommendations 8 

Executive Summary 8 

Short Term Design Enhancements 9 

Foothills-CenterGreen: The Radio Link 9 

Mixed Core (L2 + L3) 9 

Hierarchical (MultiLayer) Network Design: An Overview 11 

Overview 11 

Long Term: AS-Proposed Design 14 

Configuration Best Practices 16 

VLAN 1 16 

VLAN Trunking Protocol (VTP) 17 

Trunking Mode 17 

Spanning Tree Protocol (STP) 18 

Unidirectional Link Detection (UDLD) 20 

EtherChannel (PAgP) 21 

 

NCAR Design Review and Recommendations v1.2 3

Document Control 

Author: 

Hazim Dahir 

Advanced Services – Central Engineering 

History 

Table 1 

Revision History 

Version No. Issue Date Status Reason for Change 

1.0 20-Feb-2003 Draft, Released First release 

1.1 11-Apr-2003 Introduced Mixed Core 

1.2 24-Apr-2003 Final Updated Diagrams and added Configuration observations 

Review 

Table 2 

Revision Review 

Reviewer’s Name Version No. Date 

Tsegerada Beyen and Niels Brunsgaard 1.0 

Advanced Services 

Network Engineering Team 

NCAR 

1.1 Apr-17-2003 

Change Forecast: High 

This document will be kept under revision control. 

A printed copy of this document is considered uncontrolled. 


About The Document 

Document Purpose 

This network design review document is intended to provide an overall assessment of the design aspects of 

the network and select operational functions. The comments presented in this document are a result of 

information learned about the network from customer-documentation as well as the weekly discussions. 

This assessment is part of the Performance Engineering and Optimization services provided by the 

Central Engineering Team. This service will give a best practice assessment of the network as a complete 

system. It uses data collected about individual devices or interfaces to generate an assessment of “Campus 

Best Practices”. This assessment would consider network Availability, Scalability, Convergence, 

Modularity, Hierarchical Design and other network stability aspects. 

Business Profile 

Understanding the business goals of a company or institution is very important when analyzing a network 

design. The goal of a good network design is to empower users in meeting company objectives. The 

network should provide an acceptable level of performance and reliability while not wasting capital and 

other resources in the process of over-engineering the network. Nor should a network be under-engineered 

such that it fails to meet the service levels necessary to meet the business objectives. Many design 

decisions are a result of thoughtful risk/benefit analysis. 

The National Center for Atmospheric Research, NCAR, was established in 1960 to serve as a focus for 

research on atmospheric and related science problems and is recognized for its scientific contributions to 

our understanding of the earth system, including climate change, changes in atmospheric composition, 

Earth-Sun interactions, weather formation and forecasting, and the impacts of all of these components on 

human societies. 

With two major sites in Boulder, I.M. Pei's Mesa Laboratory and a newer Foothills Laboratory, NCAR's 

research is conducted in several principal disciplinary areas: atmospheric chemistry; mesoscale and 

microscale meteorology; solar and solar-terrestrial physics; and climate and the linking of climate with 

other environmental systems. Focused contributions are also made to national scientific initiatives. There 

are multi-disciplinary and cross-disciplinary efforts aimed at the development of a coupled climate system 

model which will simulate the complex interrelations between climate, weather, the sun, and the biosphere 

and oceans. Research on the societal interactions with atmospheric processes is an integral part of NCAR's 

program. 



Current Topology 

Mesa Lab 

UNAVCO 

Pearl Street 

ml-mr-c2-gs 

POTS 

ps-3018-c1-ts 

ps-1027a-c1- 

es 

ml-062-c1-gs 

Ml-371-c1- 

gs 

ml-50-c1-gs 

ml-mr-c2-ts 

netscm netscm2 

uv-18-c1-es 

Foothills Lab 

fl2-2076-c1-e 

s 

ps-3018-c1-es 

ps-2008-c1-es 

fl1-2152-c1-gsfl3-3087-c1-gs 

fl3-3068-c1-gs 

ml-16c-c1-gs 

netscm3 

fl4-1012-c1-gs 

fl1-2037a-c1-gs 

ml-mr-c2-es 

fl2-3095-c1-as 

fl2-3095-c1-gs 

fl2-3095-c1-ts 

ml-mr-c3-gs 

gin 

fl2-2104-c1-gs 

ml-47-c1-gs 

ml-mr-c1-ts 

ml-mr-c1-g 

s 

Center Green 

fl2-2069-c1-gs 

fl4-2060-c1-gs 

cg1-2010-c3-e 

s cg1-2036-c1-gs 

ml-243b-c1-gs 

ml-mr-c1-as 

voip1r-n2 

ml-y2k-c1-gs 

ml-y2k-c1-as 

vbnsr 

Marshall 

marshallr-n406mar-26a-c1-es fb-12-c1-es fb-12-c2-es 

cg-voipr 

cg1-3036-c1-gs 

jef-126-c1-as 

cg1-3010-c1-es 

jef-126-c1-es 

cg2-mr-c1-gs 

cg2-mr-c1-ts 

cg1-2010-c2-es 

cg1-2010-c4-es 

cg1-2010-c1-escg1-3010-c2-es 

NCAR Network 

Diagram 

A TM links 

Gigabit Ethernet links 

Fleischman Building 

JeffCo 

jef-126-c1-ts 

L3 

sw itch 

L2 

Sw itch 

ATM 

Sw it 

ch 

Terminal 

Server 

Current Design Overview 

The current design spreads over three major campuses. The largest and most populated are Mesa and 

Foothills. Center Green utilizes an L2 switch as an aggregation point. 

The existing design facilitates for several VLANs to span multiple switches as well as multiple sites. 

Although this is not recommended, at NCAR this does not present any immediate problems or issues. The 

single homing of switches to the perspective core switch and the absence of a Spanning Tree loop at the 

core provide for a stable environment. 

At the current time, NCAR is satisfied with the current “availability” model and hope to improve it in the 

future. For example, the Mesa site, acts as a transport site for all traffic exchanged between CenterGreen 



and Foothills. A total failure of the “ml-mr-c1-gs” switch will isolate all three major sites. Relying on internal 

redundancy (Dual-Supervisor and HA feature) helps reduce the chance of that type of failure. 


Overview of Recommendations 

Executive Summary 

The advent of high-speed L3 switches has moved modern Enterprise/Campus Network Design away from 

the flat L2 vlan-based model. Cisco’s current Campus Reference Design Model, commonly known as the 

Multilayer Model, features high-powered L3 switches placed in key areas of the Enterprise Network. 

This document concentrates on key design concepts required for mission critical networks. The most 

important ones are: 

- Hierarchical Network Model: Characterization of traffic flow 

- Modularity: Network made up of distinct network blocks 

- Scalability: Allow network to grow without major changes or redesign 

- High Availability: Internal, External, and path redundancy 

- Predictability: Traffic Flows, delays, bounds, fail-over paths are predictable 

- Simplicity: Satisfy network requirements with the least amount of effort or Hardware 

The following sections attempt to describe two design improvement approaches: 

1. Short Term Design Enhancement 

2. Long Term AS Proposed Design 


Short Term Design Enhancements 

Foothills-CenterGreen: The Radio Link 

NCAR is testing a Radio Link (TeraBeam) for possible deployment into the production environment to 

connect Foothills with CenterGreen. If we allow the Radio Link to act a trunk, then we are creating an 

environment that is STP dependent for convergence. That in return will force one of the Core links to be in 

the “Blocking” state. Reliability and Utilization common sense force us to “block” the Radio Link. 

All Links (including the Radio Link) are better utilized in an L3 Core. With the three major sites 

participating, this would be a full mesh. Mesa will no longer be the only link between Foothills and 

CenterGreen. 

Mixed Core (L2 + L3) 

The presence of several campus-wide VLANs requires Trunking (ISL or dot1Q) in the Core. Those 

VLANs would be handled by two trunks connecting the three sites. This is exactly how all traffic is 

handled today. 

Other VLANs that are unique to an L2 switch or to one of the sites will be cleared from the L2 trunk and 

can be routed via a separate L3 connection. This is best described by the following diagram. 

By adding a routing engine to the CenterGreen switch, three unique VLANs can be configured to represent 

the L3 core. The other L2 will only carry traffic for the VLANs requiring campus-wide configuration (all 

other VLANs must be cleared from the trunk). 

An important decision needs to be taken here regarding the Active gateway(s) for the ‘L2” VLANs. Any 

two routers in any of the three sites can handle that requirement. We can also consider M-HSRP and have 

one router active for half the VLANs and another router active for the other half. (Point of Discussion: This 

document will updated accordingly) 


Error! Reference source not found. 

Mixed Core Design 

- The L3 Core consists of the three independent point-to-point VLANs X, Y, and Z. 

- The L2 trunk illustrated by the blue lines will carry VLANs that need site-wide accessibility. 

- VLANs X, Y, and Z to be cleared from the dot1Q trunk 

- All Site-specific VLANs to be cleared from the dot1Q trunks. 


Hierarchical (MultiLayer) Network 

Design: An Overview 

Overview 

The hierarchical three tiered campus design has become the preferred architecture for most networks. The three 

tiered architecture is comprised of an access layer that directly connect network users by means of switches, 

normally placed in wiring closets positioned throughout the campus. Access layer switches are also connected to 

a distribution layer. The distribution layer sites will have a number of access switches connected to it. The 

number of access switches connected to the distribution layer is often determined by geographic proximity, such 

as all the access switches in a building homing into one distribution site for that building. The distribution sites 

normally consist of switches with layer 2 and layer 3 functions. The distribution layer switches are usually 

deployed in pairs for system redundancy. The distribution layer switches are then connected to a core layer 

switch. The core switches are also usually deployed in pairs for redundancy and may support layer 3 as well as 

layer 2 functions. An overall campus design such as this might have a numerical profile of 8000 users connected 

to 40 access switches that are then connected to 4 distribution sites that are finally connected to 1 core site. 

Access 

Distribution 

Backbone 

Core 

Building block additions 

Data Center 

WAN Internet Remote Access 


Hierarchical (MultiLayer) Network Design 

A hierarchical network design distributes networking function at each layer through a layered organization. 

Modular designs are made out of building blocks. Modules and can be added or removed without redesigning 

the network. A modular design is also easier to grow and troubleshoot. Cisco’s multilayer design is an 

example of modular hierarchical design model. The key elements of the structured hierarchy are the Core, 

Distribution and Access layer in a network. 

The Access Layer 

This layer is made up of pure L2 (layer 2) switches each with dual uplinks to two L3 distribution switches. 

There are basically two types of access vlans that can be implemented in this area and most often we see a mix 

of both. The Access Layer model use one of two methods to provide redundancy and/or load balancing: 

The HSRP Model: It does not utilise STP and has faster convergence in the face of failures. There is no regular 

routing protocol running over the uplinks, only hsrp (passive vlan interfaces). Load-balancing is achieved by 

having half of the vlans use one distribution switch as their default gateway and the other half use the other 

distribution switch. 

It should be pointed out that the standard access model does not call for STP to be turned off on the switches. 

STP should be enabled but no L2 loops should physically exist. This is a precaution to avoid the network to 

collapse in case of wrong cabling or when channeling feature misbehaves. One common perception is that a 

unique vlan must be configured on each access switch, this is not true as it is allowed and sometimes 

advantageous to configure several vlans per access switch; the only thing that is not allowed is to configure one 

same vlan on more than one access switch. 

The Spanning Tree Model: In the early days of the VLAN technology it was a common design to group clients 

and their corresponding servers within the same vlan to fully take advantage of the performance of L2 

switching. As a result, workgroup servers were usually attached at the distribution layer where all vlans would 

meet. Each of these vlans would also span the whole network thereby potentially creating extended STP 

topologies. These designs were heavily based on the vlan concept accompanied with its highly publicized 

flexibility (mobility) and therefore had quite a bit of a success. 

Although, building-wide vlans are discouraged in the multilayer model it has to be admitted that real networks 

often make heavy use of their conveniences and cannot work around them easily. Hence, another access model 

is presented with a very simple STP topology such that it can be tolerated in the multilayer model. 

In the STP/VLAN-based access model, we restrict the STP topology into a triangle. Primary and secondary 

roots are always located at the distribution layer so that each vlan blocks once on each access switch. Many 

materials explain how to design, implement and troubleshoot STP optimally for this situation. In this case, you 

want to have at least two vlans per access switch in order to use the bandwidth on all links including blocked 

uplinks. On one access switch, each vlan should block a different uplink – this can easily be achieved by using a 

different primary root at the distribution layer for each vlan. 

Helpful hints and features to deploy in the access layer: 

• Use UDLD and loop-guard to prevent the adverse effects of STP failures. 

• Keep the topology as simple as possible. Triangular shaped topologies are very well suited to implement the 

STP’s enhancements (x-fast). 

• Use multiple vlans and PVST+ to loadshare traffic across blocked ports. 

• Location of blocked ports must be very easily predicted (a consequence of simple triangular topologies). 

• The distance to the root must be kept to a minimum. 

• Carefully choose the root location. 

The Distribution Layer 



This layer interfaces the Access layer with the Core Layer; it is a major aggregation point and plays a very 

important role. The distribution layer terminates all access vlans from the layer 2 perspective. High performing 

Catalysts 6k are typically used to route the vlans efficiently at wire speed towards the backbone. The switches 

have routing intelligence to participate in any routing protocol with core routers as well as providing HSRP 

functionality to load share traffic coming from the access layer. The Distribution switches do not inject routing 

updates into the access layer in normal circumstances. 

Although it consists only of two L3 switches, the way we implement and interconnect them to other layers is not 

trivial. 

Helpful hints and features to deploy in the access layer: 

• Aggregates the Access Layer (wiring closet) switches 

• Provides uplink connectivity to the Core LayerReduces the need of high density peering with a Core-only 

Layer 

• Redundancy (Availability), Load Balancing and Capacity Planning (Provisioning) are the most important 

aspects. 

o Use Layer 3 Switching in this Layer 

o Use HSRP and HSRP-Tracking for first-hop redundancy 

The Core Layer 

The backbone of the network. Aggregates the Distribution Layer switches and plays the primary role of 

connecting other network building blocks. It should provide redundancy, rapid convergence using high-speed 

connections and high-speed Layer 3 switching. 

The Server Farm: 

A server farm is implemented as a high-capacity building block attached to the campus backbone, and is treated 

as its own distribution block. Server farm is an aggregation point for much of the traffic from the whole campus. 

As such, it makes sense to design the server farm with less over subscription of switch and trunk resources than 

the normal building block. For example, if the other campus building blocks connect to the backbone with Fast 

Ethernet, then the server farm should probably attach to the backbone with Gigabit Ethernet. 

How do we handle Campus or Enterprise-Wide VLANs 

These are definitely incompatible with the Multilayer model, which is a strictly routed environment. There are 

generally two possible ways to force these protocols on the model, either we build Campus-Wide vlans with 

complex Spanning Tree topologies spanning the whole network or we need to bridge these protocols on the 

routers. Both alternatives undermine the robustness of the network. 


Long Term: AS-Proposed Design 

This section will detail a design proposal from the AS team. The following design is hierarchical, 

redundant, and modular. If fully implemented, the design presents a solution where each of the campuses 

is totally independent from the other from a traffic and fail-over point of views. 

Mesa/Marshall 

Fleischman 

Jeffco 

Server Farm 

POTS/WAN 

10/100/1000 

L3 GEC 

MESA 

Si 

Si 

Foot 

Hills 

Gigabit Core 

Center 

Green 

Si 

Si 

Si 

Si 

FootHills/Pearl Street/UNAVCO 

Semi-Fully Meshed Core design 

A link or Switch failure at any of the buildings will not affect any other building. The design is also 

optimised for IP Telephony and Multicast and any QoS strategy needed for any technology or application. 

The above diagram provides an idea of the final design; a phased migration is provided later. 

The proposed design offers the following advantages and common features: 

At the Access Layer: 

- Switches would have unique VLANs (one or several that exist only on that switch). This in return 

eliminates dependency on Spanning Tree Protocol for convergence. 



- User ports to be configured with PortFast (See best practices section below) 

- ISL or dot1Q trunks to the Distribution layer. Allow only Access Layer VLANs on the trunk. 

- The ability to load-balance traffic over the Access-Distribution trunks. (This is done using HSRP). 

If VLANs span multiple switches or buildings, then we can achieve load balancing using STP by 

alternating “Root” configuration between Distribution Switches. 

- If STP is the choice for redundancy, the UplinkFast feature is utilized to offer almost instantaneous 

fail-over of uplinks. 

At the Distribution Layer: 

- Two Distribution switches per building to provide maximum redundancy and performance. The 

two distribution switches are also directly connected to the Core using L3 connections. 

- Terminate and limit the size of broadcast domains. With unique VLANs configured at the Access 

switches, the link between the two Distribution in every building can be an L3 connections. 

- A failure in one of the closets will only be limited to that closet. 

- Configure HSRP with alternating priority levels between the two Distribution-MSFCs to provide 

redundancy as well as load balancing of the Uplinks. 

- Easy to manage and troubleshoot 

- Root bridges are local and known. No Root is across the WAN links. 

At the Core Layer: 

- Layer 3 Core 

- Highly redundant 

- Simple Design 

- Point-to-Point L3 connections 


Configuration Best Practices 

VLAN 1 

VLAN1 has a special significance in Catalyst networks, and can be the cause of some confusion. 

The Catalyst Supervisor always uses the Default VLAN, 1, to tag a number of control and management 

protocols when trunking, such as CDP, VTP and PAgP! All ports including the internal sc0 interface are 

configured to be members of VLAN1 at initial boot up. All trunks carry VLAN1 by default, and in older 

Catalyst code, (prior to CatOS 5.3), it was not possible to block user data in VLAN1. 

Two other definitions are needed to help clarify some well-used terms in Catalyst networking: 

• The Management VLAN is where sc0 resides, and can be changed. 

• The Native VLAN is defined as the VLAN a port will return to when not trunking, and is the untagged 

VLAN on an 802.1Q trunk. 

There are several good reasons to tune a network and alter the behavior of ports in VLAN1: 

• When the diameter of VLAN1, like any other VLAN, gets large enough to be a risk to stability, 

particularly from an STP perspective, it needs to be pruned back. 

• Control plane data on VLAN1 should be kept separate from user data in other VLANs, and ideally 

the management VLAN, to simplify troubleshooting and maximise available CPU. 

• Layer-2 loops in VLAN1 must be avoided when designing Multilayer Campus networks without 

STP, yet trunking is still required to the access-layer multiple there are multiple VLANs and IP 

subnets. To do this, manually clear VLAN1 from trunk ports. 

Note: 

• CDP, VTP and PAgP updates are always forwarded with a VLAN1 tag even if VLAN1 has been 

cleared on trunks and is not the native VLAN. One implication is that when trunking to a router, an 

interface must be defined in VLAN1 to detect CDP. 

• DTP updates are forwarded with VLAN1 tags on ISL trunks, but use the native VLAN on 802.1Q. 

• 802.1Q IEEE BPDUs are forwarded untagged on the ‘Common Spanning Tree’ VLAN1 (like ISL) 

for interoperability with other vendors, unless VLAN1 has been cleared from the trunk. Cisco Per- 

VLAN Spanning tree (PVST+) BPDUs are sent and tagged for all other VLANs. Analysis 



VLAN Trunking Protocol (VTP) 

Before you create VLANs you must decide what VTP mode to use in your network. With VTP you can 

make VLAN configuration changes centrally on one or more switches and have those changes 

automatically communicated to all the other switches in the network. 

VTP is a Layer 2 messaging protocol that maintains VLAN configuration consistency by managing the 

addition, deletion, and renaming of VLANs on a network-wide basis. VTP minimizes mis-configurations 

and configuration inconsistencies that can result in a number of problems such as duplicate VLAN names, 

incorrect VLAN-type specifications, and security violations. The VLAN database built is stored in 

NVRAM, separately the configuration. 

AS recommends transparent mode. As the majority of customers do not create or delete VLANs 

frequently, when a new VLAN is needed it is not much effort to update all switches in a domain, usually 

numbering 20 or less. 

• This practice encourages good change control. 

• Limits the risk of a user error, such as deleting a VLAN, impacting the entire domain. 

• Eliminated the risk of any VTP bug affecting the entire network. 

• There is no risk from a new switch being introduced into the network with a higher VTP revision 

number and over-writing the entire domain's VLAN configuration. There is a positive and negative 

side to VTP being able to make changes very easily on a network - most enterprises prefer a cautious 

approach. 

• STP per VLAN and unnecessary flooding should be limited by explicit configuration (i.e. pruning) of 

what VLANs are propagated on what trunks. A per switch VLAN configuration also encourages this 

practice. 

• The extended VLAN range in CatOS 6.x, numbers1025-4094, can only be configured in this way. 

Trunking Mode 

Purpose: DTP is the second generation of DISL (Dynamic ISL) and exists to ensure that the different 

parameters involved in sending ISL or 802.1Q frames, like the configured encapsulation type, native 

VLAN, hardware capability, etc. are agreed by the Catalysts at either end of a trunk. This also helps protect 

against non-trunk ports flooding tagged frames, a potentially serious security risk, by ensuring ports and 

their neighbors are either in a safe trunking or non-trunking state. 

Operational Overview: DTP is a layer-2 protocol that negotiates configuration parameters between a 

switch port and it's neighbor. It uses another well-known multicast MAC address of 01-00-0c-cc-cc-cc and 

a SNAP protocol type of 0x2004. Here is a summary of the configuration modes: 

Note: ISL and 802.1Q encapsulation type can be set or negotiated - ISL will be preferred over dot1Q, but 

is recommended to be set. 

• DTP assumes point-to-point connection, and Cisco devices will support 802.1Q trunk ports that are 

only point-to-point. 

• During DTP negotiation, the ports will not participate in STP. Only after the port type becomes one of 

the three types (Access, ISL or 802.1Q), the port will be added to STP. (If PAgP is running that is the 

next process to run prior to the port participating in STP). 

• VLAN 1 will usually be there on the trunk port. If the port is trunking in ISL mode, DTP packets are 

sent out on VLAN 1, otherwise (for 802.1Q trunking or non-trunking ports) on the Native VLAN. 



• In desirable mode DTP packets transfer the VTP domain name, which must match for a negotiated 

trunk to come up, plus trunk configuration and admin status. 

• Messages are sent every 1s during negotiation, and every 30s after that. 

• Be careful that it is understood modes (on, nonegotiate, off) explicitly specify in which state the port 

will end up. A bad configuration can lead to a dangerous inconsistent state where one side is trunking 

and the other is not. A port in on, auto, or desirable sends DTP frames periodically. If a port in auto 

or desirable mode doesn't see a DTP packet in 5min it will be set to non-trunk. 

AS recommends an explicit trunk configuration of ‘Desirable-Desirable’. 

In this mode, network operators can trust syslog and command line status messages that a port is up and 

trunking, unlike the mode ‘on’ that can make a port appear up even though the neighbor is mis-configured. 

With the consistency of using the same configuration at both ends of the trunk, the ambiguous and default 

setting ‘auto’ can be avoided. 

set trunk desirable 

Importantly, also set 'trunk off' on all non-trunk ports. This helps eliminate wasted negotiation time when 

bringing host ports up (in conjunction with ‘port channel mode off’ and PortFast enabled - set port host). 

Set trunk off all 

By default, Trunking ports will propagate information about all VLANs, but AS recommends limiting that 

to the VLANs defined on the wiring closet switches. This practice has many advantages, most importantly, 

is the ability to isolate issues, broadcasts, and loops to one wiring closet instead of the entire network. 

Spanning Tree Protocol (STP) 

Spanning tree maintains a loop free layer two environment in redundant switched and bridged networks. 

Without STP, frames would loop and/or multiply indefinitely causing a network meltdown as all devices in 

the broadcast domain are interrupted by high traffic. 

As an initial observation, all Catalyst switches have STP enabled by default. AS recommends leaving the 

feature enabled for these reasons: 

• When there is a loop (induced by mis-patching/bad cable, etc.), STP will prevent detrimental effects to 

the network, potentially made much worse with multicast and broadcast data. 

• Protection against channel breaking down. 

• Most of networks are configured with STP and hence gets maximum exposure. More exposure 

generally equates to more stable code. 

• Protection against dual attached NICs misbehaving (or routing enabled on servers). 

• Many protocols are closely related to STP in the code (such as PAgP, IGMP snooping, trunking etc.) - 

running without it may lead to undesirable results. During a reported network disruption, TAC and DE 

will most likely suggest that non-usage of STP will be at the centre of the fault if at all conceivable. 



set spantree enable all 

! default 

PortFast 

PortFast is used to bypass normal spanning-tree operation on access-ports to speed up connectivity 

between end-stations and the services they need to connect to after link initialization. 

AS recommend STP PortFast be set on for all enabled host ports. Must be diligent to catch 

all ports!!! 

AS recommends using the “host” macro to configure PortFast on Access ports: 

Set port host ! macro for the following commands 

set spantree portfast enable 

set trunk off 

set channel mode off 

Note that PortFast doesn’t mean that we do not run spanning-tree at all on those ports: BPDUs are still 

sent, received and processed. It must also be understood that PortFast will not work on trunks since these 

ports are typically not access-ports. This may cause confusion on access-ports running ISL or dot1q 

connected to multi-homed trunking capable end-stations. 

PortFast BPDU-Guard provides a method for preventing loops by moving a non-trunking port into an 

ErrDisable state when a BPDU is received on that port. 

Under “normal” conditions we should never receive a BPDU packet on an access-port configured for 

PortFast, so if we for some reason should see a BPDU coming in, it indicates an invalid hence dangerous 

configuration and the action we take upon this is to shut down the access-port. When the BPDU Guard 

feature is enabled on the switch spanning tree shuts down PortFast-configured interfaces that receive 

BPDUs instead of putting them into the spanning-tree blocking state 

set spantree portfast bpdu-guard enable 

UplinkFast 

UplinkFast is a solution for Access Layer switches to move it’s blocking link almost instantaneously to 

forwarding if there is a direct link failure to the root switch. Access Layer switches using UplinkFast 

forward their CAM table over the new root port, preventing unknown unicasts to flood. UplinkFast feature 

uses uplink group, which consists of the root port and all the ports that provide an alternate connection to 

the root bridge (blocking redundant links). If the root port fails (primary uplink failure), a port from the 

uplink group is selected to immediately replace it. Uplinkfast feature is only useful when there is a 

redundant blocking link. Uplinkfast and the following Cross stack uplinkfast feature are not necessary if a 

design with no layer 2 loops is implemented 



Spanning Tree Root 

(Primary) 

Spanning Tree Root 

(Secondary) 

Distribution 

Trunk 

Access 

Gi0/2 

Gi0/1 

CatOS> (enable) set spantree uplinkfast enable 

BackboneFast 

In the above figure if the link between the two distribution layer switches failed then it’s considered an 

indirect root link failure. After an indirect failure, the secondary root bridge will start transmitting 

configuration BPDUs claiming itself to be the root. Access switch upon receiving the BPDU will 

realize that the BPDU is inferior compared to what it has on its root port. The access switch will then 

transmit a Root Link Query message on its root port to determine if the original root is still alive. Upon 

receiving a positive response from the root that it is still alive, Access switch will transmit a BPDU to 

the secondary root bridge so that it can calculate the path to root. Access switch will also transition the 

blocking port to listening and learning state then forwarding. Without backbonefast the Access switch 

won’t have reacted to the inferior BPDU until max-age (20 seconds by default) on the port expired. 

Therefore backbonefast cuts the convergence time for indirect failure by 20 seconds. The access port 

still goes through listening and learning states, which takes 30 seconds in total 

CatOS> (enable) set spantree backbonefast enable 

Unidirectional Link Detection (UDLD) 

The Uni-Directional Link Detection (UDLD) feature is intended to resolve some of the following 

issues: 

- Monitor physical cabling configurations - shutting down any mis-wired ports as ‘ErrDisabled’ 

- Protect against Uni-directional links, on Fibre-optical and copper Ethernet interfaces. When a unidirectional 

link is detected -due to media or port/interface malfunction- the affected port is 

shutdown as ‘ErrDisabled’ and a corresponding syslog message generated 

- To be clear: it is the top switch in these diagrams that see a unidirectional link and will place the 

port in Errdisable 

Spanning tree, with its steady state unidirectional BPDU flow, was an acute sufferer from the above 

failures. It is easy to see how a port may suddenly be unable to transmit BPDUs, causing an STP state 

change from 'blocking' to 'forwarding' on the neighbor, yet a loop still existing as the port is still able to 

receive. 

For maximum protection against symptoms resulting from uni-directional links, AS recommends 

enabling UDLD on point-to-point FE/GE links between “aggressive-UDLD capable” Cisco 

switches – where the message interval is configurable / set to 15 seconds default. 



By default UDLD is disabled globally, and enabled in readiness on fiber ports. As UDLD is an 

infrastructure protocol needed between switches only, it is disabled by default on copper ports as these 

tend to be used for host access. 

Set udld enable 

Set udld enable 

! once globally enabled, all FE and 

GE fibre ports have UDLD enabled 

by default. 

! for additional specific ports and 

copper media if needed. 

Set udld aggressive-mode enable ! all point to point links 

Enabling UDLD ‘aggressive-mode’ provides additional benefits: 

This feature provides enhanced protection against dangerous unidirectional link conditions in the 

following situations, and includes attempts to re-establish a connection with the neighbor upon failure 

detection: 

- One side of a link has a port stuck (either Tx and Rx). 

- One side of a link remains up while the other side of the link has gone down. This reduces the 

reliance on Layer 1 FEFI mechanisms 

- After eight failed retries, the port is transitioned to an ErrDisable state and a syslog message 

logged 

- In these cases, UDLD aggressive mode will ErrDisable both of the ports on the link and stops the 

black-holing of traffic 

Aggressive mode UDLD also allows the possibility to manually configure the UDLD probe/echo 

message interval between values ranging from 7-90 seconds, the default interval being 15 seconds. 

EtherChannel (PAgP) 

EtherChannel technologies allow the inverse multiplexing of multiple (up to eight on Cat6k) channels into 

a single logical link, and although each platform differs from the next in implementation, it is important to 

understand the common requirement 

- An algorithm to statistically multiplex frames over multiple channels 

- Creation of a logical port so that a single instance of STP can be run An algorithm to statistically 

multiplex frames over multiple channels 

- A channel management protocol. PAgP is a management protocol that will check for parameter 

consistency at either end of the link, and assist the channel in adapting to link failure or addition 

- PAgP requires that all ports in the channel belong to the same VLAN or are configured as trunk 

ports. (Because dynamic VLANs can force the change of a port into a different VLAN, they are 

not included in EtherChannel participation.) 

- When a bundle already exists and a VLAN of a port is modified, all ports in the bundle are 

modified to match that VLAN 

- PAgP does not group ports that operate at different speeds or port duplex. If speed and duplex are 

changed when a bundle exists, PAgP changes the port speed and duplex for all ports in the bundle 

Configuration Options: EtherChannels can be configured in different modes, summarized in the table 

below: 

Mode 

On 

Configurable Options: 

PAgP not in operation. The port is channeling regardless of how the 



Off 

Auto (Default) 

Desirable 

Non-silent 

(Default on 5k fiber 

FE and GE ports) 

Silent 

(Default on all 6k 

and 4k ports, and 

5k copper ports) 

peer port is configured. If the peer port mode is on, a channel is 

formed. 

The port is not channeling regardless of how the peer port is 

configured. 

Aggregation is under control of the PAgP protocol. Places a port into 

a passive negotiating state and no PAgP packets are sent on the 

interface until at least one PAgP packet is received that indicates 

that the sender is operating in desirable mode. 

Aggregation is under control of the PAgP protocol. Places a port into 

an active negotiating state, in which the port initiates negotiations 

with other ports by sending PAgP packets. A channel is formed with 

another port group in either desirable or auto mode. 

An auto or desirable mode keyword. If no data packets are received 

on the interface, then the interface is never attached to an agport 

and cannot be used for data. 

This bi-directionality check was provided for specific Cat5k hardware 

(EBC and SAINT4/5 ASICS), as some link failures result in the 

channel being broken apart and is to be avoided. More flexible 

bundling and improved bi-directionality checks are present by default 

in the 4k and 6k hardware. See section on auto-negotiation. 

F2H1A0C4 

An auto or desirable mode keyword. If no data packets are received 

on the interface, then after a 15s timeout period, the interface is 

attached by itself to an agport and can thus be used for data 

transmission. 

Silent mode also allows for channel operation when the partner can 

be an analyzer or server that never sends PAgP. 

AS recommends enabling PAgP on all switch-to-switch channel connections (ie. avoid the mode ‘on’). 

The preferred way is to set ‘desirable mode’ at both ends of a link. 

The additional recommendation is to allow the ‘silent/non-silent’ keyword to default, i.e. silent on 6ks and 

4ks, non-silent on 5k fiber ports. 

As discussed in previous sections, explicitly configuring channeling off on all other port is helpful for rapid 

data forwarding as ports come up. Waiting up to 15s for PAgP to timeout on a port that will not be used for 

channeling should be avoided, especially as the port is then handed over to STP which itself can take 30s to 

allow data forwarding, 45s in total. 

set port channel mode desirable 

set port channel mode off! where ports not in use, part of the 

! set port host macro. 


Corporate Headquarters 



San Jose, CA 95134-1706 

USA 

www.cisco.com 

Tel: 408 526-4000 

800 553-NETS (6387) 

Fax: 408 526-4100 

European Headquarters 

Cisco Systems Europe 

11 Rue Camille Desmoulins 

92782 Issy-Les-Moulineaux 

Cedex 9 

France 

www-europe.cisco.com 

Tel: 33 1 58 04 60 00 

Fax: 33 1 58 04 61 00 

Americas Headquarters 



San Jose, CA 95134-1706 

USA 

www.cisco.com 

Tel: 408 526-7660 

Fax: 408 527-0883 

Asia Pacific Headquarters 

Cisco Systems Australia, Pty., Ltd 

Level 9, 80 Pacific Highway 

P.O. Box 469 

North Sydney 

NSW 2060 Australia 

www.cisco.com 

Tel: +61 2 8448 7100 

Fax: +61 2 9957 4350 

Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on the 

Cisco Web site at www.cisco.com/go/offices. 

Argentina • Australia • Austria • Belgium • Brazil • Bulgaria • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic Denmark • Dubai, UAE 

Finland • France • Germany • Greece • Hong Kong SAR • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia • Mexico 

The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore • Slovakia • Slovenia 

South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela • Vietnam • Zimbabwe

Design Review Template - NETS

Create successful ePaper yourself

Delete template?

Save as template?