IP - MPLS Design - v1.5

Cisco Systems Advanced Services
Wateen Telecom
Converged Multi-Service Network - IP/MPLS Low Level Design
Version 1.5 DRAFT
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND
RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL
RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE
INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A
COPY.
The following information is for FCC compliance of Class A devices: This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to part 15 of the FCC rules. These limits
are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio-frequency energy and, if
not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in
which case users will be required to correct the interference at their own expense.
The following information is for FCC compliance of Class B devices: The equipment described in this manual generates and may radiate radio-frequency energy. If it is not installed in accordance with Cisco’s
installation instructions, it may cause interference with radio and television reception. This equipment has been tested and found to comply with the limits for a Class B digital device in accordance with the
specifications in part 15 of the FCC rules. These specifications are designed to provide reasonable protection against such interference in a residential installation. However, there is no guarantee that interference will
not occur in a particular installation.
You can determine whether your equipment is causing interference by turning it off. If the interference stops, it was probably caused by the Cisco equipment or one of its peripheral devices. If the equipment causes
interference to radio or television reception, try to correct the interference by using one or more of the following measures:
Turn the television or radio antenna until the interference stops.
Move the equipment to one side or the other of the television or radio.
Move the equipment farther away from the television or radio.
Plug the equipment into an outlet that is on a different circuit from the television or radio. (That is, make certain the equipment and the television or radio are on circuits controlled by different circuit breakers or fuses.)
Modifications to this product not authorized by Cisco Systems, Inc. could void the FCC approval and negate your authority to operate the product.
The following third-party software may be included with your product and will be subject to the software license agreement:
CiscoWorks software and documentation are based in part on HP OpenView under license from the Hewlett-Packard Company. HP OpenView is a trademark of the Hewlett-Packard Company. Copyright © 1992, 1993
Hewlett-Packard Company.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system.
All rights reserved. Copyright © 1981, Regents of the University of California.
Network Time Protocol (NTP). Copyright © 1992, David L. Mills. The University of Delaware makes no representations about the suitability of this software for any purpose.
Point-to-Point Protocol. Copyright © 1989, Carnegie-Mellon University. All rights reserved. The name of the University may not be used to endorse or promote products derived from this software without specific
prior written permission.
The Cisco implementation of TN3270 is an adaptation of the TN3270, curses, and termcap programs developed by the University of California, Berkeley (UCB) as part of the UCB’s public domain version of the UNIX
operating system. All rights reserved. Copyright © 1981-1988, Regents of the University of California.
Cisco incorporates Fastmac and TrueView software and the RingRunner chip in some Token Ring products. Fastmac software is licensed to Cisco by Madge Networks Limited, and the RingRunner chip is licensed to
Cisco by Madge NV. Fastmac, RingRunner, and TrueView are trademarks and in some jurisdictions registered trademarks of Madge Networks Limited. Copyright © 1995, Madge Networks Limited. All rights
reserved.
Xremote is a trademark of Network Computing Devices, Inc. Copyright © 1989, Network Computing Devices, Inc., Mountain View, California. NCD makes no representations about the suitability of this software for
any purpose.
The X Window System is a trademark of the X Consortium, Cambridge, Massachusetts. All rights reserved.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH ALL FAULTS. CISCO AND THE
ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PRACTICAL
PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST
PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY
OF SUCH DAMAGES.
AccessPath, AtmDirector, Browse with Me, CCIP, CCSI, CD-PAC, CiscoLink, the Cisco Powered Network logo, Cisco Systems Networking Academy, the Cisco Systems Networking Academy logo, Cisco Unity, Fast
Step, Follow Me Browsing, FormShare, FrameShare, IGX, Internet Quotient, IP/VC, iQ Breakthrough, iQ Expertise, iQ FastTrack, the iQ logo, iQ Net Readiness Scorecard, MGX, the Networkers logo, ScriptBuilder,
ScriptShare, SMARTnet, TransPath, Voice LAN, Wavelength Router, and WebViewer, Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo,
Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherSwitch, FastHub, FastSwitch,
GigaStack, IOS, IP/TV, LightStream, MICA, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, SlideCast, StrataView Plus, Stratm, SwitchProbe, TeleRouter, and VCO are trademarks
or registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.
All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.
(0110R).
Please refer to http://www.cisco.com/logo/ for the latest information on Cisco logos, branding and trademarks.
INTELLECTUAL PROPERTY RIGHTS:
THIS DOCUMENT CONTAINS VALUABLE TRADE SECRETS AND CONFIDENTIAL INFORMATION OF CISCO SYSTEMS, INC. AND IT’S SUPPLIERS, AND SHALL NOT BE DISCLOSED TO ANY
PERSON, ORGANIZATION, OR ENTITY UNLESS SUCH DISCLOSURE IS SUBJECT TO THE PROVISIONS OF A WRITTEN NON-DISCLOSURE AND PROPRIETARY RIGHTS AGREEMENT OR
INTELLECTUAL PROPERTY LICENSE AGREEMENT APPROVED BY CISCO SYSTEMS, INC. THE DISTRIBUTION OF THIS DOCUMENT DOES NOT GRANT ANY LICENSE IN OR RIGHTS, IN
WHOLE OR IN PART, TO THE CONTENT, THE PRODUCT(S), TECHNOLOGY OF INTELLECTUAL PROPERTY DESCRIBED HEREIN.
Converged Multi-Service Network - IP/MPLS Low Level Design

Copyright © 2003, Cisco Systems, Inc.
All rights reserved.
COMMERCIAL IN CONFIDENCE.
A PRINTED COPY OF THIS DOCUMENT IS CONSIDERED UNCONTROLLED
Contents
Contents......................................................................................................................................... 3
Figures ......................................................................................................................................... 11
About This Low Level Design .................................................................................................... 21
History..................................................................................................................................... 21
Review..................................................................................................................................... 22
About This Design Document.................................................................................................... 23
Document Purpose ................................................................................................................ 23

Intended Audience................................................................................................................. 23
Scope ...................................................................................................................................... 23
Assumptions and Caveats .................................................................................................... 24
Related Documents ............................................................................................................... 24
References.............................................................................................................................. 24
Network Overview ....................................................................................................................... 27
Network Layout: ..................................................................................................................... 28

Design Considerations.......................................................................................................... 30
Physical Network Design ........................................................................................................... 31
Physical Network Topology .................................................................................................. 31

Network Hardware Components for IP/MPLS Core:........................................................... 32
Physical and Functional Description of 12800 Series Chassis: ....................................... 36
Chassis Card Cages........................................................................................................................... 38
Upper Card Cage............................................................................................................................... 38
Lower Card Cage .............................................................................................................................. 38
Switch Fabric Overview.................................................................................................................... 39
Switch Fabric Card Functionality ..................................................................................................... 39
Clock Scheduler Cards...................................................................................................................... 40
October 2, 2006 Wateen Telecom 3

Company Confidential. A printed copy of this document is considered uncontrolled.
Switch Fabric Cards .......................................................................................................................... 40
Switch Fabric Card Cage .................................................................................................................. 40
Performance Route Processor-2 Overview ....................................................................................... 40
PRP-2 Components ........................................................................................................................... 41
PRP PCMCIA Card Slots and Status LEDs...................................................................................... 43
PRP Reset Switch ............................................................................................................................. 43
PRP Alphanumeric Message Displays.............................................................................................. 43
Physical and Functional Description of 7600 Series Chassis: ......................................... 44
SUP720 Switch Fabric Slot Allocation............................................................................................. 48
Redundancy....................................................................................................................................... 49
Fan Assembly.................................................................................................................................... 49
Supervisor Engine 720 ...................................................................................................................... 50
Supervisor Engine 32 ........................................................................................................................ 51
SIP Slot Allocation: .......................................................................................................................... 52
Logical Network Design ............................................................................................................. 54
Naming and Addressing Specifications .............................................................................. 54

BGP AS Number............................................................................................................................... 54
Naming Conventions......................................................................................................................... 54
IP Addressing Scheme ...................................................................................................................... 55
Firmware and Software Requirements ................................................................................ 56
IGP Routing: ..................................................................................................................................... 58
MP-BGP4 (Multi-protocol BGP)...................................................................................................... 58
LDP (Label Distribution Protocol): .................................................................................................. 58
Resilience.......................................................................................................................................... 58
Scalability ......................................................................................................................................... 59
MPLS Core Design ...................................................................................................................... 61
MPLS Architecture Overview................................................................................................ 61

Network Design ...................................................................................................................... 63
Sites Connectivity ............................................................................................................................. 64
MPLS Core IP Addressing..................................................................................................... 64
Autonomous System Number .............................................................................................. 70
Backbone Protocols .............................................................................................................. 70
OSPF (Open Shortest Path first) IGP................................................................................................ 70
Bi-directional Forwarding detection (BFD) ...................................................................................... 80
Cisco Express Forwarding (CEF) Switching .................................................................................... 87
MSP (Multiplexed Switching Protection) Configuration.................................................................. 87
LDP (Label Distribution Protocol).................................................................................................... 91
MP-BGP4 (Multi-protocol BGP)...................................................................................................... 96

Stateful Switchover (SSO) .............................................................................................................. 106
Non Stop Forwarding (NSF)........................................................................................................... 108
MPLS VPN Design..................................................................................................................... 111
Definition of a VPN............................................................................................................... 111

Basic VPN Configuration .................................................................................................... 111
VRF Table Definitions.......................................................................................................... 112
VRF Name ...................................................................................................................................... 112
Route-Distinguisher ........................................................................................................................ 112
Route-Targets.................................................................................................................................. 113
Route-Maps..................................................................................................................................... 113
Assigning a VRF to an interface ..................................................................................................... 114
MPLS VPN Services:............................................................................................................ 114
Intranet – Full Mesh VPN ............................................................................................................... 115
Extranet VPNs - Overlapping VPNs............................................................................................... 118
Central Services VPN (Hub And Spoke)- No Connectivity between Spokes:................................ 120
Central Services VPN (Hub and Spoke) – Connectivity between Spokes via Hub ........................ 122
Network Management VPN............................................................................................................ 124
Controlling route exports in NMS & Extranet VPN ....................................................................... 126
Customer Edge to Provider Edge Routing (CE to PE) ..................................................... 128
Routing Protocols............................................................................................................................ 128
“Connected” Routing ...................................................................................................................... 129
Static Routing.................................................................................................................................. 129
RIP v2 ............................................................................................................................................. 130
eBGP............................................................................................................................................... 131
Access to Wateen Telecom IP/MPLS Core:....................................................................... 132
VLAN Allocation............................................................................................................................ 132
Configuring HSRP on a VRF interface........................................................................................... 135
MPLS Layer 2 VPN Design ....................................................................................................... 141
AToM overview..................................................................................................................... 141

Ethernet over MPLS........................................................................................................................ 142
VC ID ........................................................................................................................................ 144
VC Label ................................................................................................................................... 144
Directed LDP Session ............................................................................................................... 144
MTU Issues ............................................................................................................................... 145
Ethernet Relay Service (ERS) – VLAN Mode .......................................................................... 145
Ethernet Wire Service (EWS) – Port Mode............................................................................... 147
VPLS / H-VPLS...................................................................................................................... 149
VPLS............................................................................................................................................... 149

VPLS Configuration example ......................................................................................................... 151
VPLS Restrictions........................................................................................................................... 152
Dial Services .............................................................................................................................. 154
Technology Primers ............................................................................................................ 154

PPP.................................................................................................................................................. 154
Layer 2 Tunneling Protocol ............................................................................................................ 155
L2TP Access Concentrator (LAC) ............................................................................................ 155
L2TP Network Server (LNS) .................................................................................................... 155
LNS Failover and Load Sharing................................................................................................ 156
AAA................................................................................................................................................ 156
Authentication ........................................................................................................................... 156
Authorization............................................................................................................................. 156
Accounting ................................................................................................................................ 156
RADIUS .................................................................................................................................... 157
Overlapping IP Address Pools................................................................................................... 157
Wateen Dial Services Overview.......................................................................................... 158
Introduction..................................................................................................................................... 158
Remote Access to MPLS VPNs ................................................................................................ 158
Backup to MPLS VPN Primary Link ........................................................................................ 158
Backup Internet Connection ...................................................................................................... 158
Bill of Quantity ............................................................................................................................... 158
Device Deployment Details ............................................................................................................ 159
Dial Service Functional Description and Call Flows ........................................................ 160
Service Functional Description ....................................................................................................... 160
Call Flow Diagrams ........................................................................................................................ 161
Dial In (MPLS VPN Remote Access) ....................................................................................... 161
Dial Backup (Internet or MPLS VPN) ...................................................................................... 162
Complete Call Flow................................................................................................................... 163
Physical Network Topology ................................................................................................ 164
Main PoP Architecture.................................................................................................................... 164
City and Satellite PoPs.................................................................................................................... 165
Physical Connectivity Details ......................................................................................................... 166
Logical Network Architecture ............................................................................................. 171
IP Addressing Details...................................................................................................................... 171
NAS IP Addressing ................................................................................................................... 171
VHG/PE Addressing ................................................................................................................. 173
Routing Protocol Design................................................................................................................. 173
Routing Architecture - NAS Device in City/Satellite PoP ........................................................ 174
Routing Architecture - NAS Device in a Main PoP .................................................................. 174

Routing Architecture – VHG/PE Device in a Main PoP ........................................................... 175
Network Address Translation (NAT).............................................................................................. 176
VHG/PE Resiliency ........................................................................................................................ 176
Dial Backup Architecture................................................................................................................ 177
Configuration Templates..................................................................................................... 178
NAS/LAC ....................................................................................................................................... 178
ISDN/Modem & PPP ................................................................................................................ 178
PPP ............................................................................................................................................ 179
L2TP.......................................................................................................................................... 180
CAR Attributes for VHG/PE Failover and Load-sharing.......................................................... 180
AAA .......................................................................................................................................... 180
RADIUS .................................................................................................................................... 180
VHG/PE .......................................................................................................................................... 181
Customer VRF’s........................................................................................................................ 181
Virtual Template........................................................................................................................ 181
BGP ........................................................................................................................................... 182
PPP ............................................................................................................................................ 182
L2TP.......................................................................................................................................... 183
AAA .......................................................................................................................................... 184
RADIUS .................................................................................................................................... 184
CAR Attribute ........................................................................................................................... 184
IOS versions ......................................................................................................................... 184
Internet Services ....................................................................................................................... 186
Internet Peering & Transit Service ..................................................................................... 186

IPv4 Route Reflectors ..................................................................................................................... 188
BGP Security .................................................................................................................................. 189
BGP TTL Security Check ............................................................................................................... 190
BGP Maximum Prefixes ................................................................................................................. 190
BGP MED....................................................................................................................................... 191
BGP Deterministic MED ................................................................................................................ 191
BGP Communities .......................................................................................................................... 192
BGP Community Formatting .......................................................................................................... 193
BGP Fast-external-fallover ............................................................................................................. 193
BGP Dampening ............................................................................................................................. 194
BGP Scaling.................................................................................................................................... 195
BGP Timers .................................................................................................................................... 197
BGP Configuration ......................................................................................................................... 198
NAT Service ................................................................................................................................... 199
Ingress Access-Lists........................................................................................................................ 200

Unicast Reverse Path Forwarding (uRPF) ...................................................................................... 200
ACL at the Internet Border Routers: ............................................................................................... 201
Data Center Connectivity................................................................................................................ 203
Internet Customers Connectivity..................................................................................................... 203
Dual Attached PEs .......................................................................................................................... 204
Transit BGP Customers .................................................................................................................. 205
Quality of Service...................................................................................................................... 206
Quality of Service Overview................................................................................................ 206

Introduction..................................................................................................................................... 206
QOS Disclaimer .............................................................................................................................. 207
Differentiated Services model – an introduction............................................................................. 207
QoS Tools ....................................................................................................................................... 214
Classification................................................................................................................................... 214
Marking........................................................................................................................................... 216
Class Queuing - LLQ ...................................................................................................................... 216
Congestion Avoidance – WRED .................................................................................................... 220
Core Router Mechanisms (12816 Router) ...................................................................................... 224
Bandwidth allocation with Deficit Round Robin ...................................................................... 224
Architecture details on Class of Service implementation on the Cisco 12000 series ................ 224
Class Queuing – MDRR............................................................................................................ 226
Congestion Avoidance – WRED............................................................................................... 228
Quality of Service MQC Examples................................................................................................. 229
QoS Architecture of the Cisco 7600 ............................................................................................... 233
Input scheduling ........................................................................................................................ 233
Cisco 7600 Classification and Marking..................................................................................... 233
Internal DSCP ........................................................................................................................... 234
QoS & MPLS traffic: Trust State .............................................................................................. 234
QoS & MPLS traffic: Marking MPLS traffic............................................................................ 235
Cisco 7600 Policing................................................................................................................... 237
Scheduling and Congestion Avoidance on LAN ports.............................................................. 237
Strict Priority Queue.................................................................................................................. 238
WRR.......................................................................................................................................... 238
Buffer Allocation....................................................................................................................... 238
Tuning the port settings ............................................................................................................. 238
LAN Port capabilities of Wateen Telecom network.................................................................. 239
WAN port capabilities............................................................................................................... 240
Wateen Telecom Provisioned Quality of Service ............................................................. 240
Classifying Traffic into Classes ...................................................................................................... 241
Categorising Traffic at the CE Ingress............................................................................................ 247

Qos at the Edge Layer..................................................................................................................... 251
Queueing at the Core layer.............................................................................................................. 259
Traffic Engineering ................................................................................................................... 263
Traffic Engineer Disclaimer ................................................................................................ 263

MPLS Traffic Engineering Overview .................................................................................. 263
MPLS TE Fast ReRoute Link Protection............................................................................ 267
Primary Tunnels.............................................................................................................................. 268
One Hop Primary Tunnels .............................................................................................................. 269
OSPF Autoroute.............................................................................................................................. 269
Primary Tunnel Labels.................................................................................................................... 270
LDP over the Primary Tunnel ......................................................................................................... 270
Backup Tunnels .............................................................................................................................. 270
TE FRR Configuration Details........................................................................................................ 271
Auto Tunnel Feature ....................................................................................................................... 273
Creation of Backup Auto Tunnels................................................................................................... 274
Creation of Primary Auto Tunnels.................................................................................................. 274
Network Management & Infrastructure Security Best Practices.......................................... 276
Overview ............................................................................................................................... 276

Simple Network Management Protocol ............................................................................. 276
SNMP Trap Settings ....................................................................................................................... 277
Cisco Discovery Protocol................................................................................................................ 278
Logging........................................................................................................................................... 279
ICMP unreachable generation......................................................................................................... 282
Disable Unneeded Services............................................................................................................. 282
Enable Protection Services.............................................................................................................. 284
Preventing Unauthorized Access .................................................................................................... 285
Controlling Access to the Device.................................................................................................... 286
Login Banners................................................................................................................................. 287
CLI Management Authentication & Logging (AAA)..................................................................... 287
HTTP Server ................................................................................................................................... 288
NTP................................................................................................................................................. 288
Out-of-band Management ............................................................................................................... 288
ACL Log ......................................................................................................................................... 289
mls rate-limiters .............................................................................................................................. 289
Control Plane Policing .................................................................................................................... 289
Guidelines.................................................................................................................................. 290
Identifying Undesirable Traffic ................................................................................................. 290
Rate ........................................................................................................................................... 291

Requirements / dependencies for CoPP..................................................................................... 291
Interaction with the built-in special case rate limiters ............................................................... 291
Conclusion................................................................................................................................. 295
Glossary ..................................................................................................................................... 296
Document Acceptance ............................................................................................................. 297

Figures
Figure 1 Network Topology Overview 28
Figure 2 Lahore topology Overview 29
Figure 3 Fiber Route 31
Figure 4 SDH Network Topology Overview 32
Figure 5 12816 Router 36
Figure 6 12816 Router Chassis/Slots 37
Figure 7 DC Power Shelf 38
Figure 8 PRP-2, Front Panel View 40
Figure 9 PRP-2 (Horizontal Orientation) 42
Figure 10 Alphanumeric Message Displays—Partial Front Panel 43
Figure 13 - Slot Allocation 48
Figure 14 Sup 720 Slot Allocation 48
Figure 15 Cisco 7609 Router Internal Airflow 50
Figure 16 Supervisor Engine 720 Front Panel Features 51
Figure 17 Supervisor Engine 32 Front Panel 51

Figure 18 SIP Slot Numbering 52
Figure 19 MPLS VPN Core Logical 61
Figure 20 Label Imposition 62
Figure 21 Label Imposition 62
Figure 22 OSPF area topology with multiple areas 72
Figure 23 OSPF Router process 73
Figure 24 OSPF router-id 73
Figure 25 OSPF Reference Bandwidth 74
Figure 26 Setting the OSPF Network type 75
Figure 27 OSPF hello-dead interval tuning 77
Figure 28 Interface Dampening 77
Figure 29 ispf configuration 78
Figure 30 OSPF LSA Generation Tuning 78
Figure 31 OSPF LSA Arrival Tuning 79
Figure 32 OSPF SPF Timers Tuning 79
Figure 33 Fast RIB update 80
Figure 34 BFD Timers 81
Figure 35 BFD Async Mode 81
Figure 36 BFD Echo Mode 82
Figure 37 BFD Configuration 83

Figure 38 Configuring OSPF Authentication 86
Figure 39 OSPF Configuration Template 87
Figure 40 Enabling CEF 87
Figure 41 APS configuration on P router 90
Figure 42 APS configuration on PE router 91
Figure 43 Layer 2 Frame with 2 MPLS Labels 92
Figure 44 Increasing the MPLS MTU 93
Figure 45 LDP Autoconfiguation on P router 93
Figure 46 LDP Configuration Template 95
Figure 47 Label Filtering Template 96
Figure 48 BGP VPN Route Distribution 97
Figure 49 BGP Route Reflectors 98
Figure 50 Wateen VPNv4 Route Reflector Topology 99
Figure 51 BGP Authentication 99
Figure 52 BGP Deterministic Med 100
Figure 53 BGP configuration template for Route Reflector 104
Figure 54 BGP configuration template for PE 105
Figure 55 RP Redundancy Configuration 108
Figure 56 OSPF NSF configuration 109
Figure 57 BGP NSF configuration 109

Figure 58 VRF Definition 113
Figure 59 VRF with Route-map 114
Figure 60 VRF Assignment 114
Figure 61 Disable TTL propagation on PE router 115
Figure 62 – Simple Full Mesh VPN Model 116
Figure 63 Sample PE Configuration for any-any Topology 116
Figure 64 - Overlapping VPNs Model 118
Figure 65 Sample PE Configuration extranet vpn Topology 119
Figure 66 – Central Services VPN Model 121
Figure 67 Sample PE Configuration Hub and Spoke Topology 122
Figure 68 - Hub and Spoke Model with Connectivity between Spoke Sites 123
Figure 69 Sample PE Configuration- Hub and 2 Spokes 124
Figure 70 Management VPN 125
Figure 71 Management VPN Configuration on Customer PE 125
Figure 71 Management VPN Configuration on Management PE 126
Figure 72 Sample customer PE Configuration with export map 127
Figure 73 Sample PE Configuration with export map at Mgmt. PE 127
Figure 74 Sample configuration “connected” routes 129
Figure 75 static routes configuration in a VRF 129
Figure 76 Configuring static routes redistribution into vrf context under BGP 130

Figure 77 route leaking example 130
Figure 78 sample eBGP configuration 132
Figure 79 HSRP configuration on Active PE router 136
Figure 80 HSRP configuration on backup PE router 136
Figure 81 – VoIP Server Connectivity in Main PoP 137
Figure 82 Portfast on a Trunk 138
Figure 83 Enabling Bpduguard 138
Figure 84 Enabling Udld 138
Figure 85 Remove vlans from a trunk 139
Figure 86 Adding vlans to a trunk 139
Figure 87 AToM overview 142
Figure 88 L2vpn configuration 144
Figure 89 Ethernet over MPLS 146
Figure 90 Interface based EoMPLS configuration 146
Figure 91 Vlan based EoMPLS configuration 147
Figure 92 Port based EoMPLS configuration 148
Figure 93 VPLS 150
Figure 94 VPLS configuration example 151
Figure 95 VPLS configuration 152
Figure 96 AAA session flow 157

Figure 97 Dial Equipment Location 159
Figure 98 Dialin Architecture 161
Figure 99 Dial Backup Architecture 162
Figure 100 Complete Call flow 163
Figure 101 Dial Infrastructure Architecture – Main PoP 164
Figure 102 Dial Infrastructure Architecture – City/Satellite PoP 165
Figure 103 NAS Routing Architecture – City/Satellite PoP 174
Figure 104 NAS Routing Architecture – Main PoP 175
Figure 105 VHG/PE Routing Architecture – Main PoP 176
Figure 106 Internet GatewayConnectivity 187
Figure 107 IPv4 Route Reflector Connectivity 188
Figure 108 IPv4 Route Reflector Configuration 189
Figure 109 BGP Authentication 189
Figure 110 BGP TTL Security Check 190
Figure 111 BGP Prefixes Limit 190
Figure 112 BGP MED Comparison 191
Figure 113 BGP Deterministic MED 191
Figure 114 BGP Community Tagging 193
Figure 115 BGP community Format 193
Figure 116 BGP Disable Fast Fallover 194

Figure 117 BGP Dampening Configuration 195
Figure 118 Hold Queue Configuration 197
Figure 119 Sample NAT Configuration 199
Figure 120 Sample NAT Port Redirection 200
Figure 121 uRPF Configuration 201
Figure 122 uRPF strict mode 201
Figure 123 uRPF Loose Mode 201
Figure 124 Sample ingress ACL Configuration for Internet Gateway 202
Figure 125 Sample egress ACL Configuration for Internet Gateway 202
Figure 126 OSPF Totally Stubby Area Configuration 203
Figure 127 IP Helper Address Configuration 203
Figure 128 Summary Route Redistribution into OSPF 204
Figure 129 Summary Route Redistribution via iBGP 204
Figure 130 Breakdown of the IP ToS Field 208
Figure 131 - CE to PE QoS mechanisms overview 213
Figure 132 - PE to P QoS mechanisms overview 213
Figure 133 - Illustration of various QoS functions 214
Figure 134 Class-maponfiguration 215
Figure 135 Class-map Match Examples 215
Figure 136 Policy map configuration 216

Figure 137 - Packet flow with PQ 217
Figure 138 - Typical VoIP packet 217
Figure 139 - Call admission control 218
Figure 140 LLQ Configuration 219
Figure 141 - WRED algorithm 220
Figure 142 WRED Configuration 223
Figure 143 - 12000 ToFab DRR and WRED 225
Figure 144 - 12000 FrFab MDRR and WRED 226
Figure 145 Shaping & CBWFQ Configuration 229
Figure 146 Maximum Bandwidth Reservation 229
Figure 147 Explicit Policing on LLQ 230
Figure 148 Example Configuration for LLQ & Explicit Policing 230
Figure 149 WRED Configuration Example 231
Figure 150 Sample Configuration for E3 and E5 232
Figure 151 - QoS Architecture of the Catalyst 7600 233
Figure 152 Output Scheduling and Congestion Avoidance 238
Figure 153 Congestion aware vs Always-on Policer Configuration 243
Figure 154 Traffic Classification at CE 248
Figure 155 Traffic Marking at CE 250
Figure 156 BW reservation Per Customer Model 251

Figure 157 Qos Configuration at PE (unmanaged CE) 253
Figure 158 Qos Configuration towards PE-P Link 254
Figure 159 Sample Qos Configuration for 1P2Q2T Linecard 257
Figure 160 Sample Qos Configuration for 1P3Q8T Linecard 259
Figure 161 Sample Qos Configuration for P-P link 260
Figure 162 TE Path Setup 266
Figure 163 Label Switch Path with Link Protection 268
Figure 164 Protected Tunnel Traffic 269
Figure 165 One Hop Primary Tunnels 269
Figure 166 LDP peering on One Hop Primary Tunnels 270
Figure 167 Sample Configuration for all TE routers 271
Figure 168 TE Link Configuration 272
Figure 169 Sample configuration- One Hop Tunnel 273
Figure 170 Sample AutoTunnel Configuration 275
Figure 171 Sample NMS Configuration 276
Figure 172 Sample Trap listing 277
Figure 173 Enabling Traps on a Router 278
Figure 174 Log Configuration on a Router 281
Figure 175 TCP Connection Tracking 284
Figure 176 Core Dump Configuration 285

Figure 177 securing console & aux ports 286
Figure 178 Limiting Access to VTYs on a Router 286
Figure 179 Login Banner Configuration 287
Figure 180 NTP Configuration 288
Figure 181 Enabling Supervisor mls rate-limiters 289
Figure 182 CoPP Configuration 293

About This Low Level Design
Author: Kashif Rashid, Cisco Systems Advanced Services

Paul Horrocks, Cisco Systems Advanced Services
Atif Awan, Cisco Systems Advanced Services
Cisco Systems, Inc.
Reference Number: AS-123269
History
Table 1 Revision History
Version No. Issue Date Status Reason for Change

1.0 August 25th, Draft Initial Version
2006
1.1 August 29th, Draft Incorporated Dial Services portion

2006
1.2 August 31st, Draft Added NAS/LAC and VHG tablular details to Dial Services
2006 section.
1.3 September 28, Draft Incorporated customer feedback for MPLS Core, updated
2006 IOS versions
1.4 October 1st, Draft Incorporated customer feedback for Dial Services,
2006 realigned Dial Service section topics to improve readability
1.5 October 2nd, Draft Modified SFP module number per 7613 Main PoP router as
2006 per BoQ

Review
Table 2 Revision Review
Reviewer’s Details Version No. Date
Change Forecast: <Medium>
This document will be kept under revision control.

A printed copy of this document is considered uncontrolled.

About This Design Document
Document Purpose
The goal of this document is to provide Wateen Telecom with detailed design and configuration template
material explaining how different services like internet, MPLS VPN and Ethernet over MPLS can run on
their new ip core network. This network (Converged Multi-service Network as referenced by the customer)
consists of various sites in Pakistan. Wateen Telecom is expected to use WiMAX technology as last mile
access to the end customers. WiMax as well as overall network services design including various voice
related services as well as data services is led by Motorola. It is expected that this Low Level Design
Document will be used as a basis for implementation for this ip core network deployment across all sites of
Wateen Telecom in various cities in Pakistan. This document is part of a larger network design which deals
with Data Center architecture, Network Security services, Network Management solution, Public Wireless
LAN, Metro Ethernet and Dialup services. Other relevant sections of various technologies encompassing
this network and previously mentioned are discussed in separate low level design documents.
Intended Audience
Wateen Telecom Architecture & Engineering Team
Wateen Network Support & Operations Team
Motorola Engineering Team
Cisco Systems Wateen Telecom Account Team and Cisco Systems Advance Services Team
Scope
The scope of the project is to provide an IP/MPLS network to satisfy the requirements laid out within this
document. The document only pertains to a low level design, required configuration & implementation of
an ip/MPLS core network deployed from ground up.

Assumptions and Caveats
It is assumed that the reader is reasonably technically competent in basic IP networking environment. The
reader should also have some basic understanding of Cisco products – both hardware & software.
Related Documents
Advanced Services Statement of work Wateen_X97743_SOW
Wateen CMSN Technical Proposal Release(x) <……..>
Finalized Bill of Quantity <……..>
Wateen Data Center and Security LLD Wateen_X97743_LLD_DCN.doc
Wateen NMS LLD Wateen_X97743_LLD_NMS.doc
Wateen Metro LLD Wateen_X97743_LLD_METRO.doc
Wateen PWLAN LLD Wateen_X97743_LLD_PWLAN.doc
References
7600 Series Router Product Overview

http://www.cisco.com/univercd/cc/td/doc/product/core/cis7600/hardware/cis_76xx/osr_over.htm#wp10681
50
Cisco Small Form-Factor Pluggable Gigabit Interface Converter

http://www.cisco.com/en/US/products/hw/modules/ps5000/products_data_sheet09186a008014cb62.html
Catalyst 6500 Series and Cisco 7600 Series Supervisor Engine 720 Compact Flash Install Note
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/cfgnotes/78_15537.htm
Cisco Catalyst 6500 Supervisor Engine 32 Architecture

http://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper0900aecd803e508c.shtml
Catalyst 6500/Cisco 7600 Series Supervisor Engine 720 Data

http://www.cisco.com/en/US/products/hw/modules/ps4835/products_data_sheet09186a0080159856.html

Cisco 7600 Series Cisco IOS Software Configuration Guide, 12.2SX
http://www.cisco.com/univercd/cc/td/doc/product/core/cis7600/software/122sx/swcg/index.htm
Release Notes for Cisco IOS Release 12.2SX on the Supervisor Engine 720, Supervisor Engine 32, and
Supervisor Engine 2
http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/122sx/ol_4164.htm
Ethernet and Gigabit Ethernet Switching Modules

http://www.cisco.com/univercd/cc/td/doc/product/lan/cat6000/6000hw/mod_gd/02ethern.htm#wp1051105
Cisco 12010, Cisco 12410, and Cisco 12810 Router Installation and Configuration Guide
http://www.cisco.com/en/US/products/hw/routers/ps167/products_installation_and_configuration_guide_b
ook09186a00801fc679.html
Performance Route Processor Installation and Configuration Note

http://www.cisco.com/en/US/partner/products/hw/routers/ps167/products_installation_and_configuration_
guide09186a00800a857a.html
Overview: Cisco 12000 Series Router SPA Interface Processors

http://cisco.com/en/US/products/hw/routers/ps167/products_installation_guide_chapter09186a0080464614
.html#wp1109706
Cisco XR 12000 and 12000 Series SPA Interface Processor-600 – Data Sheet
http://www.cisco.com/en/US/partner/products/hw/routers/ps167/products_data_sheet0900aecd8028085a.ht
ml
Cisco 5-Port, 8-Port, and 10-Port Gigabit Ethernet Shared Port Adapter – Data Sheet
http://www.cisco.com/en/US/partner/products/ps6267/products_data_sheet0900aecd8027cb99.html
Cisco Shared Port Adapters/SPA Interface Processors

http://www.cisco.com/en/US/partner/products/ps6267/prod_module_series_home.html
Cisco 2-Port OC-48c/STM-16c POS/RPR Shared Port Adapter

http://www.cisco.com/en/US/partner/products/ps6267/products_data_sheet0900aecd80350cc5.html

Cisco XR 12000 and 12000 Series Packet over SONET/SDH (POS) Line Cards [Cisco Line Cards] -
Cisco Systems
http://www.cisco.com/en/US/partner/products/hw/modules/ps2710/products_data_sheet0900aecd803fd7b9
.html
Packet-Over-SONET Line Card Installation and Configuration - Cisco Systems

http://www.cisco.com/en/US/partner/products/hw/modules/ps2710/prod_module_install_config_guide091
86a0080205a77.html#wp644336
Cisco Product Documentation

http://www.cisco.com/univercd/cc/td/doc/product/index.htm

Network Overview
Customer Wateen Telecom, a sister company of Warid Telecom has been established with the goal of
becoming a telecommunications provider in Pakistan Wateen Telecom is planning to install a major
network to achieve these objectives.
Warid Telecom (Sister Company) has a nation-wide cellular network launched in May of 2005 in 28 cities
and the major highways of Pakistan to provide high quality cellular services to Pakistan’s urban, suburban
and rural populations.
As part of the network buildout process, Wateen Telecom, would like to build an IP/MPLS core network
which can be used to offer several services to residential and business customers. This is a network being
built from scratch (green field) with last mile access technology in form of WiMAX which will be
provided by Cisco partner Motorola. The WiMAX infrastructure will consist of WiMAX radio network,
relevant CPE devices and Microwave devices (as a backhaul mechanism).
Wateen Telecom will use Cisco 12800 series routers and 7600 series routers to form the core IP/MPLS
network. The network will provide necessary infrastructure for offering services like internet, MPLS vpn
(L3vpn), Ethernet over MPLS (L2vpn). The converged multiservice network (CMSN) as referred by the
customer will also consist of:
• Metro rings. Islamabad, Karachi and Lahore are going to be covered by Ethernet metro rings. There
will be one metro Ethernet ring in Lahore, while two in Karachi and Islamabad. These rings will
provide above mentioned services while using Cisco 10720 devices and RPR (resilient packet ring)
technology.
• Data Center and Security Services: A data center will be built in Lahore which will provide data center
services like hosting services etc. The Data center will incoroporate security, content switching and
loadbalancing services for Wateen Telecom and its customers. Data Center will also host relevant
NMS elements as they pertain to Wateen Telecom infrastructure.
• PWLAN: Wateen Telecom intends to deploy public wireless LAN services in various hot spots, these
services will use IP/MPLS core as a transport mechanism.
• Voice Solution: Wateen Telecom along with Cisco partner Motorola intends to build a packetized
voice infrastructure to cater to business and residential voice requirements. IP/MPLS core provided by
Cisco systems will serve as the transport infrastructure for various voice solutions that Wateen
Telecom intends to deploy.
• Internet Peering Services: Wateen Telecom would like provide internet peering services to its
downstream customers and hence be a transit service provider.
• Content Caching & URL filtering Services: content caching services for an improved user experience
are also targeted at internet edge as well as URL filtering services.

Network Overview
While IP/MPLS and internet connectivity services are covered in this low level design document, please
refer to other low level design documents for other specific technologies and related low level design.
Network Layout:
Overall layout of the network as envisioned for connectivity and relevant transport technologies is given
below.
Figure 1 Network Topology Overview
It is understood that infrastructure used to provide various services consists of 7609 and 7613 chassis using
Supervisor720 and Supervisor 32 as P/PE devices. Core of the network will be formed by four 12816
routers which will be located in four core sites namely Lahore, Karachi, Islamabad and Faisalabad. These
four devices will act as P nodes within in IP/MPLS core and will have no customer facing ports on them.

Network Overview
Of the four core sites, Lahore will also host a Data center as well as initial internet gateway link. The data
center will provide hosting services as well as various customer portals. Some infrastructure related
network management systems will also be located in this Data center including centralized DNS, DHCP
and radius servers. The core sites are referred to as Main PoP by Motorola, the two terms are used
interchangeably in rest of the document. All other sites connecting to core sites (Main PoP) are referred to
as remote sites. Customer refers to these sites as City PoPs. There are few remote sites which connect to a
remote site (City PoP) rather than core site (Main PoP), these sites are referred to satellite sites. Customer
refers these sites to as remote PoP.
Metro rings with Cisco 10720 and RPR technology will be deployed in Lahore, Karachi and Islamabad.
Lahore will be covered by a single metro Ethernet ring whereas Karachi and Islamabad will be covered by
two metro rings. Islamabad site will also provide coverage to its sister city Rawalpindi. Similarly
Sheikhupura will be covered by Lahore where microwave links will be used to backhaul traffic to nearest
Lahore PoP. For details on Metro rings please reference a separate design document.
One dial concentrator AS5400 will be deployed in Lahore, Karachi and Islamabad. All other sites will
deploy a single AS5350XM as dial concentrator. Dial concentrators will be deployed to provide dial up
internet access as well as dial access into MPLS VPN boxes.
Customer will also deploy Public Wireless Lan services using WiFi mesh. These services will be deployed
by the customer in core sites (Lahore, Karachi, Islamabad and Faislabad) at various hotspots.
The underlying transport for this IP/MPLS core is being built on SDH rings. These rings are being
deployed on a country wide basis which will ultimately enable required transports being called out in this
design.
A more detailed view of Lahore PoP, covering ip connectivity is shown below.
Figure 2 Lahore topology Overview

Network Overview
The diagram does not cover variouis services elements deployed in data center as well as at the internet
border router. These details can be referenced in other low level design documents. Specifically low level
design document covering data center and security elements.
Design Considerations
It is desired to build this network to provide ip connectivity to interconnect various sites of the customers.
This network should be able to carry voip traffic (customer intends to offer hosted voip solution) as well as
provide basic data connectivity in form of L3 vpn (MPLS vpn), L2 vpn (Ethernet over MPLS) and internet.
With MPLS vpn services, any to any model is desired for most customers however information on other
services model is also requested by the customer as forward looking option in form of design templates.
Key factors in design include the ability to provide all aforementioned services to various customers, have
a redundant and fault tolerant network with ability to react to various changes (link failure, node failure
etc.) in the network as quickly as possible.

Physical Network Design
Physical Network Topology
Following two diagrams are included in this document for reference purpose only and are included on “as
is” basis to show physical transport and fiber connectivity. The diagrams are provided by the customer and
are meant to serve as a reference for physical topology perspective. IP/MPLS core topology is built on top
of the SDH network which is shown below.
Wateen’s Fiber Route
Peshawar
Islamabad
Bannu
Kharian
Sargodha National Fiber Route &

Bhakkar Lahore
Okara
Metro Fiber ring in 3 Cities
Quetta Multan
MNKA2084
Rahim Yar Khan MSC-02
Shikarpur
Larkana MDKA2103 MNKA2024
PTT-Pak Capital Kar-3

Dadu
metro ring in 3 cities Kar-9FIber
Kar-2Fiber
Hyderabad MSC-01
Karachi
MDKA2041
Figure 3 Fiber Route

DWDM / SDH National

Backbone
Belpat(Reg) is replaced
Quetta by D.M.jamali (ADM)
58
Ahmadi Banda is
Km
Replaced by Kark
Mach
98
Sp
Km
ur 1
01 15
Sibbi
mK
Dera Murad
50
Km
Jamali
Jacobabad Kohat
Bannu Peshawar
Kark 2nd pair 0 3
39
Larkana Shahbaz Mardan ur
Shikarpur Layyah Noshera Sp m
Km
Dadu Khel Abbotabad
Kandkot K
70 Km D.I.Khan 76
K m Kot Addu m 64 Km 60 Km 60 K
Kalari 126 66 K Rojhan
53 km 80 K m 65 K m 16 K
m 96 Taxila
m Km 71 K
Km 95 Km
m
32
95 K m Fazilpur 72 K Islamabad
Client #1 m
Jamshoro m
90
Bhakkar Rawalpindi
Km
Km
D.G. Khan 31
25
Km
82
Km
57
85
129 km Gujar Khan
57
Km 3
Mianwali 2F DWDM STM-64 (1 ?) 2nd pair
Km58 K
Reg
m
Nooriabad 2F DWDM STM-64 (1 ?) (North) Ring 03 MS-SPring

K
7
Qureshi 2nd pair Synchronization
Km
58
Ring 02 MS-SPring 83 Km Equipment

Jhelum
Chowk
m
Standby Mode
Client #4 (Central-II)
Khushab Reg 23 Km
m
22 Km Jhang 47 Km 2nd pair (North)

67 K
Muzaffar Garh Kharian

Synchronization T.T.Singh
Equipment to Extend
35 K Sargodha 91 Km 55 Km
2F DWDM STM-64 (1 ?) The clock 40 Km m 32 K 58 Km Mandi Bahauddin
Synchronization m Gojra
77 K
Equipment Ring 01 MS-SPring Spur Chiniot 2nd pair (Central-I)

Standby Mode Multan 02 33 K
Km
m
m
Client #3 22 Km
29
49 Km
m
108 Km Faisalabad
4K
Thatta NMS Wagha 2F DWDM STM-64 (1 ?)
95
Khanewal
11
Standby
26 04
K
39 Km Okara Ring 04 MS-SPring
Km
m
Lodhran 83 Km
ur
43 Km 39 Pattoki Km
Sp
Km 67 58 Gujrat
Km
62 KmRaiwind Lahore
m
Hyderabad
11
K
24
70 K
m Mailsi Spencer
5
Sahiwal 26 K
m
Km
Km
Chichawatni
Km
43 48 km
20
83 Km
99
12 Bahawalpur K m 2F STM-16 Ring 05 Pakpattan Sialkot
Km
6K m
Nawab Shah m 7K MS-SPring Lahore
10
32
33
109 K m Km Km Gujranwala
m 88 K 35 Thokar
5
101 Km Liaqatpur 43 Km Kasur
0
Vehari
ur
Amirabad
R.Y. Khan
m
Arifwala
Sp
Burewala
17 K
Sukkur Dherki
Client #2
LEGEND To Ganda Singh
NMS
Synchronization
Equipment
Active Mode
Wala
Active
DWDM MADM=06 SDH STM-16 ADM=15
DWDM ADM=38 SDH STM-4 ADM/TML=05

DWDM REG=12 Green for Warid MSC Sites
Figure 4 SDH Network Topology Overview
Network Hardware Components for IP/MPLS Core:

Core as well as provider edge devices in Wateen’s IP/MPLS backbone will consist of Cisco 7609 and
Cisco 7613, 12816 series routers as shown below:
Main (Core) PoPs:
The four core PoP sites will have following equipment for IP/MPLS PE device . These core sites are
Lahore, Faisalabad, Karachi and Islamabad. Please note metro part of the network is not included.
Following describes a single 7613 system to be used as a PE device. Two such devices with identical
configuration will be deployed at each core PoP. This means a total of 8 7613 to be used as PE devices in
four core PoPs
Product (*Quantity) Description
• 7613-2SUP720XL-2PS Cisco 7613 13-slot, Red. Sys, 2 SUP720-3BXL and 2 PS
• MEM-C6K-CPTFL128M Sup720/Sup32 Compact Flash Mem 128MB
• 4000W-DC 2700W/4000W DC Option for CISCO7609 Chassis
+
• WS-X6748-GE-TX*3 48-port 10/100/1000 GE Mod: fabric enabled, RJ-45
• WS-X6724-SFP*1 24-port GigE Mod: fabric-enabled (Req. SFPs)
• 7600-SIP-400*2 Cisco 7600 Series SPA Interface Processor-400
• SPA-2X1GE*2 2-port GE Shared Port Adapter
• SFP-GE-S*4 1000BASE-SX SFP (DOM)

+ There are 2 modules in Islamabad and Faisalabad instead of 3.
Following describes configuration of a 12816. 12816 will be a P device in Wateen Telecom network. One
12816 will be deployed in each core network.
• 12816/1280-DC Cisco 12816 Internet Router, Sixteen-Slot System, 4 DC
• PRP-2 Performance Router Processor 2 (PRP-2)
• PRP-2/R Performance Router Processor 2(PRP-2) Redundant
• MEM-PRP2-1G*2 Cisco 12000 PRP-2 1Gig DRAM Default Option
• MEM-12KRP-FD128M*2 Cisco 12000 Series 128MB PCMCIA ATA Flash Disk
++
• 4OC12X/POS-I-SC-B*3 4 PORT OC-12 POS B
• 12000-SIP-600*2 10 Gbps IP Services Engine (modular)
• SPA-2XOC48POS/RPR*2 2-port OC48/STM16 POS/RPR Shared Port Adapters
• SPA-5X1GE*2 5-port Gigabit Ethernet Shared Port Adapter
• SFP-GE-S*10 1000BASE-SX SFP (DOM)
• SFP-OC48-IR1*4 OC-48c/STM-16c IR optics
++ There are 2 modules in Lahore and Karachi instead of 3.
Each core PoP will also have a single Dial concentrator. Lahore, Karachi and Islamabad will have dial
concentrator with the following configuration:
• AS54XM-16E1-480-D AS5400XM Data; 16E1, 492 DSPs, AC RPS, IP+ IOS
• AS54XM-DC-RPS AS5400XM DC Redundant Power Supply
• MEM-1024M-AS5XM AS5350XM and AS5400XM 1024MB Main SDRAM
• MEM-128CF-AS5XM AS5350XM and AS5400XM 128M Compact Flash
Faisalabad (and all other city PoPs) will have a single dial concentrator which will have the following
configuration:
• AS535XM-4E1-120-D AS5350XM Data; 4E1, 120 DSPs, Single AC, IP+ IOS
• MEM-1024M-AS5XM AS5350XM and AS5400XM 1GB Main SDRAM
All four core PoPs have single Virtual Home Gateway (VHG) PE to allow dial in users to map to their
respective VRF. VHG has following configuration:
• CISCO7206-BB Cisco 7206 Broadband Bundle NPE-G1& 3GigE/FE/E Ports

• PWR-7200-DC+ Cisco 7200 DC (24V-60V) Power Supply Option
• PWR-7200/2-DC+ Cisco 7200 Red. DC (24V-60V) Power Supply Option
• 7206VXR/NPE-G1 7206VXR w/ NPE-G1 inc. 3GigE/FE/E Ports and IP SW
• MEM-NPE-G1-1GB 2x512MB mem modules (1GB total) for NPE-G1 in 7200
• MEM-NPE-G1-FLD128 7200 Compact Flash Disk for NPE-G1, 128 MB Option
• WS-G5484 1000BASE-SX Short Wavelen. GBIC (Multimode only)

There will also be a data center in Lahore, which will consist of two 6513 along with various services
modules, 3750 access switches, IPS 4255 appliance sensors etc.
There are also two 7609 which will act as internet gateway peering to external service providers. These
7609 will have WS-SVC-ADM-1-K9 (Anomaly Detection Module) and WS-SVC-AGM-1-K9 (Anomaly
Guard Module). These routers will also connect with WAE-7326-K9 (Wide Area Application Engine
7326) and SCE2020-4XGBE-MM (Service Control Engine, Multi Mode, 4-port GE).
Data Center equipment and other devices pertaining to different technologies like security, network
management, content switching etc. are mentioned here for completeness sake, these are discussed in more
detail in other low level design documents.
City PoPs:
All city PoP sites (excluding remote city PoPs which do not connect directly to 12816 P device) have
following equipment for MPLS/VPN deployement:
• 7609-2SUP720XL-2PS Cisco 7609 9-slot, Red. System, 2 SUP720-3BXL and 2

PS
• MEM-C6K-CPTFL128M*2 Sup720/Sup32 Compact Flash Mem 128MB
• 4000W-DC*2 2700W/4000W DC Option for CISCO7609 Chassis
• SPA-1XOC12-POS*2 1-port OC12/STM4 POS Shared Port Adapter
• SFP-OC12-IR1*2 OC-12/ STM-4 SFP, Intermediate Reach (15km)
++
++
• SFP-OC3-IR1*4 OC3/STM1 SFP, Single-mode fiber, Intermediate Reach
++ Only sites terminating STM-1 links have these SPAs and SFPs.
City PoPs will also have a single dial concentrator which will have the following configuration:
Satellite (Remote) PoPs:
Four sites (namely Bahawalpur , Okara, R.Y. Khan and Gujrat) which do not connect directly to a 12816 P
device have slightly different configuration. These sites have 7609 chassis but have different supervisor
engine as well as different linecards.
• 7609-S323B-8G-R Cisco 7609 Chassis, 9-slot, Red. SUP32-8GE-3B and PS

• MEM-XCEF720-512M*2 512MB DDR, xCEF720 (67xx interface, DFC3A/DFC3B)
• MEM-C6K-CPTFL128M*2 Cat6500 Sup720/Sup32 Compact Flash Mem 128MB
• MEM-MSFC2-512MB*2 512MB DRAM on the MSFC2 or SUP720 MSFC3
• 4000W-DC 2700W/4000W DC Option for CISCO7609 Chassis


• SFP-OC3-IR1*4 OC3/STM1 SFP, Single-mode fiber, Intermediate Reach
Satellite (Remote) PoPs will also have a single dial concentrator which will have the following
configuration:
Please note above is not comprehensive list of equipment but a summary of equipment pertaining to IP/
MPL core infrastructure. Kindly refer to latest version of BoQ for complete details of all the equipment for
this project.

Physical and Functional Description of 12800 Series

Chassis:
Figure 5 12816 Router

The line cards are distributed between two cages of 12816. This is due to power distribution. There will be
only two quad OC-12 linecards (4OC12X/POS-I-SC-B) in Lahore and Karachi (instead of three as shown
in the figure. The figure is accurate for Islamabad and Faislabad sites.

Figure 6 12816 Router Chassis/Slots
In the DC-input power configuration:
• Modules A1 and B1 provide redundant power for system load zone 1 (the upper blower module and the
upper card cage).
• Modules A2 and B2 provide redundant power for system load zone 2 (the switch fabric card cage, the lower
card cage, and the lower blower module).

Figure 7 DC Power Shelf
Caution A router configured for source DC operation must be operated with 4 DC-input PEMs installed at
all times for electromagnetic compatibility (EMC).
Chassis Card Cages

There are three integral card cages in the chassis: the upper card cage, the lower card cage, and the switch
fabric card cage.
Upper Card Cage

The upper card cage has eight user-configurable slots that support a combination of line cards, an alarm
card, and an RP.
• Alarm—The far left slot is a dedicated slot for an alarm card.
• Slots 0 through 6—Can be populated with any line cards supported by the router.
• Slot 7—The far right slot is reserved for the RP.
Lower Card Cage

The lower card cage also has eight user-configurable slots that support additional line cards, an alarm card,
and an optional, redundant RP.
Note The lower card cage is an inverted, or head-down, copy of the upper card cage, which means that
cards are installed in an inverted or head-down orientation. The orientation of the slots is opposite that of
the upper card cage.
• Slot 8—The far left slot is reserved for an optional redundant RP.
Note This slot may be used for a line card if you are not using an redundant RP.

• Slots 9 through 15—Can be populated with any line cards supported by the router.
• Alarm—The far right slot is a dedicated slot for an alarm card.
Switch Fabric Overview

The switch fabric provides synchronized gigabit-speed connections between line cards and the RP. The
switch fabric card cage is located behind the air filter door and consists of 2 clock and scheduler cards
(CSCs) and 3 switch fabric cards (SFCs). One CSC and the 3 SFCs are the active switch fabric; the second
CSC provides redundancy for the other 4 cards.
Note 10-Gbps and 40-Gbps switch fabrics do not operate in 1/4-bandwidth mode as they did in some
earlier models of the Cisco 12000 series routers. You must have at least one CSC and three SFCs for the
system to function. You can add an additional CSC for redundancy.
The combination of CSCs and SFCs make up the 2.5-Gbps, 10-Gbps, or 40-Gbps per-slot switch fabric.
Routers are identified by the switch fabrics they use:
• Cisco 12016: 2.5-Gbps switch fabric
• Cisco 12416: 10-Gbps switch fabric
• Cisco 12816: 40-Gbps switch fabric
Each SFC or CSC provides a 2.5-Gbps, 10-Gbps, or 40-Gbps full-duplex connection to each line card in
the system. For example, in a Cisco 12416 router with 16 line cards, each with 2 x 10 Gbps capacity (full
duplex), the system switching bandwidth is 16x 20 Gbps = 320 Gbps.
Note The Cisco 12000 series router supports online insertion and removal (OIR), allowing you to remove
and replace a card while the router remains powered on.
Switch Fabric Card Functionality

The core of the router is a crossbar switch fabric that provides synchronized connections between the line
cards and the RP. The switch fabric consists of 2 clock scheduler cards (CSCs) and 3 switch fabric cards
(SFCs) installed in the switch fabric card cage. One CSC and the three SFCs are the active switch fabric;
the second CSC provides redundancy for the other 4 cards.
The router also ships with a blank switch fabric card installed in the far left (non-working) slot of the
switch fabric card cage. The blank filler panel balances the air flow through the switch fabric card cage
which helps maintain proper air flow through the chassis.
Caution Do not remove the blank filler panel unless instructed to do so by a Cisco support representative.

Clock Scheduler Cards

Clock scheduler cards provide the following functionality:
• Scheduler—Handles all scheduling requests from the line cards for access to the switch fabric.
• System clock—Supplies the synchronizing signal to all SFCs, line cards, and the RP. The system clock
synchronizes data transfers between line cards or between line cards and the RP through the switch
fabric.
• Switch fabric—Carries the user traffic between line cards or between the RP and a line card. The
switch fabric on the CSC is identical to the switch fabric on the SFC.
The second CSC provides redundancy for the data path, scheduler, and reference clock. Traffic between
the line cards and the switch fabric is monitored constantly. If the system detects a loss of synchronization
(LOS), it automatically activates the data paths on the redundant CSC so data flows across the redundant
paths. The switch to the redundant CSC occurs within microseconds, with little or no loss of data.
Switch Fabric Cards

The switch fabric cards augment the traffic capacity of the router. SFCs contain switch fabric circuitry that
can only carry user traffic between line cards or between the RP and the line cards. SFCs receive all
scheduling information and the system clock signal from the CSCs.
Switch Fabric Card Cage

The router ships from the factory with 2 CSCs and 3 SFCs installed in five of the eight slots in the switch
fabric card cage.
• The 2 CSCs are installed in slot 0 (CSC0) or slot 1 (CSC1)
• The 3 SFCs are installed in slot 2 (SFC0), slot 3 (SFC1), and slot 4 (SFC2).
• Three non-working slots with no backplane connectors. These non-working slots are not labeled, but
there is a blank filler panel installed in the far left slot to help maintain proper air flow through the
chassis.
Performance Route Processor-2 Overview

The performance route processor (PRP) uses a Motorola PowerPC 7450 CPU that runs at an external bus
clock speed of 133 MHz and has an internal clock speed of 667 MHz.
Figure 8 PRP-2, Front Panel View

PRP-2 Components
The PRP-2 contains the following components:
• PowerPC processor—Motorola PowerPC 7450 central processing unit (CPU). The CPU runs at an
external bus clock speed of 133 MHz and an internal clock speed of 667 MHz.
• SRAM—2 megabytes (MB) of static random-access memory (SRAM) for secondary CPU cache
memory functions. SRAM is not user configurable or field replaceable.
• NVRAM—2 MB of nonvolatile RAM (NVRAM). NVRAM is not user configurable or field
replaceable.
• Memory—Additional memory components include onboard Flash memory and up to two Flash disks.
• Sensors—Air-temperature sensors for environmental monitoring.
The PRP-2 contains the following additional components:
• SDRAM—Up to 4 GB of Cisco-approved synchronous dynamic random-access memory (SDRAM)
on two dual in-line memory modules (DIMMs). 1 GB of SDRAM is the default shipping
configuration. SDRAM is field replaceable only when using Cisco-approved DIMMs.
Note Software releases prior to 12.0(30)S do not recognize more than 2 GB of SDRAM and will only use
the first 2 GB of the installed memory. This does not affect the functioning of the PRP-2, but commands
such as show version will indicate that only 2 GB of SDRAM are installed.
• Hard disk drive—40-GB hard disk drive can be optionally installed on the PRP-2 board.
• CF—1-GB compact flash disk can be optionally installed on the PRP-2 board.

Figure 9 PRP-2 (Horizontal Orientation)
1 Compact Flash 8 Console port

disk (optional)
2 Flash SIMM 9 Gigabit Ethernet port

(Socket number
P3)
3 Ejector lever 10 Handle
4 Flash disk slots 11 Display LEDs

(covered)
5 Ethernet ports 12 SDRAM DIMM: Bank

1 - Socket number U15
6 BITS ports1 13 SDRAM DIMM: Bank

2 - Socket number U18
7 Auxiliary port 14 Hard disk drive

(optional)

PRP PCMCIA Card Slots and Status LEDs

Two PCMCIA card slots (slot 0 and slot 1) provide the PRP with additional flash memory capacity. All
combinations of different flash devices are supported by the PRP. You can use ATA flash disks, Type 1 or
Type 2 linear flash memory cards, or a combination of the two.
Note The PRP only supports +5.2 VDC flash memory devices. It does not support +3.3 VDC PCMCIA
devices.
Status LEDs (Slot-0/Slot-1) indicate when the flash memory card in that slot is accessed, each slot has an
eject button (located behind the cover) to remove a flash card from the slot.
PRP Reset Switch

Access to the (soft) reset switch is through a small opening in the PRP front panel. To press the switch,
insert a paper clip or similar small pointed object into the opening.
Caution The reset switch is not a mechanism for resetting the PRP and reloading the Cisco IOS image. It is
intended for software development use only. To prevent system problems or loss of data, use the reset switch only
on the advice of Cisco service personnel.
Pressing the reset switch causes a nonmaskable interrupt (NMI) and places the PRP in ROM monitor
mode. When the PRP enters ROM monitor mode, its behavior depends on the setting of the PRP software
configuration register. For example, if the boot field of the software configuration register is set to:
• 0x0—The PRP remains at the ROM monitor prompt (rommon>) and waits for a user command to boot
the system manually.
• 0x1—The system automatically boots the first Cisco IOS image found in flash memory on the PRP.
PRP Alphanumeric Message Displays

The alphanumeric message displays are organized in two rows of four LED characters each ( Figure 10 ).
Figure 10 Alphanumeric Message Displays—Partial Front Panel

The alphanumeric message displays show router status messages during the boot process, and after the
boot process is complete.
• During the boot process, the message displays are controlled directly by the MBus module.
• After the boot process, the message displays are controlled by Cisco IOS software (through the
MBus).
The alphanumeric message displays also provide information about different levels of system operation,
including the status of the GRP, router error messages, and user-defined status and error messages
Note A list of all system and error messages appears in the Cisco IOS System Error Messages publication.
For further details, please refer to:

http://www.cisco.com/univercd/cc/td/doc/product/core/cis12000/cis12x16/icg/hfricgpo.htm
Physical and Functional Description of 7600 Series

Chassis:
FAN
STATUS
9 8 7 6 5 4 3 2 1
0
0
A/L
C/A
A/L
C/A
K
K
LIN
LIN
K
K
LIN
LIN
SPA-1XOC12-POS
SPA-1XOC12-POS
K
K
LIN
LIN
STA
STA
T
TUS
TUS
SE
SE
RE
RE
0
A/L
C/A
A/L
C/A
SPA-1XOC12-POS
SPA-1XOC12-POS
STA
STA
TUS
TUS
TEM TIV PW MT
TIV PW MT
MG
MG
R
R
E
E
AC
AC
EM
ST
SYS
SY
S
S
TU
TU
STA
STA
INPUT FAN OUTPUT INPUT FAN OUTPUT

OK OK FAIL OK OK FAIL
L L
AL AL
ST ST
IN UN IN UN
R R
Power Supply 1 Power Supply 2
OSR-7609

Above diagram represents a chassis where both STM-4 and STM-1 SPAs are populated in a each SIP-400.
In some instances, there is only single STM-4 SPA in each SIP-400. Please note that Slots 5 and 6 can
support primary and redundant Supervisor Engine 720s. A Supervisor Engine 720 with two GBIC uplink
ports and one 10/100/1000 Tx port. Only two of the three ports can be active at any one time.
City PoP:
Please see below for slot allocation for 7609:
Slot Module Comment
2 7600-SIP-400 Bay 0: SPA-1XOC12-

POS
Bay 1: SPA-2XOC3-
POS*
3 7600-SIP-400 Bay 0: SPA-1XOC12-
POS
Bay 1: SPA-2XOC3-
POS*
4
5 SUP-720-3BXL Sup Engine
6 SUP-720-3BXL Sup Engine
8 WS-X6748-GE-TX 10/100/1000 coppper
9 WS-X6748-GE-TX 10/100/1000 coppper
* only used in Multan, Sahilwal, Sukkhur and Gujranwala to terminate satellite-remote PoPs.
Satellite (Remote) PoP:

Sites slated to deploy supervisor engine-32 are Okara, Bahawalpur, RYKhan and Gurjat. Slot allocation
for 7609 with Sup32 is given below:
Slot Module Comment
2 7600-SIP-400 Bay0: SPA-2XOC3-POS
3 7600-SIP-400 Bay0: SPA-2XOC3-POS
5 SUP32-8GE-3B Supervisor Engine

6 SUP32-8GE-3B Supervisor Engine
8 WS-X6748-GE-TX 10/100/1000 coppper
9 WS-X6748-GE-TX 10/100/1000 coppper
Internet Gateway (Border) Router:

For two internet gateways routers (7609 series), slot allocation will be as given below:
Slot Module Comment
1 WS-X6724-SFP SFPs Required
2 7600-SIP-200 Bay0: SPA-2XT3/E3
3 WS-SVC-ADM-1-K9 Cisco Anomaly

Detection Module
4 WS-SVC-AGM-1-K9 Cisco Anomaly Guard
Module
5 SUP720-3BXL Supervisor Engine
6 SUP720-3BXL Supervisor Engine
Following is a representation of 7613 chassis populated with various cards and redundant supervisor
engines.

WS-X6724-SFP
24 PORT GIGABIT ETHERNET SFP
STATUS
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
WS-X6724-SFP
24 PORT GIGABIT ETHERNET SFP
STATUS
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
WS-X6748-GE-TX 1 11 13 23 25 35 37 47
2 12 14 24 26 36 38 48
STATUS 4 8 PORT 10/100/1000 FAB RIC ENABLED

1 2 3 4 5 6 7 8 9 GE MOD RJ45
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
WS-X6748-GE-TX 1 11 13 23 25 35 37 47
2 12 14 24 26 36 38 48

1 2 3 4 5 6 7 8 9 GE MOD RJ45
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
WS-X6748-GE-TX 1 11 13 23 25 35 37 47
2 12 14 24 26 36 38 48

1 2 3 4 5 6 7 8 9 GE MOD RJ45
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
ALL A LL
ST ST
IN IN
N
N
U
U
R
For 7613 Series router, a Supervisor Engine 720 can fit in slot 7 and an optional redundant Supervisor
Engine 720 in slot 8. Each supervisor engine provides switching, local and remote management. The
Supervisor Enigne 720 has two GBIC uplink ports and one 10/100/1000 Tx port. Only two of the three
ports can be active at any one time.
Main (Core) PoP:

Please see slot allocation for 7613 below:
Slot Module Comment
1 WS-X6724-SFP SFPs Required
3 7600-SIP-400 Bay0: 1x SPA-2X1GE
4 7600-SIP-400 Bay0: 1x SPA-2X1GE
7 SUP-720-3BXL Supervisor Engine
8 SUP-720-3BXL Supervisor Engine
10 WS-X6748-GE-TX 10/100/1000 coppper
11 WS-X6748-GE-TX 10/100/1000 coppper

12 WS-X6748-GE-TX* 10/100/1000 coppper
13
*slot number 12 is empty in Faislabad and Islamabad
SUP720 Switch Fabric Slot Allocation
Figure 13 - Slot Allocation

Figure 14 Sup 720 Slot Allocation
Above figure shows fabric channel allocation for different 7600 Series Router. Please note that Sup32 is a
bus based supervisor only. In 13 slots chassis, following modules MUST be inserted in slot 9 to 13:
WS-X6748-GE-TX, WS-X6748.

(Slot and port numbering start at 1 (top/left) , 7609 with vertical line card ,NEB chassis, it starts from right)
Redundancy
The Cisco 7609 and 7613 series routers have these redundancy features:
• Ability to house two hot-swappable supervisor engines
• Ability to house two fully redundant, AC-input or DC-input, load-sharing power supplies
• Redundant hot-swappable fan assemblies containing multiple fans
Fan Assembly
The system fan assembly, located in the chassis, provides cooling air for the supervisor engine and
the switching modules
Following figure shows Cisco 7609 router show the direction of airflow into and out of the router. Sensors
on the supervisor engine monitor the internal air temperatures. If the air temperature exceeds a preset
threshold, the environmental monitor displays warning messages.
If an individual fan within the assembly fails, the FAN STATUS LED turns red.

Figure 15 Cisco 7609 Router Internal Airflow
Supervisor Engine 720

Supervisor Engine 720 Front Panel Features :

Figure 16 Supervisor Engine 720 Front Panel Features
Reset Button
The Reset button allows you to restart the system.
Supervisor Engine 32
Supervisor Engine 32 (WS-SUP32-GE-3B) Front Panel Features
Figure 17 Supervisor Engine 32 Front Panel

Flash Memory on a Supervisor Engine 32
The Supervisor Engine 32 has:
• disk0:—One external CompactFlash Type II slot (supports CompactFlash Type II Flash PC cards)
• sup-bootdisk:—256 MB internal CompactFlash Flash memory (from ROMMON, it is bootdisk:)
Supervisor Engine 32 Ports
The console port for the Supervisor Engine 32 port is an EIA/TIA-232 (RS-232) port. The
Supervisor Engine 32 also has two Universal Serial Bus (USB) 2.0 ports that are not currently enabled.
WS-SUP32-GE-3B ports 1 through 8 have small form-factor pluggable (SFP) connectors and port 9 is a
10/100/1000 Mbps RJ-45 port. Port 9 can be used as regular port for traffic or for management purposes.

SIP Slot Allocation:

Following shows slot allocation for SIP-200 or SIP-400 cards.
Figure 18 SIP Slot Numbering
1 SIP subslot 0 4 SIP subslot 3
2 SIP subslot 1 5 Chassis slots 1-9 (numbered from right to left)
3 SIP subslot 2

For further details please see:

http://www.cisco.com/en/US/products/hw/routers/ps368/module_installation_and_configuration_guides_c
hapter09186a0080440138.html

Logical Network Design
Naming and Addressing Specifications
BGP AS Number
Wateen Telecom has applied to APNIC for publicly announceable AS number. In the interim a
private AS number from the range of 64512 to 65535 will be chosen. Customer has advised to use
65501 AS number until assignment of public AS number.
Naming Conventions
Customer has advised to use the following naming convention.
LHE-7600-PE1-GE0/0
Where LHE represents airport code of Lahore, 7600 is the platform name and PE01 is device role.
It has been advised to use airport code for cities where available to abbreviate a city’s name.
Devices included in this design and their hostnames are listed below.
Number Site Device Role Location Device Name

Abb.
1 Lahore 12816 P LHE Lhe-12816-P1
7613 PE Lhe-7613-PE1
7613 PE Lhe-7613-PE2
7609 Internet Lhe-7609-Intgw1
Gatewy
7609 Internet Lhe-7609-Intgw2

Gatewy
6513 DC Lhe-6513-DC1
Switch
6513 DC Lhe-6513-DC2
Switch
2 Sialkot 7609 PE SKT Skt-7609-PE1
3 Gujranwala 7609 P/PE GUJ Guj-7609-PE1
4 Gujrat 7604 PE GJT Gjt-7609-PE1
5 Karachi 12816 P KHI Khi-12816-P

7613 PE Khi-7613-PE1
7613 PE Khi-7613-PE2
6 Quetta 7609 PE QTA Qta-7609-PE1
7 Hyderabad 7609 PE HYD Hyd-7609-PE1
8 Sukkhur 7609 P/PE SKR Skr-7609-PE1
9 R.Y.Khan 7604 PE RYK Ryk-7609-PE1
10 Faisalabad 12816 P FSD Fsd-12816-P1

7613 PE Fsd-7613-PE1
7613 PE Fsd-7613-PE2
11 Sargodha 7609 PE SGD Sgd-7609-PE1
12 Multan 7609 P/PE MUL Mul-7609-PE1
13 Bahawalpur 7609 PE BWP Bwp-7609-PE1
14 Sahiwal 7609 P/PE SWL Swl-7609-PE1
15 Okara 7604 PE OKR Okr-7609-PE1
16 Islamabad 12816 P ISB Isb-12816-P1

7613 PE Isb-7613-PE1
7613 PE Isb-7613-PE2
17 Peshawar 7609 PE PES Pes-7609-PE1
18 D.I.Khan 7609 PE DIK Dik-7609-PE1
19 Jehlum 7609 PE JEH Jeh-7609-PE1
20 Abottabad 7609 PE ABT Abt-7609-PE1
IP Addressing Scheme
For infrastructure, publicly assigned ip addressing will be used. Customer has advised that APNIC
has allocated following public ip address space: 58.27.128.0 - 58.27.255.255 (/17). Please see
subsequent section for IP address allocation of core mpls infrastructure.

Firmware and Software Requirements
Cisco 7600 Series run 12.2S series software image specifically built for 7600/Catalyst 6500 series
platform. The current software image has various releases which are indicated by 12.2(18)SX. A
maintenance build of a particular release is noted as 12.2(18)SX. The latest release for 12.2(18)SX
at the time of writing this document is 12.2(18)SXF and is required for 7609 with Sup32 chassis.
Currently 12.2SXF has a maintenance build of 12.2(18)SXF5 and is recommended release for all
7609/7613 series chassis.
There are some features which are only available in a recently released train called 12.2(33)SRA
internally code named Cascades. While the release is available, it is suggested that either customer
should test this release extensively before deployment or wait for some field exposure of this
train.
On 12816, latest version on 12.0S train is 12.0(32)SY at the time of writing this document,
however it is suggested that customer deploy with 12.0(32)S4 as there are no features which are
required on customer network and only available in 12.0(32)SY.
Chassis Version File (Flash/RAM Mb) Comment
12816 12.0(32)S4 c12kprp-k4p-mz.120-32.S4.bin P Router image
7609 s72033-advipservicesk9_wan- PE Router image

12.2(18)SXF6 mz.122-18.SXF6.bin
7609 12.2(18)SXF6 s3223-advipservicesk9_wan- PE Router image with
mz.122-18.SXF6.bin Superivsor 32
7613 12.2(18)SXF6 s72033-advipservicesk9_wan- PE Router image
mz.122-18.SXF6.bin
For 12.2(18)SXF5 with Sup720:

Software Feature Set: ADVANCED IP SERVICES SSH
Minimum Memory needed: 512
Minimum Flash Size 128
Date Released: 22-SEP-2006
Platform: 7600-SUP720/MSFC3
The image can be downloaded here:
http://www.cisco.com/cgi-bin/Software/Iosplanner/Planner-
tool/iosresult.cgi?get_crypto=&data_from=&hardware_name=CAT6000-
SUP720%2FMSFC3&software_name=ADVANCED%20IP%20SERVICES%20SSH&release_na
me=12.2.18-SXF5&majorRel=12.2&state=:HW:RL:SW&type=Early%20Deployment
Release notes for 12.2(18)SXF5 release are available here:

http://www.cisco.com/en/US/products/hw/switches/ps708/prod_release_note09186a00801c8339.h
tml
For 12.2(18)SXF6 with Sup32:

Software Feature Set: ADVANCED IP SERVICES SSH
Date Released: 22-SEP-2006
Platform: 7600-SUP32/MSFC2A
http://cco/cgi-bin/Software/Iosplanner/Planner-
tool/iosresult.cgi?get_crypto=&data_from=&hardware_name=7600-
SUP32%2FMSFC2A&software_name=ADVANCED%20IP%20SERVICES%20SSH&release_n
ame=12.2.18-SXF5&majorRel=12.2&state=:RL:HW:SW&type=Early%20Deployment
Release notes for 12.2(18)SXF5 release are available here:

http://www.cisco.com/en/US/products/hw/switches/ps708/prod_release_note09186a00801c8339.h
tml
For 12.0(32)S4
Software Feature Set: SERVICE PROVIDER/SECURED SHELL 3DES
Date Released: 18-AUG-2006
Platform: 12000-PRP
http://cco/cgi-bin/Software/Iosplanner/Planner-
tool/iosresult.cgi?get_crypto=&data_from=&hardware_name=12000-
PRP&software_name=SERVICE%20PROVIDER%2FSECURED%20SHELL%203DES&release
_name=12.0.32S3&majorRel=12.0&state=:HW:RL:SW&type=Early%20Deployment
Release notes are available here:

http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/relnote/xprn120s/120scavs.htm
#wp2065691

IGP Routing:
After several discussions, customer has chosen OSPF as routing protocol of choice as IGP for the
core network. The decision is driven primarily by folks responsible for supporting the network
feel more comfortable with OSPF. Further details about OSPF are given under MPLS Core
Design section.
MP-BGP4 (Multi-protocol BGP)
Route Reflectors:
The two PE routers i.e. 7613 in Lahore and Karachi will be configured as redundant route
reflectors for vpnv4 traffic in absence of dedicated route reflectors as an interim measure:
The second pair of P routers i.e.12816 in Lahore and Faisalabad will be configured as redundant
route reflectors for ipv4 traffic in absence of dedicated route reflectors as an interim measure.
LDP (Label Distribution Protocol):

LDP Protocol to be used, further discussion regarding this topic can be found under MPLS Core
Design section.
Resilience
Resilience is achieved by using redundant route processors and redundant power supplies at the
device level. At the network level this is achieved by providing backup paths in case of a link
failure in the core network consisting of 12816 devices. On the edge, in core PoPs redundant
Gigabit Ethernet links are used to attach to 12816 devices. All remote PoP 7609 routers
connecting to 12816 have APS protection to ADM (add drop Mux). Same protection is used at
other side of the link, where 12816 link has APS protection to ADM. Similarly remote PoP 7609
connecting via STM-1 also have APS protection to ADM. While APS provides some protection
loca to the PoP, it would be much preferred to have redundant uplinks to core P (12816) device. It
is understood that due to underlying SDH ring and related optical transport constraint, it is not
possible to have redundant uplinks at this stage.
IGP will be used to reroute traffic in case of a link failure where a redundant link is available.
It is understood that in current Wateen Telecom network design, single P routers (12816s) are
single point of failure, e.g. in case if a 12816 router in Lahore goes down, it will isolate
connectivity to Lahore and all the infrastructure residing in Lahore. Same is true for all other core
(Main) PoPs. It is understood that this is a design choice Wateen Telecom made in early design
cycles, it is understood that possible impact of P routers going down is well understood by the
customer.

Scalability
Scalability of solution is restricted by various factors in the network. These include forwarding
capabilities of each device with various features turned on as required in our solution. It also has to do
with maximum number of interfaces and vlans that can be terminated on each element, number of
pseudowires that can be terminated, number of vpn routes that PE and Route Reflectors can carry as
well as maximum number of vpns that can be defined on a given platform, maximum number of
interfaces available on a platform, maximum memory supported etc.
This is something to be considered once network has achieved substantial growth. It is advised that as a
precautionary measure routes accepted in a vpn be limited to a specific number to help scale mpls vpn
service and avoid any potential misconfiguration issues. This is shown in the example below:
ip vrf cust1
rd 65501:10
route-target export 65501:10
route-target import 65501:10
maximum routes 500 warning-only #if number of routes exceeed 500 in this
vpn, issue a warning
Should EBGP be used as the PE-CE protocol, it can also be accomplished in

BGP config:
router bgp 65501

<standard iBGP config here>
address-family ipv4 vrf VRF1
neighbor 192.168.6.6 maximum-prefix 500 ! limit number of prefixes to 500
only
It is understood that Wateen Telecom will use maximum 500 prefixes as initial number for limiting
number of prefixes received in a customer vpn.
There may also be some platform specific information that must be taken into consideration for overall
scalability as well. This information will vary from platform to platform. Following section describes
some factors to be taken into account for 7600 series routers.
7600 Restrictions
The Cisco 7609/7613 Series routers based on supervisor 720 are able to support 1024 MPLS L3 VPN
Virtual Routing and Forwarding (VRF) instances. The first 511 VRFs are supported with optimal
forwarding performance, the rest may experience some performance degredation due to recirculation in
the Cisco Supervisor Engine 720-3BXL.
Layer 3 LAN ports, WAN interfaces and subinterfaces, and some software features use internal
VLANs in the extended range. It is not possible to use an extended range VLAN that has been
allocated for internal use. Internal VLANs and user-configured VLANs share the 1006 to 4094 VLAN
spaces. A "first come, first served" policy is used in allocating these spaces.
By default internal vlan allocation is done in an ascending manner i.e.internal VLANs are allocated
from 1006 to 4094. A knob has been added to set the vlan internal allocation policy. The command
allows one to configure the allocation direction of the internal VLAN in descending order i.e internal
VLAN allocation is done from 4094 to 1006. The global config command is “vlan internal allocation

policy descending”. Internal vlan allocation can be verified by the command “show vlan internal
usage”.
For more details, please see: http://www.cisco.com/en/US/products/hw/routers/ps368/products_configuration_guide_chapter09186a0080160ed0.html
Wateen Telecom will use global command: “vlan internal allocation policy descending” to ensure
that internal vlan allocation is done in descending order. A system reload is required for change to take
effect. Vlan allocation as planned by the customer will leave a vlan range of 3500-4094 set aside for
internal usage initially.
These factors should be considered when designing the scaling factors for the maximum number of
VPNs supported in the system.
In certain cases, the PFC3BXL or PFC3B provides the capability to recirculate the packets.
Recirculation can be used to perform additional lookups in the ACL or QoS TCAMs, the Netflow
table, or the FIB TCAM table. Recirculation is necessary in these situations:
• To push more than three labels on imposition
• To pop more than two labels on disposition
• To pop an explicit null top label
• When the VPN Routing and Forwarding (VRF) number is more than 511
• For IP ACL on the egress interface (for nonaggregate (per-prefix) labels only)
Packet recirculation occurs only on a particular packet flow; other packet flows are not affected.The
rewrite of the packet occurs on the modules; the packets are then forwarded back to the PFC3BXL or
PFC3B for additional processing.
Please see below for further details:
http://www.cisco.com/univercd/cc/td/doc/product/core/cis7600/cfgnotes/optical/122sx/mpls.htm#wp1327282
It must also be noted that it is recommend to configure no more than 2,000 Layer 3 VLAN interfaces
on 7600 platform.
Please se below for further details: http://www.cisco.com/univercd/cc/td/doc/product/core/cis7600/software/122sx/swcg/layer3.htm
While it is understood that Wateen Telecom anticipates to have a mix of L3 and L2 vpn as well as
internet (ipv4 only) services on 7600 platform, above limits should be kept in mind when considering
overall capacity of the platform.

MPLS Core Design
MPLS Architecture Overview

The figure below provides a logical view of the various components required to build the Wateen
Telecom’s IP/MPLS core network and how the various components are interconnected. The
intention of the diagram is to reflect part of the network covered in this low level design
document, along with products and relevant naming convention.
MPLS VPN Core Logical Layout
Figure 19 MPLS VPN Core Logical

MPLS Core Design
At the edge of the MPLS network there will be access switches (or RPR ring(s) populated with
10720 devices). Access switches will connect CE and perform a layer 2 function. In our case 7600
series routers will act as P/PE routers, providing customer facing trunk ports or access ports.
The PEs can classify traffic for different customers and apply the appropriate labels in a process
known as label imposition. The labels applied by the PE indicate the destination PE for each
packet, and also identify the VPN to which the device is connected or subscribes to (for example,
VPN1).
Two labels are applied to the IP packet, the first level, or top label defines the destination PE,
whilst the second level label defines the interface on the destination PE that the packet will be
forwarded to.
As the MPLS network is frame based (carries IP packets not ATM cells), the first level label and
second level labels are imposed as part of the frame that carries the IP packet. This is done by
inserting or stacking the labels between the layer 2 (link layer) and layer 3 (network layer)
headers. The following figure illustrates this.
Destination Customer
PE Destination
IP Packet
L2 Header Label 1 Label 2 L3 Header Data
Frame
Figure 20 Label Imposition
In terms of details of the MPLS header, it is located between the Layer 3 (IP) header and Layer 2
header. The EXP bits and the TTL field of the MPLS header can be copied from the IP header.
The S bit indicates whether there is more than one MPLS label in this frame.
Layer 2 Header + MPLS Header + Layer 3 Header
LABEL EXP S TTL

20 bits 3 1 8
S = Bottom of stack
TTL = Time to live
EXP=Experimental Bits (CoS) Direct COS support in MPLS

Figure 21 Label Imposition
The labelled frames are usually then forwarded to either a P device (12816 in the core) or a PE
(7613/7609) device depending on the routing metric and the shortest path as determined via the IP
routing table. In our case traffic will almost always pass through one of the core routers to reach
destination PE (depending on the traffic flow).

MPLS Core Design
Once the labelled frames have left the PE routers, they are switched through the P network to the
destination PE. It is important to understand that P routers have no knowledge of layer 3 traffic. It
only looks at the first level label.
As the labelled frames are forwarded to the appropriate egress PE, each P router in the path will
examine the incoming label and replace it with an outgoing label to direct it to the appropriate
egress interface. This process is referred to as label swapping.
The last router in the path to the PE,, known as the penultimate hop (the hop before the last), will
remove the level 1 label. This is known as penultimate hop popping. Therefore when the frame
arrives at the destination PE, only the level 2 (or bottom of the stack) label remains.
The level 2 label is then examined to determine the egress interface on the PE, which will be the
link to the customer site. The label is removed from the frame and the original IP packet is then
forwarded to the destination customer network.
The underlying protocols to allow this process to operate: OSPF; Label Distribution Protocol; and
MP-BGP; are explained in further sections.
Network Design
The network design consists of a core consisting of point to point STM-16/STM-4 links between
12816 devices. These devices will be “P” devices and will not have any customer facing
links/ports. As described earlier these core sites are Lahore, Karachi, Islamabad and Faisalabad.
All four sites are cross connected to each other. Connectivity between Lahore, Karachi and
Islamabad consists of STM-16, while Faisalabad is connected via STM-4 link to all other devices.
It has been advised that connectivity to Faislabad had to be reduced to STM-4 (from STM-16, for
a symmetrical core) due to facilities reasons.
Gigabit Ethernet link exist between 7613 PEs in core sites and 12816 routers. Two 7613 PEs are
deployed in each core PoP. All remote PoPs connect with 12816 routers via STM-4 links. These
links are APS protected to respective ADMs. Some remote (satellite) sites connect via STM-1 link
to remote PoPs, these too are protected with ADM links. Only core sites have redundant PEs and
redundant links (albeit to same device) to 12816 P device.
All customer facing ports are Ethernet ports, these ports will connect to Microwave/WiMax
equipment which will provide last mile access to the customer via Ethernet. All customer facing
Ethernet ports on PEs are on LAN based cards (67xx or 65xx series cards).
There are also metro rings (layer 3 based RPR rings with 10720 as PE devices) which will be
deployed in three core sites, Lahore, Karachi and Islamabad. The interconnect between metro
rings and core city PEs (7613) is via point to point GE links. These GE links are cross connected
(i.e. between two 10720 and two 7613s). Further details can be found in separate low level design
which covers metro ring design.
Lahore site will also host a data center, the data center will have two 6513 chassis populated with
various services modules to provide content switching, security services and among others will
also host NMS systems for IP/MPLS infrastructure core.
Internet connectivity will be available in Lahore via two 7609 series routers. These routers will
connect via point to point GE links to data center switches as well as two 7613 core PE devices in
Lahore. Internet connectivity will be achieved by peering with different service providers and
learning full internet routing table via BGP. These routes will be carried in the core via IBGP and
offered to potential clients who are interested in multi homed internet connectivity to multiple
service providers. This service will establish Wateen Telecom as transit service provider in
addition to other services being offered to the customers.

MPLS Core Design
Sites Connectivity
All sites are interconnected via either point to point Gigabit Ethernet links (core PEs) or STM-16
(between three core 12816s), STM-4 (remote PoPs and 12816 in Faislabad) and STM-1 (satellite
remote PoPs) links. .
MPLS Core IP Addressing

It is recommended that a single OSPF area be used for IP routing for IP/MPLS core*; no
summarization of IP addresses anywhere is required at this time.
** 58.27.191.0/24 & 58.27.192.0/22used for point to point addressing from public ip address
scheme.
Table 3 Infrastructure Address Blocks

MPLS Core Design
Address Assigned to (Loopback)

58.27.190.1 /32 Lhe-12816-P1
58.27.190.31/32 Khi-12816-P1
58.27.190.57/32 Isb-12816-P1
58.27.190.83/32 Fsd-12816-P1
58.27.190.3/32 Lhe-7613-PE1
58.27.190.5/32 Lhe-7613-PE2
58.27.190.13/32 Lhe-7609-Intgw1
58.27.190.15/32 Lhe-7609-Intgw2
58.27.190.17/32 Lhe-6513-DC1
58.27.190.19/32 Lhe-6513-DC2
58.27.190.9/32 Skt-7609-PE1
58.27.190.7/32 Guj-7609-PE1
58.27.190.11/32 Gjt-7609-PE1
58.27.190.33/32 Khi-7613-PE1
58.27.190.35/32 Khi-7613-PE2
58.27.190.39/32 Qta-7609-PE1
58.27.190.37/32 Hyd-7609-PE1
58.27.190.41/32 Skr-7609-PE1
58.27.190.43/32 RYK-7609-PE1
58.27.190.85/32 Fsd-7613-PE1
58.27.190.87/32 Fsd-7613-PE2
58.27.190.91/32 Sgd-7609-PE1
58.27.190.89/32 Mul-7609-PE1
58.27.190.95/32 Bwp-7609-PE1
58.27.190.93/32 Swl-7609-PE1
58.27.190.97/32 Okr-7609-PE1
58.27.190.59/32 Isb-7613-PE1
58.27.190.61/32 Isb-7613-PE2
58.27.190.63/32 Pes-7609-PE1
58.27.190.65/32 Dik-7609-PE1
58.27.190.67/32 Jeh-7609-PE1
58.27.190.69/32 Abt-7609-PE1

MPLS Core Design
Link Addressing is shown below:
Point to Point Address Assigned to Link

Block (/30)
58.27.191.0 Lahore P -- Karachi P
58.27.191.1 -- 58.27.191.2
58.27.191.4 LahoreP -- Islamabad P
58.27.191.5 -- 58.27.191.6
58.27.191.8 Lahore P -- Faislabad P
58.27.191.9 -- 58.27.191.10
58.27.191.12 Lahore P -- Lahore PE1: Link1
58.27.191.13 -- 58.27.191.14
58.27.191.17 -- 58.27.191.18
58.27.191.21 -- 58.27.191.22
58.27.191.25 -- 58.27.191.26
58.27.191.28 Lahore PE1 – Lahore PE2
58.27.191.29 -- 58.27.191.30
58.27.191.32 Lahore PE1 -- LHE-10720-1
58.27.191.33 -- 58.27.191.34
58.27.191.36 Lahore PE1 -- LHE10720-2
58.27.191.37 -- 58.27.191.38
58.27.191.40 Lahore PE2 -- LHE-10720-1
58.27.191.41 -- 58.27.191.42
58.27.191.44 Lahore PE2 -- LHE10720-2
58.27.191.45 -- 58.27.191.46
58.27.191.48 Lahore PE1 -- Int-GW1
58.27.191.49 -- 58.27.191.50
58.27.191.52 Lahore PE1 -- Int-GW2
58.27.191.53 -- 58.27.191.54
58.27.191.56 Lahore PE2 -- Int-GW1
58.27.191.57 -- 58.27.191.58

MPLS Core Design
58.27.191.60 Lahore PE2 -- Int-GW2

58.27.191.61 -- 58.27.191.62
58.27.191.64 Lahore PE1 -- LHE-6513-DC1
58.27.191.65 -- 58.27.191.66
58.27.191.68 Lahore PE1 -- LHE-6513-DC2
58.27.191.69 -- 58.27.191.70
58.27.191.72 Lahore PE2 -- LHE-6513-DC1
58.27.191.73 -- 58.27.191.74
58.27.191.76 Lahore PE2 -- LHE-6513-DC2
58.27.191.77 -- 58.27.191.78
58.27.191.80 Lahore PE1 -- LHE-5400-AS1
58.27.191.81 -- 58.27.191.82
58.27.191.84 Lahore PE2 -- LHE-5400-AS1
58.27.191.85 -- 58.27.191.86
58.27.191.88 Lahore P1 – Gujranwala
58.27.191.89 -- 58.27.191.90
58.27.191.92 Lahore P1 – Sialkot
58.27.191.93 -- 58.27.191.94
58.27.191.96 Gujranwala PE1 – Gujrat PE1
58.27.191.97 -- 58.27.191.98
58.27.191.100 Karachi P1 – Faisalabad P1

58.27.191.101 -- 58.27.191.102
58.27.191.104 Karachi P1 – Islamabad P1
58.27.191.105 -- 58.27.191.106
58.27.191.108 Karachi P1 – KHI PE1: Link1
58.27.191.109 -- 58.27.191.110
58.27.191.113 -- 58.27.191.114
58.27.191.117 -- 58.27.191.118
58.27.191.121 -- 58.27.191.122
58.27.191.124 Karachi PE1 – Karachi PE2
58.27.191.125 -- 58.27.191.126
58.27.191.128 Karachi PE1 -- KHI-10720-1
58.27.191.129 -- 58.27.191.130
58.27.191.132 Karachi PE1 -- KHI-10720-2
58.27.191.133 -- 58.27.191.134

MPLS Core Design
58.27.191.136 Karachi PE2 -- KHI-10720-1

58.27.191.137 -- 58.27.191.138
58.27.191.140 Karachi PE2 -- KHI-10720-2
58.27.191.141 -- 58.27.191.142
58.27.191.144 Karachi PE1 -- KHI-10720-1-R2
58.27.191.145 -- 58.27.191.146
58.27.191.148 Karachi PE1 -- KHI-10720-2-R2
58.27.191.149 -- 58.27.191.150
58.27.191.152 Karachi PE2 -- KHI-10720-1-R2
58.27.191.153 -- 58.27.191.154
58.27.191.156 Karachi PE2 -- KHI-10720-2-R2
58.27.191.157 -- 58.27.191.158
58.27.191.160 Karachi P1 – Hyderabad PE1
58.27.191.161 -- 58.27.191.162
58.27.191.164 Karachi P1 – Quetta PE1
58.27.191.165 -- 58.27.191.166
58.27.191.168 Karachi P1 - - Sukkhur PE1
58.27.191.169 -- 58.27.191.170
58.27.191.172 Sukkhur PE1 - - RYKhan PE1
58.27.191.173 -- 58.27.191.174
58.27.191.176 Islamabad P1 - - Faislabad P1

58.27.191.177 -- 58.27.191.178
58.27.191.180 Islamabad P1 – ISL PE1: Link1
58.27.191.181 -- 58.27.191.182
58.27.191.185 -- 58.27.191.186
58.27.191.189 -- 58.27.191.190
58.27.191.193 -- 58.27.191.194
58.27.191.196 Islamabad PE1 – Islamabad PE2
58.27.191.197 -- 58.27.191.198
58.27.191.200 Islamabad PE1 – ISB-10720—R1-1
58.27.191.201 -- 58.27.191.202
58.27.191.204 Islamabad PE1 -- ISB-10720—R1-2
58.27.191.205 -- 58.27.191.206
58.27.191.208 Islamabad PE2 – ISB-10720-R1-1

MPLS Core Design
58.27.191.209 -- 58.27.191.210
58.27.191.212 Islamabad PE2 -- ISB-10720-R1-2

58.27.191.213 -- 58.27.191.214
58.27.191.217 -- 58.27.191.218
58.27.191.221 -- 58.27.191.222
58.27.191.225 -- 58.27.191.226
58.27.191.229 -- 58.27.191.230
58.27.191.232 Islamabad P1 – Peshawar PE1
58.27.191.233 -- 58.27.191.234
58.27.191.236 Islamabad P1 – DIKhan PE1
58.27.191.237 -- 58.27.191.238
58.27.191.240 Islamabad P1 – Jehlum PE1
58.27.191.241 -- 58.27.191.242
58.27.191.244 Islamabad P1 – Abottabad PE1
58.27.191.245 -- 58.27.191.246
58.27.191.248 Faislabad P1 – FSL PE1 Link1

58.27.191.249 -- 58.27.191.250
58.27.191.252 Faislabad P1 – FSL PE1: Link2
58.27.191.253 -- 58.27.191.254
58.27.192.1 -- 58.27.192.2
58.27.192.5 -- 58.27.192.6
58.27.192.8 Faislabad PE1 – Faisalabad PE2
58.27.192.9 -- 58.27.192.10
58.27.192.12 Faislabad P1 – Multan PE1
58.27.192.13 -- 58.27.192.14
58.27.192.16 Faislabad P1 – Sargodha PE1
58.27.192.17 -- 58.27.192.18
58.27.192.20 Faislabad P1 – Sahiwal PE
58.27.192.21 -- 58.27.192.22
58.27.192.24 Multan PE1 – Bahawalpur PE1
58.27.192.25 -- 58.27.192.26
58.27.192.28 Sahiwal PE1 – Okara PE1
58.27.192.29 -- 58.27.192.30

MPLS Core Design
*Data Center and Metro Ethernet addressing excluded
Autonomous System Number

An autonomous system (AS) number is required for MP-BGP peering as well as propagating ipv4
routes to rest of IP/MPLS core. By convention this value is also used in Route Distinguishers to
create IP-VPNv4 addresses and route-targets, although it is not necessary that they be the same as
the AS number.
As mentioned earlier, as publicly announceable AS number is currently being applied for, per the
customer following AS number will be used in the interim.
AS Number Notes
65501 For use in MP-iBGP peerings, Route Distinguishers, Route Targets
Backbone Protocols
This section deals with the protocols that the Label Switch Routers require to provide an MPLS
backbone and associated IP services, for example, Virtual Private Networks (VPNs). The
following routing protocols will operate in the MPLS backbone:
• OSPF will function as the Interior Gateway routing Protocol (IGP) providing IP connectivity
amongst all Label Switch Routers (P & PE).
• LDP (Label Distribution Protocol) is responsible for distributing label information for IP
destinations within the IGP routing table only. That is, P and PE addresses not part of a VPN,
such as link and loopback addresses.
• MP-BGP4 (Multi-protocol Border Gateway Protocol V4) is used to distribute VPN routing
information amongst the Edge LSRs (PE). It also distributes VPN-related labels to the PEs. It
will also carry internet ipv4 prefixes.
OSPF (Open Shortest Path first) IGP

Generally speaking every router in a large network runs an interior gateway protocol (IGP). The
connectivity amongst all edge devices and Core devices is maintained through an Interior
Gateway Protocol. In other words, the IGP provides next hop reachability through the core
network.
As in any other large network, the role of IGP in the Wateen Telecom’s network is to exchange
internal routes (router loopback and link addresses) and compute paths inside the network, which
primarily enables the BGP peers to reach all next hop addresses defined in their BGP table.
Specifically, the IGP generally does not carry any customer prefixes (defined in a vpn). Instead,
BGP, which usually needs to be run on every provider edge router, is used to exchange customer
(vpn) routing information.
To provide MPLS VPN service, Wateen Telecom will need to implement MP-BGP between the
PE routers to exchange the VPN prefixes. An Interior Gateway Routing Protocol (IGP) must be
operating amongst the P and PEs before the label distribution (LDP) and VPN routing distribution
can execute.

MPLS Core Design
OSPF is one of the most highly scalable and widely deployed link state protocol. OSPF has been
chosen by the customer as preferred IGP protocol primarily due to familiarity with the protocol in
comparison to ISIS. It will be used to build the internal routing table, which MPLS then uses to
build the Label Forwarding Information Base, LFIB table.
LDP assigns and distributes labels only for the IGP learnt routes. Therefore IGP must be running
for MPLS to be successfully enabled in the network.
Global Routing Table

The global routing table (GRT) exists in all P and PE routers. It generally contains all routes that
do not belong to a VPN. This includes all loopback addresses and numbered link addresses. The
GRT contains reachability information to allow routing in the P/PE network.
Customer routes are kept in separate routing tables, known as VPN Routing and Forwarding
tables (VRFs). Generally, routes in the global routing table are not reachable by routes in VRFs,
unless explicitly configured using the “global” routing command. This is discussed further in the
section entitled “Static Routing”.
OSPF area design: Single Area vs. Multi-area

OSPF creates hierarchy through areas to provide a scalable solution. One motivation behind a
multi-area design is less computational work for the routers that are internal to the area during a
topology change in any remote OSPF area of the network. Most of the SPF algorithm cost is
proportional to the number of links in an area. In general, the fewer the number of links in an area,
the fewer resources needed to run SPF computations.
If summarization is used between areas, instability in one area can be completely hidden from
other areas. So from the computational and stability perspective, it is evident that by using areas,
OSPF can scale to larger numbers of nodes in the network.
Several things must be considered when using multiple areas:

• Route summarization must never be done on the loopback address range of the P and PE
routers as this will cause the label switching path to terminate where the summarization
occurs. This will result in loss of connectivity.
• There are many restrictions running MPLS traffic engineering between OSPF areas since
some link information is lost when sending route advertisements (LSAs) between areas
Because the current roll out of Wateen Telecom’s IP/MPLS core network is not very large in
terms of the number of devices & links; a single area design is recommended. This allows for
simple management and operations support and allows for area definition depending on growth of
the network.

MPLS Core Design
Area 1
Area 2 Area 3
Area 0
Core
Area 4 Area 5
Figure 22 OSPF area topology with multiple areas
With an OSPF design as shown above it is crucial to remember that any future network links may
only be within in area, or interconnecting an area to the backbone Area 0.
If a connection from Area 2 to Area 1 was required then either the area would need to be merged
into a single area; or a virtual link would be required. Virtual links are to be avoided if at all
possible.
For Wateen’s MPLS/IP network, a single area design will allow for maximum flexibility and will
scale for current set of requirements for IP/MPLS Core.*
OSPF Area Numbering

The single area for IP/MPLS will be numbered area zero (0), which is the backbone area in OSPF
routing protocol. *
[*Only accurate for IP/MPLS core, see Data Center Connectivity section later in the document and Metro Ethernet low level design for additional
details
]
OSPF Network Addressing
Table 4 MPLS Core Addressing Structure
Address Mask Comment

58.27.190.x /32 P/PE Router Loopback Addresses
58.27.191-192.x /30 P/PE Point-to-Point Link Addresses
OSPF Area Summarization

No summarization of addresses is required, or is possible using a single area design and is not
needed.

MPLS Core Design
*Exception Data Center and Metro Ethernet devices
As with other routing protocols, enabling OSPF requires that you create an OSPF routing instance
by creating a OSPF process, specify the range of IP addresses to be associated with the routing
process, and assign area IDs to be associated with that range of IP addresses. By default, OSPF
process ID is chosen as the OSPF domain ID. Since, any process number can be chosen. e.g. 1
router ospf 1
network <address> <wildcard-mask> area 0
passive-Interface loopback 0
log-adjacency-changes
Figure 23 OSPF Router process
There are multiple ways to define OSPF network statements to activate OSPF on an interface or
interfaces. By specifying a subnet to encompass multiple interfaces, keep in mind that other
interfaces will build neighbor adjacencies if the interface is active with a valid IP address
configured. This can be avoided by specifying individual host or subnets specific to an interface.
“Passive-Interface” disables the ability to create OSPF adjacencies over specified interfaces.
Clearly the loopback interface is not required to establish adjacencies.
The “log-adjacency-changes” command provides a high level view of adjacency state specifically
neighbors going up or down events. Include the detail keyword to cause syslog messages of all
state changes. This can assist in troubleshooting network issues in the future.
OSPF Router ID
Every router running OSPF within a network must have a unique router ID. The router ID is used
by the OSPF link-state database (LSDB) as a method of tracking each router within the AS and
the links associated with it. To assign the router ID, by default OSPF uses the highest IP address
of one of the router's active interfaces. If a loopback interface (loopback0) is configured with an
IP address, the Cisco IOS software will use this IP address as its router ID, even if other interfaces
have larger IP addresses. Since loopback interfaces are not affected by circuit failure, greater
stability in the routing table is achieved. Should there be other loopback addresses defined on the
router it is recommended to specifically set the router-id to loopback0 to remove any confusion as
to the ip address to used as the router id.
Cisco IOS provides an option of manually setting the OSPF router ID with the “ip ospf router-id”
router configuration command. We recommend setting the router-id manually as shown in Figure
24.
router ospf 1
router-id <address>
Figure 24 OSPF router-id
OSPF Reference Bandwidth

The default value for OSPF metrics is based on normalized bandwidth. By default OSPF will
calculate the OSPF metric for an interface according to the bandwidth of the interface. For
example, a 64K link will get a metric of 1562, while a T1 link will have a metric of 64.

MPLS Core Design
The OSPF metric is calculated as ref-bw divided by bandwidth, with ref-bw equal to 108 by
default, and bandwidth determined by the bandwidth command. The calculation gives FDDI a
metric of 1.
Granted that Wateen Telecom has multiple interface with high bandwidth (i.e. STM-16, one
gigabit Ethernet etc), one must use a larger number to differentiate the cost on those links vs other
links (if and when such links should be available, e.g. ten gigabit Ethernet or instances where Fast
Ethernet may be deemed sufficient).
If the reference bandwidth is left at its default, then the cost for a Fast Ethernet, Gigabit Ethernet
and ten gigabit Ethernet links would be 1, providing no differentiation to OSPF for the various
link speeds.
To overcome this, the reference bandwidth should be changed on all routers to the fastest link that
might exist in the network. Setting reference bandwidth to 10 Gigabit Ethernet should be
sufficient for foreseeable future, therefore the reference bandwidth should be changed using the
following command:
router ospf 1
auto-cost reference-bandwidth 10000
Figure 25 OSPF Reference Bandwidth
This will set all links with a cost relative to the 10 Gigabit Ethernet as shown in the following
table.
The value set by the ip ospf cost command overrides the cost resulting from the ospf auto-cost
command.
Table 5 OSPF link costs
Line Speed Mbps Designator OSPF Cost

100 100BaseX 100
155 STM-1 65
622 STM-4 16
1000 Gigabit Ethernet 10
2488 STM-16 4
10000 Ten Gigabit Ethernet 1
The reference bandwidth must be set on all routers participating in OSPF to provide consistent
cost allocation across the network. Also it should be assured that correct bandwidth of interface is
reflected for the physical interface.
OSPF Network Type:

Gigabit Ethernet interface are broadcast interfaces by default. For Wateen Telecom IP/MPLS
core network, it is recommended to configure Gigabit Ethernet interface as point-to-point network
type between PE and P devices (where the link is point to point) to prevent OSPF from
performing a designated router selection, Gigabit Ethernet connections running OSPF should be

MPLS Core Design
set as point-to-point network type as long as there are only two routers on a single link as shown
in Figure 26.
interface <interface type-number>

ip ospf network point-to-point
Figure 26 Setting the OSPF Network type
OSPF Convergence
Resiliency and redundancy to circuit failure is provided by the convergence capabilities of OSPF
at layer 3. There are two components to OSPF routing convergence; detection of topology
changes and recalculation of routes.
Detection of topology changes are supported in two ways by OSPF. The first, and quickest, is a
failure or change of status on the physical interface, such as Loss of Carrier. The second is a
timeout of the OSPF hello timer. An OSPF neighbor is deemed to have failed if the time to wait
for a hello packet exceeds the dead timer, which defaults to four times the value of the hello timer.
On a Serial, Fast Ethernet or Gigabit Ethernet interface, the default hello timer is set to 10
seconds, therefore the dead timer is 40 seconds
Recalculation of routes is done by each router after a failure has been detected. A link-state
advertisement (LSA) is sent to all routers in the OSPF area to signal a change in topology. This
causes all routers to recalculate all of their routes using the Djikstra (SPF) algorithm. This is a
CPU intensive task, and a large network, with unreliable links, could cause a CPU overload.
A summary of OSPF protocol timers is given below. They are further discussed in detail in the
following section. It is suggested that in the initial deployment of the Wateen Telecom network,
initial timers should be left to default values as shown in table below.
These values can then be further adjusted and behavior of the network monitored depending on
results required. Section below discusses various options and suggested values that can be used
for fast convergence. These values can be tuned further based on the convergence requirements
and overall stability of the network. Some values have been suggested in subsequent sections.
Table 6 OSPF Protocol Timers

Timer Default Value
ip ospf dead-interval 4 x hello
interval
(40 sec)
ip ospf hello-interval 10 sec
ip ospf retransmit-interval 5 sec
ip ospf transmit-delay 1 sec
Timers spf spf-delay 5 sec
Timers spf spf-holdtime 10 sec

MPLS Core Design
LSA Generation Interval 0.5 sec
Each of these timers will affect the performance of OSPF and can be tuned to effect both
convergence time and network resource utilization. While it is possible to use rather very small
values for hello and dead intervals (subseconds), these values need to be evaluated against overall
network stability and overall resource utilization (please see below for futher details) of the
network.
It is also worth noting that tuning hello and dead interval for the edge sites has no substantial
value as they are connected via POS links and are singly attached to the core. In case of quickly
reacting to the changes in network may mean quick reaction to bad news but no alternate path for
traffic.
I. Link Failure Detection

A. Carrier Delay
IOS uses carrier-delay as a dampening mechanism against link flaps. Default value for carrier
delay is 2 seconds, It can be set to zero (for fast convergence). However if there is SDH protection
on the circuit, it may be desireable to set it to non zero value, otherwise it can be set to zero msec.
This is offcourse with an understanding that underlying transport infrastructure (Layer 1) is stable
and an occasional failure should be reacted to as quickly as possible. The interface command
required is “carrier-delay msec 0”.
For Wateen Telecom it is understood that core links between all 12816 P nodes (STM-16 and
STM-4) links carrier delay should be set to zero. Similarly carrier-delay should be set to zero for
all Gigabit Ethernet interfaces interconnecting 7613 PE devices to 12816 P nodes as well as 7613
PE devices to 10720 metro devices.
It is suggested that this value should be left unchanged for STM-4 and STM-1 links to remote PE
devices. This is primarily due to single point of connectivity for these devices. A quick reaction to
failed link will not lead to an alternate path. However for consistency reasons carrier delay may be
set to zero on all interfaces.
B. Auto Negotation
To deal with with unidirectional failures it is recommended that all Gigabit Ethernet interfaces be
configured with auto-negotiation, for all supported Gigabit Ethernet interfaces on all devices
within IP/MPLS core. The router detecting the LoS will signal this fact to the other router thereby
quickly bringing the link to the down state.
C.Hello and Dead Interval

It is desirable to use POS links whenever possible in order to avoid relying on RouterDeadInterval
timer, as when using POS links, failure detection is very quick. On multi-access link Hello and
Dead Interval need to be tuned. To improve the convergence time in the network, one option can
be to decrease the value of the hello timer. The timer should not be set too low as this may cause
phantom failures, hence unnecessary topology recalculations. Remember that these timers are
used to detect failures that are not at the physical level. For example, carrier still exists but there is
some sort of failure in the intermediate network.
A keepalive timer is also associated with the interface that will detect failure at a level lower than
OSPF. The default for this timer is 10 seconds.

MPLS Core Design
While it is possible to set hello timer value in msec in recent releases of IOS it is recommended
that the minimum value of Hello should be set to no less than 1 sec and therefore the minimum
value of RouterDeadInterval should be set to 4 sec. The following commands can be configured
on the link
!
interface GigabitEthernet1/2
ip address <ip address> <subnet mask>
ip ospf hello-interval 1
ip ospf dead-interval 4
Figure 27 OSPF hello-dead interval tuning
POS interfaces provide for timely (sub 10ms) and reliable link failure management. Therefore the
tuning of OSPF hello timers is not necessary and should remain at their default settings.
For all the core links and core PEs it is suggested to rely on alternate mechanism like BFD
(discussed later) for dead peer detection as a worst case. BFD is very fast and allows for a failure
detection in 150 msec. Fore all remote PE devices (city and satellite PoP), it is suggested that BFD
should not be used (as all these devices have only single uplink). However should it be deemed
necessary, it is suggested to use above mentioned OSPF hello and dead interval values.
II. Interface dampening

When there is a instability due to a link flap, back-off algorithm for LSA generation and SPF
computation will exponentially grow until reaching the maximum value therefore a flapping link
will make exponential timers to their maximum and delay the convergence time.
Interface dampening is a feature that dampens a flapping link until it becomes stable. It improves
convergence times and stability throughout the network by isolating failures so that disturbances
are not propagated which reduces the utilization of system processing resources by other devices
in the network and improves overall network stability. Following interface command enable
interface dampening:
!
dampening
Figure 28 Interface Dampening
For Wateen Telecom, it is recommended that interface dampening be enabled on all interfaces
which have redundant uplink paths. This will include all core links on 12816 P devices as well
as Gigabit Ethernet links to core PEs, and interconnect of core 7613 PEs with 10720 metro
devices.
Interface dampening should not be enabled for remote PE links where only single connection
exists to the core network.
III.Incremental SPF (iSPF)

When there is a change to type 1 or type 2 LSA, the whole SPF tree is computed from scratch
even if in some cases the SPF tree has not been changed. The SPF run time is the time to compute

MPLS Core Design
the SPF tree and is a function of the number of nodes and number of links in the overall network
topology.
Incremental SPF optimises the SPF re-computation by only considering part of the Shortest Path
Tree (SPT) that has changed rather than a Full SPF computation from scratch.
iSPF decreases the SPF run time and leads to a faster convergence. The more nodes and links are
there in the network, the better results of the iSPF computation in terms of noticeable
improvement.
!
router ospf 1
ispf
Figure 29 ispf configuration
For Wateen Telecom it is recommended to enable incremental SPF on all routers participating in
OSPF backbone area.
IV.LSA Generation & Propagation Time

Any change in adjacency with a neighbor, redistribution and and change in network reachability
are triggers for LSA generation. This trigger is managed by exponential backoff timers to assure
rapid response to an isolated event, yet at the same time maintaing stability in the case of
substantial churn.
Prior to the OSPF LSA Throttling feature, LSA generation was rate-limited for 5 seconds. That
meant that changes in an LSA could not be propagated in milliseconds, so the OSPF network
could not achieve millisecond convergence.
Timers associated with LSA Throttling are following:
Start: Default value of 0 msec i.e. LSA is generated immediately
Hold: Default value of 5 sec. Hold time between consecutive LSA regeneration
For interoperability reasons, one may need to set Hold value equal to MinLSArrival (timers lsa
arrival) as defined in RFC2328 (default is 1 sec). LSA instances received at higher frequencies
than MinLSArrival are discarded. If interoperability is not an issue, one can consider to set Hold
and Arrival values to tens of msec to ensure that if all the failures have not been advertised by the
first LSA, they will be captured by the second.
Maximum: default value is 5000msec. The subsequent LSAs generated for the same LSA are
rate-limited until the maximum interval is reached.
LSA generation can be tuned in order to react quickly to a change and delay LSA generation in
the event of network instability. Following values are most aggressive in terms of
recommendation.
Router ospf 1
timers throttle lsa all 1 200 5000
Figure 30 OSPF LSA Generation Tuning
We will generate a LSA immediately and exponentially delay by 200msec until reaching 5 sec
maximum value. Please note that MinLSArrival is by default 1 sec therefore it needs to be
changed on all routers otherwise any LSA generation before 1 sec will be ignored. The “timers
lsa arrival” command controls the minimum interval for accepting the same LSA. If an instance

MPLS Core Design
of the same LSA arrives sooner than the interval that is set, the LSA is dropped. It is
recommended that the arrival interval be less than or equal to the hold-time interval of the timers
throttle lsa all command.
router ospf <pid>

timers lsa arrival 200
Figure 31 OSPF LSA Arrival Tuning
For Wateen Telecom it is recommended to leave the default values initially. Throttle and LSA
arrival timers can be tuned to above recommended vaules. For a network with lot of churn can
cause excessive CPU burden on the router, hence these values should be tuned keeping overall
stability of the network in mind. It is suggested that these values be kept consistent across the
entire OSPF backbone area and hence should be applied to all routers participating in backbone
OSPF area.
V. Alternative Path Time (SPT Computation)

When a LSA is received, an SPF is scheduled. The actual SPF execution takes place after initial
SPF time. Once the SPF is run and other LSA(s) are received, the SPF execution is delayed for a
hold time SPF.
Once a topology change has been detected, recalculation of the routes will not occur until the spf
timer has expired. The default value of this timer is 5 seconds (initial SPF time). An spf hold time
is also used to delay consecutive SPF calculations (to give the router some breathing space). The
default for this value is 10 seconds (hold time).
For fast convergence, it is possible to tune the SPF timers to increase the convergence time. Hold
time SPF is the time which controls the SPF in case of instability in the network (link flap)
therefore initial SPF time can be set to a low value.
SPT throttling allows SPF execution to be schedule in milliseconds intervals. At the same time it
can allow SPF execution to be delayed during network churn. SPF calculations occur at the
interval set by the “timers throttle spf” command.
SPF Start interval : Initial SPF schedule delay. By default initial SPF schedule delay is 5
seconds (5000 msec).
SPF Hold Time: (i.e. wait interval): The amount of time to wait until the next SPF calculation
occurs. Each wait interval after the “wait interval” calculation is twice as long as the previous one
until maximum wait time is reached. Default value for spf hold time is set to 10 seconds (10000
msec).
SPF maximum wait : maximum wait time: once the maximum wait time is reached, the wait
interval remains the same until the topology stablizes and no event is received in that interval.
Default value for maximum hold time is set to 10 seconds (10000 msec).
SPF computation can be tuned in order to react fast to a topology change and delay SPF
computation in the event of network instability. Following values are suggested:
router ospf 1
timers throttle spf 50 200 5000
Figure 32 OSPF SPF Timers Tuning

MPLS Core Design
The first computation will be delayed by 50msec in order to flood all changes to the neighbours
before running the SPF, next SPF will be delayed exponentially by 200 msec until reaching the
maximum value of 5 sec.
Please note in earlier versions of IOS use “timers spf” command to tune SPF values (allowed
two arguments, initial value and hold value). This command has been replaced by “timers
throttle spf” command.
For Wateen Telecom it is recommended to leave the default values initially. SPF throttling can be
tuned to above vaules which are most aggressive values suggested. For a network with lot of
churn can cause excessive CPU burden on the router, hence these values should be tuned keeping
overall stability of the network in mind. It is suggested that these values be kept consistent across
the entire OSPF backbone area and hence should be applied to all routers participating in the
backbone OSPF area.
VI. RIB Update:

When a link goes down, the less efficient Routing Information Base (RIB) process is
automatically triggered to delete all prefixes from the RIB that have the next hop on the interface
which went down. When the process works through a large routing table, it can consume many
CPU cycles and increase convergence time. A more efficient way of accomplishing this is
possible with the following knob in IOS using global configuration mode:
ip routing protocol purge interface
Figure 33 Fast RIB update
Please note that any changes made should be configured across the entire backbone area not only
limited to 7600 and 12816 devices. i.e. any device (e.g. 10720) participating in backbone area
should have consistent configuration as noted above to achieve deterministic behaviour.
Bi-directional Forwarding detection (BFD)
Bi-Directional Forwarding (BFD) is a lightweight protocol that is designed to provide quick

detection of failures in the path between 2 adjacent forwarding routers.
BFD is a single mechanism that can be used for failure detection over any media (POS, GE etc)
and at any protocols layer, it has a range of configuration detection times and overhead.
The fast detection of a failure provides immediate reaction in the event of a failed link or node,
whilst link down detection is normally used to determine this level of failure, BFD has the
additional advantage of being able to detect forwarding plane failures that may not be seen by the
control plane.
BFD packets use a UDP Datagram with a destination port of 3784 and a source port between
49252 and 65535, because this is designed as a rapid response protocol the output features are
bypassed for locally generated BFD control packets.

MPLS Core Design
The main applications for BFD are to detect that the forwarding plane is operational
BFD uses two timers that are negotiated between the adjacent nodes, a transmit timer and a
receive timer the system that reports the slower times is used to determine the transmission rate.
Figure 34 BFD Timers
As can be seen above, the router on the left has TX/RX rates of 100/50ms and the remote site has
TX/RX Rates of 40/60ms.
The left router will pick the slowest out of his transmit rate and the other side receive rate, and the
same vice versa.
The right node here is transmitting at 50ms (even though the desired transmit rate was set to
40ms), this is because the desired receive rate on the adjacent node was set for 50ms.
BFD has two main modes of operation: Asynchronous and Echo mode:
BFD Asynchronous Mode
In Asynchronous mode BFD sends control packets in each direction informing the adjacent node
that it is still alive.
Figure 35 BFD Async Mode

MPLS Core Design
These control packets are sent on a periodic basis, if a number of packets (default 3) are not
received by the adjacent node then it assumes there is a link / forwarding plane issues and the
session is declared down. This is the only mode available today
BFD Echo Mode
In Echo mode BFD sends packets to it’s own IP address, the packets are sent to the adjacent node
and looped back.
Figure 36 BFD Echo Mode
As before control packets are sent on a periodic basis and a number of lost echo packets will cause
the session to be declared down, echo mode also has the benefit of testing the remote forwarding
plane as well.
In both modes BFD will immediately notify routing process that interface is down and thus
‘shortcut’ the legacy, ‘hello/holdtime’ timers.
Asynch mode is the only avalaible mode.

BFD is not supported on port channel
With IOS 12.2(18) SXF on 7600 series, BFD support notification only for ISIS, EIGRP and
OSPF http://www.cisco.com/en/US/partner/tech/tk365/technologies_white_paper0900aecd80244005.shtml

MPLS Core Design
interface GigabitEthernet1/1/0
ip address <ip address> <subnetmask>
ip ospf bfd disable !selectively disable bfd on an interface
bfd interval 50 min_rx 50 multiplier 3
!
router ospf 1
bfd all-interfaces
!
!on PE devices where there are many interfaces and bfd is not required on
most it is prefferable to enable BFD on a per interface basis
!
no ip directed-broadcast
ip ospf bfd
negotiation auto
bfd interval 50 min_rx 50 multiplier 3
Figure 37 BFD Configuration
Following table shows the interfaces for each device where BFD is recommended to be enabled in
Wateen Telecom’s network.
Table 7 BFD Interface Application
Device Name Interface Description BFD Configured

Lhe-12816-P1 All Core links (STM-16 Y
and STM-4)
GigE Links to Lhe-7613- Y
PE1 & Lhe-7613-PE2
STM-4 links to remote N

PEs
Lhe-7613-PE1 GigE Links to Lhe- Y

12816-P, 10720 and Lhe-
7613-PE2
Lhe-7613-PE1 Access Ports N
Lhe-7613-PE2 GigE Links to Lhe- Y

12816-P, 10720 and Lhe-
7613-PE1
Lhe-7613-PE2 Access Ports N
Lhe-7609- GigE links to Lhe-7613- Y

Intgw1 PE1 and Lhe-7613-PE2
Lhe-7609- E3 link for upstream N

MPLS Core Design
Intgw1 internet connectivity
Lhe-7609- GigE links to Lhe-7613- Y

Intgw1 PE1 and PE2
Lhe-7609- E3 link for upstream N

Intgw1 internet connectivity
Lhe-6513-DC1 All interfaces N

Lhe-6513-DC2 All interfaces N
Slk-7609-PE1 All interfaces N
Guj-7609-PE1 All interface N
Guj-7609-PE1 All interfaces N
Khi-12816-P All Core links (STM-16 Y

and STM-4)
GigE Links to Khi-7613- Y
PE1 & Khi-7613-PE2

PEs
Khi-7613-PE1 GigE Links to Khi-12816- Y

P, 10720 & Khi-7613-
PE2
Khi-7613-PE1 Access Ports N
Khi-7613-PE2 GigE Links to Khi-12816- Y
P, 10720 & Khi-7613-
PE1
Khi-7613-PE2 Access Ports N
Qta-7609-PE1 All interfaces N
Hyd-7609-PE1 All interfaces N

Skr-7609-PE1 All interfaces N
Ryk-7609-PE1 All interface N
Fsd-12816-P1 All Core links ( STM-4) Y

GigE Links to Fsd-7613- Y
PE1 & Fsd-7613-PE2

PEs
Fsd-7613-PE1 GigE Links to Fsd 12816- Y

P1 and Fsd-7613-PE2
Fsd-7613-PE1 Access Ports N

Fsd-7613-PE2 GigE Links to Fsd 12816- Y
P1 and Fsd-7613-PE2
Fsd-7613-PE2 Access Ports N
Sgd-7609-PE1 All interfaces N

MPLS Core Design
Mul-7609-PE1 All interfaces N
Bwp-7609-PE1 All interfaces N

Swl-7609-PE1 All interface N
Okr-7609-PE1 All interfaces N
Isb-12816-P1 All Core links ( STM-16 Y

and STM-4)
GigE Links to Isb-7613- Y
PE1 & Isb-7613-PE2
PEs
Isb-7613-PE1 GigE Links to Isb-12816- Y
P1, 10720 and Isb-7613-
PE2
Isb-7613-PE1 Access Ports N

Isb-7613-PE2 GigE Links to Isb-12816- Y
P1, 10720 and Fsd-7613-
PE2
Isb-7613-PE2 Access Ports N

Pes-7609-PE1 All interfaces N
Dik-7609-PE1 All interfaces N
Jeh-7609-PE1 All interfaces N
Abt-7609-PE1 All interface N
As mentioned earlier, BFD can be enabled on all interfaces for consistency reasons, in case of
stable network this will not pose any problems. However as precaustionary measure it is
suggested to follow above table as a guideline.
OSPF Security
It is recommended to authenticate the OSPF packets such that routers can participate in routing
domains based on predefined passwords. By default, a router uses a Null authentication, which
means that routing exchanges over a network are not authenticated. Two other authentication
methods exist: Simple password authentication and Message Digest authentication (md5).
Authentication does not need to be set, but it is strongly recommended for security purposes. And
it is recommended that MD5 be used as the authentication method since it provides higher
security than the plain text authentication method.
Message Digest Authentication is a cryptographic authentication. A key (password) and key-id are
configured on each router. The router uses an algorithm based on the OSPF packet, the key, and
the key-id to generate a “message digest” that gets appended to the packet. Unlike the simple
authentication, the key is not exchanged over the wire. A non-decreasing sequence number is also
included in each OSPF packet to protect against replay attacks.
This authentication not only provides for protection against Denial of Service attacks; but it also
assists with prevention of downtime by preventing routes advertised from a misconfigured node
entering the routing table of core devices.

MPLS Core Design
It is possible to configure a unique password for every subnet in the network; however this may be
difficult to administer so a single password can be used for all interfaces in the core network.
interface <interface type-number>

ip ospf message-digest-key 1 md5 cisco
router ospf 1
area 0 authentication message-digest
Figure 38 Configuring OSPF Authentication
OSPF Deployment Recommendations

For the Wateen Telecom IP/MPLS network:
• OSPF will run on all PE/P routers in the network, and will provide layer 3 reachability for all
routers.
• All IP network interface addresses should be numbered with an IP address.
• All routers must be allocated a loopback address, which will be used as the next hop by MP-
BGP when exchanging VPN and IPv4 information.
• Set the auto-cost reference-bandwidth to 10000 on all routers.
• Assure that bandwidth on all physical interface is reflected according to their speed.
• Set the OSPF router-id to loopback0. OSPF will use the highest IP address on any interface as
its router-id. If a loopback interface (loopback0) is configured with an IP address, the Cisco
IOS software will use this IP address as its router ID, even if other interfaces have larger IP
addresses. Since loopback interfaces are not affected by circuit failure, greater stability in the
routing table is achieved. Since there may be other loopback addresses defined in the future it
would be prudent to specifically set the router-id to loopback0 using the “router-id a.b.c.d”
sub-command. This will remove any confusion as to the ip address to use as the router id.
• Hello and Dead timers should be left unchanged to default values initially and can be tuned in
future if required. It is understood that BFD will be used to detect dead peer.
• OSPF timers value may be tuned to improve convergence time, suggested values should be
used as starting point.
• MD5 authentication should be used to authenticate all OSPF packets received on the network.
• BFD will be used as a last resort dead peer detection mechanism
OSPF Configuration
The following OSPF configuration is an example for a PE/P router in the Wateen Telecom’s
network. The same configuration can be used for the other PE routers – just the router-id needs to
be changed along with identifying correct network interfaces. SPF timers are shown as default,
these may be tuned to suggested values documented earlier.

MPLS Core Design
interface Loopback0
ip address <ip-address> 255.255.255.255
interface Ethernet0/0
bandwidth 1000000
ip ospf message-digest-key 1 md5 7 110A1016141D
!
router ospf 1
router-id <loopback-ip-address>
auto-cost reference-bandwidth 10000
bfd all-interfaces
area 0 authentication message-digest
network <network address> <wild-card mask> area 0
Figure 39 OSPF Configuration Template
Cisco Express Forwarding (CEF) Switching
Cisco Express Forwarding (CEF) is advanced Layer 3 IP switching technology. CEF optimizes
network performance and scalability for networks with large and dynamic traffic patterns by
essentially distilling the routing information into a forwarding database known as the FIB,
Forwarding Information Database. Cisco Express Forwarding or CEF switching is a pre-requisite
for MPLS to function properly. Therefore CEF must be configured on all the PE and P devices in
the Wateen Telecom’s network.
This is default switching mode for all Cisco 7600 Series routers. It is possible that due to memory
or some other software related issues, CEF may get disabled. In that scenario, CEF can manually
be enabled as shown in Figure 40.
ip cef [distributed]
Figure 40 Enabling CEF
* show ip cef and show cef line can be used to verify if CEF is operational
cef forwarding should not be turned off even if a platform allows for turning it off.
MSP (Multiplexed Switching Protection) Configuration

The network survivability schemes used in SONET/SDH are known as automatic protection
switching (APS) for SONET and multiplexed switching protection (MSP) for SDH. APS and
MSP are fundamentally similar and depend on protection signaling via the K1/K2 line overhead

MPLS Core Design
bytes. Cisco routers with PoS interfaces can receive and send appropriate protection signals to
connecting ADMs. The Cisco PoS protection scheme can be set up for situations where protect
and working interfaces are different ports on the same router or even on the same line card in the
same router. These scenarios, however, provide protection for only router interface or link failure.
APS refers to the mechanism of using a "protect" POS interface in the SONET network as the
backup for "working" POS interface. When the working interface fails, the protect interface
quickly assumes its traffic load. On the protect circuit, the K1 and K2 bytes from the line
overhead (LOH) of the SONET frame indicate the current status of the APS connection and
convey any requests for action. This signalling channel is used by the two ends of the connection
to maintain synchronization.
The working and protect circuits themselves, within the router or routers in which they terminate,
are synchronized over an independent communication channel, not involving the working and
protect circuits. This independent channel may be a different SONET connection, or a lower-
bandwidth connection. In a router configured for APS, the configuration for the protect interface
includes the IP address of the router (normally its loopback address) that has the working
interface.
Cisco routers also use a proprietary protocol, known as the Protect Group Protocol, between
working and protect routers to complement SONET/SDH protection signaling that occur with
ADMs. The Protect Group Protocol is IP-based and uses User Datagram Protocol (UDP) transport
(UDP port 172). Using this protocol, the process controlling the protect circuit directs the process
containing the working circuit whether to activate or deactivate the working circuit, in the case of
degradation or loss of channel signal, or manual intervention. If communication between the two
processes is lost, the working router assumes full control of the working circuit as if no protect
circuit existed.
1 + 1 linear APS and MSP implementation requires the ADMs to forward the same data signal to
both protect and working circuits. In the 1 + 1 protection scheme, the working PoS interface
selects the data signal when there are no failures or severe errors. Signal selection is at a protected
interface when there are problems on the working circuit.
There are two modes of operation, unidirectional and bidirectional. In bidirectional mode, the
receive and transmit channels are switched as a pair. In unidirectional mode, the transmit and
receive channels are switched independently. For example, in bidirectional mode, if the receive
channel on the working interface has a loss of channel signal, both the receive and transmit
channels are switched.
Cisco 12000 and 7600 sereies routers do not provide true unidirectional mode of operation, where
the transmit interfaces are bridged and the receive interface is chosen independently. Instead, Tx
and Rx are tied to the same interface (the Active one). The router uses alarms like LAIS to force
ADM switchover to the appropriate interface. It is a hack for ADMs which do not provide
bidirectional support. It is recommended to configure the ADM in bidirectional mode.
The W (working) and P (protect) interfaces receive the same payload from the ADM - but only
one is selected, or currently active. Only the selected interface actually processes the payload. The
deselected interface is held in a "line protocol is down" state and cannot participate in routes or
adjacencies. That is, the currently-deselected interface is completely removed from the layer 3
picture. A protection switch forces the APS routers to remove adjacencies and routes involving
the now-deselected interface, and form new adjacencies over the now-selected interface. In other
words, IP traffic begins to flow on the new W interface only after routing-protocol convergence.
Thus, although the APS switch itself requires less than 50 ms to complete, as required, all this
means is that the choice of which interface is to be selected is changed, which affects at most two
routers (W and P). Full restoration of IP traffic via the newly-selected interface requires that new
adjacencies be formed between the newly-selected interface and the remote router, and that the
resulting routes be disseminated to all the routers directly connected to either W or P.
Reflector mode provides improved routing convergence. Reflector mode enhances routing
convergence because the remote router now has early notice of a switch between W to P, and can

MPLS Core Design
tear down its now-outdated adjacency with the now-deselected system, and need not wait for a
timeout. The convergence enhancement does not depend on whether the aps reflector command is
configured. The W(working), P(protect), and R(remote) routers must support reflector mode
requirements. Reflector mode requires an interface capable of reflector mode at the remote end of
the SONET path .
Open Shortest Path First (OSPF) supports APS reflector mode via DDTS CSCdr57673.
A value of "(null)" in the Remote APS configuration field of the show controller pos command
indicates that the local end has not received reflector channel information from the remote PTE. If
the remote PTE supports the reflector channel capability, a problem probably exists between the
remote PTE and remote ADM.
As noted above APS protection is always configured between a router and ADM, in Wateen
Telecom’s deployment both links (working and protect) terminate on same router.It is
recommended that these links be placed on separate linecards. Remote PEs (7609) connect to
their respective core P (12816) device, connecting via POS. These links are expected to be APS
protected to their respective ADM device on both ends. Following diagram represent the
connectivity between different routers and ADMs.
Configuration for core P (12816) device would look like following:

MPLS Core Design
interface Loopback0
ip address 100.1.1.1 255.255.255.255
!
interface POS1/0
ip address 10.1.1.1 255.255.255.0
no keepalive
crc 32
pos framing sdh
pos report rdool
pos report lais
pos report lrdi
pos report pais
pos report prdi
pos report sd-ber
pos flag s1s0 2
aps group 10
aps working 1
pos ais-shut
!
interface POS1/1
ip address 10.1.1.3 255.255.255.0
carrier-delay ms 0
no keepalive
crc 32
no keepalive
pos framing sdh
pos report rdool
pos report lais
pos report lrdi
pos report pais
pos report prdi
pos report sd-ber
pos flag s1s0 2
aps group 10 !allows multiple protect/working int on one Router
aps protect 1 100.1.1.1
pos ais-shut
Figure 41 APS configuration on P router
* Permitted group numbers are 0-255. A sequential group number approach is suggested.
* aps revert 1 command allows for revertive aps i.e. it enables automatic switchover (1 minute in
this case) from the protect interface to the working interface after the working interface becomes
available. It is recommended that revertive APS should not be used as for every n failures one
gets 2n switches and therefore 2n times to reconverge.
It is also possible to use same ip address space for both working and protect interfaces. Using the
same IP address obviously conserves IP address space. Using different IP addresses provides with
a bit more granularity while testing - i.e by pinging different IP addresses one can confirm which
link (working or protect) is currently active. Alternatively this info can also be obtained via "sh
aps pos x/y" command as well. It is suggested that different IP addresses be used for ease of
troubleshooting unless insufficient IP addresses are available.
The pos report commands as shown above are optional and allow reporting of POS alarms to the
console.
The pos ais-shut is an important command which triggers an Alarm Indication Signal (AIS) if the
interface is placed in administrative shutdown state.

MPLS Core Design
While the configuration on a 7609 PE device would look like the following:
interface Loopback0
ip address 200.1.1.1 255.255.255.255
!
interface POS3/0
ip address 10.1.1.2 255.255.255.0
no keepalive
crc 32
pos framing sdh
pos report rdool
pos report lais
pos report lrdi
pos report pais
pos report prdi
pos report sd-ber
pos flag s1s0 2
aps group 10
aps working 1
pos ais-shut
!
interface POS3/1
ip address 10.1.1.4 255.255.255.0
no keepalive
crc 32
pos framing sdh
pos report rdool
pos report lais
pos report lrdi
pos report pais
pos report prdi
pos report sd-ber
aps group 10
aps protect 1 200.1.1.1
pos ais-shut
!
Figure 42 APS configuration on PE router
The pos delay triggers commands “pos delay triggers line” and “pos delay triggers path”
provide the option to delay IOS to set the line protocol to down when a POS alarm triggers the
line protocol to go down.
It is recommended not to use Line triggers on interfaces that participate in APS, because Line
triggers interfere with APS operation. The “pos delay triggers line n” command does not allow
the line to go down on the brief LOS that comes from internally protected DWDM gear, from the
time an internal DWDM protection switch occurs. If the defect clears during the holdoff period, it
is like the defect never occurred.
LDP (Label Distribution Protocol)

LDP is responsible for distributing the labels associated with every IP destination prefix in the
MPLS network. Labels will be assigned to every address that is in the global routing table. This
global routing table is created and maintained by the IGP, which in this case is OSPF. Essentially

MPLS Core Design
all IP destination prefixes will be either a loopback or network interface address and normally
there should be no customer IP addresses in the global routing table.
The core P routers only have an understanding of labels that are associated with IP destinations in
the OSPF internal routing table. They have no knowledge of labels related to routes in customer
VPNs as these are created and distributed by the MP-BGP process between PEs only. Therefore,
in the P network, a labeled IP packet is switched to the next-hop based only on the outer label (the
one allocated by LDP from the global routing table) until it finally reaches its destination. In an
MPLS-VPN network, this final destination will always be the PE that originated the VPN route.
The IOS CLI accepts “tag” and “mpls” command interchangeably for most cases. For example,
“show tag tdp neighbor” and “show mpls ldp neighbor” produce identical output. In some
cases, there is only an “mpls” command such as “mpls label protocol ldp”. Wateen Telecom is
advised to use only the newer “mpls” form of the commands.
Prior to activating Label Switching on a router Cisco Express Forwarding must be enabled (on
7600 series router this is default switching mode), plus the mpls ip command must be issued on
each IP-enabled interface of P and PEs that are connected. mpls ip command should not be
enabled on PE-CE connections or on PE interfaces where hosts will reside in a VRF.
As with OSPF, MD5 based authentication should be enabled where LDP will be used to prevent
any DoS attacks, and to help avoid configuration errors.
Controlling Label Distribution

MPLS labels are only required for the loopback addresses of the PE routers in Wateen Telecom’s
network, whereas by default they would be distributed for every LDP enabled subnet. Optionally
it is possible to reduce the amount of information carried; IOS offers a feature to limit which
addresses labels are allocated for.
This can be enabled using the “mpls ldp advertise labels for” command along with an access-
list.
Interface MTU size

As mentioned earlier, two levels of labels are used to deliver MPLS-VPN services. The first level
label is distributed by the LDP protocol, whilst the second level label is created by MP-BGP for
VPN distribution.
When these two labels are placed into the frame they increase the frame size by 8 bytes (4 bytes
per label). This can be problematic particularly on Ethernet interfaces which have a default
Maximum Transmission Unit of 1500 bytes; bigger frame sized packets will be dropped if packets
arrive with no fragment bit set. Note that with Ethernet encapsulation without dot1q encap, the
actual layer 2 frame size is 1518 while with dot1q encap, the actual layer 2 frame size is 1522.
With two label imposition, the actual layer 2 frame size becomes 1526 (or 1530 with dot1q encap)
as shown in the Figure 43.
Figure 43 Layer 2 Frame with 2 MPLS Labels
To support label switching over an interface where the MTU size is default (1500bytes), the MTU
size for MPLS frames must be increased with the following command on every interface in the
core on P and PE routers.

MPLS Core Design
The command shown in Figure 44 can be configured on all gigabit Ethernet interfaces in the
Wateen Telecom’s network. The default MTU on STM-1/STM-4/STM-16 interfaces is 4470
bytes and does not need to be changed.
mpls mtu 1530
Figure 44 Increasing the MPLS MTU
Upto six labels have been allowed in MPLS to cater for other services on the network such as
traffic engineering and Carrier Supporting Carrier. Each service may require an increase in the
label stack from 2 to something greater. Given that there may be requirements to roll out traffic
engineering and other services, it may be desireable to set mpls mtu to a higher value to
accommodate maximum number of labels.
LDP Autoconfiguration
LDP autoconfiguration enables one to globally enable LDP on every interface associated with an
IGP instance. The IGP supported for LDP autoconfiguration are OSPF and ISIS. Support for
OSPF has been there in IOS 12.0S train since 12.0(30)S release.
Under current software releases, this feature can only be used for devices which are targeted to
run 12.0S release i.e. 12816 routers. This feature is currently not available for 7600 series
platform. Its support is slated for a subsequent release (i.e. a release scheduled to ship after
12.2(33)SRA).
Once OSPF configuration is complete, LDP autoconfiguration can be invoked under “router
OSPF x” process with following configuration:
mpls ip
mpls label protocol ldp
router ospf process-id
network ip-address wildcard-mask area area-id
mpls ldp autoconfig [area area-id]
Figure 45 LDP Autoconfiguation on P router
Once mpls ldp autoconfig command is configured under a routing process, all the interfaces that
belong to an OSPF area are enabled for LDP.
If it is desired to remove LDP from an interface, command: “no mpls ldp igp autoconfig” can be
used on the interface to disable LDP from the interface. It is suggested to use LDP
autoconfiguration on 12816 series P routers for convenience and consistency.
LDP MD5 Global Configuration
MPLS LDP MD5 Global configuration allows one to enable LDP MD5 globally instead of on a
per-peer basis. Using this feature one can set up password requirements for a set of LDP

MPLS Core Design
neighbors to help prevent unauthorized peers from establishing LDP sessions and to block
spoofed TCP messages. Currently this feature is only available for our P devices on the latest
12.0(32)SY release. The feature is slated for an upcoming release for 7600 series platform. While
this feature is not available on our current target release for GSR platform, this may be of interest
once Wateen Telecom has decided to upgrade to this version of the code. For further details,
please see:
http://www.cisco.com/en/US/products/ps6566/products_feature_guide09186a00805f24da.html
LDP Configuration and Design Recommendations

• To activating Label Switching on a router the mpls ip command must be issued on each
interface that connects P and PE routers together. This should not be enabled on PE-CE
connections or PE connections defined in a vpn.
• It is recommended to explicitly enable LDP globally by entering mpls label protocol ldp
command globally using Cisco IOS CLI
• For proper operation of MPLS, LDP chooses an IP address as a router-id. It is important to
note that the IP address chosen as router-id is routable, otherwise LDP will not be able to
form the neighbor relationship with the adjacent nodes. On Cisco router, LDP router ID is, by
default, determined as follows:
o The IP addresses of all operational interfaces are examined.
o If these IP addresses include loopback interface addresses, the largest such
loopback address is selected as the LDP router ID.
o Otherwise, the largest IP address pertaining to an operational interface is
selected as the LDP router ID.
However, the normal (default) method for determining the LDP router ID may result in a
router ID that is not usable in certain situations. For example, an IP address selected as the
LDP router ID might not be advertisable by the routing protocol to a neighboring router,
therefore it is recommended to manually set the router-id by entering the mpls ldp router-id
<interface> command. The specified interface must be operational for its IP address to be
used as the LDP router ID. In addition, force keyword should be entered to make sure the
router-id takes effect immediately upon entering this command. However, care should be
taken using this command since all the existing LDP sessions will be torn down if the router-
id of the existing sessions is different from the newly selected ID.
• logging of LDP neighbor state change is the default and can be re-enabled using “mpls ldp
logging neighbor-changes” command. It is recommended to keep the default.
• As with OSPF, MD5 based authentication should be enabled on each link where LDP will be
used to prevent any DoS attacks, and to help with configuration errors. It is recommended to
use LDP authentication.
• The tag-switching advertise-tags for <ACL> command should be used to limit the number
of prefixes for which labels are allocated.
• LDP autoconfig may be used on P routers.
• It is possible that at some point when IGP reconverges, it will reconverge before LDP has
completed label exchange, it is therefore possible to black hole labled traffic for a very short
period of time. Based on the current size of Wateen Telecom network this is not a major
concern at this stage. However as the network grows, a directed LDP session should be
considered between the neighbors where redundant paths are available. Please note that by
default directed LDP session does not timeout, this shields the LDP neighbor relationship
being torn down and visibility into LDP relationship from operations and support perspective.
There are currently two features which are meant to address this specific situation:
MPLS LDP Session Protection:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/12
0s30/fssespro.htm

MPLS Core Design
This feature is currently available on 12.0S train starting 12.0(30) and hence available on
12000 series platform. This feature has recently been incoporated in IOS train available on
7600 series routers. This feature is available in 12.2(33)SRA release on 7600 series platform.
MPLS LDP-IGP Synchronization:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios120/120newft/120limit/120s/12
0s30/fsldpsyn.htm
This feature is also available on 12.0S train starting 12.0(30)S. However this feature is not
available on 7600 series platform and is scheduled for a subsequent release (i.e. a release after
12.2(33)SRA).
Please note that as indicated above, these features can be used in future depending on various
factors including additional redundancy and growth of the network.
LDP Configuration Template

It is highly advisable to associate the router-id for the TDP/LDP session with a loopback address
as shown below. Below is a configuration for one of the routers in the network for LDP, this
configuration may be used for all MPLS P/PE routers however with a change of the router-id and
correct neighbor ip addresses. Please also note how some commands while entered as “mpls ldp”
commands are shown in configuration as tag-switching commands. Similarly “mpls ip” command
is entered and tag-switching ip command is shown in configuration.
hostname <insert hostname>

!
mpls label protocol ldp

mpls ldp neighbor <ip-address> password <password>
tag-switching tdp router-id Loopback0 force
!
!
!
interface Loopback0
!
bandwidth 1000000
ip address <ip-address> <subnet-mask>
tag-switching ip
!
Figure 46 LDP Configuration Template
With filtering LDP labels for PE loopbacks only, following additional commands will need to be
added:

MPLS Core Design

no tag-switching advertise-tags
tag-switching advertise-tags for PE_loopback
ip access-list standard PE_loopback
permit <prefix> <wild-card mask>
!
Figure 47 Label Filtering Template
The “mpls ip” command merely enables LDP on an interface, and it does not control whether a
packet is switched using the MPLS Label at all – this is determined by the Ethertype on an
incoming frame and via various forwarding tables.
MP-BGP4 (Multi-protocol BGP)

BGP is necessary to provide VPN services in Wateen Telecom’s network because it is used to
propagate VPN routing information. It is also required for Wateen Telecom to provide internet
transit services to other customers for ipv4 traffic.
In the MPLS network each defined VPN consists of VPN Routing and forwarding tables (VRFs)
which are associated with each services interface. The VRF tables consist of unique VPN-IPv4
addresses, (they are unique as each is prefixed with the Route Distinguisher). Since these are not
IPv4 addresses, BGP provides multi-protocol extensions that allow the distribution of these VPN-
IPv4 routes.
BGP propagates VPN-IPv4 information using the BGP multi-protocol extensions for handling
these extended addresses. (See RFC 2283, Multi-protocol Extensions for BGP-4.) It propagates
reachability information, expressed as VPN-IPv4 addresses, among the PE routers only. The
reachability information for a given VPN is propagated only to other members of that VPN. The
BGP multi-protocol extensions identify the valid recipients for VPN routing information.
VPN Route Distribution Operation

The following figure shows the process of route distribution using MP-BGP between a VPN
(labeled vpn1), which terminates on a 7600 PE at site1 and to another 7600 PE at siteN.
update X update X
CE1 P1 P2 PE2 CE3

PE1
x
CE2 CE4
x
MP-iBGP session
update X update X
VPN-IPv4 updates are

translated into IPv4
VPN-IPv4 update: VPN-IPv4 update: address and inserted into
RD1:X, Next-hop=PE1 RD2:X, Next-hop=PE1 the VRF corresponding to
RT=RED, Label=10 RT=ORANGE, Label=12 the RT value

MPLS Core Design
Figure 48 BGP VPN Route Distribution
The steps are as follows:
Step1: Customer routes are injected into the VRF table at the Site1 PE using redistribute
connected
Step 2: At the Site1 PE, the routes in the customer VRF are then exported into MP-BGP,
updates contains RD for unicity of IPv4 + one or more RT for VRF identification.
Step 3: These exported routes are then sent across the MPLS backbone between the BGP peers.
This process would be carried out for any other BGP peers that had members in the
same VPN. Note that this step shows a logical connection between the two BGP peers
(actually MP-BGP update is sent to a Route-Reflector then RR reflect to other MP-
BGP peers, RR are explained further)
Step 4: The routes are imported into the correct VRF at the other end in respective PE which
also have same VRF defined, (RT configured in VRF define in which VRF the routes
should be imported).
Step 5: These routes are then accessible for users from VPN at respective site.
BGP Route Reflectors - General Operation

BGP Route Reflectors are not essential for an MPLS network to function but the following should
be kept in mind.
Without route reflectors, whenever a new PE is introduced, each existing PE in the Wateen
Telecom’s network will need to have an extra BGP neighbor commands added pointing to the
new PE. This is due to a requirement in BGP, that updates received by a peer in the same
autonomous system (AS) are not allowed to be forwarded on within the AS. Therefore a BGP
network must be fully meshed, with all peers seen as directly adjacent to one another, as far as
BGP routing updates are concerned.’
If the number of PE’s becomes too great to make this operation practical (that is, adding neighbor
commands in every PE) then BGP Route Reflectors should be introduced. Route Reflectors
obviate the need to fully mesh the BGP peers and avoid the adding neighbor commands to each
PE.
With Route Reflectors, the PE’s would only require neighbors defined for each route reflector.
Any updates, including VRF information, would be sent to the Route reflectors only. The Route
Reflectors are then responsible for propagating any information received from PE’s to all other
PE’s. Each time a new Edge LSR is added, only the Route Reflectors would need to be updated
with neighbor statements.
Route Reflectors are also useful for a customer that has connections to several PE’s (dual
homing). In the situation where a route change occurs in the customer network, the Edge LSR
that terminates that part of the customer network, would have to update every Edge LSR peer
participating in that VPN. Route Reflectors would remove the burden of BGP updates from the
Edge LSR.
Figure 49 shows the effect of Route Reflectors. A BGP network without Route Reflectors
requires 5 (n-1) peers. Adding Route Reflectors in this example reduce the PE peerings to 2.

MPLS Core Design
RR
Without RR RR
With RR
Figure 49 BGP Route Reflectors
BGP Route Reflectors - VPN route distribution

As discussed in the previous section, it would not be practical or good practice to create a full
mesh of BGP peers in a large network. Given the current number of PE devices in Wateen
Telecom’s network, e..g connectivity of 20 devices would mean 19 BGP peering sessions in each
of the PE devices if no route reflectors (RR) were used, this will not scale and is prone to
configuration errors.
Using RR’s in an MPLS network comes with its own set of constraints. When RRs are used to
peer with the PEs in a MPLS/BGP network, the RRs will hold all the VPNV4 routes advertised by
all the PE’s. It other words every route in every customer network must be held by the RR for
distribution to all PE’s or RR’s. This has scalability implications as RR’s would potentially
introduce memory resources as a bottle neck in a MPLS network with a large set of customers and
routes. This however is not a concern for Wateen Telecom’s network in its current phase of
deployment. It is suggested that number of prefixes accepted by each customer be limited to a
reasonable number, in order to help scale the service. In addition, it is suggested to have dedicated
route reflectors sized according to Wateen Telecom’s requirements depending on growth and the
amount of prefixes that need to be carried by Wateen Telecom route reflectors.

MPLS Core Design
Figure 50 Wateen VPNv4 Route Reflector Topology
As shown above all PE routers will peer with two VPNv4 route reflectors. The interim route
reflectors are two PE devices (7613) in Lahore and Karachi. These two routers will peer with each
other as well as peer with all PE devices, treating them as vpnv4 route reflector clients.
It should also be noted that two vpnv4 route reflector clients are also PE devices and hence will
carry full internet BGP routing table. Therefore vpnv4 route reflectors will also peer with two
ipv4 route reflectors. This is not a recommended solution, instead an interim measure. This
necessity can be eliminated in case of dedicated route reflectors.
Please refer to Interet gateway section for ipv4 route reflector description.
BGP MD5 Authentication

As OSPF and LDP, BGP by default does not use encryption for the sessions established, it is
however recommended to ensure that BGP is configured with neighbour passwords.
While the sessions may still be internal and the requirement exists for both-ends of the session to
be configured to ensure BGP session establishment it is still recommended to use MD5
authentication on the sessions for maximum security
BGP Passwords are disabled by default and are configured on a per-neighbour basis
!
router bgp 65501
neighbour <ip address / peer group> password <password>
!
Figure 51 BGP Authentication

MPLS Core Design
BGP deterministic MED

If bgp deterministic med is not enabled, the order in which routes are received from peers could
impact MED based decisions. This may occur when the same route is received from different
neighbours in the same AS with the same path length but different MED values.
For example, if we receive the following routes in this order:
1. ASPATH 1, MED 100, Internal, IGP Metric 10

3. ASPATH 3, MED 200, External
If the possibilities for a prefix were received in this order without deterministic MED then the
node would install path 2 over path 1 due to a lower IGP metric, it would then prefer path 3 to
path 2 due to the fact that it is an external route.
However, with deterministic MED this removes any order dependencies of MED based decisions,
it ensures that the best MED comparison is made across all routes from the same AS, in this case
path 1 would be chosen due to it having the lowest MED.
router bgp 65501

bgp deterministic-med
Figure 52 BGP Deterministic Med
BGP path selection process is described at the following URL

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml
BGP Convergence
Several BGP timers can be used to tune the iBGP convergence in an IP MPLS / VPN Network.
• Advertisement-interval: To set the interval between the sending of BGP routing updates. The
default interval is 5 seconds for iBGP peers.
• Keepalive: Frequency, in seconds, with which the Cisco IOS software sends keepalive
messages to its peer. The default is 60 seconds.
• Holdtime: Interval, in seconds, after not receiving a keepalive message that the software
declares a peer dead. The default is 180 seconds.
• Scan-time: it defines the frequency at which the router validates routes in BGP table. It
validates the locally originated network and next-hop within the BGP table. Valid values used
for selecting the desired scanning interval are from 5 to 60 seconds. The default scan time is
60 seconds. (Per address-family use “bgp scan-time {5-60}”)

MPLS Core Design
• Scan-time import: this timer is relevant for VPNv4 address-family. It defines the frequency at
which routes received from another PE through MP-BGP are imported into a VRF. This timer
is not relevant for next-hop deletion that is processed based on the “normal” BGP scan-time.
Default value is 15 seconds. (Per address-family “bgp scan-time import {5-60}”)
The following table summarizes the default values for the iBGP timers.
Table 8 BGP Default Timers
BGP Timers iBGP Default value eBGP Default value

Advertisement-interval 5 sec 30 sec
Keepalive 60 sec 60 sec
Holdtime 180 sec 180 sec
Scan-time 60 sec 60 sec
Scan-time import 15sec 15sec
• iBGP convergence follows IGP convergence, This means that when a PE router or its
uplinks fail, BGP next-hop reachability will be lost after the IGP converges. The BGP
scanning process will detect the loss of the next-hop for all relevant BGP and MP-BPG
routes and stop selecting it in the BGP RIB.
• It is recommended initially to leave the timers at their default.
• In current release on 7600, BFD is not available for BGP. BFD will be really relevant for
eBGP sessions
As discussed in the IGP routing section, fast IGP convergence and related values that should be
tuned have already been provided under the event of an internal link or node failure.
This means that when a PE router or it’s uplinks fail, BGP next-hop reachability will be lost after
the IGP converges. The BGP scanning process will detect the loss of the next-hop for all relevant
BGP and MP-BPG routes and stop selecting it in the BGP RIB.
It is recommended initially to leave the timers at their default.
Table below shows potentially tuned BGP tuned times, these should be modified with care and
only after ensuring stability of the network infrastructure
BGP Timers iBGP Tuned value

Advertisement-interval 5 sec
Keepalive 10 sec
Holdtime 30 sec
Scan-time 30 sec
Scan-time import 5sec
Table 9 - Potential Modified values for iBGP timers

MPLS Core Design
BGP monitors the next hop of installed routes to verify next-hop reachability and to select, install,
and validate the BGP best path. By default, the BGP scanner is used to poll the RIB for this
information every 60 seconds. During the 60 second time period between scan cycles, Interior
Gateway Protocol (IGP) instability or other network failures can cause black holes and routing
loops to temporarily form.
BGP Support for Next-Hop Address Tracking
BGP monitors the next hop of installed routes to verify next-hop reachability and to select, install,
and validate the BGP best path. By default, the BGP scanner is used to poll the RIB for this
information every 60 seconds. During the 60 second time period between scan cycles, Interior
Gateway Protocol (IGP) instability or other network failures can cause black holes and routing
loops to temporarily form.
The BGP Support for Next-Hop Address Tracking feature is enabled by default when a supporting
Cisco IOS software image is installed. BGP next-hop address tracking is event driven. BGP
prefixes are automatically tracked as peering sessions are established. Next-hop changes are
rapidly reported to the BGP routing process as they are updated in the RIB. This optimization
improves overall BGP convergence by reducing the response time to next-hop changes for routes
installed in the RIB. When a bestpath calculation is run in between BGP scanner cycles, only
next-hop changes are tracked and processed.
BGP Next hop address tracking feature is available on 12816 series router staring 12.0(30)S
(actual release of the feature is in 12.0(29)S). However this feature only showed up in latest
release of 7600 i.e. 12.2(33)SRA. It is suggested that default scan-time value be used once 7600
series routers have been updgraded to 12.2(33)SRA.
ip tcp path-mtu-discovery
Every TCP session has a limit in terms of how much data it can transport in a single packet. This
limit is defined as the Maximum Segment Size (MSS) and is 536 bytes by default. This means
TCP will take all of the data in a transmit queue and break it up into 536 byte chunks before
passing packets down to the IP layer. Using a MSS of 536 bytes ensures that the packet will not
be fragmented before it gets to its destination because most links have a MTU of at least 1500
bytes. The following command will allow you to check the MSS of all BGP peers:
Router#show ip bgp neighbors | include max data

Datagrams (max data segment is 536 bytes):
The problem is that using such a small MSS value creates a large amount of TCP/IP overhead,
especially when TCP has a lot of data to transport like it does with BGP. The solution is to
dynamically determine how large the MSS value can be without creating packets that will need to
be fragmented. This is accomplished by enabling "ip tcp path-mtu-discovery" (PMTU). PMTU
allows TCP to determine the smallest MTU size among all links between the ends of a TCP
session. TCP will then use this MTU value minus room for the IP and TCP headers, as the MSS
for the session. If a TCP session only traverses Ethernet segments then the MSS will be 1460
bytes. If it only traverses POS segments then the MSS will be 4430 bytes. The increase in MSS
from 536 to 1460 or 4430 bytes reduces TCP/IP overhead, which helps BGP converge faster.
Once we have enabled this knob and reset all of our BGP sessions via a reboot or "clear ip bgp *"
we can see the expected increase in MSS:

MPLS Core Design

Route Reflectors - Deployment Recommendations

The Wateen Telecom’s network will be deployed with two route reflectors to carry VPN routing
information between the PE routers. The route reflectors will be the two core city PE devices
7613 series routers located at Lahore and Karachi as an interim measure. Wateen Telecom is
suggested to use dedicated vpnv4 route reflectors.
Similarly there will be two route reflectors for ipv4 traffic to carry all ipv4 prefixes learnt via
EBGP. The route reflectors will be second set of two core P devices 12816 series routers located
in Lahore and Faisalabad.
Recommendation of RR design for Wateen Telecom’s network:
• The RRs will peer with each other forming a full mesh.
• All PEs will peer with both RRs for vpnv4 and with second set of RR for ipv4 traffic.
• The AS number used between BGP peers will be 65501.
• The two Pes chosen as vpnv4 RRs will only reflect VPNv4 routes, there are separate RRs for
IPv4 routes. It is always recommended to have dedicated route refelectors, separate route
reflector for ipv4 and vpnv4 are recommended.
• Normally RRs are not required to enable MPLS, as they are not a transit point for any traffic,
therefore MPLS need not be configured on them. All routers will peer, using the BGP
neighbor statement, to the loopback address of the adjacency. The loopback address is used to
make sure the IP address of an interface stays up and is independent of an interface that might
be unstable.
• Passwords with MD5 authentication will be used on all BGP peers.
Configuration at the Route Reflector

The route reflectors will send the VPN routes between the PE routers. In order to make this
process as efficient as possible a BGP peer group will be used for all PE router sessions except for
route-reflector peering amongst themselves. The major benefit of specifying a BGP peer group is
that it reduces the amount of system resources (CPU and memory) used in an update generation. It
also simplifies BGP configuration since it allows the routing table to be checked only once, and
updates to be replicated to all other in-sync peer group members. Depending on the number of
peer group members, the number of prefixes in the table, and the number of prefixes advertised,
this can significantly reduce the load.
Peer groups have the following requirements:
• All members of a peer group must share identical outbound announcement policies (such as
distribute-list, filter-list, and route-map), except for default-originate, which is handled on a
per-peer basis even for peer group members.
• Inbound update policies may be customized for any member of a peer group.
• A peer group must be either internal (with internal BGP (iBGP) members) or external (with
external BGP (eBGP) members).

MPLS Core Design
• Members of an external peer group have different autonomous system (AS) numbers.
Following is a configuration example of route reflector.

!
router bgp 65501
bgp router-id <loopback0 ip address>
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor vpnclients peer-group
neighbor vpnclients remote-as 65501
neighbor vpnclients password cisco
neighbor vpnclients update-source Loopback0
neighbor vpnclients version 4
neighbor <PE1 ip address> peer-group vpnclients
neighbor <Peer RR ip address> remote-as 65501

neighbor <Peer RR ip address> description RR & PE at Khi
neighbor <Peer RR ip address> password <password>
neighbor <Peer RR ip address> update-source Loopback0
neighbor <Peer RR ip address> version 4
!
address-family vpnv4
neighbor vpnclients activate
neighbor vpnclients send-community both
neighbor vpnclients route-reflector-client
neighbor <Peer RR ip address> activate
neighbor <Peer RR ip address> send-community both
exit-address-family
!
Figure 53 BGP configuration template for Route Reflector
Configuration at the PE Routers

A simpler configuration is required at each PE router as it only needs to peer to the two route-
reflectors. Again, a peer group is used to make this simpler. The configuration is exactly the
same on every PE router.

MPLS Core Design
Hostname <insert hostname>

router bgp 65501
bgp router-id <loopback 0 ip address>
neighbor refvpn peer-group
neighbor refvpn peer-group
neighbor refvpn remote-as 65501
neighbor refvpn password cisco
neighbor refvpn update-source Loopback0
neighbor refvpn version 4
!
neighbor refip peer-group
neighbor refip peer-group
neighbor refip remote-as 65501
neighbor refip password cisco
neighbor refip update-source Loopback0
neighbor refip version 4
!
neighbor <RR1-vpnv4 ip address> peer-group refvpn
neighbor <RR2-vpnv4 ip address> peer-group refvpn
!
neighbor <RR1-ipv4 ip address> peer-group refip
neighbor <RR2-ipv4 ip address> peer-group refip
!
address-family ipv4
neighbor refip activate
neighbor refvpn activate
neighbor <RR1-ipv4 address> peer-group refip
neighbor <RR2-ipv4 address> peer-group refip
no auto-summary
no synchronization
exit-address-family
!
neighbor refvpn activate
neighbor refip activate
neighbor refvpn send-community extended
neighbor <RR1-vpnv4 address> peer-group refvpn
neighbor <RR2-vpnv4 address> peer-group refvpn
exit-address-family
!
address-family ipv4 vrf voip
redistribute connected
no auto-summary
no synchronization
exit-address-family
!
Figure 54 BGP configuration template for PE
General Best Practices for BGP Configuration
The following parameters are suggested to be enabled on all BGP speaking routers.
• Use the description field to add the hostname of the neighbour in the configuration

MPLS Core Design
• Neighbour addresses will always be loopback addresses (for iBGP peering)

• Loopback address will be used as the update-source locally (for iBGP peering)
• BGP version should be set to 4
• All neighbours will use a peer-group
• BGP neighbour changes will be logged
• BGP will be enabled to use new format community strings
• All neighbours should pass on community information
The following BGP features should be disabled (if not disabled by default):
• Auto-summarization
• IGP Synchronization
Stateful Switchover (SSO)

Traditionally, core routers protect against network faults using router redundancy and mesh
connections that allow traffic to bypass failed network elements.
SSO provides protection for network edge devices with dual Route Processors (RPs) that
represent a single point of failure in the network design, and at which point an outage might result
in loss of service for customers.
The SSO feature takes advantage of available RP redundancy with in the dual-processor
environments (7600, 12000 and 10000 series routers etc). The protocol establishes one of the RPs
as the active processor while the other RP is designated as the standby processor, synchronizing
critical state information between the two devices occurs automatically and ensures that upon an
active failure the standby is initialised and ready to take over operations.
Following an initial synchronization between the two processors, SSO dynamically maintains RP
state information between them.
A switchover from the active to the standby processor occurs when the active RP fails, is removed
from the networking device, or is manually taken down for maintenance. Specifically following
conditions will cause a switchover:
• Hardware failure on the active supervisor engine
• Clock synchronization failure between supervisor engines
• Manual switchover
Without having a SSO enabled solution when a switchover of a RP occurs the state of the layer 2
control is lost.
With SSO when a switchover occurs the standby RP maintains the session state for a number of
protocols: Frame Relay, ATM, PPP, MLPPP, HDLC. With these sessions maintained there is no
protocol bounce, no interface flap so downstream traffic can continue to be forwarded as if there
is no interruption
In contrast to SSO, other operating mode that are supported on redundant supervisor engine in
7600 series chassis are RPR and RPR+. RPR is slowest form of redundancy supported, in this
mode, a switchover can take upto 2 or more minutes. In RPR mode, redundant supervisor engine
come out of reset mode but are not operational.
In RPR+ mode, supervisor engine switch over time can be 30 seconds or more. In this mode the
redundant supervisor engine is fully initialized and configured.

MPLS Core Design
RPR+ Restrictions:
These guidelines and restrictions apply to RPR+:
• Network services are disrupted until the redundant supervisor engine takes over and the
router recovers.
• The Forwarding Information Base (FIB) tables are cleared on a switchover. As a result,
routed traffic is interrupted until route tables reconverge.
• Static IP routes are maintained across a switchover because they are configured from entries
in the configuration file.
• Information about dynamic states maintained on the active supervisor engine is not
synchronized to the redundant supervisor engine and is lost on switchover.
These are examples of dynamic state information that is lost at switchover:

• All terminated PPP sessions
• All terminated TCP and other connection-oriented Layer 3 and Layer 4 sessions
• BGP sessions
• All Automatic Protection System (APS) state information
SSO provides a faster switchover relative to HSA, RPR, and RPR+ by fully initializing and fully
configuring the standby RP, and by synchronizing state information, which can reduce the time
required for routing protocols to converge due to the uptime gained by not re-initialising linecards
and interfaces
For 7600 series platform, please note that during switchover, system control and routing protocol
execution is transferred from the active supervisor engine to the redundant supervisor engine. The
switch requires between 0 and 3 seconds to switchover from the active to the redundant supervisor
engine.
Redundancy Restrictions:
Configuration changes made through SNMP are not synchronized to the redundant supervisor
engine. A work around is that after configuring the router through SNMP, the running-config file
should be copied to the startup-config file on the active supervisor engine to trigger
synchronization of the startup-config file on the redundant supervisor engine and with RPR+,
reload the redundant supervisor engine and MSFC.
Supervisor engine switchover takes place after the failed supervisor engine completes a core
dump. A core dump can take up to 15 minutes. A core dump configuration should only be used
when troubleshooting a specific problem. Under normal operations, core dump shouldn’t be
taken.
* With Sup720 and Release 12.2(18)SXF and later releases, if a fabric synchronization error
occurs, the default behavior of the router is to switchover to the redundant supervisor engine. In
some cases, a switchover to the redundant supervisor engine may be deemed more disruptive than
powering down the module that caused the fabric synchronization error. The “no fabric error-
recovery fabric-switchover “command should be used to disable the switchover and power down
the module with the fabric synchronization error. The command should be used in case a problem
is detected and confirmation is made that problem is due to fabric synchornization error.
Hardware Restrictions:
• Cisco IOS running on the supervisor engine and the MSFC supports redundant configurations
where the supervisor engines and MSFC routers are identical. If they are not identical, one
will boot first and become active and hold the other supervisor engine and MSFC in a reset
condition.

MPLS Core Design
• Each supervisor engine must have the resources to run the router on its own, which means all
supervisor engine resources are duplicated, including all Flash devices.
• Separate console connections are required to each supervisor engine.
• Both supervisor engines must have the same system image
Configuration Mode Restrictions:

The following configuration restrictions apply during the startup synchronization process:
• One cannot perform configuration changes during the startup (bulk) synchronization. If an
attempt to make configuration changes occurs during this process, the following message is
generated: “Config mode locked out till standby initializes”
• If configuration changes occur at the same time as a supervisor engine switchover, these
configuration changes are lost.
Redundancy Configuration:
!For SSO configuration
redundancy
mode sso
!
!For RPR+ configuration
redundancy
mode rpr-plus
Figure 55 RP Redundancy Configuration
It is recommended to use SSO mode should be deployed on all P, PE and Data Centre devices,
this will help maintain uptime in the eventuality of a RP switchover.
Non Stop Forwarding (NSF)

Usually, when a networking device restarts, all routing peers of that device detect that the device
went down and then came back up. This transition results in what is called a routing flap, which
could spread across multiple routing domains. Routing flaps caused by routing restarts create
routing instabilities, which are detrimental to the overall network performance.
NSF helps to suppress routing flaps in SSO-enabled devices, thus reducing network instability.
Non-Stop Forwarding in IOS allows for the forwarding of data packets to continue along known
routes while the routing protocol information is being restored following a failover. With NSF,
peer networking devices do not experience routing flaps. During failover, data traffic is forwarded
through line cards while the standby Route Processor (RP) assumes control from the failed RP.
When an OSPF NSF-capable router performs an RP failover, it must perform two tasks to
resynchronize its link-state database with its OSPF neighbours.
First, it must relearn the available OSPF neighbours on the network without causing a reset of the
neighbour relationship.
Second, it must reacquire the contents of the link-state database for the network.

MPLS Core Design
After an RP failover, the NSF-capable router sends an OSPF NSF signal to neighbouring NSF-
aware devices. This signal is in the form of a link-local LSA generated by the failed-over router.
Neighbour networking devices recognize this signal as a cue that the neighbour relationship with
this router should not be reset. As the NSF-capable router receives signals from other routers on
the network, it can begin to rebuild its neighbour list.
After neighbour relationships are re-established, the NSF-capable router begins to resynchronize
its database with all of its NSF-aware neighbours. At this point, the routing information is
exchanged between the OSPF neighbours. After this exchange is completed, the NSF-capable
device uses the routing information to remove stale routes, update the RIB, and update the FIB
with the new forwarding information. OSPF on the router as well as the OSPF neighbours are now
fully converged.
Devices which have a single processor (e.g. 10720, 7200, 3800 series router etc) will not
experience RP switchover in the event of a fault; however, these devices can be NSF aware,
meaning that they run NSF software and can maintain session information with a peer device (a
Cisco 7600, 10000, or 12000 series router) following a switchover of the peer device.
BGP, OSPF, and IS-IS all have been enhanced with NSF-capability and awareness, which means
that routers running these protocols can detect a switchover and take the necessary actions to
continue forwarding network traffic and to recover route information from the peer devices. NSF
is a routing “knob”. Therefore it is configured under the routing portion of a routers configuration
router ospf 1
nsf
network 10.0.0.0 0.0.0.255 area 0
!
!
Figure 56 OSPF NSF configuration
router bgp 65501

bgp graceful-restart
!
Figure 57 BGP NSF configuration
It is recommended that Wateen Telecom deploy NSF as most of the infrastructure is singly
attached and some protocols may benefit from NSF in case of RP switchover. Please note that in
case of 7600 BFD is not sso aware and hence will cause routing protocol adjacencies to be reset
during RP switchover as documented below.
OSPF NSF requires that all neighbor networking devices be NSF-aware. If an NSF-capable router
discovers that it has non-NSF -aware neighbors on a particular network segment, it will disable
NSF capabilities for that segment. Other network segments composed entirely of NSF-capable or
NSF-aware routers will continue to provide NSF capabilities
7600 series platform has following restriction when deploying NFS & SSO:
• For NSF operation, SSO configuration is required on the device

MPLS Core Design
• NSF with SSO supports IP Version 4 traffic and protocols only.

• The Hot Standby Routing Protocol (HSRP) is not SSO-aware, meaning state information is
not maintained between the active and standby supervisor engine during normal operation.
HSRP and SSO can coexist but both features work independently. Traffic that relies on HSRP
may switch to the HSRP standby in the event of a supervisor switchover.
• Multiprotocol Label Switching (MPLS) is not suported with Cisco NSF with SSO; however,
MPLS and NSF with SSO can coexist. If NSF with SSO is configured in the same chassis
with MPLS, the failover performance of MPLS protocols will be at least equivalent to RPR+
while the supported NSF with SSO protocols still retain the additional benefits of NSF with
SSO.
• OSPF NSF for virtual links is not supported
• Bidirectional forwarding detection (BFD) is not SSO-aware and is not supported by NSF with
SSO

MPLS VPN Design
The section details the MPLS VPN design to be used by Wateen Telecom. Some overview of
MPLS VPN will be given while detailing the sample configuration to be used by Wateen Telecom
for its initial deployment phase.
Definition of a VPN
A VPN is a set of systems, which are allowed to communicate with each other. Such a system is
defined by a set of administrative policies. The path between two systems in a VPN, and the
characteristics of that path may also be determined (wholly or partially) by policy. Whether a
system in a particular VPN is allowed to communicate with systems that are not in the same VPN
is also a matter of policy.
In MPLS, a VPN generally consists of a set of sites, but it is also possible to apply different
policies to different systems that are located at the same site. A given set of systems may be in
one or more VPNs. A VPN can consist of sites or systems which are all from the same enterprise
"intranet,” or from different enterprises "extranet;” it may consist of sites or systems which all
attach to the same MPLS backbone, or to different service provider backbones.
The mechanisms for stating the policies are very general, and do not require that two sites in the
same VPN be associated with a common Route Distinguisher or a common Route-Target
extended community attribute.
Basic VPN Configuration

In MPLS VPN terminology the term PE (Provider Edge) refers to a router where the VRF tables
are defined and the CE (Customer Edge device) connects to.
All MPLS VPN configurations are done at the PE router. The rest of the network merely switches
labels and is not aware of the VPN structure and separation of customers. The core network is
referred to as the P network in an MPLS VPN. In Wateen Telecom’s case core consists of four
12816 series routers located in four core cities, namely Lahore, Karachi, Islamabad and
Faisalabad. For Wateen Telecom, the PE vrf ports are fanned out using ME3400 series access
switches. These access switches are physically not expected to be collocated at sites where PE
devices are located. Traffic from these access switches may terminate on PE devices via
microwave media which may be used for backhauling purposes. It has been advised that
physically this will be GigabitEthernet links from microwave device to access switch as well as
microwave device to PE device.
It is understood that last mile access to the customer will be via WiMAX technology, however it
has been advised by the customer that it is transparent to PE devices. From PE device perspective,
traffic will be mapped to different vlans, these vlans will be transported from access switches via
.1Q trunk. Scope of this document does not cover details of access methodology from PE device

MPLS VPN Design
to customer’s end including access switch. It is understood that these design details may be
covered by Cisco partner Motorola in a separate document.
In order to enable MPLS VPN there are several steps to be detailed:

• VPN Routing and Forwarding Table Definitions
• PE to CE Routing Definition (where required)
• MP-BGP Configuration
The following sections discuss each of these areas.
VRF Table Definitions

The VRF is the basis for providing VPN services in MPLS. There will be a VRF table defined on
every interface that connects to one of the edge devices in the network.
Each VRF definition includes the following components;
• A case sensitive name which identifies the VRF
• A Route Distinguisher to uniquely identify IPv4 routes coming in on this customer interface
• Import and export route targets defining export and import attributes for routes
• Optional route-maps to further define the granularity of route export/imports
VRF Name
The VRF name is simply a unique name used to identify the VRF and the routes it contains. It is
suggested the name always be lower case to avoid any potential compatibility issues with some
NMS platforms that may only work with lower case names.
Based on the requirements put forth by the customer and information provided thus far, following
vpns will be created at the rollout phase. It is understood that nms vpn will be created at each PE,
therefore this vpn can also be considered as a test vpn to aid with troubleshooting and ongoing
maintenance and support of the IP/MPLS network.
Table 10 VRF Naming
VRF Name Function

nms All WiMAX, Microwave, access switch devices which can
be managed via snmp
Voip All hosted voip traffic.
Based on information provided by the customer so far, voip vpn may be split int several vpns. For
further details please refer to access section of this document.
Route-Distinguisher
Each VRF must have a unique RD (Route Distinguisher) which will have the following format:
<AS>:<Unique Number>
Where:

MPLS VPN Design
<AS> is the 16-bit autonomous system number allocated to the network. This will be set to
the value 65501 and will also be used for all MP-BGP configurations
<Unique Number> A unique 32 bit number within the AS that is allocated by Wateen Telecom.
(Decimal maximum of 4294967295)
The situation where multiple sites of the same customer are connecting to the same PE router,
each sub-interface should use the same VRF definition as they would normally be part of the same
routing policy. This would give the customer sites peer access to each other via the PE.
Route-Targets
Route targets define which routes will be imported into a particular VRF and how the routes
exported into the VPN-IPv4 address space, which MP-BGP operates on, will be tagged.
The import and export values for route-targets can match the RD value for the VRF although as
mentioned earlier they are unrelated values. The RD uniquely identifies customer IPv4 routes and
the route-targets define import export policies for routes into and out of the VRF. Using the same
route-target and RD values simplifies the configuration and management of it.
The default route-target for Wateen Telecom VRFs will be equal to the RD. Additional route-
targets may be required given the route import & export policies needed.
Once the name, route descriptor & route-targets are assigned for a VRF it can be defined from the
command line in global configuration mode:
ip vrf nms
rd 65501:11
Figure 58 VRF Definition

RD number should be tracked per vpn.
Route-Maps
There may be occasion where just using route-targets may not provide sufficient granularity for
importing or exporting routes. Finer route control can be achieved by using route-maps which can
import or export routes based on many different characteristics. This can be accomplished by
using import map <route-map name> or export map <route-map name>
To identify a route-map for a particular VRF table, a recommended approach would be to precede
it with IN_ for import route-maps and OUT_ for export route maps. For example,
IN_Preferred_Gateway.
An example of vrf route-map is given below:

MPLS VPN Design
ip vrf svc
rd 65501:903
import map in-svc-imp
!
route-map in-svc-imp permit 10
match ip address 101
access-list 101 permit <ip|tcp|udp> network wildcard mask wildcard
Figure 59 VRF with Route-map
Assigning a VRF to an interface

When a VRF has been defined it can be assigned to multiple interfaces on the router. Given the
basic configuration above an interface would be assigned to the LAN VRF as follows:
Interface vlan10
description NMS VLAN 10
ip vrf forwarding nms
ip address 172.24.100.1 255.255.255.0
Figure 60 VRF Assignment
For Wateen Telecom each access vlan will be mapped to either a subinterface or SVI (switched
virtual interface) which will represent a routed interface for VLAN. This will in turn be assigned
to a vrf. Based on the information provided so far, each customer site will be mapped to a specific
vlan, this in turn would mean a subinterface or an svi can then be mapped to a specific vpn.
In the case, where ip telephony servers are connected to a vpn, it may make sense to configure
these as an svi, rather than configuring subinterfaces. Unless a specific requiremen exists, it is
generally suggested to use svi interface rather than suninterfaces.
When an interface is assigned a VRF, any existing IP address assignments on that interface are
automatically deleted and must be re-entered.
Please note that an interface can only belong to a single VRF.
MPLS VPN Services:

MPLS VPN allows for a great number of different routing policies to be defined within a given
VPN; and also for the sharing of routing information between different VPN or VRFs. These may
be broadly defined as follows:
• Intranet – Full Mesh VPN – How to provide access between different sites of the same
customer/service

MPLS VPN Design
• Extranet - Overlapping VPN – How to provide access between different customer/services

• Central Services VPN (Hub and Spoke) – How to define hub and spoke topologies for
centralized customers
• Network Management VPN – How to provide management access to CE, P/PE routers
It is understood that initial phase of deployment Wateen Telecom will be deploying intranet – full
mesh vpn form of services, this pertains to nms as well as voip vpn. Other forms of services are
mentioned as forward looking options.
The following sections will discuss these services.
Intranet – Full Mesh VPN

This is the classical model. It essentially consists of all sites of the same customer/service directly
peering with each other. From the CE device perspective, all of the sites appear one hop away
from each other. In order to assure that core topology is not visible to customer, following
command can be used in global configuration mode on all core (P) and edge (PE) devices:
no mpls ip propagate-ttl [forwarded]
Figure 61 Disable TTL propagation on PE router
By this configuration command, traceroute on host devices attached to PE will show other PE
devices to be just one hop away. In reality an IP packet may transit more than one core node,
though the host will not see this. It is also possible to use modified version of above mentioned
command where “forwarded” key word is added. With this option, while hosts see everything
one hop away, local troubleshooting is still possible on the PE router and all intermediate hops are
shown for traffic generated by the PE router.
All the sites of a VPN can communicate directly with each other in this model. All sites export
and import routes using the same RT.
Figure below shows a VPN with three sites. All the sites can communicate with each other. All
the sites use the same RT (65501:1) to export the routes and the same RT (65501:1) to import the
routes.

MPLS VPN Design
VRF RED
RD 65501:1 MPLS Network
RT export 65501:1
RT import 65501:1
PE1
RED
RED PE2 CE2
CE1 PE3
10.0.0.0/24 11.0.0.0/24
VRF RED VRF RED

RD 65501:1 RD 65501:1
RT export 65501:1 RT export 65501:1
RT import 65501:1 RT import 65501:1
CE3
12.0.0.0/24
RED
Figure 62 – Simple Full Mesh VPN Model
The design rules for a Simple – Full Mesh VPN are:

• Use 1 VRF per VPN (per PE).
• Use 1 RD per VPN.
• Use 1 Import / Export RT per VPN, identical to the RD.
ip vrf test
rd 65501:1
!
router bgp 65501
bgp router-id <ip-address>
<standard iBGP neighbour configuration>
!
address-family ipv4 vrf nms
redistribute <connected / static / rip>
no auto-summary
no synchronization
exit-address-family
Figure 63 Sample PE Configuration for any-any Topology
Same configuration can be repeated on all PEs and any to any connectivity will be achieved for all
devices with in a single vpn . Please note in this case only a single best path will be forwarded by

MPLS VPN Design
the route reflector. Please see PE-CE routing protocol section for further details on routing
protocol between PE-CE link.

MPLS VPN Design
Extranet VPNs - Overlapping VPNs

An overlapping VPNs scenario is useful for example for companies where central sites need to
participate in a corporate network and at the same time in an extranet. Another application could
be a company with several security-conscious departments that exchange sensitive data between
their servers. Figure below provides an example of an overlapping VPN scenario.
VRF RED
Rd 65501:1
RT exp 65501:1 - VPN RED
VP
VPN RT imp 65501:1 - VPN RED
RE
RED
CE1
10.0.0.0/24
10.0.0.0/24
MPLS Network CE2 VPN
VPN
RE
RED
RE
PE1 PE2 11.0.0.0/24

11.0.0.0/24
VP PE3
GREE CE1
GREEN
CE2 VPN
12.0.0.0/24
VRF BO TH
GREEN
GREE
Rd 65501:3
RT exp 65501:1 13.0.0.0/24
RT exp 65501:2
RT imp 65501:1
RT imp 65501:2 VRF GREEN
CE3 Rd 65501:2
Site in both RT exp 65501:2 - VPN GREEN
GREEN and RT imp 65501:2 - VPN GREEN
RED VPNs
14.0.0.0/24
Figure 64 - Overlapping VPNs Model
Obviously, the IP addresses that are being used in the sites that belong to overlapping VPNs need
to be unique in both VPNs.
The way to implement an overlapping VPNs infrastructure is to:
• Have each VPN use its own Import / Export RT that the sites that participate in that VPN will
export and import.
• Have the sites that participate in more than one VPN import routes with route targets from any
VPN in which they participate and export routes with route targets for all the VPNs in which
they participate.
The design rules for an Overlapping VPNs scenario are:
• Use 1 VRF per VPN and per overlapping VPN instance (per PE).
• Use 1 RD per VPN and per overlapping VPN instance.

MPLS VPN Design
• Use 1 Import / Export RT per VPN. The sites that belong to multiple VPNs will use multiple
Import / Export RTs.
Sample configuration is given below:
! VPN-PE1
!
ip vrf RED
rd 65501:1
ip vrf GREEN
rd 65501:2
!
router bgp 65501
< BGP configuration goes here >
VPN-PE2
!
ip vrf RED
rd 65501:1
ip vrf GREEN
rd 65501:2
!
router bgp 65501
VPN-PE3
!
ip vrf BOTH
rd 65501:3
router bgp 65501

Figure 65 Sample PE Configuration extranet vpn Topology

MPLS VPN Design
Central Services VPN (Hub And Spoke)- No Connectivity between

Spokes:
A central services VPN scenario is a VPN topology where there are client (Spoke) sites and server
(hub) sites. The server sites can communicate with the all of the client and server sites. The client
sites can only communicate with the server sites. This is a model which is applicable to ASPs and
various central services providers.
Obviously, this model assumes that client (spoke) IP addresses will not overlap, as this is a single
VPN where this should not be the case.
The way to implement a central services VPN infrastructure is to:

• Each client site should import a specific RT and export a different RT (these RTs need to be
different).
• Have each server site use their own Import / Export RT. In addition, each server site will
import the exported routes from the client sites, and export the server routes to the client sites.

MPLS VPN Design
The design rules for a Central Services Hub and Spoke VPN scenario are:
• Use 1 VRF per client site.

• Use 1 VRF for the server VPN (per PE).
• Use 1 RD per client site (per PE, since non-overlapping address space is assumed between the
client sites).
• Use 1 RD for the server VPN.
• Use 1 Import / Export RT for the server VPN (if multiple server sites).
• Use another Import / Export RT pair (Import and Export RT must be different) to mutually
export and import routes between the client sites and the server sites.
Figure 66 provides an example of a central services VPN scenario.
VRF SPOKE
Rd 65501:2
RT Exp 65501:2 MPLS Network
RT Imp 65501:1
VPN-PE1
Regional
CE 1
VPN-PE2 CE 2 Site2
10.0.0.0/24
10.0.0.0/24 VPN-PE3
Regional 11.0.0.0/24
VRF HUB VRF SPOKE
Site1
Rd 65501:1 Rd 65501:2
RT exp 65501:1 RT exp 65501:2
RT imp 65501:2 RT imp 65501:1
CE 3
12.0.0.0/24
Central
Site
Figure 66 – Central Services VPN Model
Example configuration is given below :

MPLS VPN Design
VPN-PE1
!
ip vrf SPOKE1
rd 65501:5
!
router bgp 65501
VPN-PE2
!
ip vrf SPOKE2
rd 65501:8
!
router bgp 65501
VPN-PE3
!
ip vrf HUB
rd 65501:6
!
router bgp 65501
Figure 67 Sample PE Configuration Hub and Spoke Topology
This sample configuration will not allow spoke1 to talk to spoke 2 even if they are on the same
PE, this is because the routes are imported into a different VRF, both VRF’s will receive copies of
the Hub routing table as they are importing the Hub routes (65501:6)
Central Services VPN (Hub and Spoke) – Connectivity between

Spokes via Hub
This VPN model is used for VPNs that have a Central Site and Regional Sites and the Central Site
controls the connectivity between the Regional Sites. For e.g. the Central Site might have a
Firewall and all the traffic between the Regional Sites must pass through the Firewall.
Figure below shows a VPN with one central site and two regional sites. Communication between
the two regional sites is through the central site. The central site is connected to MPLS-VPN
Network with two different VRFs in order to implement this solution. VRF HUB-IMPORT is
used to import all the routes from the Spoke sites and the second VRF HUB-EXPORT is used to
export all the routes (including the routes from the Central Site) to the Spoke sites.

MPLS VPN Design
Data forwarding Path from Regional Site1 to Regional Site2
VRF SPOKE1
RD 65536:1
RT export 65536:1
MPLS Network Regional
RT import 65536:2 VPN-PE2 CE2 Site2
VPN-PE1
11.0.0.0/24
CE1 VPN-PE3 VRF SPOKE2
10.0.0.0/24
RD 65536:2
RT export 65536:1
Regional RT import 65536:2
Site1 VRF HUB-IMPORT VRF HUB-EXPORT
RD 65536:3
RD 65536:4
RT import 65536:1 RT export 65536:2
CE3 CE4
Firewall
12.0.0.0/24
Central Site
Figure 68 - Hub and Spoke Model with Connectivity between Spoke Sites
All the Spoke sites will be assigned unique Route Distinguishers. This scheme will help to avoid
the problem of importing the PE-CE links in some of the spoke sites.
Sample configuration for the PE’s in this scenario is shown below:

MPLS VPN Design
This is a Sample configuration for hub-and-spoke with two VRF on the hub site.
VPN-PE1
ip vrf SPOKE
rd 65536:5
export map SPOKE
VPN-PE2
ip vrf SPOKE
rd 65536:8
export map SPOKE
VPN-PE3
ip vrf HUB-EXPORT
rd 65536:7
export map HUB-EXPORT
!
ip vrf HUB-IMPORT
rd 65536:6
export map HUB-IMPORT
!
router bgp 65501
Figure 69 Sample PE Configuration- Hub and 2 Spokes
Please note that Hub-PE site must have two separate interfaces connecting to the hub site.
Network Management VPN

The network management VPN is really just an extranet VPN in that IP access is provided
between VPN instances. It is primarily needed where managed CE service is offered and visibility
to each vpn CE is needed. Granted that Wateen Telecom may consider offering both managed and
unmanaged services, network management is discussed here for completeness sake. It is
understood that Wateen Telecom has not procured any management tools for managed CE service
offering. It is also understood that Wateen Teleocm will use global routing table for management
of PE and P devices. A separate nms vrf (providing any to any connectivity) is created for
managing WiMax and Microwave infrastructure. However should managed CE service offering
be included, a separate management vpn will be needed.
By creating a management VRF, all CE routers can be managed from a single spot, and by virtue
of the transitivity rule routing separation is guaranteed between CE routers; this means that the

MPLS VPN Design
common management VRF can reach all customer VRFs; but they cannot reach each other. The
management workstations will originate from this VRF and the VRF will have visibility of each
VRF to be managed. Conversely, each customer VRF must contain the address of the
management workstation to allow two way communications between the Management
workstation and the CE router.
The figure below illustrates the importing and exporting of routes from the management VRF.
Figure 70 Management VPN

The example below shows what the configuration might look like to achieve this result on a PE
router connecting to a customer VRF. Note that the management VRF need not be defined on the
customer facing PE – since the import of its routes is controlled via the route-target
ip vrf vpn_customer2
rd 65501:10
route-target import 65501:4 # Import routes from the Management VRF
rd 65501:10
route-target import 65501:4 # Import routes from the Management VRF
Figure 71 Management VPN Configuration on Customer PE

MPLS VPN Design
This configuration will import all routes from the management VRF into the customer VRF,
though with no control over exactly which routes are imported.
Similarly on the PE connecting to the Management VRF the configuration would be:
ip vrf management
rd 65501:4
route-target import 65501:10 !Import routes from the Customer VRF1
ip vrf management
rd 65501:4
Figure 72 Management VPN Configuration on Management PE
Again this configuration would mean all management hosts can access all hosts in the customer
VPN above.
Controlling route exports in NMS & Extranet VPN

If more control is required over which routes are accessible between VRFs then route-maps can be
used to control the exporting of routes. Route-maps are very useful if one wants to avoid
populating customer/services VRF with unnecessary management routes. This assists in
conserving memory and provides a basic form of security (no route means no access).
Each customer VRF will have its standard route-targets to import/export routes for their Intranets.
These are the first two route-target commands shown in the configuration for each VRF.
Each VRF can have a export map defined. The export map can set a specific route-target value
(referred to as an extended community attribute in BGP) for the Extranet route defined. For
example the PE router connected to Customer 2 will set the route-target value of 65501:106501 to
the route prefix of interest
The management VRF will import any routes that have the route-target value 65501:106501
using the command route-target import 65501:106501. Doing this allows the management VRF
to receive the Extranet route of Customer 2.
The VRF connected to Customer 2 will import the Extranet route of the management VRF in a
similar fashion.
By using route-targets, we can selectively import only the routes the CE needs to participate in an
Extranet. Individual host addresses could also be explicitly specified and exported using route-
maps much in the same way.
The configuration required to achieve this on the PE connecting to the customer is shown below.

MPLS VPN Design
rd 65501:10600
route-target import65501:10600
export map OUT_Extranet_Cust2 ! Export the Extranet subnet
route-target import 65501:107501 ! Customer 2 Extranet In
!
ip prefix-list Customer2_Prefix seq 10 permit <prefix/mask>
!
route-map OUT_ Extranet_Cust2 permit 10
match ip address prefix-list Customer2_Prefix
set extcommunity rt 65501:106501
Figure 73 Sample customer PE Configuration with export map
A similar configuration would be used at the PE connecting to the management VRF
ip vrf management
rd 65501:10
export map OUT_Extranet_Mgmt ! Export the Extranet subnet
route-target import 65501:106501 ! Customer 2 Extranet In
!
ip prefix-list Mgmt_Prefix seq 10 permit <prefix/mask>
!
route-map OUT_Extranet_Mgmt permit 10
match ip address prefix-list Mgmt_Prefix
set extcommunity rt 65501:107501
!
Figure 74 Sample PE Configuration with export map at Mgmt. PE

MPLS VPN Design
Customer Edge to Provider Edge Routing (CE to PE)
Routing Protocols
A routing protocol may be used between CE and the PE if the CE is a router and dynamic routing
information exchange between PE and CE is required.
A VPN Routing and Forwarding table (VRF) is associated with each CE interface on the PE and
contains the routing information associated with that site. A PE/CE routing protocol is necessary
so that the PE table can be populated with the customer’s routes.
The following routing protocols can operate between the CE and PE for an MPLS network;
• Connected
• Static
• RIPv2
• eBGP
• OSPF
• EIGRP
The above listed dynamic routing protocols have been modified to understand VRF tables by the
use of a feature called address families. Address families define the VRF contexts that the routing
protocol will operate in.
Note that the routing protocol that operates between the PE/CE is independent of any IGP that
may be operating in the VPN network. These routes may be redistributed into the PE/CE routing
protocol to populate the VRF. Therefore an edge device could be running EIGRP in the private
LAN, and redistributing to RIPv2 running between the PE/CE to populate the VRF.
It is important to understand that no special MPLS configurations are needed at the Customer
Edge devices. Only standard IOS routing commands are required. The Customer Edge is unaware
of the MPLS cloud it is connecting to and only requires the configuration of the appropriate
PE/CE routing protocol and redistribution commands into the IGP, if necessary.
Based on Wateen Telecom’s current requirements, it is understood that in most cases for small
customers only static routing will suffice. While for medium sized customers RIP and for large
enterprises BGP may be used as PE-CE routing protocol.
Therefor routing PE/CE routing protocols that may be used in Wateen Telecom’s environment can
be Connected, Static, RIPv2, and eBGP. As discussed while it is possible to offer all routing
protocols as potential PE-CE routing protocols, standardizing on specific protocols allows
customer to gain expertise and build an appropriate operations support model for these routing
protocol. Hence only connected, static, rip and eBGP will be discussed in the subsequent sections.
It is also suggested that in order to keep the changes in configuration of a VPN-PE to the
minimum, static routes may be used on the VPN-PE to propagate the VPN Routes where dynamic
routing information provides no additional value (e.g. single point of exit) or if a site has less than
3 subnets to advertise.

MPLS VPN Design
“Connected” Routing
“Connected” routing is not really a routing protocol as such, but it merely implies that the PE
router will inject the subnets of active interfaces directly connected to the VPN into the VRF
routing table.
The PE router can be set to redistribute the connected interface subnet addresses into the VRF test
for example; the routes should then be redistributed into MP-BGP to other PEs where vrf test is
defined.
!
router bgp 65501
address-family ipv4 vrf test
!
Figure 75 Sample configuration “connected” routes

The “redistribute connected” will be predominantly used in Wateen Telecom’s environment
where no CE exists (e.g. voip servers, NMS vpn for access to Microwave and WiMax devices
etc.)
It is also important to note that in order to achieve multiple equal cost routes in a vrf, following
command is required under each vrf context configuration in BGP process:
maximum-paths ibgp 2
This allows upto two equal cost paths to be available to each vrf. This command can be used
under each vrf context in BGP configuration where equal cost paths are desired. There have been
no customer requirements for such a scenario, it is documented here for situation where equal cost
paths are available and may be considered for use for achieving quicker convergence.
Static Routing
Static routing is normally used when a VPN site is small and the IP addressing of devices is
unlikely to change. The CE router would have a default route pointing in the direction of the
MPLS cloud, whilst the PE would require a similar static route inserted into the appropriate VRF
table associated with the CE remote interface.
For example, on a PE device following static route is added for vrf test :
ip route vrf test 192.168.1.0 255.255.255.0 172.16.1.2
Figure 76 static routes configuration in a VRF
The static route needs to redistributed as shown below:

MPLS VPN Design
Router bgp 65501

bgp router-id <ip-address>
<standard ibgp configuration here>
address-family ipv4 vrf test
redistribute static ! need to redistribute static routes for this vpn for
! other sites to have visibility
no auto-summary
no synchronization
exit-address-family
Figure 77 Configuring static routes redistribution into vrf context under BGP
Where 192.168.1.0/24 is a prefix on CE device attached to a PE. By adding this static route and
redistributing static routes in vrf configuration, other sites in test vpn can reach this network.
It is also possible to have vrf routes made visible into global routing table. This process for
sharing routes between a VPN routing table and the global routing table is known as “route-
leaking”.
configuration example for route-leaking is given below:
router ospf 1
redistribute static subnets
!
router ospf 2 vrf management
redistribute static subnets
!
ip route <network > <mask> interface NH ip address !interface for vrf, NH
(next hop address) is
needed on multipoint
interface
!
ip route vrf test <network> <subnet mask> next-hop address global
Figure 78 route leaking example
It is highly recommended that route-leaking should not be used until and unless absolutely
required.
RIP v2
RIP v2 can be used as the routing protocol on the CEs that do not support BGP or where the the
number of prefix are small.
Passwords can be used to authenticate RIP v2 sessions between VPN-PE and CE. The password
will be unique for each VPN.
Sample configuration is given below:

MPLS VPN Design
router rip
!
address-family ipv4 vrf cust1
network 172.16.0.0
no auto-summary
version 2
exit-address-family
For small sites (sites that do not require the complete routing table of the VPN) and stub sites
(sites that have only one network behind the CE router), RIP updates from the VPN-PE to the CE
can be suppressed. (passive interface on the VPN-PE sub-interface) Default route pointing to the
VPN-PE should be added to the routing table on the CE.
eBGP
Private AS Numbers (64512 to 65535.) can be used for the BGP session towards the CEs where
the CE does not have a globally assigned AS number.
The same AS Number can be used for all the sites of a single VPN to conserve the number of AS
Numbers this also allows for VPNs that have more than 1024 sites. AS-override will be used on
the VPN-PEs in order to reuse the same AS Number for all the sites.
In order to prevent loops, a BGP extended community Site-Of-Origin (soo) should be used to
identify each site. soo has the same format as the RD or the RT (X:Y) and uses 64 bits. Each site
must be assigned a unique soo.
The following design rules will apply to the eBGP session between the VPN-PE and CE:
1. Private AS numbers will be used for the BGP session on the CE Router
2. as-override will be used to conserve the private AS numbers.
3. Each Customer site belonging to a Central ServiceVPN will be assigned a unique SOO
attribute.
4. soo will be used to detect and prevent loops for multi-homed sites.
5. All the eBGP timers will be left to their default value in this phase of the project.
6. Passwords will be used to authenticate eBGP session between VPN-PE and CE. The
password will be unique for each VPN.
7. Peer-Groups will be used on the VPN-PE routes for eBGP sessions with CE Routers that
belong to the same VPN. This will help to reduce the size of the configuration and also
help in trouble-shooting.
The following IOS commands shows the configuration on a PE router that uses BGP as the
Routing Protocol with the CE. This configuration also uses as-override and soo. The soo in this
example is set to 65536:4

MPLS VPN Design
address-family ipv4 vrf vrf1

redistribute rip metric 1
neighbour <CE-ip address> remote-as 65001
neighbour <CE-ip address> password vrf1
neighbour <CE-ip address> update-source Serial3/0:0
neighbour <CE-ip address> activate
neighbour <CE-ip address> as-override
neighbour <CE-ip address> route-map VRF1-SOO in
no auto-summary
no synchronization
exit-address-family
route-map VRF1-SOO permit 10

set extcommunity soo 65536:4
Figure 79 sample eBGP configuration
Access to Wateen Telecom IP/MPLS Core:
On the 7600 PEs, separate SVI interfaces (mapped to different customer/services) will be
configured for each vpn. The vpn can be L2vpn or L3vpn. Alternatively SVI can belong to global
routing table (internet access).
Assuming only several VLANs for MPLS VPN and global routing, multiple interfaces will need
to be configured.
Vlan Dot 1Q interface or SVI

Vlan A ip address in global routing table for internet access
Vlan B ip address for NMS, this VLAN will have all devices which are
manageable and belong to nms vpn
Vlan C ip address for voip customers/servers.
Vlan D ip address for customer vpn1
Vlan E Possible L2 VLAN for interserver communication for voip servers etc.
VLAN Allocation
Different VLAN ranges can be reserved for different services on 7600 PE. Preliminary
assignment as provided by the customer is given below.
Table 11 Customer VLAN Assignment

MPLS VPN Design
VLAN Number Total Assigned to Customer

VLANs
1 1 System use, not recommended for customer
usage
2-460 459 Allocated for IMS
461 - 960 500 Business Data
1000 1 WiMAX Management VLAN
1001 1 Microwave Management VLAN
1011-2510 1500 Business Internet Subscribers
2511-2750 240 Business Voice Subscribers
3011-3250 240 Residential Internet Subscribers
3301-3500 200 Future Expansion
3501-4094 500 Reserved for System Internal Use
Customer has been advised to allocate a range of vlans set aside for internal use, coupled with
system configuration to assure internal vlans are allocated in descending order. It is also advised
that customer consider overall system scalability guidelines including a limit of 2K svis and
recommended maximum number of 1K vrfs along others.
Customer has devised an allocation scheme for NMS (management) VLAN and voip vlans. This
allocation is separately documented by the customer in a document called “VLAN-VPN Map
140806.pdf” and should be referenced independently.
The vlan allocation scheme uses non-overlapping vlans for regional PoPs.
Customer has also requested following vpns to be configured everywhere for voip network and
management vpn. Suggested vpn names are given below.
Table 12 VoIP VPN Breakdown
Name VPN Name
SBC Access VPN sbc

VoIP Core Signalling ims
IMS+AS VPN
VoIP Core Class 4 VPN classfour
VoIP Core HTTP VPN vchttp
VoIP Core Management vcmgmt
VPN
Access Infra Nms
Management

MPLS VPN Design
It is understood that SBC (session border controllers) are dual homed systems which will
participate in both sbc and ims vpns.
Interaction between these vpns is described by the customer is given below:
• IMS+AS VPN should be able to communicate to Class 4 VPN
• Management VPN should should be able to communicate to Class 4 VPN
In the absence of any further details, it may make sense to collapse three vpns into a single vpn. It
is expected that futher details will be provided by the customer. Connectivity requirements for all
vpns are any to any connectivity.
Following IP addressing scheme is understood for WiMAX AP and subscriber Module within
nms vrf context:
Table 13 NMS VRF Addressing
Region Network Allocation for AP-SM

Lahore 10.128.0.0-10.135.255.255
Islamabad 10.136.0.0-10.143.255.255
Faislabad 10.144.0.0-10.151.255.255
Karachi 10.152.0.0-10.159.255.255
It is understood that these subnets will be assigned in nms vrf to all access infrastructure
elements.
Similary allocations for other VPNs are provided by the customer on regional basis:
Region Network Allocation for SBC

Access
Lahore 10.16.0.0 - 10.31.255.255
Islamabad 10.32.0.0 - 10.47.255.255
Faislabad 10.48.0.0 - 10.63.255.255
Karachi 10.64.0.0 - 10.95.255.255
It is understood that customer will use same ip addresses in global routing table as well as in sbc
access vpn. This requirement is stated by the customer and per the customer driven by the
requirement of customer edge device.
Region Network Allocation for VoIP Core

HTTP
Lahore 192.168.0.0 - 192.168.15.255

MPLS VPN Design
Islamabad 192.168.16.0 - 192.168.31.255
Faislabad 192.168.32.0 - 192.168.47.255
Karachi 192.168.48.0 - 192.168.63.255
IP addressing scheme for these vpns is documented by Motorola in a separate document titled: “IP
Summary 140806.pdf” and should be referenced for all vpn assignments, a subset of which is
shown above.
Configuring HSRP on a VRF interface

To provide high availability for voip servers, it has been requested that HSRP should be
configured on vlans where these servers reside and redundant routers are available. This is
currently applicable to core sites i.e. Lahore, Karachi, Islamabad and Faisalabad. All other sites
have single PE routers and hence HSRP configuration will not be required.
An HSRP group for each vlan/subnet will be created. HSRP provides a default gateway address to
the hosts on a segment which is a virtual mac-address. PEs that are participating in an HSRP
group communicate to each other via a multicast UDP based hello packet, these hello packets are
sent to multicast address 224.0.0.2 using UDP port 1985.
During the startup, or through the use of the priority and preempt commands, one of the routers is
chosen to be the active router and the second router is designated as the backup. If the backup
router fails to receive the hello packet of the active router, the backup router assumes control of
the virtual MAC and the protocol addresses. By default, this equals to three hello packets from the
active router having been missed. The HSRP hellotime timer defaults to 3 and the holdtime timer
defaults to 10. These values can be tuned to sub-second values, however it is suggested that
initially these values be set to 1 and 3 seconds and then reduced further if needed.
For design specific to Wateen Telecom, it has been advised that hosts (voip servers, either single
attached or dual attached via nic teaming) will connect directly to 7613 PE devices i.e. there is no
access switch providing distinction of access and distribution layer. There will be a dedicated link
between two 7613 PEs. This link will provide layer-2 connectivity between two switches for
specific access vlans on which voip servers will connect.
In this scenario, the HSRP configuration will track the priority depending on uplink availability,
should both uplink fail on active HSRP 7613 PE router, the virtual mac-address ownership will
automatically move to the backup 7613 PE router. Default priority of an HSRP router is 100, and
loss of link defaults to decrement of priority by 10.
For Wateen Telecom active router will have a priority of 120, since the PE is dual attached to the
single 12816 core router, loss of one link will not switch over virtual mac address ownership to
the backup router. In case when both uplinks are lost, HSRP priority will be decremented to 95 so
the backup router will takeover the virtual mac-address ownership and become the active router.
Once the uplink on active HSRP router is restored, the virtual mac-address ownership will be
restored back to the active router.
HSRP example configuration templates for both active and backup PE routers are shown below:

MPLS VPN Design
!
interface ethernet2/0
description Access Segment for voip server
bandwidth 1000000
ip vrf forwarding voip
ip address <interface ip address> <subnet-mask>
standby 2 ip <virtual ip-address>
standby 2 priority 120
standby 2 timers 1 4
standby 2 preempt
standby 2 track ethernet0/0 <uplink 1, priority decrement default of 10>
standby 2 track ethernet4/0 15 <uplink2, priority decrement of 15>
Figure 80 HSRP configuration on Active PE router
While the configuration on Backup PE router will be as given below:
!
description Access segment for voip servers
bandwidth 1000000
ip vrf forwarding voip
ip address 172.24.5.2 255.255.255.0
standby 2 ip 172.24.5.254
standby 2 timers 1 4
standby 2 preempt
Figure 81 HSRP configuration on backup PE router
A second scenario can be considered where yet another dedicated link is configured between two
7613-PE devices (via GE-SPA ports), this will be a layer-3 link will be included in IGP updates
and will provide reachability to a PE in case when both uplinks have been lost. Should this
scenario be chosen, we can potentially do away with HSRP link tracking requirements, as we will
always have an uplink path either directly to 12816 or via the adjacent 7613-PE device. The
HSRP configuration will be stay the same, except no uplinks will be tracked. The added benefit of
this approach is that only time HSRP timers kick into failure condition detection is when a
complete node is lost. This approach is preferred, however it adds complexity to the design and
uses additional ip addresses, precisious WAN ports etc.
The HSRP group assignment is typically done with odd vlans active on one PE e.g. PE1 while
even VLAN active on PE2. This is standard practice to distribute traffic and VLANs across
distribution routers (in our case these routers are also Provider Edge devices) from access
switches.

MPLS VPN Design
Figure 82 – VoIP Server Connectivity in Main PoP
HSRP pre-empt delay can also be tuned best on the overall boot time of the host to avoid
possibility of having hsrp become active before the 7600 is ready ( Layer-3 with IGP
convergence) . This however needs to be tuned with some real life data and can be set using
following command: “standby 1 preempt delay minimum <sec>”
Overall access architecture (in terms of access to the customer) is lead by Cisco partner, Motorola.
It is therefore not covered in this document and is considered out of scope for the purpose of this
low level design. However based on discussion with Wateen Telecom and Motorola, following is
understood to be a high level representation of access network.
Following are generic recommendations based on above mentioned scenario. It is suggested that
customer consider these recommendation within the context of access architecture and only apply
relevant features/configurations where deemed appropriate. It has been advised that Cisco
Systems should consider access as a simple Ethernet link, specifically a .1Q trunk terminating on
a PE device via Gigabit Ethernet links.
Set VTP mode Transparent

It is higly recommended to use transparent mode for VTP (vlan trunking protocol). The global
configuration command: “vtp mode transparent”

MPLS VPN Design
Enable PortFast on trunk ports

While port fast is typically is used for access ports, as long as it is assured that in PoPs there is no
spaning-tree loop, this feature can move a port from blocking state to forwarding state very
quickly, this should be configured for all trunk ports.
Portfast can be configured using following commands:
spanning-tree portfast trunk
Figure 83 Portfast on a Trunk
It is required that we assure no spanning tree loops exist. This implies no layer2 loop when
connecting PE devices to access infrastructure.
Enable bpdu guard

For remote or satellite PoP 7609 PEs, assuming there is no spanning tree required on access ports
and no device running spanning tree will be plugged into those, it is recommended to use bpdu
guard on access ports only if these assumptions are valid..
Bpdugaurd can be enabled using following command on a per port basis:
spanning-tree bpduguard enable
Figure 84 Enabling Bpduguard
Enable UDLD
UDLD in aggressive mode should be configured when connecting to access switch infrastructure
to deal with unidirectional link failures. Following syntax can be used to enable UDLD on an
interface:
udld port aggressive
Figure 85 Enabling Udld
UDLD should be enabled on 7600 PE ports connecting to access switches for unidirectional link
failure.
Defining what VLANS are allowed on a trunk

When configuring a switchport as a 802.1q trunk, by default all VLANS will be allowed. Both
from a security as well as from a scalability point of view it is very important to clear out VLANs
that are not needed on a specific interface.
This goes in general especially for dot1q trunk connections towards customers

MPLS VPN Design
There are a couple of ways to change what VLANs are allowed on a trunk:
PE(config-if)#switchport trunk allowed vlan ?
WORD VLAN IDs of the allowed VLANs when this port is in trunking mode
add add VLANs to the current list
all all VLANs
except all VLANs except the following
none no VLANs
remove remove VLANs from the current list
Figure 86 Remove vlans from a trunk
When adding new VLANs to an existing trunk it is very important to use the “ADD” keyword. If
that is not done, all currently allowed VLANs will be removed from the allow list.
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-299,350,401-4094
switchport mode trunk
no cdp enable
!
PE(config-if)#switchport trunk allowed vlan except 1300-3500

!
PE(config-if)#do sh run int gig 1/0/1
Building configuration...
Current configuration : 195 bytes

!
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-1299,3501-4094 Å-Note the change
switchport mode trunk
no cdp enable
end
Figure 87 Adding vlans to a trunk
Enhance VLAN Security

It is also recommended to change the native vlan to an unused vlan enhance vlan security. This
can be accomplished using the interface command: “switchport trunk native vlan <unused vlan
number>” .
An alternative to this command is to assure that 7600 admits only 802.1Q tagged frames on
802.1Q trunks, dropping any untagged traffic, including untagged traffic in the native VLAN.
This can be accomplished with global command: “vlan dot1q tag native”. Care must be taken
when using this command, as

MPLS VPN Design
• At least one VLAN must be available for native VLAN tagging (vlan dot1q tag native
option). If all the available VLANs are used and then trying to enable the vlan dot1q tag
native option, the option will not be enabled.
• By enabling this command on 7600 (SP core switch), tag native VLAN egress traffic and
drop untagged native VLAN ingress traffic
• If this option is enabled on one switch and disabled on another switch, all traffic is
dropped; all access switches must have this option configured the same on each switch.
For simplicity and to avoid any interoperability/misconfiguration issues (at the access layer), it is
recommended to use first option, i.e. dedicating an unused vlan as native vlan in each trunk port.
Enable Host Macro for Host Deivces

Finally for ethernet ports connecting directly to hosts (e.g. servers) on 7600 platform following
additional macro is recommended:
“switch port host” on the interface connecting directly to a host.

MPLS Layer 2 VPN Design
The section details the Ethernet over MPLS design to be used by Wateen Telecom. The basic
premise is that a customer will be provided layer-2 connectivity across Wateen Telecom’s
IP/MPLS backbone core to other sites of the same customer. This customer is not interested in
Layer-3 vpn services (i.e. doesn’t peer with serive provider’s PE device) and instead is only
interested in building a layer-2 pipe to its multiple destinations. This form of connectivity is point
to point. i.e. an Ethernet link is provided between any two sites of the customer using a tunneling
mechanism by Wateen Telecom’s IP/MPLS core.
AToM overview
Virtual Private Wire Services (VPWS) allow the carriage of common layer 2 protocols across an
MPLS network in a point-to-point topology. This includes layer 2 protocols such as Ethernet,
Frame Relay (FR) etc. This service is commonly referred to as a Pseudo Wire service as it
provides a virtual direct neighbouring between CE devices. VPWS is enabled in Cisco PE routers
by the feature Any Transport over MPLS (AToM).
Any Transport over MPLS (AToM) provides a common framework to encapsulate and transport
supported Layer 2 traffic types over an MPLS network core.
AToM supports the following like-to-like transport types:
• ATM AAL5 over MPLS
• ATM Cell Relay over MPLS
• Ethernet over MPLS (VLAN and port modes)
• Frame Relay over MPLS
• PPP over MPLS
• HDLC over MPLS
How Any Transport over MPLS Works:

Figure 88 AToM overview
Bi-directional Label / VCID mapping exchange
The following process shows a Layer 2 packet traveling from Customer Edge 1 (CE1) across the
MPLS network to CE 2:
1- L2 transport route entered on ingress PE
2- PE1 starts LDP session with PE2 if one does not already exist
3- PE1 allocates a VC label for the new interface & binds to configured VCID
4- PE1 sends a label mapping message containing VC FEC TLV & VC label TLV
5- PE2 receives VC FEC TLV & VC label TLV that matches local VCID
6- PE2 repeats steps 1-5 so that bi-directional label/VCID mappings are established
Ethernet over MPLS

EoMPLS is an IETF standard-track protocol that allows SPs to incorporate Layer 2 VPN services
over an existing MPLS backbone. It leverages the MPLS network by mapping Ethernet services
directly onto the MPLS VCs and extends the Ethernet from the MPLS edge to a customer site.
Using EoMPLS, an SP can connect two disparate enterprise Ethernet networks over the SP cloud
and have them appear as a single logical Ethernet or VLAN domain. However, it should not be
confused with traditional LAN bridging. That is, EoMPLS does not perform any Layer 2 lookup
to determine if the destination MAC address resides on the local or remote segment and does not
perform any Layer 2 address learning, like traditional LAN bridging does. Instead, EoMPLS is
more analogous to the transport of Layer 2 Frame Relay packets through an ATM backbone.
Some of the key characteristics of EoMPLS are:

• Establishing an EoMPLS circuit requires that the SP customer be assigned a specific physical
port on a Label Edge Router (LER) device, such as the Cisco 7600 (a physical port here
means a sub-if). The identification of that physical port is a critical element in the binding of
the MPLS Label assigned to the customers EoMPLS VC.
• A customer may have more than one EoMPLS VC per physical port as long as the Ethernet
traffic transmitted from the customer site to the PE device can be distinguished as having
specific 802.1Q headers for each EoMPLS VC by the PE device. This requires coordination
between the SP and customer in the provisioning of the EoMPLS service. Different customers

cannot use overlapping VLAN ranges, unless customer-provided 802.1Q headers are
themselves tunneled into SP-only significant 802.1Q headers.
• EoMPLS VCs as defined in the Martini draft are point-to-point transmissions only.
• Traffic sent between the imposition/disposition routers (between LERs) over an EoMPLS VC
takes the same path across the IP/MPLS backbone. The Label Switched Path (LSP) may
change due to routing changes inside the provider network.
• Adding or removing a point-to-point Layer 2 VC requires configuration of the two VC
endpoints (at the two LERs), as each VC is unidirectional. Provisioning a VC involves
defining an endpoint of a Layer 2 VC at each of the VLAN interfaces at the PE router on the
interface that connects to the CE.
• The two LERs at the ingress/egress points of the IP/MPLS backbone (the N-PE routers) are
the only routers with knowledge of the Layer 2 transport VCs. All other LSRs have no table
entries for the Layer 2 transport VCs. This implies that only the PEs requires software with
EoMPLS functionality, making it very easy to scale an EoMPLS-based network.
The technical requirements for EoMPLS LERs are:

• Targeted LDP sessions between peers—The ingress PE needs a tunnel label from its
downstream IGP neighbor to route the packets it receives from the ingress interface of the
ingress PE to the egress PE. The ingress PE and its IGP neighbor are local label distribution
peers and maintain a LDP session over which tunnel labels are distributed. They must all
have the FEC that contains the host route for the egress PE and the tunnel label is bound to
that FEC. Additionally, one targeted LDP session is required between the ingress PE and
egress PE, which are remote label distribution peers, to exchange VC labels.
• Two level labeling—The Layer 2 transport service over MPLS is implemented through the
use of two-level label switching between the edge routers. The label used to route the packet
over the MPLS backbone to the destination PE is called the tunnel label. The label used to
determine the egress interface is referred to as the VC label. The egress PE allocates a VC
label and binds the Layer 2 egress interface to the VC in question, then it signals this label to
the ingress PE via the targeted LDP session.
• Label imposition/disposition—When the Layer 2 PDU arrives at the ingress interface of the
ingress PE, the router must perform label imposition and switch the packet to the appropriate
outgoing MPLS interface, which routes the packet to the egress LER for the VC in question.
When the egress PE receives the packet, it receives the packet with only the VC label because
its neighbor (known as the penultimate router) pops the tunnel label prior to forwarding the
packet. The egress PE uses the VC label to perform disposition and switch the packet to the
appropriate egress interface.
• Ethernet over MPLS works by encapsulating Ethernet PDUs in MPLS packets and
forwarding them across the MPLS network. Each PDU is transported as a single packet.
There are two ways to configure Ethernet over MPLS :
o Subinterface (VLAN mode), which transports Ethernet traffic from a source 802.1Q
VLAN to a destination 802.1Q VLAN over a core MPLS network.
o Port mode, which allows a frame coming into an interface to be packed into an
MPLS packet and transported over the MPLS backbone to an egress interface. The
entire Ethernet frame is transported without the preamble or FCS as a single packet.
802.1Q VLAN Range

The IEEE 802.1Q standard provides for 8 bits in the Vlan Id field giving 4096 individual Vlans
per interface. Most of these are available for customer use however there are some reserved Vlans
Table 14 802.1q Vlans

Vlan Use
0 Reserved by Standard
1 Default Vlan (Native)
2 – 1001 Usable Vlans
1002 – 1005 Cisco defaults for FDDI
and Token Ring
1006 - 4094 Usable Vlans
4095 Reserved by Standard
It should not be assumed that if a device supports the full range of Vlan ID’s that it also supports
their use concurrently. Most access ethernet switch devices support a smaller number of
concurrent active Vlan ID’s. The exact number is device dependent. In case of 7600, following
should be observed:
o It is possible to use Vlan 1 , but it cannot be deleted. It is highly recommended that vlan
1 should not be used for data traffic (customer or internal).
o Extended system ID should be enabled to use extended range of VLANs ( 1006-4094)
using the command: “spanning-tree extend system-id” in global configuration mode.
o While it is understood that there are no switches running Catalyst OS in the access side,
please note that switches running the Catalyst operating system do not support
configuration of VLANs 1006-1024.
o 7600 uses internal vlans for L3 interfaces etc; these should be accounted for when
determining maximum vlans available.
Following components are needed when configuring a pseudowire:
VC ID
This is a unique identifier that is shared between the two PE routers. The vcid is a 32-bit
identifier.
The combination of the peer-router-id and the VC ID must be a unique combination on the router.
Two circuits cannot use the same combination of peer-router-id and VC ID.
!
xconnect 192.168.2.4 10 pw-class port-to-vlan Å 10 is used as vcid here
Figure 89 L2vpn configuration
VC Label
AToM uses one MPLS label to identify the AToM VCs (VC label). Therefore, the minimum
MPLS label stack is 1 for directly connected AToM PEs, which are PE routers that do not have a
P router between them
Directed LDP Session

A directed LDP session will be setup automatically between the source and destination routers
when the first Virtual Circuit between that pair of PE routers is configured. This session will

carry all VC Label (Tunnel) information between the pair for all subsequent VCs. Please see LDP
section for configuration details.
MTU Issues
MTU values at each end of an EVC must match for the EVC to transition to UP state.
Ethernet Relay Service (ERS) – VLAN Mode
For this service type, point-to-point circuits between customer routers (CEs) are established
through an Ethernet over MPLS (EoMPLS) core. The customer only needs one physical
connection to the Service Provider (SP), but is able to direct traffic to different destination routers
via the appropriate VLAN-ID (a sub interface will be used on the 7600 for each vlan).
In an EoMPLS-based Layer 2 VPN implementation, a customer’s point-to-point traffic based
upon a specific VLAN-ID is mapped to an MPLS label switched path where it extends across the
MPLS backbone. If the customer access port is an 802.1Q trunk, in the PE there will be as many
EoMPLS VCs as there are 802.1Q VLAN IDs. In other words, one physical Ethernet customer
UNI can have different destinations for each point-to-point connection identified by a VLAN-ID.
In each PE a mapping is made between each VLAN and Virtual Circuit (VC) which is tunneled
through a Label Switched Path (LSP). As specified in the IETF Martini draft, EoMPLS
technology provides point-to-point connections only. As long as the system is provisioned so that
there are only two endpoints to the circuit, traffic will be properly carried across the network.
For this Ethernet Relay service, the customer CPE equipment is ideally a router. This offers the
benefit of having only one MAC address on each end of the point-to-point circuit; i.e., only one
sub interface or svi has to be provisioned per neighboring router.
It should also be noted that ERS gives the possibility of creating arbitrary topologies as per
customer request, e.g. full mush or partial mesh.
The ERS service is less suitable if the customer has no CE router or has the requirement of
connecting directly the SP’s CPE a server or host Layer 2 network with no routing capability.
VLAN mode (subinterface)

Each interface is processed individually for example on the below configuration, two
subinterfaces are transported over MPLS and the third one terminates on the PE router.
The PE would only forward Ethernet PDU to the other PEs through the MPLS network. It is not
possible to send traffic over the Vlan on the local PE.

Figure 90 Ethernet over MPLS
Sample of configuration on PE-1
! interface gigabitethernet x/y.xx

no ip address
encapsulation dot1q xx
xconnect <IP addr of remote PE-2> <VC-ID-1> encapsulation mpls
!
interface gigabitethernet x/y.yy
no ip address
encapsulation dot1q yy
xconnect <IP addr of remote PE-3> <VC-ID-2> encapsulation mpls
!
interface gigabitethernet x/y.zz
encapsulation dot1q zz
ip vrf forwarding <vrf name>
IP address <IP address / submask
!
Figure 91 Interface based EoMPLS configuration
*VC ID must be unique and the same on both sides
This mode of operation does not require any OSM or SIP line cards, and can be used with a
regular switched interface as the core facing interface.
Even though the VLAN IDs are configured as subinterfaces, VLAN ID numbers must be unique
per the whole device. It is impossible to use the same VLAN ID on separate trunks, and it is
impossible to create an SVI interface with the same VLAN number.
VLAN mode (SVI/Vlan Interfaces)

In previous section we have described how to use L2VPN with subinterfaces on a trunk port. This
mode does not allow performing local layer 2 switching between a VLAN configured on the same
trunk while the trunk uses subinterface with other VLANs.
In order to allow local VLAN switching, a different approach has to be taken, and the use of SVI
interfaces (interface Vlan) should be introduced.
In this approach the trunk port is configured a regular switchport, and dot1q trunking is enabled
on it.
Every vlan has to be created in the vlan database and an “interface vlan” would be created. All the
relevant configuration would be done on the “interface vlans”:

• For L3 VPN, the “ip vrf forwarding” command would be configured on the SVI.
• For L2 VPN, the “xconnect” command would be configured on the SVI. No ip address is
configured in such a case.
• For locally layer 2 switched vlans there is no need for configuring an SVI interface, but a vlan
has to be created in the vlan database by using the “vlan <vlan-id>” command.
The configuration of the EoMPLS PW would be using SVIs on the PE:
vlan 2000
name customer1
interface Vlan 2000
no ip address
xconnect <remote PE> <VC ID> encapsulation mpls
!
Figure 92 Vlan based EoMPLS configuration
In order for the above configuration to work the core facing uplink link MUST be over a WAN
connection of a SIP line card. The CE facing interface can be a regular switch line card.
When configuring the xconnect command on a SVI interface, the following error message may
appear:
%C6K_MPLS_COMMON-3-L3_CONFIG_NOT_RECOMMENDED: LAN interfaces have
MPLS configured.
Do not configure xconnect on interface vlans.
This is a warning message which is used to make sure the uplink is a SIP enabled interface. It
would appear if there is any interface with “mpls ip” enabled which is not a SIP/OSM interface.
Similarly layer 3 vpns would be configured using an SVI interface as indicated in access section.
This method of configuration is recommended for Wateen Telecom in case that local layer 2
switching between two branches of a customer e.g. site A and site B is required. This will only
work if uplink (i.e. core facing interface) is on a SIP linecard. This also means that in case of both
uplink failures at core PoPs, interconnect between two 7613s should be on a SIP line card (instead
of LAN based card (WS-67xx series)) otherwise this configuration will no longer work.
It is recommended to use this method from day one if local switching is planned, as migrating
between the two different methods would require a service interruption. It is also recommended to
keep internconnect between two 7613 chassis to be on SIP-line cards if this method is chosen.
Ethernet Wire Service (EWS) – Port Mode

The important characteristic of this service is that it is aimed at emulating the behavior of a wire.
Also this service can be classified to fall under the VPWS category. Any number of customer
VLANs can be tunneled together through a single POINT-TO-POINT circuit across an EoMPLS
core. The traffic is encapsulated in an SP VLAN, hence there is no danger of customers
overlapping VLANs. By using P routers in the MPLS core, the SP VLAN can be different on both
sides of the EoMPLS core, therefore the VLAN address space can overcome the 4094 VLAN-IDs
independency limitation. Again, to simplify the configuration, the SP will always connect the
customer through a U-PE prior to handing the traffic to a PE (N-PE).

The technique that allows a customer’s VLAN space to have some independence from the SPs
VLAN space is called Q-in-Q. This feature is set-up by configuring interfaces on the customers’
CEs as VLAN trunks. The corresponding interfaces on the SP PE edge are deliberately configured
as non-trunking with the provider VLAN IDs (native VLANs). As a result, the data traffic gets
VLAN tagged on both the CE and PE, and is therefore double-tagged within the SP’s switching
infrastructure. For this solution the SP network is not capable of looking into the Q-in-Q frame
and working with the double-encapsulated frame. Therefore all the frames with the same outer
VLAN tag are switched everywhere in the SP’s network. Frames that seem to be going to the
wrong VLAN are dropped when they reach the egress PE and the outer tag is popped. At this
point, unless there is a match with the inner Customer VLAN tag at that port, traffic is dropped.
The customers’ control traffic (namely CDP, VTP, and Spanning Tree PDUs) is tunneled through
the EoMPLS cloud by enabling the Protocol Tunneling feature on Q-in-Q ports.
If configured in conjunction with EoMPLS in the core, Q-in-Q may increase the VC’s space. It
can thus be seen to be a special case of the EMS service, however currently no true EWS service
is possible due to the inability of the QinQ network to emulate the transparency of a wire with
respect to protocols such as PagP, etc.
With port mode the full interface is forwarded on the remote PE through the MPLS backbone.
The PE would not be able to send any traffic by itself over the interface, and all traffic would flow
only over the PW.
Sample of configuration on the PE:
interface gigabitethernet x/y

xconnect <IP addr of remote PE> <VC-ID> encapsulation mpls
!
Figure 93 Port based EoMPLS configuration
VC ID must be unique and the same on both sides
7600 restrictions
• Ensure that the maximum transmission unit (MTU) of all intermediate links between
endpoints is sufficient to carry the largest Layer 2 packet received.
• EoMPLS supports VLAN packets that conform to the IEEE 802.1Q standard. The 802.1Q
specification establishes a standard method for inserting VLAN membership information into
Ethernet frames.
• If QoS is disabled globally, both the 802.1p and IP precedence bits are preserved. When the
QoS is enabled on a Layer 2 port, either 802.1q P bits or IP precedence bits can be preserved
with the trusted configuration. However, by default the unpreserved bits are overwritten by
the value of preserved bits. For instance, if you preserve the P bits, the IP precedence bits are
overwritten with the value of the P bits. PFC3BXL or PFC3B mode provides a new command
that allows you to trust the P bits while preserving the IP precedence bits. To preserve the IP
precedence bits, use the no mls qos rewrite ip dscp command.
• EoMPLS is not supported with private VLANs.

• The following restrictions apply to using trunks with EoMPLS:

• To support Ethernet spanning tree bridge protocol data units (BPDUs) across an EoMPLS
cloud, you must disable the supervisor engine spanning tree for the Ethernet-over-MPLS
VLAN. This ensures that the EoMPLS VLANs are carried only on the trunk to the customer
router. Otherwise, the BPDUs are directed to the supervisor engine and not to the EoMPLS
cloud.
• The native VLAN of a trunk must not be configured as an EoMPLS VLAN.
• In PFC3BXL or PFC3B mode, all protocols (for example, CDP, VTP, BPDUs) are tunneled
across the MPLS cloud without conditions.
• ISL encapsulation is not supported for the interface that receives EoMPLS packets.
• Unique VLANs are required across interfaces. You cannot use the same VLAN ID on
different interfaces.
• EoMPLS tunnel destination route in the routing table and the CEF table must be a /32 address
(host address where the mask is 255.255.255.255) to ensure that there is a label-switched path
(LSP) from PE to PE.
• For a particular EoMPLS connection, both the ingress EoMPLS interface on the ingress PE
and the egress EoMPLS interface on the egress PE have to be subinterfaces with dot1Q
encapsulation or neither is a subinterface.
• 802.1Q in 802.1Q over EoMPLS is supported if the outgoing interface connecting to MPLS
network (the user facing interface) is a port on a Layer 2 card and the core facing interface is
on an OSM card.
• Shaping EoMPLS traffic is not supported if the egress interface connecting to an MPLS
network is a Layer 2 LAN port (a mode known as PFC-based EoMPLS).
• EoMPLS based on a PFC3BXL or PFC3B does not perform any Layer 2 lookup to determine
if the destination MAC address resides on the local or remote segment and does not perform
any Layer 2 address learning (as traditional LAN bridging does). This functionality (local
switching) is available only when using OSM and FlexWAN modules as uplinks.
• The AToM control word is not supported.
• EoMPLS is not supported on Layer 3 VLAN interfaces (It is possible to configure EoMPLS
over SVI interfaces, but the same SVI interface cannot become a Layer 3 interface at the same
time).
• Point-to-point EoMPLS works with a physical interface and subinterfaces.
VPLS / H-VPLS
VPLS
It is understood that with suggested release of code:12.2(18)SXF, Wateen Telecom cannot deploy
VPLS on their network. This is due to the fact that VPLS support for SIP-400 is not available in
12.2(18)SXF5 release. An upgrade to 12.2(33)SRA (internally codenamed cascades) is required
for this functionality to be supported. Wateen Telecom has agreed not to deploy VPLS at this
stage. Generic information and sample configurations are included for reference and potentially
forward looking requirement.
VPLS enables Ethernet multipoint services over a packet-switched network infrastructure. VPN
users get an emulated LAN segment that offers a Layer 2 broadcast domain. End users perceive

the service as a virtual private Ethernet switch that forwards frames to their respective destination
within the VPN. The next figure shows the logical view of a VPLS connecting three sites. Each
customer edge (CE) device requires a single connection to the network to get full connectivity to
the remaining sites. A multipoint technology allows a user to reach multiple destinations through a
single physical or logical connection, which requires the network to make a forwarding decision
based on the destination of the packet. Within the context of VPLS, this means that the network
makes a forwarding decision based on the destination MAC address of the Ethernet frame. From
the end customer's perspective, a multipoint service is attractive because fewer connections are
required to get full connectivity between multiple points. An equivalent level of connectivity
based on a point-to-point technology requires a much larger number of connections or the use of
suboptimal packet forwarding.
Figure 94 VPLS
In its simplest form, a VPLS consists of a collection of sites connected to a number of provider
edge (PE) devices implementing the emulated LAN service. A virtual switching instance (VSI) is
used at each PE to implement the forwarding decisions of each VPLS. The PE devices make the
forwarding decisions between sites and encapsulate the Ethernet frames across a packet-switched
network using an Ethernet virtual circuit (VC) or pseudo-wire. PEs use a full mesh of Ethernet
VCs to forward the Ethernet frames between PEs. VPLS relies on the same encapsulation defined
for point-to-point Ethernet over MPLS. The frame preamble and frame check sequence (FCS) are
removed, and the remaining payload is encapsulated with a control word, a VC label, and an
Interior Gateway Protocol (IGP) or transport label. VPLS has been initially specified and
implemented over an MPLS transport.
PEs would automatically populate the VSI with the forwarding information required to switch
frames within the VPLS. PEs acquire this information using the standard MAC address learning
and aging functions used in Ethernet switching. The VSI forwarding information is updated with
the MAC addresses learned from physical ports and from the virtual circuits. These functions
imply that all broadcast, multicast, and destination unknown MAC addresses are flooded over all
ports and VCs associated with a VSI. PEs use split-horizon forwarding on the VCs to form a loop-
free topology. In this way, the full mesh of VCs provides direct connectivity between the PEs in a
VPLS, and there is no need to use more resource-intensive protocols to generate a loop-free
topology (for example, Spanning Tree Protocol, or STP).

VPLS Configuration example
Figure 95 VPLS configuration example
Configuration on the PE
Creation of the virtual switch instances (VSIs) and associated VCs.
For PE 1
!
l2 vfi <PE-VPLS-A> manual
vpn id <ID number>
neighbor <PE2 loopback IP> encapsulation mpls
!
interface loopback <number>
ip address <IP address> <mask>
For PE 2
!
vpn id <ID number>
!
!
For PE 3
!
vpn id <ID number>

!
Configuration of the CE interface side (there can be multiple Layer 2 interfaces in a VLAN).
On all the CE
Interface <any Ethernet interface>

switchport
switchport mode dot1qtunnel
switchport access vlan <vlan number>
Configuration of the attachment circuit (VLAN) associated with the VSI and enablement of the
layer 2 VLAN instance.
On all the PE
Interface vlan <vlan number>
no ip address
xconnect vfi <PE-VPLS-A>
vlan <vlan number>

state active
Figure 96 VPLS configuration
VPLS Restrictions
The following restrictions pertain to all transport types under VPLS:

• Split horizon is the default configuration to avoid broadcast packet looping and to isolate
Layer 2 traffic. With split horizon, a packet coming from a WAN interface never goes back to
another WAN interface (it always get switched to a Layer 2 interface). Split horizon prevents
packets received from an emulated VC from being forwarded into another emulated VC. This
technique is important for creating loop-free paths in a full-meshed network.
• The Cisco 7600 series routers support a maximum of 60 peer PEs and a maximum of 15,000
VCs. For example, you can configure 15,000 VCs as 1,000 VFIs with 15 VPLS peers per
VFI.
The 60 peer PEs are distributed between the MPLS edge and the core; do not assume there are 60
peer PEs on each side.
• No software-based data plane is supported.
• No auto-discovery mechanism is supported.
• Load sharing and failover on redundant CE-PE links are not supported.
• The addition or removal of MAC addresses with label distribution protocol (LDP) is not
supported.

• On the Cisco 7600 series router, the Virtual Forwarding Instance (VFI) is supported only with
the interface vlan command.
7600 restriction
• The core facing interface must be installed on an OSM module
• The Customer facing interfaces are all Ethernet/ Fast Ethernet/ Gigabit Ethernet interfaces
based on Layer 2 Catalyst LAN ports

Dial Services
This section details Wateen Telecom’s Dial-up network infrastructure. The Dial infrastructure will be
used to provide the following services:
Remote access to MPLS VPNs
As a backup to MPLS VPN primary link
Backup internet connection
Before discussing Wateen Telecom’s Dial design specifics it is believed that the reader should be familiar
with certain technologies and protocols that are integral to the overall operation of dial services. The next
section will discuss each of these protocols and technologies in sufficient detail in order to facilitate
satisfactory understanding of Wateen Telecom’s Dial Services architecture.
Technology Primers
For the purpose of this section the terms NAS/LAC are used interchangeably with LAC, VHG/PE with
LNS and CAR with RADIUS.
PPP
PPP is a symmetrical peer-to-peer protocol used for transporting Layer 2 and Layer 3 traffic over point-
to-point links. There are three main components:
Encapsulation
Link Control Layer (LCP)
Network Control Layer (NCP)
First off, the datagram is encapsulated in PPP. Secondly, there is the Link Control Layer (LCP) part,
which allows for configuration options to be negotiated allowing Link Establishment and Network
Control Protocols (NCP), which are negotiated for each of the Layer 3 protocols that will run on the link.
During the life of a PPP session, there are four distinct phases the link goes though:
Link Establish
Authentication
Network Layer
Link Termination

Dial Services
As part of the Link Establish phase, PPP uses an LCP function that must be completed and declared open
prior to entering the Authentication phase (this is an optional phase dependant on end-user configuration)
and negotiating the opening of the Network Layer. LCP is also used to terminate the PPP link.
The Authentication phase is implementation-specific and not a mandatory requirement to move from
LCP to NCP. If negotiated and agreed during the LCP phase, then the remote peer must identify itself and
pass the agreed authentication method prior to PPP moving to Network Layer.
Network Control Protocol (NCP) negotiation ensures both peers agree on the characteristics of the Layer
3 protocol. In the case of IP, the Control Protocol is called IPCP. In addition to the negotiation between
peers, there is also an element of assignment. This is common with Windows-type remote access clients
who have no allocated IP address and rely on the Service Provider allocating on connection.
The Link termination phase can be entered anytime during the lifecycle of the call. LCP is used to deliver
the Termination request.
Layer 2 Tunneling Protocol
L2TP extends the Point to Point nature of PPP by providing an encapsulation method for sending
tunneled PPP frames, thereby allowing the PPP endpoints to be tunneled over a packet switched network.
This is most commonly deployed in Remote Access type scenarios using the internet to offer intranet type
services; a concept of a Virtual Private Network (VPN).
The two primary physical elements of L2TP are the L2TP Access Concentrator (LAC) and the L2TP
Network Server (LNS).
L2TP Access Concentrator (LAC)

The LAC is a peer to the LNS acting as one side of the tunnel endpoint. The LAC terminates the remote
PPP connection and sits between the remote and the LNS. Packets are forwarded to and from the remote
over the PPP connection and packets to and from the LNS are forward over the L2TP tunnel.
L2TP Network Server (LNS)

The LNS is a peer to the LAC acting as one side of the tunnel endpoint. The LNS is the termination point
for the LAC PPP tunneled sessions. This is used to aggregate the multiple LAC tunneled PPP sessions
and ingress into the Private Network.
There are two different message types used in L2TP.
Control Message
Data Message
L2TP passes Control and Data messages over separate Control and Data channels. The in-band Control
channel is used to pass sequenced control connection management, call management, error reporting and
session control messages. Initiation of the Control Connection is not specific to either the LAC or the
LNS rather the tunnel originator and receiver that has relevance in the Control Connection Establishment.
A shared secret challenge authentication method is used between the tunnel endpoints
Data Messages are used to encapsulate the PPP frames that are sent into the L2TP tunnel
L2TP uses the registered UDP port 1701; the whole L2TP packet is encapsulated within the UDP
datagram. As per normal UDP operation, the tunnel initiator selects an available UDP port and sends port
number 1701 to the UDP destination. In the reply, the destination port number is the same as the source
port number used in the incoming UDP header; the source port is set based on free port found. Once the
source and destination ports are established, they must remain the same for the duration of the tunnel. In
Cisco IOS the source and destination port numbers are always set to UDP port number 1701.

Dial Services
LNS Failover and Load Sharing

When Multiple Tunnel-Server-Endpoints are defined in a Tagged Attribute Group (TAG) the NAS/LAC
interprets this as equal cost load balancing. By using Tunnel-Preference – Attribute 83 – it is possible to
specify load balancing and failover within different priority groups. When the Tunnel-Preference values
of the different TAG's are the same, the Tunnel-Server-Endpoint of those groups are considered to have
the same priority. When the Tunnel-Preference values are higher (they have a lower preference) than
other attribute groups, their Tunnel-Server-Endpoint attributes will have higher priority values. When an
attribute group has a higher priority value, that attribute group will be used for failover in case the
attribute groups with lower priority values are unavailable for the connections.
AAA
AAA describes the access control and accounting requirements to allow control of who is allowed to
access the network and what services they are allowed to use and then how to report on that particular
user’s activity for the lifetime of that connection. Various methods are employed to perform these
functions and network security services. Commonly these are set up and administered on a centralized
database. To send the information to the centralized database and forward the response to the respective
network device a protocol is required.
Authentication
Provides a method of identifying users - login and password. Using challenge and response to allow and
refuse the connection. Authentication is the way a user is identified prior to being allowed access to the
network and network services.
Authorization
Provides a method for remote access control, including one-time authorization or authorization for each
service, per-user account list and profile, user group support amongst many others. AAA authorization
works by assembling a set of attributes that describe what the user is authorized to perform.
Accounting
Provides a method for collecting and sending security server information used for billing, auditing, and
reporting, such as user identities, start and stop times, executed commands (such as PPP), number of
packets, and number of bytes. Accounting enables you to track the services users are accessing as well as
the amount of network resources they are consuming.

Dial Services
Figure 97 AAA session flow
RADIUS
The RADIUS protocol is an access server authentication and accounting protocol. Communication
between a client and the RADIUS server is based on the User Datagram Protocol (UDP). Generally, the
RADIUS protocol is considered a connectionless service. Issues related to server availability,
retransmission, and timeouts are handled by the RADIUS-enabled devices rather than the transmission
protocol. RADIUS is a client/server protocol. The client passes user information to the configured
RADIUS servers and acts on the response that is returned. RADIUS servers receive user connection
requests, authenticate the user, and then return the configuration information necessary for the client to
deliver services to the user.
Overlapping IP Address Pools

Overlapping IP Address Pools allow the creation of overlapping IP address pool groups to create different
address spaces and concurrently use the same IP addresses. The feature has the concept of an IP address
group to support multiple IP address spaces and still allow the verification of non-overlapping IP address
pools within a pool group. Pool names must be unique within the VHG/PE. The pool name carries an
implicit group identifier because that pool name can be associated only with one group.
The motivation behind using overlapping address pools comes from applications like MPLS VPNs where
each customer VPN can potentially use the same address space.

Dial Services
Wateen Dial Services Overview
Introduction
As mentioned before Wateen Telecom’s dial infrastructure will be used to provide the following services:
Remote access to MPLS VPNs
As a backup to MPLS VPN primary link
Backup internet connection
Remote Access to MPLS VPNs

Using a combination of NAS/LAC and VHG/PE devices Wateen Telecom will be able to provide a
scalable and complete end-to-end VPN service to its customers. This service will allow remote access
directly into the MPLS VPN network which is a significant value add to an already rich service portfolio
of Wateen Telecom. Using this service mobile users will be able to leverage Wateen’s vast NAS/LAC
deployment (20 cities) and dial into their respective corporate VPNs as if they were dialing to connect to
the internet.
Backup to MPLS VPN Primary Link

Wateen’s diverse service offering also includes MPLS VPNs. A Layer-3 MPLS VPN enables a customer
to peer with the service provider at L3 at any/all of its sites. The routing between all of the customer sites
is carried in an isolated Virtual Routing and Forwarding (VRF) table whereby forming a secure virtual
private network (VPN) for that customer. The customer gets the advantage of any-to-any connectivity
between all of its sites without having to provision a full mesh of connections.
Large corporations or financial organizations will have redundancy requirements whereby some of the
sites will be required to have a backup mechanism of maintaining connectivity to the MPLS VPN in case
the primary link fails. To cater for such requirements Wateen’s dial design will provision for the use of a
dial backup mechanism. The Customer Equipment (CE) will be configured to initiate dial out in case the
primary CE to PE link fails. Based on the authentication mechanism, the respective NAS/LAC will
decide which VHG/PE to establish the L2TP tunnel with and once VHG/PE completes authentication the
session will be allocated to the respective customer’s VPN.
Backup Internet Connection

.Wateen Telecom does not plan to provide residential dial internet services. However, they do want dial
internet to be provided to corporates as a backup to their primary internet connection. This service is
somewhat similar to the Backup VPN connection service, the key difference being that via the internet
backup services users will land on public internet after successful authentication rather than in a specific
VPN.
Even though Wateen has opted not to deploy residential dial internet services, the same can be offered
without any change of design / infrastructure in the future, if required.
Bill of Quantity
The following figure details the equipment that has been ordered for the dial network and its distribution:

Dial Services
Figure 98 Dial Equipment Location
Device Deployment Details
As part of the Wateen Telecom network buildout process Cisco Access Servers AS5400XM and
AS5350XM’s will be deployed throughout the Main, City and Satellite PoPs to provide dial up internet
access as well as dial access into MPLS VPNs. A Cisco 7206 router will be deployed in each Main PoP to
offer Virtual Homegateway/Provider Edge (VHG/PE) support. Cisco Access Registrar (CAR) has been
added in the Data Center at Lahore to authenticate, authorize and account the dialin users and assign them
to the relevant VRF at the Home Gateway
The following table summarizes the hostname, type, location, and deployment role of various dial
components as applicable to Wateen’s network. The hostnames mentioned in the following table will be
used throughout the documentation, whenever a reference has to be made to dial components.
Deployment Device
Device Hostname Deployed As
Location Model
Khi-7206-VHG1 Karachi 7206VXR VHG (Virtual Home Gateway) / PE (LNS)
Khi-AS5400-AS1 Karachi AS-5400XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Hyd-AS5350-AS1 Hyderabad AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Qta-AS5350-AS1 Quetta AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Skr-AS5350-AS1 Sukkhur AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Ryk-AS5350-AS1 R.Y.Khan AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Lhe-7206-VHG1 Lahore 7206VXR VHG (Virtual Home Gateway) / PE (LNS)

Lhe-AS5400-AS1 Lahore AS-5400XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Skt-AS5350-AS1 Sialkot AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Guj-AS5350-AS1 Gujranwala AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Gjt-AS5350-AS1 Gujrat AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)

Dial Services
Isb-7206-VHG1 Islamabad 7206VXR VHG (Virtual Home Gateway) / PE (LNS)

Isb-AS5400-AS1 Islamabad AS-5400XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Pes-AS5350-AS1 Peshawar AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Dik-AS5350-AS1 D.I.Khan AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Jeh-AS5350-AS1 Jehlum AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Abt-AS5350-AS1 Abottabad AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Fsd-7206-VHG1 Faisalabad 7206VXR VHG (Virtual Home Gateway) / PE (LNS)

Fsd-AS5350-AS1 Faisalabad AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Sgd-AS5350-AS1 Sargodha AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Mul-AS5350-AS1 Multan AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Swl-AS5350-AS1 Sahiwal AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Bwp-AS5350-AS1 Bahawalpur AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Okr-AS5350-AS1 Okara AS-5350XM NAS / LAC (Network Access Server / L2TP Access Concentrator)
Table 15 Dial Device Deployment Details
Please refer to the Layer 2 Tunneling Protocol sub-section for details on LAC and LNS functionality.
Additionally, the above table does not show Cisco Access Registrar (CAR). A pair of CAR servers will
be deployed at Wateen’s Lahore Data Center for providing AAA services.
Dial Service Functional Description and Call Flows

The following paragraphs summarize the sequence of events that take place when a remote user dials into
Wateen’s network. This summary will be followed by illustrations with explanatory notes to enhance
reader understanding of the dial service implementation.
Service Functional Description
As part of the initial connection sequence when a user connects they are prompted for an authentication
password. When the authentication response is received this triggers the NAS/LAC to send the CAR an
Access-Request. The Access-Request message contains various attributes including the Domain Name in
the User-Name attribute, NAS/LAC-IP-Address and Acct-Session-Id. Each NAS/LAC uniquely identifies
itself to the CAR by the source address of the IP packets that it generates will match a corresponding
entry in the CAR clients file.
If the Access-Request is invalid, the CAR will respond to the NAS/LAC with an Access-Reject, this will
cause the NAS/LAC to terminate PPP and disconnect the user. Otherwise, the CAR returns Access-
Accept to the NAS/LAC. The Access-Accept for an MPLS user contains various Tunnel attributes that
allows the NAS/LAC to build an L2TP tunnel to the appropriate VHG/PE router. For an Internet user the
Access-Accept will contain information that will allow the user to terminate on the NAS/LAC, be
allocated an IP address and connect to the internet.
As part of the L2TP protocol the NAS/LAC signals the negotiated LCP state to the VHG/PE thus
allowing the VHG/PE to bring up PPP LCP and continue the PPP Authentication phase and if successful
the NCP phase (see technology briefs below for detailed protocol explanation).
Once the L2TP session is established the NAS/LAC sends an Accounting-Start to the CAR accounting
server. Amongst others the following attributes are included - User-Name, Acct-Session-Id and Tunnel-
Server-Endpoint.

Dial Services
When the VHG/PE accepts the tunnel/session it sends an Access-Request to the CAR then completes the
remote user's authentication. The VHG/PE creates a unique virtual access interface for each dialin user,
completes PPP NCP, and terminates the PPP session into an IP connection.
The remote user's session becomes part of the appropriate VRF. An Accounting-Start packet is now sent
and subsequent interim accounting packets are sent at a configured interval.
Packets now flow from and to the remote user. When the user disconnects the PPP session will terminate
and the L2TP session will disconnect then both the VHG/PE and the NAS/LAC will send Accounting-
Stops.
For the internet users the NAS/LAC sends Accounting-Start, Interim and Stop packets throughout the
lifetime of the call.
Call Flow Diagrams
Dial In (MPLS VPN Remote Access)
Figure 99 Dialin Architecture
1. The remote user initiates a PPP connection to the NAS/LAC using either analog POTS or ISDN.
2. NAS/LAC accepts the connection and a PPP link is established.
3. The NAS/LAC partially authenticates the user with CHAP or PAP. The Domain name is used to
determine whether the user is an MPLS customer. The NAS/LAC queries the CAR for tunnel information.
The CAR returns the address of a VHG/PE.

Dial Services
4. If an L2TP tunnel does not exist, the NAS/LAC initiates a tunnel to the VHG/PE. The NAS/LAC and the
VHG/PE authenticate each other before any sessions are attempted within a tunnel.
5. Once the tunnel exists, a session within the tunnel is created for the remote user and the PPP connection is
extended to terminate on the VHG/PE.
6. The NAS/LAC propagates all available PPP information (the LCP negotiated options and the partially
authenticated CHAP/PAP information) to the VHG/PE.
7. The VHG/PE associates the remote user with a specific customer MPLS VPN. The VPN's VRF (routing
table and other information associated with a specific VPN) has been preinstantiated on the VHG/PE.
8. The VHG/PE completes remote user's authentication with the CAR.
9. The VHG/PE obtains an IP address for remote user.
10. The remote user becomes part of the customer VPN. Packets flow from and to the remote user.
Dial Backup (Internet or MPLS VPN)
Figure 100 Dial Backup Architecture
The dial backup connection sequence is identical to the dial in model however for the dial backup
solution the CPE has two connections into the network and loses the primary interface.
1. The CPE loses the primary interface and initiates backup PPP connection to the NAS/LAC using
either analog POTS or ISDN.
2. NAS/LAC accepts the connection and a PPP link is established.
3. The NAS/LAC partially authenticates the user with CHAP or PAP. The Domain name is used to
determine whether the user is an MPLS customer or Internet customer; the absence of a domain in the
username represents internet users. The NAS/LAC queries the CAR for tunnel information. If the user

Dial Services
is an Internet customer authentication continues on the NAS/LAC. If the user is an MPLS customer
the CAR returns the address of a VHG/PE.
4. If an L2TP tunnel does not exist, the NAS/LAC initiates a tunnel to the VHG/PE. The NAS/LAC and
the VHG/PE authenticate each other before any sessions are attempted within a tunnel.
5. Once the tunnel exists, a session within the tunnel is created for the remote user and the PPP
connection is extended to terminate on the VHG/PE.
6. The NAS/LAC propagates all available PPP information (the LCP negotiated options and the partially
authenticated CHAP/PAP information) to the VHG/PE.
7. The VHG/PE associates the remote user with a specific customer MPLS VPN. The VPN's VRF
(routing table and other information associated with a specific VPN) has been preinstantiated on the
VHG/PE.
8. The VHG/PE completes remote user's authentication with the CAR.
9. The VHG/PE obtains an IP address for remote user via an IP Pool or delivered from CAR as Framed-
IP-Address.
10. The remote user becomes part of the customer VPN. Packets flow from and to the remote user.
11. When the primary link is restored the primary route is also restored and the backup is terminated by
the remote user
12. The backup route is deleted by the VHG/PE
Complete Call Flow
The complete call flow is detailed below, showing the initial PPP connection through to the final ISP call
acceptance.
Figure 101 Complete Call flow

Dial Services
Physical Network Topology
In this section we will discuss the dial component physical connectivity details as they pertain to each
type of Wateen PoP; Main PoP, City PoP, and Satellite PoP.
One AS5400XM will be deployed in the Lahore, Karachi and Islamabad PoPs. All remaining PoPs will
be deployed with a single AS5350XM.
A 7206/NPE-G1 will be deployed as VGH/PE in the Lahore, Karachi, Islamabad and Faisalabad PoPs.
Main PoP Architecture

As discussed earlier (Device Deployment Details section), each of the Main PoPs (with the exception of
Faisalabad) will be equipped with an AS5400XM access server. The AS5400XM in a Main PoP will
connect to both local PE devices via Gigabit Ethernet UTP interfaces.
The following diagram shows what the Dial infrastructure of a Main PoP will look like. Although the
figure has adopted the Lahore Main PoP as an example please note that from Dial perspective both
Karachi and Islamabad will have exactly the same connectivity. Faisalabad will also follow a similar
connectivity model with the difference being in the type of the acces server deployed there; Faisalabad
will be equipped with an AS5350XM as opposed to an AS5400XM.
Figure 102 Dial Infrastructure Architecture – Main PoP

Dial Services
City and Satellite PoPs
All City and Satellite PoPs will be equipped with one AS5350XM access server. Within a City or
Satellite PoP the AS5350XM access server will connect to the local PE via a Gigabit Ethernet UTP
interface.
The following diagram shows what the Dial infrastructure of a City and a Satellite PoP will look like. For
the sake of simplicity, only Gujranwala has been chosen as the reference City PoP while Gujrat has been
chosen as the reference Satellite PoP. All City and Satellite PoPs will follow the same NAS/LNS
connectivity model as depicted in the following figure:
Figure 103 Dial Infrastructure Architecture – City/Satellite PoP

Physical Connectivity Details
Local Remote
Local Device Local Interface Local Interface IP Remote Remote Remote Interface
Interface Interface Comments
Name Type Address Device Name Interface Type IP Address
ID ID
Karachi Core and Associated City / Satellite PoPs
Loopback 0 Virtual - Public 58.27.190.55 / 32 - N/A - - N/A - - N/A - - N/A - Loopback interface for management functions
Gi 0 / 1 GE - Fiber - MM 58.27.194.194 / 30 Khi-7613-PE1 Gi 1/22 GE - Fiber - MM 58.27.194.193 / 30 Provides L3 interconnect to local PE1
Khi-7206-VHG1
Gi 0 / 2 GE - Fiber - MM 58.27.194.198 / 30 Khi-7613-PE2 Gi 1/22 GE - Fiber - MM 58.27.194.197 / 30 Provides L3 interconnect to local PE2
Gi 0 / 3 GE - Fiber - MM - Not Used - - N/A - - N/A - - N/A - - N/A - This interface will not be used
Loopback 1 Virtual - Private 192.168.64.1 / 32 - N/A - - N/A - - N/A - - N/A - Provides address to unnumbered interfaces
Khi-AS5400-AS1 Loopback 2 Virtual - Private 172.24.0.1 / 23 - N/A - - N/A - - N/A - - N/A - Used to originate summary route for configured dial pool
Gi 0/0 GE - Copper 58.27.194.130 / 30 Khi-7613-PE1 Gi 12/46 GE - Copper 58.27.194.129 / 30 Provides L3 interconnect to local PE1
Gi 0/1 GE - Copper 58.27.194.134 / 30 Khi-7613-PE2 Gi 12/46 GE - Copper 58.27.194.133 / 30 Provides L3 interconnect to local PE2
Hyd-AS5350-AS1 Loopback 2 Virtual - Private 172.24.2.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.194.138 / 30 Hyd-7609-PE1 Gi 8/46 GE - Copper 58.27.194.137 / 30 Provides L3 interconnect to Local PE
Gi 0/1 GE - Copper - Not Used - - N/A - - N/A - - N/A - - N/A - This interface will not be used
Qta-AS5350-AS1 Loopback 2 Virtual - Private 172.24.2.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.194.142 / 30 Qta-7609-PE1 Gi 8/46 GE - Copper 58.27.194.141 / 30 Provides L3 interconnect to local PE
Skr-AS5350-AS1 Loopback 0 Virtual - Public 58.27.190.51 / 32 - N/A - - N/A - - N/A - - N/A - Loopback interface for management functions

Dial Services
Loopback 2 Virtual - Private 172.24.3.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.194.146 / 30 Skr-7609-PE1 Gi 8/46 GE - Copper 58.27.194.145 / 30 Provides L3 interconnect to local PE
Ryk-AS5350-AS1 Loopback 2 Virtual - Private 172.24.3.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.194.150 / 30 Ryk-7609-PE1 Gi 8/46 GE - Copper 58.27.194.149 / 30 Provides L3 interconnect to local PE
Lahore Core and Associated City / Satellite PoPs
Loopback 0 Virtual - Public 58.27.190.29 / 32 - N/A - - N/A - - N/A - - N/A - Loopback interface for device management
Gi 0 / 1 GE - Fiber - MM 58.27.192.194 / 30 Lhe-7613-PE1 Gi 1/22 GE - Fiber - MM 58.27.192.193 / 30 Provides L3 interconnect to local PE1
Lhe-7206-VHG1
Gi 0 / 2 GE - Fiber - MM 58.27.192.198 / 30 Lhe-7613-PE2 Gi 1/22 GE - Fiber - MM 58.27.192.197 / 30 Provides L3 interconnect to local PE2
Lhe-AS5400-AS1 Loopback 2 Virtual - Private 172.24.32.1 / 23 - N/A - - N/A - - N/A - - N/A - Used to originate summary route for configured dial pool
Gi 0/0 GE - Copper 58.27.192.130 / 30 Lhe-7613-PE1 Gi 12/46 GE - Copper 58.27.192.129 / 30 Provides L3 interconnect to local PE1
Gi 0/1 GE - Copper 58.27.192.134 / 30 Lhe-7613-PE2 Gi 12/46 GE - Copper 58.27.192.133 / 30 Provides L3 interconnect to local PE2
Skt-AS5350-AS1 Loopback 2 Virtual - Private 172.24.34.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.192.138 / 30 Skt-7609-PE1 Gi 8/46 GE - Copper 58.27.192.137 / 30 Provides L3 interconnect to local PE1
Guj-AS5350-AS1 Loopback 2 Virtual - Private 172.24.34.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.192.142 / 30 Guj-7609-PE1 Gi 8/46 GE - Copper 58.27.192.141 / 30 Provides L3 interconnect to local PE1
Gjt-AS5350-AS1 Loopback 0 Virtual - Public 58.27.190.27 / 32 - N/A - - N/A - - N/A - - N/A - Loopback interface for management functions

Dial Services
Loopback 2 Virtual - Private 172.24.35.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.192.146 / 30 Gjt-7609-PE1 Gi 8/46 GE - Copper 58.27.192.145 / 30 Provides L3 interconnect to local PE1
Islamabad Core and Associated City / Satellite PoPs
Gi 0 / 1 GE - Fiber - MM 58.27.196.194 / 30 Isb-7613-PE1 Gi 1/22 GE - Fiber - MM 58.27.196.193 / 30 Provides L3 interconnect to local PE1
Isb-7206-VHG1
Gi 0 / 2 GE - Fiber - MM 58.27.196.198 / 30 Isb-7613-PE2 Gi 1/22 GE - Fiber - MM 58.27.196.197 / 30 Provides L3 interconnect to local PE2
Isb-AS5400-AS1 Loopback 2 Virtual - Private 172.24.64.1 / 23 - N/A - - N/A - - N/A - - N/A - Used to originate summary route for configured dial pool
Gi 0/0 GE - Copper 58.27.196.130 / 30 Isb-7613-PE1 Gi 11/46 GE - Copper 58.27.196.129 / 30 Provides L3 interconnect to local PE1
Gi 0/1 GE - Copper 58.27.196.134 / 30 Isb-7613-PE2 Gi 11/46 GE - Copper 58.27.196.133 / 30 Provides L3 interconnect to local PE2
Pes-AS5350-AS1 Loopback 2 Virtual - Private 172.24.66.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.196.138 / 30 Pes-7609-PE1 Gi 8/46 GE - Copper 58.27.196.137 / 30 Provides L3 interconnect to local PE1
Dik-AS5350-AS1 Loopback 2 Virtual - Private 172.24.66.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.196.142 / 30 Dik-7609-PE1 Gi 8/46 GE - Copper 58.27.196.141 / 30 Provides L3 interconnect to local PE1
Jeh-AS5350-AS1 Loopback 2 Virtual - Private 172.24.67.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.196.146 / 30 Jeh-7609-PE1 Gi 8/46 GE - Copper 58.27.196.145 / 30 Provides L3 interconnect to local PE1

Dial Services
Abt-AS5350-AS1 Loopback 2 Virtual - Private 172.24.67.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.196.150 / 30 Abt-7609-PE1 Gi 8/46 GE - Copper 58.27.196.149 / 30 Provides L3 interconnect to local PE1
Faisalabad Core and Associated City / Satellite PoPs
Gi 0 / 1 GE - Fiber - MM 58.27.198.194 / 30 Fsd-7613-PE1 Gi 1/22 GE - Fiber - MM 58.27.198.193 / 30 Provides L3 interconnect to local PE1
Fsd-7206-VHG1
Gi 0 / 2 GE - Fiber - MM 58.27.198.198 / 30 Fsd-7613-PE2 Gi 1/22 GE - Fiber - MM 58.27.198.197 / 30 Provides L3 interconnect to local PE2
Fsd-AS5350-AS1 Loopback 2 Virtual - Private 172.24.96.1 / 25 - N/A - - N/A - - N/A - - N/A - Used to originate summary route for configured dial pool
Gi 0/0 GE - Copper 58.27.198.130 / 30 Fsd-7613-PE1 Gi 11/46 GE - Copper 58.27.198.129 / 30 Provides L3 interconnect to local PE1
Gi 0/1 GE - Copper 58.27.198.134 / 30 Fsd-7613-PE2 Gi 11/46 GE - Copper 58.27.198.133 / 30 Provides L3 interconnect to local PE2
Sgd-AS5350-AS1 Loopback 2 Virtual - Private 172.24.96.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.198.138 / 30 Sgd-7609-PE1 Gi 8/46 GE - Copper 58.27.198.137 / 30 Provides L3 interconnect to local PE1
Mul-AS5350-AS1 Loopback 2 Virtual - Private 172.24.97.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.198.142 / 30 Mul-7609-PE1 Gi 8/46 GE - Copper 58.27.198.141 / 30 Provides L3 interconnect to local PE1
Swl-AS5350-AS1 Loopback 2 Virtual - Private 172.24.97.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.198.146 / 30 Swl-7609-PE1 Gi 8/46 GE - Copper 58.27.198.145 / 30 Provides L3 interconnect to local PE1

Dial Services
Bwp-AS5350-AS1 Loopback 2 Virtual - Private 172.24.98.1 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.198.150 / 30 Bwp-7609-PE1 Gi 8/46 GE - Copper 58.27.198.149 / 30 Provides L3 interconnect to local PE1
Okr-AS5350-AS1 Loopback 2 Virtual - Private 172.24.98.129 / 25 - N/A - - N/A - - N/A - - N/A - Will be used in future if dynamic routing is enabled
Gi 0/0 GE - Copper 58.27.198.154 / 30 Okr-7609-PE1 Gi 8/46 GE - Copper 58.27.198.153 / 30 Provides L3 interconnect to local PE1
Table 16 Dial Component IP Addressing and Physical Connectivity Detail

Dial Services
Logical Network Architecture

This section will cover the logical design aspects of Wateen’s dial infrastructure.
IP Addressing Details
IP Addressing for both NAS/LAC and VHG/PE devices is described separately in the following sub-
sections:
NAS IP Addressing
NAS devices will require IP Addresses for:
Physical and virtual interfaces
Address pools for internet users
For physical and logical interface IP Addressing please refer to Table 16.
A separate IP Address pool has been allocated to each NAS device for internet dialup users. In order to
conserve public IP Address space RFC 1918 addresses have been used for this purpose. The size of the
pool assigned to each NAS depends on the maximum number of simultaneous users the NAS can
support; this in turn depends on the number of ordered PRIs.
The private pool 172.24.0.0 / 16 has been reserved for Dialup users. This pool has been further subnetted
to allocate each NAS the required number of addresses while still maintaining regional hierarchy. The
following table shows how this prefix has been subnetted.
172.24.0.0/16 Master Dial Pool

172.24.0.0/17
172.24.0.0/19 Assigned to Karachi Region
172.24.0.0/20
172.24.0.0/21
172.24.0.0/23 Assigned to Khi-AS5400-AS1
172.24.2.0/23
172.24.2.0/25 Assigned to Hyd-AS5350-AS1
172.24.2.128/25 Assigned to Qta-AS5350-AS1
172.24.3.0/25 Assigned to Skr-AS5350-AS1
172.24.3.128/25 Assigned to Ryk-AS5350-AS1
172.24.4.0/23 Reserved for future use
172.24.32.0/19 Assigned to Lahore Region
172.24.32.0/20
172.24.32.0/21
172.24.32.0/23 Assigned to Lhe-AS5400-AS1
172.24.34.0/23
172.24.34.0/25 Assigned to Skt-AS5350-AS1
172.24.34.128/25 Assigned to Guj-AS5350-NAS1
172.24.35.0/25 Assigned to Gjt-AS5350-NAS1

Dial Services

172.24.64.0/19 Assigned to Islamabad Region
172.24.64.0/20
172.24.64.0/21
172.24.64.0/23 Assigned to Isb-AS5400-AS1
172.24.66.0/23
172.24.66.0/25 Assigned to Pes-AS5350-AS1
172.24.66.128/25 Assigned to Dik-AS5350-AS1
172.24.67.0/25 Assigned to Jeh-AS5350-AS1
172.24.67.128/25 Assigned to Abt-AS5350-AS1
172.24.96.0/19 Assigned to Faisalabad Region
172.24.96.0/20
172.24.96.0/21
172.24.96.0/23
172.24.96.0/25 Assigned to Fsd-AS5350-AS1
172.24.96.128/25 Assigned to Sgd-AS5350-AS1
172.24.97.0/25 Assigned to Mul-AS5350-AS1
172.24.97.128/25 Assigned to Swl-AS5350-AS1
172.24.98.0/23
172.24.98.0/25 Assigned to Bwp-AS5350-AS1
172.24.98.128/25 Assigned to Okr-AS5350-AS1
Table 17 Dialup User Pool Division
Using the above ip schema as the base the following table has been formulated to provide details such as
maximum number of concurrent users, number of PRIs, assigned pool, etc. for each NAS:
Max. # of
NAS/LAC No. of PRIs Concurrent Dialup Suggested Pool
Location Model Assigned Pool
Hostname (Per BoQ) Users (As per # of Size
PRIs)
Khi-AS5400-AS1 Karachi AS-5400XM 16 480 / 23 172.24.0.0 / 23

Hyd-AS5350-AS1 Hyderabad AS-5350XM 4 120 / 25 172.24.2.0 / 25
Qta-AS5350-AS1 Quetta AS-5350XM 4 120 / 25 172.24.2.128 / 25
Skr-AS5350-AS1 Sukkhur AS-5350XM 4 120 / 25 172.24.3.0 / 25
Ryk-AS5350-AS1 R.Y.Khan AS-5350XM 4 120 / 25 172.24.3.128 / 25
Lhe-AS5400-AS1 Lahore AS-5400XM 16 480 / 23 172.24.32.0 / 23

Skt-AS5350-AS1 Sialkot AS-5350XM 4 120 / 25 172.24.34.0 / 25
Guj-AS5350-AS1 Gujranwala AS-5350XM 4 120 / 25 172.24.34.128 / 25
Gjt-AS5350-AS1 Gujrat AS-5350XM 4 120 / 25 172.24.35.0 / 25

Dial Services
Isb-AS5400-AS1 Islamabad AS-5400XM 16 480 / 23 172.24.64.0 / 23

Pes-AS5350-AS1 Peshawar AS-5350XM 4 120 / 25 172.24.66.0 / 25
Dik-AS5350-AS1 D.I.Khan AS-5350XM 4 120 / 25 172.24.66.128 / 25
Jeh-AS5350-AS1 Jehlum AS-5350XM 4 120 / 25 172.24.67.0 / 25
Abt-AS5350-AS1 Abottabad AS-5350XM 4 120 / 25 172.24.67.128 / 25
Fsd-AS5350-AS1 Faisalabad AS-5350XM 4 120 / 25 172.24.96.0 / 25

Sgd-AS5350-AS1 Sargodha AS-5350XM 4 120 / 25 172.24.96.128 / 25
Mul-AS5350-AS1 Multan AS-5350XM 4 120 / 25 172.24.97.0 / 25
Swl-AS5350-AS1 Sahiwal AS-5350XM 4 120 / 25 172.24.97.128 / 25
Bwp-AS5350-AS1 Bahawalpur AS-5350XM 4 120 / 25 172.24.98.0 / 25
Okr-AS5350-AS1 Okara AS-5350XM 4 120 / 25 172.24.98.128 / 25
Table 18 NAS Dialup Pool Assignment Details
VHG/PE Addressing
Each VHG/PE device will require IP Addresses for:
Physical and Logical interfaces
Address pools per customer VPN.
For physical and logical interface (management loopback only) addressing details please refer to Table
16.
Address pools per MPLS VPN will have to be defined by Wateen in consultation with the customers. One
option could be to use one unique pool per VHG/PE and reuse it across all customer VPNs. Wateen will
have to ensure that this does not overlap with customer’s internal addressing. For example, Wateen can
configure VHG/PE devices as:
Karachi VHG/PE Æ Use 172.24.4.0 / 23 for MPLS VPNs
Lahore VHG/PE Æ Use 172.24.36.0 / 23 for MPLS VPNs
Islamabad VHG/PE Æ Use 172.24.68.0 / 23 for MPLS VPNs
Faisalabad VHG/PE Æ Use 172.24.100.0 / 23 for MPLS VPNs
These pools can be reused per VRF using the ‘overlapping address pools’ feature described earlier. Please
note that this is just a recommendation. Wateen Telecom can adopt this recommendation in case they
have not formulated a policy of their own on the subject matter.
It shall be noted that for each customer VPN on loopback interface will need to be created per VHG/PE.
The purpose of this interface is to preinstantiate the customer routes in the VPN so that a user dialing in
does not experience unnecessary delay (due to BGP convergence) in having connectivity established. It is
recommended that a single /32 address be used on all such loopbacks (all customer VRFs) to conserve
address space.
Routing Protocol Design

This section will describe how the routing infrastructure will look like for each of the following device
deployment models:
NAS device in a City/Satellite PoP

Dial Services
NAS device in a Main PoP

VHG/PE router in a Main PoP
Routing Architecture - NAS Device in City/Satellite PoP

NAS servers installed in City or Satellite PoPs will be connected to the local PE via a single Gigabit
Ethernet link (UTP). Since there is only a single entry/exit point for these devices it does not make sense
to introduce any dynamic routing protocol to achieve reachability to/from these devices. Consequently it
is recommended that all NAS devices deployed in City/Satellite PoPs should have a static default route
pointing to the next-hop PE. Similarly the local PE router will have two static routes configured as
follows:
Route for the respective NAS pool and pointing to the local NAS as the next hop.
Route for the local NAS management loopback address and pointing to the local NAS as the
next hop.
Each PE will perform a controlled redistribute static in order to advertise the management loopback of the
NAS to the rest of the IGP domain. There is no requirement to redistribute the NAS pool into OSPF as all
users will be undergo NAT at the PE when initiating connections to the outside world.
The following figure shows the routing design in a City / Satellite PoP in a graphical manner:
Figure 104 NAS Routing Architecture – City/Satellite PoP
We have used Hyderabad City PoP as an example; however, the same routing architecture applies to all
Wateen City/Satellite PoPs.
Routing Architecture - NAS Device in a Main PoP

NAS servers installed in Main PoPs will have two Gigabit Ethernet interfaces, one connected to each
local PE. This dual-homed architecture necessitates the use of a dynamic routing protocol to effectively
Dial Services
utilize the dual entry/exit points both during normal conditions and during periods of a single link failure.
Consequently each NAS device installed in a Main PoP will be configured to run OSPF. The Core OSPF
Area 0 will be extended to these NAS devices by configuring them to include their Gigabit Ethernet
interfaces (connecting to local PE) in the OSPF Area 0. Additionally, the Loopback 2 interface of these
NAS devices will also be included in the OSPF Area 0 in order to send a summary of the configured dial-
up towards the rest of the network.
Since a default route is not being propagated in the core of the network (core PEs have visibility to full
internet routes) we need a mechanism to add a default route to the dual-homed NAS devices even though
they are configured to run OSPF with the core. Keeping simplicity in mind it is suggested to configure
two static default routes per NAS device installed in a Main PoP. One default will point to local PE1
device while the other default will point to the local PE2 device. ECMP (Equal Cost Multipath) load
sharing will be performed by the NAS device using the underlying CEF per destination load sharing
scheme.
The following figure is a graphical representation of the routing architecture of NAS devices located at
Wateen Main PoPs:
Figure 105 NAS Routing Architecture – Main PoP
The digram uses Lahore Main PoP as an example; however, the routing architecture of all Main PoPs will
follow a similar design from NAS connectivity perspective.
Routing Architecture – VHG/PE Device in a Main PoP
VHG/PE devices located in Main PoPs will also be dual-homed; one Gigabit Ethernet link will connect
each such device to each of the local PE routers. Since the purpose of the VHG/PE routers is to provide
remote access to MPLS VPNs, they will require Multiprotocol BGP (MP-BGP) configuration for vpnv4
addresses. The VHG/PE routing architecture can therefore be summarized by the following points:
OSPF Area 0 to extend to the VHG/PE Gigabit Ethernet interfaces connecting to the local PE
routers.

Dial Services
Management loopback of VHG/PE to be advertised via OSPF by including it in area 0 as a

passive interface
Multiprotocol-BGP configuration for vpnv4 to allow VHG/PE to accommodate creation of
customer VPNs and exchange vpnv4 routes with other PE routers in the core.
There is no requirement to carry a default route in VHG/PE devices. If, however, this
requirement changes in the future, default routes can be accommodated using the same
approach as used in the case of NAS devices installed in Main PoPs (two static defaults, one
pointint to each local PE)
Figure 106 VHG/PE Routing Architecture – Main PoP
The figure uses Lahore VHG/PE deployment as an example only. All VHG/PE deployments will follow
the same routing architecture as depicted in Figure 105.
Network Address Translation (NAT)

All internet dialup users will be assigned addresses from RFC 1918 based address pools as allocated per
NAS in the ‘NAS IP Addressing’ section. Consequentlu a means to represent these privately addressed
users via publicly accessible address space is required. Following the same architecture that is being
provisioned for Wimax subscribers, NAT for dialup internet users will also be done at the respective local
PE device(s). No configuration specific to NAT is required on the NAS itself.
VHG/PE Resiliency
VHG/PE redundancy is achieved by passing tunnel attributes from CAR to the NAS/LAC. By specifying
(on a per domain basis) multiple tunnel endpoints and tunnel preference values it is possible to achieve
failover and load balancing. The recommended failover and load balancing scheme will be for the
NAS/LAC’s to be returned tunnel attributes that allow the primary VGH/PE (that is the geographically
closest) to be the primary tunnel endpoint. The remaining three VHG/PE’s will then have equal priority in
the next priority group. An example being the NAS/LAC in Lahore will be returned the VGH/PE in
Lahore as its primary tunnel endpoint. Should this VHG/PE not be available the NAS/LAC will failover
to the next priority group that will have the endpoints of the VHG/PE’s in Karachi, Islamabad and

Dial Services
Faisalabad. The subsequent connections from this particular NAS/LAC will then be shared amongst the
three VHG’s above, until the primary endpoint becomes available again. The following table provides a
map for VHG/PE resiliency showing which VHG/PE will be primary and which VHG/PEs will be
backup for a particular NAS.
NAS/LAC
Location Model Primary VHG/PE Backup VHG/PE Group
Hostname
Khi-AS5400-AS1 Karachi AS-5400XM
Hyd-AS5350-AS1 Hyderabad AS-5350XM Lhe-7206-VHG1
Qta-AS5350-AS1 Quetta AS-5350XM Khi-7206-VHG1 Isb-7206-VHG1
Lyp-7206-VHG1
Skr-AS5350-AS1 Sukkhur AS-5350XM
Ryk-AS5350-AS1 R.Y.Khan AS-5350XM
Lhe-AS5400-AS1 Lahore AS-5400XM

Khi-7206-VHG1
Skt-AS5350-AS1 Sialkot AS-5350XM
Lhe-7206-VHG1 Isb-7206-VHG1
Guj-AS5350-AS1 Gujranwala AS-5350XM Lyp-7206-VHG1
Gjt-AS5350-AS1 Gujrat AS-5350XM
Fsd-AS5350-AS1 Faisalabad AS-5350XM

Sgd-AS5350-AS1 Sargodha AS-5350XM
Isb-7206-VHG1
Mul-AS5350-AS1 Multan AS-5350XM
Lyp-7206-VHG1 Khi-7206-VHG1
Swl-AS5350-AS1 Sahiwal AS-5350XM Lhe-7206-VHG1
Bwp-AS5350-AS1 Bahawalpur AS-5350XM
Okr-AS5350-AS1 Okara AS-5350XM
Isb-AS5400-AS1 Islamabad AS-5400XM

Pes-AS5350-AS1 Peshawar AS-5350XM Lhe-7206-VHG1
Dik-AS5350-AS1 D.I.Khan AS-5350XM Isb-7206-VHG1 Khi-7206-VHG1
Lyp-7206-VHG1
Jeh-AS5350-AS1 Jehlum AS-5350XM
Abt-AS5350-AS1 Abottabad AS-5350XM
Table 19 VHG/PE Resiliency Map
There is no special configuration required on the NAS or the VHG/PE to implement this resiliency
scheme. The tunnel attributes will be returned dynamically from the CAR and the NAS will always
initiate a tunnel to the primary VHG/PE. If the primary VHG/PE is unavailable then the NAS will attempt
to establish a tunnel with one of the three backup VHG/PE routers.
For the VHG/PE resiliency functionality to be effective it is reqiured that VPNs requiring resiliency are
pre-configured on all four VHG/PE routers. To ensure that backup VHG/PE devices have adequate
IPaddress resources to accommodate the clients normally associating with the primary VHG/PE, it is
recommended to over-provision the dial pools per customer.
Dial Backup Architecture

Wateen will provide a Dial Backup service to it’s MPLS VPN customers whereby a customer will utilize
a low-speed link (dialup) to maintiain connectivity to the corporate VPN even in case of a failure of the
primary link. The basic architecture of such a service is very similary to that of ‘Remote Access to MPLS
VPN’ sevice. The difference is that in case of Dial Backup the CAR will return an additional attribute
‘Framed-route’ to the VHG/PE which will insert a static route in the appropriate VRF pointing to the
backup link connecting to the CE. No special configuration is required on the VHG/PE for this
functionality; all required attributes are downloaded dynamically from CAR on successful user
authentication.
Dial Services
Configuration Templates
In this section an attempt has been made to provide configuration snippets that will be used to provision
various components of Wateen Telecom’s Dial infrastructure. Please note that these templates are strictly
for the dial configuration parameters; it is assumed that other configuration, specific to device security
and ip connectivity, will co-exist with these templates.
NAS/LAC
ISDN/Modem & PPP

The following is the generic required Modem and ISDN configuration
spe 1/0 1/9

firmware location system:/ucode/mica_port_firmware
The slot/SPE number will be auto generated for the specific access server build
!
modem country mica
The required country code to be specified here
!
isdn switch-type primary-net5
The required isdn switch type to be specified here
!
modemcap entry Wateen_MODEMCAP: MSC=&F&D2S0=0S29=6S21=3S40=10S10=50
Further details on modemcap can be found at the below CCO links
http://www.cisco.com/en/US/tech/tk801/tk36/technologies_tech_note09186a008009428b.shtml
http://www.cisco.com/univercd/cc/td/doc/product/access/acs_serv/as5400/sw_conf/atnextpt.htm
http://www.cisco.com/warp/public/471/recc_modemcaps.html
Suggested modemcap, can be optimized to reflect local conditions during NFRU

!
controller E1 0
clock source line primary
pri-group timeslots 1-31
!
controller E1 1
clock source line secondary 1
pri-group timeslots 1-31
!
interface Serial0:15
no ip address
encapsulation ppp
no logging event link-status
dialer rotary-group 1
no snmp trap link-status
isdn incoming-voice modem

Dial Services
!
interface Serial1:15
no ip address
encapsulation ppp
isdn incoming-voice modem
Repeat the above for the number of configured PRI interfaces
!
interface Group-Async1
ip unnumbered loopback 1
encapsulation ppp
dialer in-band
async mode dedicated
peer default ip address pool internet_users
ppp authentication chap pap callin
group-range 1 XX
The group-range should cover all modems in the access server
!
interface Dialer1
ip unnumbered Loopback0
encapsulation ppp
peer default ip address pool internet_users
!
line 1 XX
The line number should cover all modems in the access server
no flush-at-activation
modem InOut
modem autoconfigure type Wateen_MODEMCAP
no modem log rs232
no exec
transport input all
transport output none
PPP
Generic config for the IP local-pool, this is defined on the NAS/LAC to allocate Internet users IP
addresses as they connect. The pool name can be configured on the physical interface or downloaded
from RADIUS.
ip local pool internet_users A.B.C.2 A.B.C.254
It is recommended that the pool range is summarized as an aggregate route to be advertised by OSPF.

Dial Services
L2TP
The following is the generic required L2TP configuration; the per-user tunneling information will be
retrieved from the CAR.
vpdn enable
vpdn source-ip A.B.C.D (Management Loopback Address)
vpdn search-order domain
CAR Attributes for VHG/PE Failover and Load-sharing

The attributes below are an example of the attributes required on the CAR to assign an MPLS user
(based on domain) a primary tunnel endpoint in the highest priority group and three equal priority
tunnel endpoints on the next priority group.
Customer_1 Password = "cisco" Service-Type = Outbound,

Tunnel-Type = :1:L2TP,
Tunnel-Preference = :1:1,
Tunnel-Client-Auth-ID = :1:"NAS/LAC",
Tunnel-Server-Endpoint = :1:"A.B.C.D",
Tunnel-Password = :1:"xxxx"
Tunnel-Type = :2:L2TP,
Tunnel-Preference = :2:2,
Tunnel-Client-Auth-ID = :2:"NAS/LAC",
Tunnel-Server-Endpoint = :2:"E.F.G.H",
Tunnel-Server-Endpoint = :2:"I.J.K.L",
Tunnel-Server-Endpoint = :2:"M.N.O.P ",
Tunnel-Password = :2:"xxxx"
Where:
Customer_1 is the domain name of the MPLS VPN user sent to LNS by LAC; however, CAR has no
notion of domain names so for CAR Customer_1 is just another username in the database.
AAA
The following is the generic required AAA and RADIUS configuration.
aaa new-model
aaa authentication ppp default group radius
aaa authorization network default group radius
aaa accounting delay-start
aaa accounting update periodic xx
The required interval for sending interim accounting packets
aaa accounting network default start-stop group radius
RADIUS
ip radius source-interface Loopback <#> (Management Loopback)

radius-server host A.B.C.1 auth-port <#> acct-port <#> (For CAR_1)

Dial Services

radius-server source-ports extended
radius-server timeout 3
radius-server deadtime 10
radius-server key <key>
VHG/PE
As previously discussed the VHG/PE router sends an Access-Request to the CAR. CAR then returns an
Access-Accept that completes the remote user's authentication and delivers, as part of network
authorization, an unnumbered loopback interface (that places the interface into the required VRF) and ip
pool or framed-ip-address. The VHG/PE then creates a unique virtual access interface using the ip
address of the unnumbered loopback interface, completes PPP NCP, allocates the remote user an IP
address from the local pool or the delivered framed-ip-address and terminates the PPP session into the
delivered VRF.
Customer VRF’s
The following is the generic required VRF configuration to define the customer VRF’s and Loopback
associated with VRF’s
ip vrf Customer_1
rd 10:10
!
ip vrf Customer_2
rd 20:20
!
interface Loopback10
description Customer_1
ip vrf forwarding Customer_1
ip address A.B.C.D 255.255.255.255
!
interface Loopback20
description Customer_2
ip vrf forwarding Customer_2
ip address A.B.C.D 255.255.255.255
Virtual Template
Configure the Virtual-Template that will be used as the generic clone block for creating a virtual access
interface for each dialin user and assigning them to the respective VRF.
interface Virtual-Template1
no ip address
ip verify unicast reverse-path

Dial Services
no peer default ip address

The per-user IP address can be downloaded via radius as a framed- ip-address or the name of the ip local pool
BGP
Generic configuration for BGP and address family.
router bgp <AS number>

no synchronization
neighbor <bgp peer address> remote-as <as number>
neighbor <bgp peer address> update source loopbackX
neighbor <bgp peer address> activate
no auto-summary
neighbor <ip address of bgp peer> activate
neighbor <ip address of bgp peer> next-hop-self
neighbor <ip address of bgp peer> send-community extended
exit-address-family
!
address-family ipv4 vrf Customer_1
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf Customer_2
no auto-summary
no synchronization
exit-address-family
PPP
Below is a generic config for the overlapping IP address pools, this is defined on the VHG/PE to
allocate MPLS users IP addresses as they connect. The pool names are downloaded from RADIUS.
ip local pool pool1_Customer_1 10.1.1.1 10.1.1.50 group Customer_1

With the above configuration the pools the below is the sample output showing the overlapping
addresses.
VHG/PE#show ip local pool
Pool Begin End Free In use

Dial Services
** pool <pool1_Customer_1> is in group <Customer_1>

pool1_Customer_1 10.1.1.1 10.1.1.50 50 0
pool2_Customer_1 10.1.1.100 10.1.1.110 11 0
pool3_Customer_1 10.1.2.1 10.1.2.30 30 0
pool1_Customer_2 10.1.1.1 10.1.1.40 40 0
pool2_Customer_2 10.1.1.50 10.1.1.70 21 0
It is recommended that the pool range is summarized as an aggregate route to be advertised by MP-
BGP.
In order to accommodate the VHG/PE Failover and Load Sharing scheme as discussed in the
NAS/LAC configuration template the below is an example showing the ip local pool configuraion for
two MPLS customers, Customer_1 & Customer_2 (assuming a /24 requirement). Both have Khi as
the primary VHG/PE with Lhe, Isb and Fsd in a failover group all with equal priority.
Khi-7206-VHG1
Lhe-7206-VHG1
Isb-7206-VHG1
Fsd-7206-VHG1
In this example one must consider, if Khi-7206-VHG1 fails and then Lhe-7206-VHG1 & Isb-7206-
VHG1 fail then Fsd-7206-VHG1 must be able to terminate all users for Customer_1 & Customer_2.
This configuration will allow Customer_1 or Customer_2 clients to dial into any NAS across Pakistan
and be connected to their private VPN. The actual IP Address allocated to a client will depend on the
VHG that terminates the L2TP tunnel; it will be the VHG closest to the NAS the user lands on
L2TP
The following is the generic required L2TP configuration; the per-user tunneling information will be
retried from the CAR.
vpdn enable
vpdn source-ip A.B.C.D
vpdn authen-before-forward
!
vpdn-group 1

Dial Services
! Default L2TP VPDN group

accept-dialin
protocol l2tp
virtual-template 1
lcp renegotiation on-mismatch
If the proxied LCP options do not match those on the VHG/PE the LCP state machine is collapsed and restarted
l2tp tunnel password <xxxxxxxx>
The password on the NAS/LAC should be the same as the password configured on the VHG/PE. The password to
be used for Wateen’s Deployment is available in the document WateenTelecom_X97743_LLD_Default_Passwords
(AS-122691)
AAA
The following is the generic required AAA and RADIUS configuration.
aaa new-model
aaa authentication ppp default group radius
aaa authorization network default group radius
aaa accounting delay-start
aaa accounting update periodic 60
aaa accounting network default start-stop group radius
RADIUS
ip radius source-interface Loopback <#> (Management Loopback)
radius-server source-ports extended
radius-server timeout 3
radius-server deadtime 10
radius-server key <key>
CAR Attribute
The attributes below are an example of those required on the CAR to assign the user to the correct VRF
and allocate the respective IP Pool or Framed IP address.
Cisco:Avpair = "ip:ip-unnumbered=Loopback10"
Cisco:Avpair = "ip:addr-pool=Customer_1" or Framed-IP-Address = A.B.C.D
IOS versions
Below are the IOS versions that will be deployed on the NAS and VHG devices.
Chassis Version File Comment

Dial Services
AS5350XM 12.4(7b) c5350-ik9s-mz.124-7b.bin NAS/LAC
AS5450XM 12.4(7b) c5400-ik9s-mz.124-7b.bin NAS/LAC
7206 12.4(2)T c7200-advipservicesk9-mz.124-2.T5.bin VHG/PE

Internet Services
The section details the internet connectivity, with upstream service provider(s). It details current plans of
Wateen Telecom for multihomed internet connectivity. The section also coveres internet as a customer
service offering and related infrastructure details.
Internet Peering & Transit Service

It is understood that Wateen Telecom will initialy connect with PTCL PIE infrastructure. This
connectivity is based on the assumptions that:
Wateen Telecom will have publicly assigned IP address space (which PTCL will be able to route)
Wateen Telecom will have their own publicly assigned AS number.
As previously discussed, Wateen Telecom is currently in process of acquiring this information.
It is understood that connectivity to PTCL means the following:
• SP will provide the /30 subnet per transit link. The IP address will be assigned once customer link
has been tested at layer1
• SP’s AS number is 17557
• SP will not less accept prefix length shorter than /24
• For single link, SP does neighborship on directly connected interfaces
• If multiple links are there on single router then SP can peer using loopback addresses
• SP will accept MED and can use it in best path within their AS
• SP does not run IGP with customers (for redundant router solution at both ends)
• Customer has the option to terminate multiple links on different routers; subject to availability of
ports: link addresses will be used to peer in this case
It is understood that Wateen Telecom will like to peer with multiple internet service providers (and be
dual homed). Wateen Telecom has plans for two external peering points. Currently it is envisaged that
possible peering point will be located in Lahore. However it is possible that Wateen may expand the
number of external peering points beyond two, it is also possible that peering point location may be
extended to other cities or possibly countries. In addition to previous diagram depicting Lahore PoP,
following represents internet connectivity for internet border router (these devices are labeled internet
gatew router and the two terms can be used interchangeably).

Internet Services
Figure 107 Internet GatewayConnectivity
The internet gateway routers are singly attached to respective PE routers due to SCE (service control
engine requirements). SCE requires symmetrical flow of traffic and hence cross connects are not used.
The Gigabit Ethernet links are on LAN based cards (67xx) on both gateways as well as PE devices.
From services perspective Wateen Telecom will like to provide transit internet connectivity to their
customers, this would mean it will be necessary to carry full internet routing tables in many parts of the
network.
iBGP will be used to propagate BGP learnt internet routes within Wateen Telecom’s network. iBGP has a
limitation that an iBGP peer never advertises an iBGP learnt route to another iBGP neighbour. As
previously discussed, this means creating a full iBGP mesh between all iBGP speakers. While this may
be possible for a very small number of nodes, this does not scale for any sizeable network. Therefore
IPv4 route reflectors are required to avoid a full iBGP mesh (please refer to see MP-BGP4 section for
details).
Possible options for relaxing this requirement are either using confederations or route reflectors.
Confederations require splitting the network into several autonomous systems (AS), route reflectors are
designed to be hierarchical and are a much better solution. Route Reflector approach is most widely
deployed approach.
While it is possible to provide BGP Next-Hop reachability via MPLS in the core, and hence eliminate the
need to run BGP in the core of the network (i.e. maintain a hollow core). As an alternative we can run
BGP in the core, assuming that core network is rather stable and there are no foreseeable plans to grow
the core.
For a hollow core, all the network elements within the core are not required to run BGP and carry full
internet routes. It becomes mandatory in this case to modify the next-hop at the border gateway router,
when peering with iBGP speakers.
For Wateen Telecom, both options are possible and administrative overhead of running BGP everwhere is
rather small. (P devices in this case will need to carry all interner routes). It is suggested that initially
BGP be used everywhere, and Wateen Telecom can eventually move to hollow core depending on growth
of the network and customer requirements.

Internet Services
IPv4 Route Reflectors

Separate ipv4 route reflectors (from vpnv4 route reflectors) are recommended for scalability and security
reasons. It is also recommended that Wateen Telecom use dedicated route reflectors. As an interim
measure two P nodes (12816) devices in Lahore and Faisalabad will be used.
While potentially any other PE device can be used as an interim measure, since PEs are also going to be
vpnv4 clients or reflectors, using P device allows for simplicity as there will no vpnv4 sessions
configured on P routers.
Figure 108 IPv4 Route Reflector Connectivity
As noted in the diagram above, remote (satellite) PE devices(7609) routers can be configured such that
full internet routes are not required and hence IPv4 configuration of BGP may be omitted for these
devices.
Granted that these PoP devices have scaled down version of Supervisor engine in 7609 series router
(Sup32 instead of Sup720-3BXL), it is suggested to make optimal use of resources by not carrying full
internet routing table in these devices. Instead it is suggested that
• A default route be statically configured pointing to upstream devices’s loopback address
• While it is possible to originate default route elsewhere in the network and distribute via IGP, in
satellite(remote) PoPs, there is little value to this approach as only single exit to upstream device
exists. This also assures that any unknown prefixes are only carried upto upstream PEs before being
discarded rather than being carried thoughout the core.
• Any customers in these PoPs be provided internet routing table by allowing them to peer directly
with an upstream PE (by using multihop knob). Disadvantage of this approach is that every time a
BGP session flaps, complete internet routing table will be sent across STM-1 link. However it is
expected that few customers will subscribe to this service in satellite PoPs. Wateen Telecom will
always have the option of transitioning these PoPs to follow same configuration as other city PoP
PEs.
• As indicated earlier, VPNv4 peering session will still be configured on satellite PoP PEs.
Internet Services
• These PoP PEs will be configured identically to other PEs for IGP, LDP and any internet related
services.
Following is a sample configuration of ipv4 route reflector

!
router bgp 65501
bgp router-id <loopback0 ip address>
neighbor clients peer-group
neighbor clients remote-as 65501
neighbor clients password cisco
neighbor clients update-source Loopback0
neighbor clients version 4
neighbor <PE1 ip address> peer-group clients
neighbor <Peer RR ip address> remote-as 65501

neighbor <Peer RR ip address> description RR & at FSD
neighbor <Peer RR ip address> password <password>
neighbor <Peer RR ip address> update-source Loopback0
neighbor <Peer RR ip address> version 4
!
address-family ipv4
neighbor clients activate
neighbor clients send-community both
neighbor clients route-reflector-client
neighbor <Peer RR ip address> activate
neighbor <Peer RR ip address> send-community both
exit-address-family
!
Figure 109 IPv4 Route Reflector Configuration
For all other PE devices, please see MP-BGP4 (Multi-protocol BGP) section earlier in this design
document.
Following section replicates some information which is common for ipv4 iBGP session as well as eBGP
configuration (be it for upstream provider or downstream node) from MP-BGP4 section.
BGP Security
BGP by default does not use encryption for the sessions established, it is however recommended to
ensure that BGP is configured with neighbour passwords.
While the sessions may be internal and the requirement exists for both-ends of the session to be
configured to ensure BGP session establishment it is still recommended to use MD5 authentication on the
sessions for maximum security
BGP Passwords are disabled by default and are configured on a per-neighbour basis
router bgp 65501

neighbour <ip address / peer group> password <password>
!
Figure 110 BGP Authentication

Internet Services
It is also suggested that all external BGP sessions be secured with a unique password mutually agreed
with upstream provider or with a customer.
BGP TTL Security Check

By default BGP sends packets out with a TTL of <x>, the BGP TTL Security Check ensures that eBGP
sessions from a directly connected peer cannot be spoofed or faked.
When the BGP session is established, it is told (on both routers) to implement a TTL of 255, the control
plane of the peering router then checks that the packet coming in has a TTL of 254 (decremented by one
hop).
In order to configure this the following configuration should be applied to eBGP neighbours
!
router bgp 65501
neighbor <IP address> ttl-security hops <hop count>
!
Figure 111 BGP TTL Security Check

Where <hop count> is the number of hops to the directly connected peer, Note: it is not recommended to
use this for eBGP multihop peerings.
This should also be used in conjunction with MD5 to limit the scope of the attack to only the directly
connected neighbour.
BGP Maximum Prefixes

There is also an ability to determine the maximum number of routes that are advertised by a peer. This
has been designed to assist in the eventuality of either mis-configuration of peering routers or by blatant
attacks on ISP core BGP routing tables.
!
router bgp 65501
neighbor <IP Address> maximum-prefix 150000 <threshold> <warning-only>
!
Figure 112 BGP Prefixes Limit
This command allows the user to configure the max numbers we receive from a peer or peer-group, when
the number of routes learnt exceeds the maximum number configured the default action is to terminate
the peering, the peering is then only re-established by manual intervention with a “clear ip bgp
neighbour” statement. We can use the keyword warning-only to generate a warning message to the
console and the network management platforms to indicate there has been a violation of the peering
policy.
The threshold value is set to indicate an initial warning that we are receiving a quantity of routes
approaching the pre-defined limit, this default is 75% this can be modified by inserting a integer value in
the <threshold> value with defining the maximum-prefix.

Internet Services
Wateen Telecom will use a maximum-prefix value of 200000 when peering with upstream BGP
neighbors for eBGP connectivity.
BGP MED
When there is no Multi-Exit Discriminator (MED) attribute set to a route, BGP IOS assumes a value of 0
for this route, in some circumstances other routing vendors have assumed absence of MED to mean a
value of 4294967295 (2^32-1), this difference caused possible potential to cause routing loops between
vendors with differing BGP implementations.
A belated IETF draft concluded that absence of a MED should cause it to have an infinite value (as
shown above), this would make a route without the MED attribute the least preferred, this is the inverse
generic behaviour in Cisco IOS where lack of MED indicates the best route available.
An additional command was introduced to ensure compatibility with other vendors using the IETF
implementation of MED.
!
router bgp 65501
bgp bestpath med missing-as-worst
!
Figure 113 BGP MED Comparison
This will cause IOS to act in the same way as described by the IETF, however this should not be
provisioned without due consideration.
BGP Deterministic MED

If bgp deterministic med is not enabled, the order in which routes are received from peers could impact
MED based decisions. This may occur when the same route is received from different neighbours in the
same AS with the same path length but different MED values.
For example, if we receive the following routes in this order:

6. ASPATH 3, MED 200, External
If the possibilities for a prefix were received in this order without deterministic MED then the node would
install path 2 over path 1 due to a lower IGP metric, it would then prefer path 3 to path 2 due to the fact
that it is an external route.
However, with deterministic MED this removes any temporal dependencies of MED based decisions, it
ensures that the best MED comparison is made across all routes from the same AS, in this case path 1
would be chosen due to it having the lowest MED.
router bgp 65501

bgp deterministic-med
Figure 114 BGP Deterministic MED

Internet Services
BGP path selection process is described at the following URL

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094431.shtml
BGP Communities
BGP Communities are community tags that can be applied to a BGP route and carried throughout the
network of iBGP peerings.
These communities can identify a route or be used to apply policies applicable to only routes with certain
community tags.
It is proposed that Wateen Telecom use communities to classify the network routes:
• Lahore Subnets
• Karachi Subnets
• Islamabad Subnets
• Faisalabad Subnets
• National Subnets
• International Internet Subnets
Each edge node with eBGP peerings to customers connecting to the Wateen Telecom network would use
their own prefix so that it could be identified as coming from a certain PoP.
Table 20 Proposed BGP Communities
Region Community Value
Lahore 65501:42001
Karachi 65501:42002
Islamabad 65501:42003
Faisalabad 65501:42004
National Prefixes 65501:42005
International 65501:42000
Prefixes
These BGP communities can then easily be checked with ACL’s to match Subnets from a major core
PoP, national or international traffic etc. A route can optionally be tagged with multiple communities if
necessary such as a regional and a national community, however care should then be taken when
configuring policies that are applied based on communities.
This community should be set outbound when peering from the Edge / Border router to the route-
reflectors,
Internet Services
!
router bgp 65501
neighbour <RR IP Address> route-map setcommunity out
!
route-map setcommunity permit 10
Set community 65501:xxxx additive
!
Figure 115 BGP Community Tagging
Note, the keyword additive adds the community to any existing communities that exist on the route
already, without the additive keyword any existing communities would be overwritten by the single
community being set in the route-map.
BGP Community Formatting

Since the release of IOS 12.0 it has been possible to configure BGP communities in three different
formats: Decimal, Hexadecimal and AA:NN. Cisco IOS defaults to the Decimal format by default.
To utilise the new style formatting where AA represents the AS number and NN is a 2 bytes community
value we need to configure the router.
!
ip bgp-community new-format
!
Figure 116 BGP community Format
This will show the communities (as seen in a show running-config) in the new format, however data can
still be entered in any of the three supported formats.
BGP Fast-external-fallover
BGP fast-external-fallover is an option that may be disabled on eBGP peering routers (or on routers with
known unstable interfaces) where the Wateen Telecom network peers with outside service providers or
large clients with eBGP peerings. Typically with eBGP peers (as opposed to iBGP peers) the peering is
achieved using link addresses, rather than loopback addresses.
With fast-external-fallover, if the directly connected link for BGP peering fails then the session is
immediately reset. This then means that when the link recovers all the BGP routes have to be re-
advertised again causing CPU spikes and unnecessary potentially large BGP updates.
Fast-external-fallover is by default, enabled on Cisco routers, to disable this the syntax is:

Internet Services
!
router bgp 65501
no bgp fast-external-fallover
!
Figure 117 BGP Disable Fast Fallover
Usage of this knob (i.e. disabling fast-external-fallover) is dependent on the link stability and facilities
condition, in a stable environment the feature may be deployed to deal with a short hit on the link.
BGP Dampening
BGP route dampening is a feature that is designed to minimise the propagation of flapping routes
throughout the Internet.
If a link between differing ISPs flaps then the routes that that may be advertised will be withdrawn, this
has a rippling effect as these withdraws are propagated through the Internet, then during this propagation
the link returns and re-advertises the routes. When this process occurs several times in a short period of
time we have a number of routes that are de-advertised and then re-advertised, this propagated through
the Internet can cause routing instabilities and cause higher than normal CPU utilisation.
BGP route dampening uses a number of configurable options to enable dampening to a peer.
Table 21 BGP Route Dampening Parameters
Terminology Description Default Value

Suppress Value When we start to suppress a route 2000
Half-Time (or Half-life) Time after which the penalty is halfed 15 (Minutes)
Reuse Value When we start to reuse a route 750
History State Maximum time to suppress a stable route 4 x Half Life (in
Minutes)
Penalty Penalty assigned when a route flaps 1000
Damp State Current state of a dampened route N/A
BGP route dampening operates in the following fashion (as defined by RIPE229),
1. For each flap, a penalty (1000 points) is incrementally added to the routes advertised.
2. As soon as the penalty exceeds the suppress value the advertisement of the route will be
suppressed.
3. The penalty is then exponentially decayed using the pre-configured half-life value.
4. Once the value has fallen below the reuse value then the route is re-advertised.
The penalty is decayed at a granularity of 5 seconds and the entries are un-suppressed with a granularity
of 10 seconds.
Note that external routes that have been learnt through iBGP are not dampened (these are routes within
our own network and hence do not affect external peers and the Internet in general).

Internet Services
!
router bgp 65501
bgp dampening 15 750 2000 60
!
Figure 118 BGP Dampening Configuration
The mathematical equation for calculating the maximum penalty is:
MAX − SUPRESS − TIME

MAX − PENALTY = ( REUSE − LIMIT * 2)^ ( )
HALF − LIFE
The following shows some examples, good and bad, of BGP dampening configuration:
• Bgp dampening 30 2000 3000 60 – These are valid BGP dampening values, the maximum
suppress time is 60 minutes with a reuse-value of 2000, the maximum penalty is 8000
(2000*2)^(60/30) == 4000*2 == 8000. The example suppress limit of 3000 means that penalties
can easily exceed this limit. If the maximum penalty is incurred then the penalty will decay from
8000 to 4000 after one half-life, which is 30 minutes, and to 2000 after two half-lifes, which is
one hour.
• Bgp dampening 15 750 3000 45 – These are again valid BGP dampening values, the maximum
suppress time is 45 minutes, with a reuse-value of 750, the maximum penalty is 6000
(750*2)^(45/15) == 1500*2*2 == 6000, the suppress limit of 3000 is easily reachable and
penalties can exceed this value. If the maximum penalty is incurred then the penalty will decay
from 6000 to 3000 after one half-life (15 minutes)
• Bgp dampening 15 500 2500 30 – This is an example of an illegal set of dampening values that
will cause no routes to be dampened. The maximum suppress time is 30 minutes, so working the
equation out to reach the re-use limit of 500 we will need a route with a max penalty of 2000 to
be decayed through two half lives (2000 -> 1000 ->500) in order to be reused. However
considering we do not start suppressing routes until a penalty of 2500 then these are invalid
combinations.
This shows the importance of making the suppress limit LESS than the maximum possible value that the
route can achieve, otherwise no suppression will take place. It is important to note that the prefixes will
only be suppressed when a neighbour advertises them and it has a flap-history and a penalty value which
will cause them to be suppressed. A prefix will attract the 1000 penalty when the WITHDRAW is heard
from the neighbour eBGP peer. However because the prefix has been withdrawn it is not present in the
BGP table apart from the history entry. The penalty will start to decay, when the prefix is reannounced if
the penalty is still above the prefix limit the prefix will be suppressed, if the penalty has fallen below the
reuse limit it will not be suppressed, as an example:
• Bgp dampening 30 751 3000 60 – will generate a maximum penalty of 3004, which is only 4
higher than the suppress limit, the fastest the prefix would re-appear in the BGP table is one
minute later – when the BGP process runs, during that intervening time it would have attracted a
delay enough to have the penalty value below 3000 and not cause any dampening, if we had a
more realistic re-use limit of around 800 then the decay would be still high enough to cause the
prefix to be suppressed.
BGP Scaling
Peer Groups:

Internet Services
Update generation for BGP peers without the use of Peer Groups requires that the entire BGP table is
walked for every peer, the selected prefixes to be selected are then filtered through any outbound policies.
Then the updates are generated and sent to one peer. This process is repeated for each peer making this a
CPU intensive operation.
With peering groups the BGP table is walked only once, this is then sent to the peer-group leader after
applying all outbound filtering policies. These updates are then replicated for each peer-group member;
this ensures the peers are synchronised and receive the same information.
Peer-Group members are defined as being synchronised if all updates sent to the leader have also been
sent to the particular member. If the peer-groups are in a constant synchronised state then BGP can
replicate the update messages, this replication process is significantly faster then formatting an update
because this does not require a walk of the BGP table and the application of performing outbound routing
policies.
Peer groups may become out of synchronisation if the router in question is receiving slow TCP
throughput; differing CPU speeds or if the peer has a high CPU utilisation performing other tasks.
It is worth noting that the use of peer groups in a route-reflector design offers between 35% and 50% in
BGP scalability, allowing us to push more routes to more peers with less CPU utilisation in a shorter
period of time.
ip tcp path-mtu-discovery
Every TCP session has a limit in terms of how much data it can transport in a single packet. This limit is
defined as the Maximum Segment Size (MSS) and is 536 bytes by default. This means TCP will take all
of the data in a transmit queue and break it up into 536 byte chunks before passing packets down to the IP
layer. Using a MSS of 536 bytes ensures that the packet will not be fragmented before it gets to its
destination because most links have a MTU of at least 1500 bytes. The following command will allow
you to check the MSS of all BGP peers:

The problem is that using such a small MSS value creates a large amount of TCP/IP overhead, especially
when TCP has a lot of data to transport like it does with BGP. The solution is to dynamically determine
how large the MSS value can be without creating packets that will need to be fragmented. This is
accomplished by enabling "ip tcp path-mtu-discovery" (PMTU). PMTU allows TCP to determine the
smallest MTU size among all links between the ends of a TCP session. TCP will then use this MTU
value minus room for the IP and TCP headers, as the MSS for the session. If a TCP session only
traverses Ethernet segments then the MSS will be 1460 bytes. If it only traverses POS segments then the
MSS will be 4430 bytes. The increase in MSS from 536 to 1460 or 4430 bytes reduces TCP/IP overhead,
which helps BGP converge faster. Once we have enabled this knob and reset all of our BGP sessions via
a reboot or "clear ip bgp *" we can see the expected increase in MSS:
It is recommended to turn on TCP MTU discovery on all 12xxx and 7x00 network routers.
SPD Headroom
Traditionally the default SPD headroom has been 100 packets. In later releases this has been changed to
1000. Starting 12.0(22)S it is changed to 1000 and hence need not be changed. This should be confirmed
by typing sho spd or sho ip spd in privileged exec mode.
Internet Services
For more details please see:

http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a008012fb87.shtml
Input Queue Size:

Other ways to improve scalability and BGP convergence is to modify both the TCP windowing and input
queue sizes, this prevents a rush of ACKs from peers towards the Route Reflector filling the default input
queue (75 packets), increasing these input queues allows us to more than double the numbers of peers we
can push routes to.
It is recommended to increase the Interface Hold-Queue (on each interface) to 1500 (an increase from the
default of 75) by using following CLI:
!
interface pos 6/0
hold-queue 1500 in
!
Figure 119 Hold Queue Configuration
The Hold-queue refers to the buffer size on the GRP/Sup etc that is used to hold packets that have been
punted from the linecards, typically in the GSR’s distributed environment this is a relatively small
number of packets (IGP/BGP updates etc).
It is recommended to increase the interface hold-queue on the interfaces of all Routers.
BGP Timers
Several BGP timers can be used to tune the iBGP convergence in an IP MPLS / VPN Network.
• Advertisement-interval: To set the interval between the sending of BGP routing updates. The default
interval is 5 seconds for iBGP peers.
• Keepalive: Frequency, in seconds, with which the Cisco IOS software sends keepalive messages to its
peer. The default is 60 seconds.
• Holdtime: Interval, in seconds, after not receiving a keepalive message that the software declares a
peer dead. The default is 180 seconds.
• Scan-time: it defines the frequency at which the router validates routes in BGP table. It validates the
locally originated network and next-hop within the BGP table. Valid values used for selecting the
desired scanning interval are from 5 to 60 seconds. The default scan time is 60 seconds.
• Scan-time import: this timer is relevant for VPNv4 address-family. It defines the frequency at which
routes received from another PE through MP-BGP are imported into a VRF. This timer is not
relevant for next-hop deletion that is processed based on the “normal” BGP scan-time. Default value
is 15 seconds.
The following table summarizes the default values for the iBGP timers.
Table 22 – Default BGP timers

Internet Services
BGP Timers iBGP Default value eBGP Default value

Advertisement-interval 5 sec 30 sec
Keepalive 60 sec 60 sec
Holdtime 180 sec 180 sec
Scan-time 60 sec 60 sec
Scan-time import 15sec 15sec
• iBGP convergence follow IGP convergence, This means that when a PE router or its uplinks fail,
BGP next-hop reachability will be lost after the IGP converges. The BGP scanning process will
detect the loss of the next-hop for all relevant BGP and MP-BPG routes and stop selecting it in
the BGP RIB.
• It is recommended initially to leave the timers at their default.
• In current release on 7600, BFD is not available for BGP BFD will be really relevant for eBGP
sessions and is not deemed significant for Wateen Telecom at this stage.
As discussed in the IGP routing section, the IGP convergence has already been tuned to provide fast
convergence under the event of an internal link or node failure.
This means that when a PE router or it’s uplinks fail, BGP next-hop reachability will be lost after the IGP
converges. The BGP scanning process will detect the loss of the next-hop for all relevant BGP and MP-
BPG routes and stop selecting it in the BGP RIB.
It is recommended initially to leave the timers at their default, unless there is a specific convergence
requirement for BGP.
Table below shows potentially tuned BGP tuned times, these should be modified with care and only after
ensuring stability of the network infrastructure
Table 23 - Potential Modified values for iBGP timers
BGP Timers iBGP Tuned value

Advertisement-interval 5 sec
Keepalive 10 sec
Holdtime 30 sec
Scan-time 30 sec
Scan-time import 15sec
BGP Configuration
• The BGP AS# for Wateen Telecom will be 65501 (interim AS number as indicated by the customer)
• iBGP Neighbour addresses will always be to the public loopback addresses

Internet Services
• BGP Router ID should be configured as per the public loopback address

• The Public Loopback interfaces will be used as the update-source locally
• To consider the Multi-Exit-Discriminator (MED) attribute in comparison of routes from the same
AS#
• To consider the absence of a MED attribute as an infinitely large value and, therefore de-prefer
routes without MED in comparison to those with
• To disable synchronisation between BGP and the IGP forwarding table
• To Log changes of neighbour state on both EBGP and IBGP sessions
• To allow BGP (MP-BGP) to propagate both IPV4 unicast and VPN-V4 address families
• BGP version should be set to 4
• BGP will be enabled to use new format community strings
• BGP peer-groups (and templates) should be used where appropriate
• All internal neighbours should pass on community information
• Next hop-self to be used for PE’s with EBGP peering
• Auto-summarisation should be disabled
• All BGP sessions should be protected with a password to encrypt the traffic (password encryption
will be MD5)
• BGP TTL Security hack should be configured on all directly connected eBGP sessions (not
applicable for eBGP Multihop)
• Path MTU Discovery (PMTUD) should be enabled for all peers, this can be selectively disabled for
eBGP peers if the interconnects cannot support high MTU.
• Fast Session Deactivation should be disabled for directly connected eBGP peers to avoid small flaps
causing tear-down of BGP sessions
NAT Service
For end user internet access (i.e.customers not doing BGP peering), it is expected that most of the
internet users will be using rfc1918 private ip address space. These users will be NATed (doing Port
Address Translaton since N to 1 mapping) to an interface address using following CLI:
ip nat inside source list 1 interface pos0/0 overload
int pos0/0
ip nat outside
int vlan10
ip nat inside
access-list 1 permit <source ip adddress> <wild-card mask>

Figure 120 Sample NAT Configuration
It is suggested to use a separate loopback address for NATing all the internet traffic, hence keeping a
clear distinction with infrastructure traffic vs user internet traffic. However in the interest of conserving ip
address space it is possible to NAT the users to one of the egress interfaces of PE device.
Please note that with above configuration, all CPE devices which are getting NATed by the PE device,
will no longer be reachable (as they are being NATed) without access to PE device first. Should this be a
concern, a static NAT mapping can be done per device, such that various devices can be reached directly.
This may be required for a NMS application managing CPE devices which are only in the global routing
table. i.e. a Customer’s CPE which has only subscribed to internet service.
Following example would allow telnet access to NATed hosts at different ports of public ip address:

Internet Services
ip nat inside source static tcp <interface-ip-address> 2001 inside ip-address1 23

ip nat inside source static tcp <interface-ip-address> 2002 inside ip-address2 23
.
.
.
ip nat inside source static tcp <interface-ip-address> 2xxx inside ip-addressN 23
Figure 121 Sample NAT Port Redirection
For Wateen Telecom, another alternative to this approach is to bypass NAT for certain source and
destination prefiexes using extended ACLs (access-lists) while performing NAT. This approach is
simpler and recommended, as long as both prefixes are reachable within Wateen autonomous system via
IGP or BGP.
Ingress Access-Lists
For customer facing ports, it is suggested to deploy access-lists filtering certain traffic which can
commonly be exploited. e.g. permitting all traffic except icmp traffic etc.
Unicast Reverse Path Forwarding (uRPF)

Traditionally it was a challenge stopping attacks where the source of the IP packet was altered. In Cisco
IOS there is a feature called uRPF that makes dealing with these ip packets whose source ip address has
been forged much easier.
The concept of uRPF is to verify all packets when they enter the router, and not only to a forwarding
lookup, but also validate the interface the traffic was received on.
There are two different flavors of uRPF:
Strict, will verify that the received traffic is coming in on the Interface that the router think is the optimal
path from the source
Loose, will verify that the source address of the packet exists with a valid next hop in the routing table. If
the next hop is equal to null, the packet will be seen as non-valid and accordingly it will be dropped.
On the Cisco 7600 router uRPF processing is done in hardware, resulting in no performance impact by
enabling this feature.
If uRPF is configured in combination with ACLs, the ACLs are checked before and after the reverse path
validation as explained in the following:
Step 1 - Check input ACLs configured on the inbound interface.
Step 2 - uRPF validates the packet has a return path through the inbound interface by checking the CEF
table (CEF or dCEF).
Step 3 - CEF table (FIB) lookup is carried out for packet forwarding.
Step 4 - Output ACLs are checked on outbound interface.
Step 5 - Packet is forwarded.
Unicast RPF’s advantage when used for IP source address validation is that it dynamically adapts to
changes in the routing tables, including static routes. Additionally it requires less operational maintenance
than traditional approaches which use IP access or extended-access lists
Internet Services
To configure uRPF, use the command:

ip verify unicast source reachable-via {rx | any} allow-self-ping
Figure 122 uRPF Configuration
The rx keyword enables uRPF in strict mode, the any keyword enables uRPF in loose mode. It is
important to add the “allow-self-ping” at the end of this command. If allow-self-ping is not present, the
uRPF will filter out “self-ping” packets from the router, which can be a useful troubleshooting tool.
In the Wateen Telecom network uRPF should be used on all edge interfaces.
There will be two types of external access connections, single home and dual homed links.
• Single homed connection. In this scenario, a “simple” connection exists from SP core to the customer
edge i.e. traffic will always be synchronous to and from this customer. This is due to the fact that the
customer only has one uplink, and no “dual-homing” is present. This is expected to be majority of
Wateen Telecom’s customer access scenario.
• Multi homed connections: In this scenario, because of redundant links we can not ensure symmetric
routing. All Internet connections are classified as connections where traffic flow may be asymmetric,
as well as all customer connections where the customer has multiple connections to one or multiple
service providers .
For single homed connections, the following command should be applied on the customer facing
interface:
ip verify unicast source reachable-via rx allow-self-ping
Figure 123 uRPF strict mode
For multi homed connections, the following command should be applied on the customer facing
interface:
ip verify unicast source reachable-via any allow-self-ping
Figure 124 uRPF Loose Mode
ACL at the Internet Border Routers:

At the Internet borders it is recommended to implement access-lists that stop some specific traffic, both
inbound and outbound. The main purpose of these ACL's is to prevent traffic to and from non-valid IP
addresses.
Inbound traffic:
In the inbound direction (traffic coming from the Internet towards the Wateen Telecom network) an ACL
should be deployed that stops traffic with an source IP address that is a RFC1918 or RFC 3330 address,
or that have a Wateen Telecom owned subnet as the source address. For more details on Cisco
recommendation on Infrastructure ACL, please refer to public white paper at
http://www.cisco.com/en/US/tech/tk648/tk361/technologies_white_paper09186a00801a1a55.shtml

Internet Services
!--- Deny special-use address sources.

!--- Refer to RFC 3330 for additional special use addresses.
access-list 177 deny ip host 0.0.0.0 any

access-list 177 deny ip 255.255.255.255 any
access-list 177 deny ip 127.0.0.0 0.255.255.255 any
!--- Filter RFC 1918 space.

!--- Deny your space as source from entering your AS.

!--- Deploy only at the AS edge.
!--- Permit transit traffic.
access-list 177 permit ip any any

!
!
int pos 1/0
ip access-group 177 in
!
Figure 125 Sample ingress ACL Configuration for Internet Gateway
Outbound traffic:
For outbound traffic towards the Internet ACL’s should be configured to stop known “non-valid” source
addresses, like RFC1918 and RFC 3330 address.
!--- Deny special-use address sources.

!--- Refer to RFC 3330 for additional special use addresses.
access-list 178 deny ip host 0.0.0.0 any

!--- Filter RFC 1918 space.

!--- Permit transit traffic.

!
!
int pos 1/0
ip access-group 178 out
!
Figure 126 Sample egress ACL Configuration for Internet Gateway

Internet Services
Data Center Connectivity

As previously indicated there will be a data center in Lahore (Main PoP). The data center will house two
6513 switches. These switches will connect to IP/MPLS core as show in Lahore PoP diagram with dual
cross connected Gigabit Ethernet links.
These 6513 will participate in global routing table via participating in OSPF. It is understood that
carrying a larg routing table will not be very useful as the routers only need a default route back to
upstream devices. It is therefore suggested that a totally stubby area be introduced to connect these Data
Center devices to IP/MPLS core. By configuring totally stubby area, LHE-7613-PE1 and LHE-7613-PE2
both will introduce a redundant default route for totally stubby area.
It is recommended that totally stubby area be numbered as area 1. Following configuration will be
required on both PEs and 6513 DC switches for area 1 configuration.
router ospf 1
area 1 stub no-summary
!
Figure 127 OSPF Totally Stubby Area Configuration
The “no-summary” keyword is only required at ABR i.e. Lhe-76130-PE1&2. For further details on Data
Center architecture and related services please refer to relevant LLD document.
Internet Customers Connectivity

Based on discussions with the customers, it is understood that customer edge device (referred by the
customer as IAD) will need to request an ip address via DHCP on bootup. This ip address will be
assigned by CNR (Cisco Network Registerar) which will be located in Data Center. The DHCP request
will comein via global routing table. Given that most customers will be addressed via RFC1918 address
space (as documented earlier), it is understood that each interface on 7600 series PE router will have an ip
address as well as two helper address pointing to two CNR servers in Data center.
interface vlan10
ip address <interface ip address>
ip helper-address <dhcp (CNR) server-1>
ip helper-address <dhcp (CNR) server-2>
!
Figure 128 IP Helper Address Configuration

In order for customer edge device to get a DHCP address, the ip address of interface vlan10 should be
visible to CNR and vice versa.
It is recommended to summarize all interfaces on each 7600 PE and this summary in turn should be
advertised via BGP or IGP. It is considered best practice to carry customer prefixes in BGP to help scale
customer services. However it must be noted that BGP is meant to scale very well but can’t converge as
quickly as IGP.
It is recommended to configure a summary for all possible prefixes and advertise this summary into
OSPF by redistributing static. Following shows an example of such a summary in OSPF.

Internet Services
ip route 194.168.0.0 255.255.0.0 Null0

router ospf 1
redistribute static metric-type 1 subnets
!should there be multiple static routes on the router, route-map can be
!added to filter only desired summary to be redistributed in OSPF
redistribute static metric-type 1 subnets route-map stat-sum

route-map stat-sum permit 10
!access-list 121 matches desired summary prefix to be adv by OSPF
access-list 121 permit ip 194.168.0.0 0.0.255.255 255.255.0.0 0.0.255.255
Figure 129 Summary Route Redistribution into OSPF
By allowing summarized prefixes into IGP, individual subnets are no longer required to be advertised in
area 0. This approach would allow IGP to scale while providing optimal reachability.
Same objective can be accomplished via advertising the summary into BGP. Care must be taken such that
these RFC1918 addresses are not advertised externally. In order to achieve this objective, these summary
prefixes can be tagged with a no-export community string. IF BGP was used to carry these prefixes,
following configuration will be required.
ip route 194.168.0.0 255.255.0.0 Null0
router bgp 65501
<standard BGP configuration for a PE>
address-family ipv4
!configure community to be sent to BGP Peers
neighbor refip send-community both
neighbor refip route-map setcom out
neighbor 10.100.0.1 activate
neighbor 10.100.0.4 activate
no auto-summary
no synchronization
network 194.168.0.0 mask 255.255.0.0
exit-address-family
route-map setcom permit 10
set community no-export
access-list 112 remark rfc1918 adv via BGP with community set
access-list 112 permit ip 194.168.0.0 0.0.255.255 255.255.0.0 0.0.255.255
Figure 130 Summary Route Redistribution via iBGP
Once the Customer Edge (CE) device has obtained an ip address via DHCP, any traffic destined for
internet will be NATed by respective 7600 PE device.
Dual Attached PEs

For Faislabad PEs where there are two PEs connecting directly to WiMAX infrastructure (unlike Lahore,
Karachi and Islamabad, where only metro rings are anticipated for customer connectivity), it has been
advised by Cisco partner Motorola that there is an access architecture which allows for dual connectivity
to each PE from access switch using Microwave infrastructure. It has been advised that we should treat
this infrastructure as if access switch is dual connected to both PEs carrying same VLANs on both links.
Granularity of the details in terms of how access infrasturcure works from access switch to PE devices is
primed by Motorola. It has been advised that from core perspective it is as if access switch is connected
via two cables, to both PEs in Faislabad PoP. Each cable is a trunk carrying same set of vlans to both PEs.

Internet Services
In this case it is anticipated that each prefix for a customer facing network will span both PE devices. i.e.
same vlan/prefix (e.g. vlan 10 will terminate on PE-1 and PE-2, prefix assigned to vlan10 e.g.192.168.1.0
will appear on both PEs). It is understood that layer-2 connectivity will be provided by access switch (to
avoid any spanning tree issues). In this case, both PEs will advertise static summaries into area 0. This is
in line with other PEs (which will be advertising static summaries). Since it is possible that one of the PEs
may loose an access link and continue to advertise a summary, a separate routing process (e.g. RIP) will
be configured between both PEs over a dedicated link (or a dedicated vlan). This link will advertise
specific prefixes/subnets to its neighbor so as to avoid blackholing any traffic.
An alternative to this approach is to define a separate area between two PEs, with a dedicated link
between the two PEs. This way both PEs will become ABR (area border routers) and OSPF can advertise
summaries from each PE for non backbone area into area 0. ABR will summarise their respective
prefixes using the router ospf command: “area area-id range address mask”. By using this
summarization technique the discard route (summary pointing to null0) is automatically generated by
default by OSPF process. Either approach will work for ipv4 traffic. Based on the information available
to date for dual attached PE scenario, later approach is suggested.
Transit BGP Customers

Wateen Telecom will provide transit BGP peering services to it customers also. Transit services will be
offered based on the fact that customer already has a publicl automous system and ip address space and
wants to be dual/multi homed using BGP. In this case Wateen Telecom will provide eBGP service to this
customer providing its own prefixes as well as all internet prefixes. For such a customer following will be
observed:
• eBGP session will be setup with a customer router
• IP address for point to point link to the customer will be provided by Wateen Telecom
• This IP address (a /30 prefix) will be routed via IGP within Wateen Telecom towards border
edge router (from Wateen Telecom public ip address space)
• BGP will be used to learn customer prefix(es)
• BGP will be used to limit the number of prefixes (as agreed with the customer)
• Ingress/egress ACLs will be used on the customer interface as shown in Figure 120 and 121
(ACLs at Internet Border router)
• uRPF will be used on customer facing interface
• BGP TTL Security Check will be enforced for directly connected neighbors
• BGP Dampening will be used (assuming stable underlying transport infrastructure)
• BGP Fast externall fallover will be disabled (assuming stable underlying transport infrastructure)
• BGP neighbor will use password to enforce authenticated session
• BGP description command will be used to identify the customer and related description

Quality of Service
Quality of Service Overview
Introduction
In order to fulfill Wateen Telecom requirements of having 5 distinct classes of service, each with their
specific service characteristics, QoS mechanisms will be deployed in the core of the network. Wateen
Telecom has requested for following classes:
• Real Time Traffic (LLQ, voice)
• Video Traffic (potential future requirement)
• Management Traffic (snmp, management, signaling)
• Business Data (Premium Traffic)
• Best Effort (Internet Traffic)
It is to be noted that indeed, the CE to PE link is the most bandwidth constrained and as a result,
congestion will occur most likely either on the CE (egress to the PE ) or on the PE (egress to the CE).
It is understood that WiMAX, WiMAX aggregaton and Microwave infrastructure can provide equivalent
five classes of traffic with appropriate prioritization to keep end to end Qos commitment and SLA to the
end customer.
Characteristics of each class are given below:
• Real-Time (VoIP) traffic. A minimum bandwidth together with minimum loss, delay and jitter needs
to be guaranteed. It is advised that the VPN customer implements some kind of Call Admission
Control method for their Voice traffic. This is required because, in the event that a VPN customer
exceeds the amount of Voice traffic that has been allocated in their Voice Class of Service, Voice
traffic could be dropped in case of congestion, irrespective of to which voice call this traffic belongs.
This would then potentially lead to degradation in Voice quality for all voice calls.
• Video Traffic (potential future requirement) : Streaming Video is considered a class with minimum
bandwith / service gaurntee. Streaming video applications have more lenient QoS requirements
because they are delay-insensitive and are largely jitter-insensitive (because of applicationbuffering).
• Business/Premium traffic. This is mission critical traffic for which the VPN customer requires
preferential treatment. A minimum bandwidth needs to be guaranteed to see SLA.

Quality of Service
• Manage\ment Class: This is a class carrying all inband management traffic and is protected with
minum bandwidith gaurntee.
• Best Effort traffic. This traffic requires no special of preferential treatment. This would typically be
E-mail and Internet type traffic. A (low) minimum bandwidth needs to be guaranteed.
Note: typically QOS is described as IP Precedence, however in MPLS this maps to MPLS EXP bits
as this is how the QOS is delivered in an MPLS environment, IP Precedence is or can be maintained
throughout the network even though the SP can modify the MPLS EXP bits to suit their internal
QOS policies.
QOS Disclaimer
Since no clear requirements were given by Wateen Telecom in terms of expected traffic patterns, or the
strategy with regards to what type of QoS packaging that should be offered, what qos guarntees should be
made for each class, amount of expected link utilization etc. The QoS section in this document will
contain the following sections
• Generic QoS recommendation
• QoS architectures for the Cisco Platforms deployed in Wateen IP/MPLS core
• Generic configuration templates for the Cisco devices for QoS
The following assumptions will be made:

• 40 % of the available link bandwidth will be reserved for the real time class
• 20 % of the available link bandwidth will be reserved for the “video traffic”
• 10 % of the available link bandwidth will be reserved for the business traffic
• 10% of the available link bandwidth will be reserved for the management traffic
• The remaining bw will be allocated for the best effort traffic (Internet traffic)
These assumptions will be applied on all CORE Links, CORE-DIST, DIST-DIST and DIST-ACCESS
links.
Differentiated Services model – an introduction
This section is intended as an introduction to the Differentiated Services (DiffServ) reference model.

Quality of Service
DiffServ is a new model by which traffic is treated by intermediate systems with relative priorities based
on the type of services (ToS) or Differentiated Services Code Point (DSCP) field. Defined in RFC’s 2474
and 2475, the DiffServ standard supersedes the original specification for defining packet priority
described in RFC 791.
The new DiffServ standard proposes a new way of interpreting a field that has always been part of an IP
packet. In the DiffServ standard, the ToS field will be renamed to Differentiated Services Code Point
(DSCP) and will have new meaning. The DiffServ standard proposes to increase the number of definable
priority levels by re-allocating bits of an IP packet for priority marking.
The ToS field describes one entire byte (eight bits) of an IP packet. Precedence refers to the three most
significant bits of the ToS field---that is, [XXX]XXXXX. There may be some confusion because people
occasionally use the term ToS to refer to the next three bits---XXX[XXX]XX. To be consistent with the
original RFC 791 specification, this document uses the term ToS to refer to the entire eight bits.
The three most significant bits of the ToS field---the precedence bits---define the IP packet priority under
RFC 791. The precedence portion of the ToS field---[XXX]XXXXX---defines the priority or importance
of the field. The following diagram and tables illustrate the old method used to interpret this eight-bit
portion of the IP packet.
Figure 131 Breakdown of the IP ToS Field
Table 24 - TOS Field
0-2 3 4 5 6 7
Precedence Delay Throughput Reliability Reserved = 0 Reserved = 0
XXX00000 Bits 0,1,2 = Precedence, where:

111 = Network Control = Precedence 7
110 = Internetwork Control = Precedence 6
101 = CRITIC/ECP = Precedence 5
100 = Flash Override = Precedence 4
011 = Flash = Precedence 3
010 = Immediate = Precedence 2

Quality of Service
001 = Priority = Precedence 1

000 = Routine = Precedence 0
000XXX00 Bits 3, 4, 5:
Bit 3 = Delay [D] (0 = Normal; 1 = Low)
Bit 4 = Throughput [T] (0 = Normal; 1 = High)
Bit 5 = Reliability [R] (0 = Normal; 1 = High)
000000XX Bits 6,7: Reserved for future use
This one-byte ToS field has been almost completely unused since it was proposed almost 20 years ago.
Only in the last few years have Cisco and other router companies begun utilising the Precedence bits for
making forwarding decisions.
The DiffServ standard follows a similar scheme to RFC 791, but utilises more bits for setting priority.
The new standard maintains backward compatibility with RFC 791 implementations, but allows more
efficient use of bits 3, 4, and 5. (Bits 6 and 7 will still be reserved for future development.) With the
additional 3 bits, there are now a total of 63 classes instead of the previous 7 classes.
RFC 2475 defines Per Hop Behaviour (PHB) as the externally observable forwarding behaviour applied
at a DiffServ-compliant node to a DiffServ Behaviour Aggregate (BA).
With the ability of the system to mark packets according to DSCP setting, collections of packets with the
same DSCP setting and sent in a particular direction can be grouped into a BA. Packets from multiple
sources or applications can belong to the same BA.
In other words, a PHB refers to the packet scheduling, queuing, policing, or shaping behaviour of a node
on any given packet belonging to a BA, as configured by a service level agreement (SLA) or a policy
map.
The following sections describe the four available standard PHBs:

• Default PHB (as defined in RFC 2474)
• Class-Selector PHB (as defined in RFC 2474)
• Assured Forwarding (AFxy) PHB (as defined in RFC 2597)
• Expedited Forwarding (EF) PHB (as defined in RFC 2598)
Default PHB
The default PHB essentially specifies that a packet marked with a DSCP value of 000000 (recommended)
receives the traditional best-effort service from a DS-compliant node (that is, a network node that
complies with all of the core DiffServ requirements). Also, if a packet arrives at a DS-compliant node,
and the DSCP value is not mapped to any other PHB, the packet will get mapped to the default PHB.
For more information about default PHB, refer to RFC 2474, Definition of the Differentiated Services
Field in IPv4 and IPv6 Headers.

Quality of Service
Class-Selector PHB:
To preserve backward-compatibility with any IP Precedence scheme currently in use on the network,
DiffServ has defined a DSCP value in the form xxx000, where x is either 0 or 1. These DSCP values are
called Class-Selector Code Points. (The DSCP value for a packet with default PHB 000000 is also called
the Class-Selector Code Point.)
The PHB associated with a Class-Selector Code Point is a Class-Selector PHB. These Class-Selector
PHBs retain most of the forwarding behaviour as nodes that implement IP Precedence-based
classification and forwarding.
For example, packets with a DSCP value of 110000 (the equivalent of the IP Precedence-based value of
110) have preferential forwarding treatment (for scheduling, queuing, and so on), as compared to packets
with a DSCP value of 100000 (the equivalent of the IP Precedence-based value of 100). These Class-
Selector PHBs ensure that DS-compliant nodes can coexist with IP Precedence-based nodes.
The DiffServ standard utilizes the same precedence bits (the most significant bits: 0, 1, and 2) for priority
setting, but further clarifies their functions/definitions, plus offers finer priority granularity through use of
the next three bits in the ToS field. DiffServ reorganizes (and renames) the precedence levels (still
defined by the three most significant bits of the ToS field) into the following categories:
Table 25 – Precedence Levels
Precedence 7 Stays the same (link layer and routing

protocol keep alive)
Precedence 6 Stays the same (used for IP routing protocols)
Precedence 5 Class 5
Precedence 0 Best effort
For more information about class-selector PHB, refer to RFC 2474, Definition of the Differentiated
Services Field in IPv4 and IPv6 Headers.
Assured Forwarding PHB

Assured Forwarding PHB is nearly equivalent to Controlled Load Service available in the integrated
services model. AFxy PHB defines a method by which BAs can be given different forwarding assurances.
For example, network traffic can be divided into the following classes:
• Gold: Traffic in this category is allocated 35 percent of the available bandwidth.
• Silver: Traffic in this category is allocated 40 percent of the available bandwidth.
• Bronze: Traffic in this category is allocated 25 percent of the available bandwidth.

Quality of Service
Further, the AFxy PHB defines four AF classes: AF1, AF2, AF3, and AF4. Each class is assigned a
specific amount of buffer space and interface bandwidth, according to the SLA with the service provider
or policy map.
Within each AF class, you can specify three drop precedence (dP) values: 1, 2, and 3. Assured
Forwarding PHB can be expressed as shown in the following example:
AFxy
In this example, x represents the AF class number (1, 2, or 3) and y represents the dP value (1, 2, or 3)
within the AFx class.
In instances of network traffic congestion, if packets in a particular AF class (for example, AF1) need to
be dropped, packets in the AF1 class will be dropped according to the following guideline:
dP(AFxy) <= dP(AFxy) <= dP(AFxy)
where dP (AFxy) is the probability that packets of the AFxy class will be dropped. In other words, y
denotes the dP within an Afx class.
In the following example, packets in the AF13 class will be dropped before packets in the AF12 class,
which in turn will be dropped before packets in the AF11 class:
dP(AF13) <= dP (AF12) <= dP(AF11)
The dP method penalises traffic flows within a particular BA that exceed the assigned bandwidth. Packets
on these offending flows could be re-marked by a policer to a higher drop precedence.
An AFx class can be denoted by the DSCP value, xyzab0, where xyz can be 001, 010, 011, or 100, and ab
represents the dP value.
Bits 3 and 4 of DiffServ field allow further priority granularity through the specification of a packet drop
probability for any of the defined classes. Collectively, Classes 1-4 are referred to as Assured Forwarding
(AF). The following table illustrates the DSCP coding for specifying the priority level (class) plus the
drop percentage. (Bits 0, 1, and 2 define the class; bits 3 and 4 specify the drop percentage; bit 5 is always
0.)
Using this system, a device would first prioritize traffic by class, then differentiate and prioritize same-
class traffic by considering the drop percentage. It is important to note that this standard has not specified
a precise definition of "low," "medium," and "high" drop percentages. Additionally, not all devices will
recognize the DiffServ bit 3 and 4 settings. Remember also that even when the settings are recognized,
they do not necessarily trigger the same forwarding action to be taken by each type of device on the
network---each device will implement its own response in relation to the packet priorities it detects. The
DiffServ standard is meant to allow a finer granularity of priority setting for the applications and devices
that can make use of it, but it does not specify interpretation (that is, action to be taken).

Quality of Service
Table 26 - Dropping Percentage
Class 1 Class 2 Class 3 Class 4

Low Drop Percentage 001010 010010 011010 100010
Medium Drop 001100 010100 011100 100100

Percentage
High Drop Percentage 001110 010110 011110 100110
Expedited Forwarding PHB

Resource Reservation Protocol (RSVP), a component of the integrated services model, provides a
Guaranteed Bandwidth Service. Applications such as Voice over IP (VoIP), video, and online trading
programs require this kind of robust service. The EF PHB, a key ingredient of DiffServ, supplies this kind
of robust service by providing low loss, low latency, low jitter, and assured bandwidth service.
EF PHB is ideally suited for applications such as VoIP that require low bandwidth, guaranteed
bandwidth, low delay, and low jitter.
The recommended DSCP value for EF PHB is 101110.
For more information about EF PHB, refer to RFC 2598, An Expedited Forwarding PHB.
It is Cisco’s experience that a Quality of Service design and deployment is never a straightforward
process – after an initial deployment, a performance assessment phase and subsequent tuning of the QoS
deployment is a necessity. Therefore, we strongly recommend a tuning phase while beta customers are
connected.
The following table and figure gives an overview of the various QoS mechanisms that can be used on the
CE (for traffic flowing from CE to PE) and on the PE (for traffic flowing from PE to CE).
Table 27 - Qos Mechanism at CE
Traffic DSCP Prec/ Classification Marking Class queuing Congestion

Class EXP avoidance
Real-Time EF 5 CE: ACL or CE: class- CE: LLQ REAL_TIME
(DSCP “match ip rtp” based PE: MDRR – queue low-
40) PE: precedence latency
Business AF41 4 CE: ACL CE: class- CE: LLQ BUSINESS WRED
(DSCP PE: precedence based PE: MDRR – queue 1
34)

Quality of Service
Best Effort DSCP 0 0 CE: ACL CE: class- CE: LLQ WRED
PE: precedence based BEST_EFFORT
PE: MDRR – queue 0
Routing DSCP 48 6 CE: ACL BGP CE: LLQ RP
Protocol PE: precedence precedence Min (8 Kbps, 1% of
6 access)
PE: MDRR – queue 3
Table 28 - CE to PE QoS mechanisms overview
Bottleneck Bottleneck
CE-Ingress PE-Ingress PE-Egress CE-Egress
classification classification classification classification

marking queuing queuing marking
queuing congestion avoidance congestion avoidance queuing
congestion avoidance congestion avoidance
Figure 132 - CE to PE QoS mechanisms overview
The following table and figure gives an overview of the various QoS mechanisms that can be used on the
PE and P routers, for traffic flowing between PE and P routers, and P to P routers.
Table 29 - PE to P and P to P QoS mechanisms overview
Traffic Class Prec/EXP Classification Class queuing Congestion

avoidance
Real-Time 5 Precedence / MDRR – queue low-
EXP latency
Business / Management 2,3,4, 6 Precedence / MDRR – queue 1 WRED
/ Routing Protocols EXP
Best Effort 0,1 Precedence / MDRR – queue 0 WRED
EXP
Bottleneck Bottleneck
PE-Ingress P PE-Egress
classification classification classification classification
queuing queuing queuing queuing
congestion avoidance congestion avoidance congestion avoidance congestion avoidance
Figure 133 - PE to P QoS mechanisms overview

The various QoS mechanisms and their detailed configuration will be discussed in detail in subsequent
sections. Sample configuration templates will be provided. It should be understood that the IP addresses,
DLCI numbers, VPI / VCI numbers, ACL numbers, etc, have been taken for the sake of examples and
should be adapted to the specific QOS requirements of the Wateen Telecom’s network.

Quality of Service
QoS Tools
There are several types of tools that are used to provide and deploy a QoS enabled network, be it local to
a network element or across the network:
• Classification and Marking
• Congestion Avoidance
• Congestion Management
• Traffic Conditioning
• Signaling
• Link Efficiency Mechanisms
The following diagram provides a visual representation of various QoS functions that are performed in a
network.
.
Classification and Scheduling, Bandwidth

Classification, Marking
Queuing Management and
and Policing
Congestion Avoidance
Policer Drop Scheduler Drop
Figure 134 - Illustration of various QoS functions
Classification
On the CE, packets will be classified through the use of extended access lists (ACLs). These ACLs can
match packets on IP Source or Destination address, protocol type, and UDP/TCP port numbers.
Dependent on the actual signalling method used (packet sizes), speed of the access links and the number
of concurrent voice call set-ups that need to be supported, 2 possible design options can be taken with
regards to the queuing method used.
• Queue the voice signalling packets in the same PQ as the actual voice bearer packets. This will result
in a simpler design but could delay the transmission of some of the voice bearer packets (dependent
on voice signalling packet size, access link speed and number of concurrent voice call set-ups). This
could than have an impact on the voice delay / jitter.
• Queue the voice signalling packets in another normal class queue. This should ideally be a separate
class queue from the ones that are used for regular data traffic to ensure delivery of the voice
signalling packets. This will result in a more complicated design where bandwidth needs to be
allocated for the voice signalling class. Also, voice signalling packets might be delayed through the
network resulting in a delay in the voice call set-up process. The advantage is that the actual voice
quality will not be impacted as no voice signalling packets will travel in the PQ.

Quality of Service
Testing has indicated that, without cRTP (Compressed Real Time Protocol) enabled, the effect of
mapping VoIP signalling packets together with the VoIP bearer packets in the same priority queue is
negligible. The signalling packets have little effect on the latency nor do they cause any drops due to the
default bust size of 200ms that has been built into the priority queue. Therefore, the design
recommendation is to match the VoIP signalling packets with ACL <x> and queue them together with the
VoIP bearer packets in the priority queue.
It should however be understood that VoIP signalling implementations differ and that some might have a
negative effect on the performance of the priority queue. In that event, the VoIP signalling traffic needs to
be mapped in another class queue (Business, Management class for example).
The classified traffic will subsequently be mapped in their respective classes. A maximum of 1024
classes can be defined on a single router, 256 classes per policy map and 256 policy maps maximum not
to exceed the 1024 classes.
For example:
class-map match-all REAL_TIME

match access-group <x>
class-map match-all RP
match access-group <x>
Figure 135 Class-maponfiguration
Note: the above samples would need to be modified to include both real ACL’s that reflect real business
traffic along with real class-maps that define the classes within the network. Following lists all possible
options in a class-map on a 7600 series router:
7606-8c(config)#class-map test
7606-8c(config-cmap)#match ?
access-group Access group
any Any packets
atm Match on ATM info
bgp-index BGP traffic index value
class-map Class map
cos IEEE 802.1Q/ISL class of service/user priority values
destination-address Destination address
discard-class Discard behavior identifier
dscp Match DSCP in IP(v4) and IPv6 packets
fr-de Match on Frame-relay DE bit
fr-dlci Match on fr-dlci
input input attachment circuits
input-interface Select an input interface to match
ip IP specific values
mpls Multi Protocol Label Switching specific values
not Negate this match result
packet Layer 3 Packet length
precedence Match Precedence in IP(v4) and IPv6 packets
protocol Protocol
qos-group Qos-group
source-address Source address
vlan VLANs to match
Figure 136 Class-map Match Examples

Quality of Service
Marking
After classification, packets need to be marked with their appropriate IP precedence or DSCP value. The
following marking mechanism will be used here.
• Class Based Marking. This is the currently recommended method of marking or colouring traffic.
Class Based Marking is part of the Modular QoS Command Line Interface (or MQC) method of
configuring QoS parameters. Class Based Marking will be used for the Best Effort, Real-Time, and
Business classes of traffic. The BGP traffic will be marked with precedence 6 by the router.
Other marking mechanisms (not used here) are.

• Class Based Marking through Class Based Policing. This could be required to introduce in / out
contract traffic parameters. This cannot be achieved through pure Class Based Marking.
• Local Policy Routing (LPR). This was a legacy method used to mark packets locally generated by a
router. As of IOS release 12.2(5), locally generated packets can be marked through Class Based
Marking.
The following is the required configuration for Class Based Marking of the Best Effort traffic. Best Effort
traffic will be coloured with a DSCP value of 0.
!
Policy-map customer_profile
class best_effort
set ip prec 0
Figure 137 Policy map configuration
Class Queuing - LLQ

Before priority scheduling can happen, voice packets will need to be classified and marked with the
appropriate DSCP value. This can happen with the same mechanisms as described earlier for the data
classes.
Once traffic has been classified, the flow can be placed into an interface egress queue that meets its
handling requirements. Voice over IP, because of its extremely low tolerance for packet loss and delay,
should be placed into a Priority Queue (PQ).
LLQ combines the use of a PQ with a class-based weighted fair queuing scheme. Classes are defined with
classification admission schemes. Traffic flows have access to either the PQ, one of the class-based
queues, or a default weighted fair queue. LLQ, the recommended queuing scheme for all low-speed links,
allows up to 1024 traffic classes with the ability to specify such parameters as priority queuing behaviour
for voice,
As depicted in then following figure, when a Priority Queuing class is configured, the PQ has direct
access to the transmit (TX) ring. This is, of course, unless interleaving is configured, in which case
interleaving occurs prior to placing the PQ traffic onto the TX-ring.

Quality of Service
Figure 138 - Packet flow with PQ

The maximum configured bandwidth in the PQ and class-based queues cannot exceed the minimum
available amount of bandwidth on the WAN connection. The bandwidth as configured in the priority
queue (“priority” under “class voice”) will be the minimum bandwidth guaranteed under congestion. An
important remark is that in current IOS versions, this bandwidth should be the “uncompressed”
bandwidth (i.e. without cRTP).
As illustrated in the following figure, a VoIP packet consists of the payload, IP header, User Datagram
Protocol (UDP) header, Real-time Transport Protocol (RTP) header, and Layer 2 Link header. The IP
header is 20 bytes, the UDP header is 8 bytes, and the RTP header is 12 bytes. The link header varies in
size according to media.
Figure 139 - Typical VoIP packet

At the default packetization rate of 20 ms, VoIP packets have a 160-byte payload for G.711 or a 20-byte
payload for G.729. The bandwidth consumed by VoIP streams is calculated by adding the packet payload
and all headers (in bits), then multiplying by the packet rate per second (default of 50 packets per
second). The following table details the bandwidth per VoIP flow at a default packet rate of 50 packets
per second (pps). This does not include Layer 2 header overhead and does not take into account any
possible compression schemes, such as compressed Real-time Transport Protocol (cRTP). While it is
possible to configure the sampling rate above 30 ms, this usually results in very poor voice quality.
CODEC Sampling rate Voice payload Packets per Bandwidth per

(msec) (bytes) second (pps) conversation
(kbps)
G.711 20 160 50 80
G.711 30 240 33 53
G.729A 20 20 50 24
G.729A 30 30 33 16
Table 30 - Bandwidth consumption - voice payload only
A more accurate method for provisioning is to include the Layer 2 headers in the bandwidth calculations,
as shown in the following table. Again, these bandwidth rates are without any voice compression schemes
such as cRTP.
CODEC Ethernet – 14 PPP – 6 bytes ATM – 53 bytes Frame Relay – 4

bytes header header cells – 48 bytes bytes header
(kbps) (kbps) payload (kbps) – excluding
(kbps) RFC 2427 (2 bytes)
G.711 at 50 pps 85.6 82.4 106 81.6
G.711 at 33 pps 56.5 54.4 70 54
G.729A at 50 pps 29.6 26.4 42.4 25.6
G.729A at 33 pps 19.5 17.4 28 17
Table 31 - Bandwidth consumption - header included

Quality of Service
For example, a G.729A codec at 50 pps would generate 25.6Kbps. 5% should be added to this to include
the VoIP signalling overhead (RTCP / H.225 / H.245). As discussed before, the VoIP signalling traffic
will be mapped together with the VoIP bearer traffic in the priority queue.
Call admission control is another important issue that needs to be considered. Call admission control is a
mechanism for ensuring that voice flows do not exceed the maximum provisioned bandwidth allocated
for voice conversations. After doing the calculations to provision the network with the required
bandwidth to support voice, data, and possibly video applications, it is important to ensure that voice does
not oversubscribe the portion of the bandwidth allocated to it. While most QoS mechanisms are used to
protect voice from data, call admission control is used to protect voice from voice. This is illustrated in
the following figure, which shows an environment where the network has been provisioned to support
two concurrent voice calls. If a third voice call is allowed to proceed, the quality of all three calls is
degraded. Call admission control should be external to the network.
Figure 140 - Call admission control

Queuing within the classes will be implemented through Low latency Queuing (LLQ). LLQ is in fact the
combination of Class Based Weighted Fair Queuing (CBWFQ) and Priority Queuing (PQ). The PQ is
used for delay sensitive traffic such as VoIP. LLQ will be configured through MQC.
Different traffic classes – a maximum of 64 traffic classes can be defined on a single router – can be
combined in a service policy. This is kind of a traffic profile. Each of the classes in the service policy will
be assigned a minimum bandwidth according to the service contract that has been agreed with the
customer. The minimum bandwidth that can be configured is 8 Kbps. Under congestion, each of the
traffic classes will have this minimum bandwidth available. Also, other parameters like congestion
avoidance and control parameters can be configured on a per-class basis. This will be discussed further
on.
The sum of the minimum bandwidths reserved for the customer traffic classes (Real-Time, Business and
Best Effort) needs to be lower than the total link bandwidth. Indeed, some bandwidth needs to be reserved
for management traffic and routing traffic. If Wateen Telecom decides to offer a managed service, it
needs to keep control over the CEs, even under congestion circumstances. Also the routing traffic –
which is eBGP in this case – needs to have some minimum bandwidth available. It is recommended that
both for the management and routing protocol a minimum bandwidth of 8 Kbps (or 1 %, whatever is
smaller) is configured. Obviously, the sum of all the minimum reserved bandwidths cannot be larger than
the total link bandwidth.
It should also be understood that the actual minimum bandwidths configured through MQC include the
following layer 2 overhead, in contrast with CAR and Police which only includes pure layer-3 IP
bandwidth. Overhead added by the hardware (CRC, flags) is not included in the MQCLI bandwidths.
• The 8 bytes of SNAP/LLC overhead and 4 bytes of the 8-byte AAL5 trailer for ATM interfaces (the
remaining 4 bytes of the AAL5 trailer CRC are not taken into account). AAL5 padding is equally not
taken into account. The ATM cell overhead (5 bytes per cell payload of 48 bytes) is not taken into
account.
• The 4-byte Frame Relay overhead for Cisco Frame Relay encapsulation (additional overhead due to
possible FRF.12 headers is not taken into account). CRC and flags overhead is not taken into
account.
• The 2 bytes of PPP encapsulation overhead.
After defining the service policy in a policy-map, it needs to be applied on an interface (service-policy).

Quality of Service
By default, on the non-distributed router platforms, the sum of the minimum bandwidths needs to be
lower than 75 % of the configured access bandwidth. Since the actual configured sum of minimum
bandwidths will probably be larger (Real-Time + Business + Best Effort + 8 Kbps management + 8 Kbps
routing protocol), this default parameter setting can be changed (maximum-reserved-bandwidth) to 100
%. However, it is also a very good design practice not to push the design boundaries to the edge without
allowing for any margin of error or unexpected traffic patterns. Therefore, it is still recommended to keep
the sum of all minimum bandwidths below 100 %. Keeping the sum of all minimum bandwidths around
90 % will allow for unaccounted traffic such as layer 2 overhead, layer 2 keepalives, LMI (in the case of
Frame Relay), etc.
The following is the required configuration for LLQ class queuing. The order in which the classes are
configured under the policy-map is important as traffic will be queued in the first queue that matches the
defined ACL.
!
!
policy-map CUSTOMER_PROFILE
class REAL_TIME
set ip prec 5
priority <bps>
class BUSINESS
set ip prec 3
bandwidth <bps>
class BEST_EFFORT
set ip prec 0
bandwidth <bps>
!
interface Serial0/1
bandwidth 2048
encapsulation ppp
max-reserved-bandwidth 90
service-policy input CUSTOMER_PROFILE
Figure 141 LLQ Configuration

Quality of Service
Congestion Avoidance – WRED
When congestion occurs in the class queues, packets will be dropped. By default, tail dropping (First In
First Out – FIFO) will occur. Tail dropping is in general not a desired behaviour. TCP stacks react to
multiple packet drops by severely reducing their window sizes, thus causing severe interruption to higher
layer applications. In addition, multiple packet drops can lead to a phenomenon called “global
synchronisation” whereby TCP stacks become globally synchronised resulting in a wave-like traffic flow.
This is also not desirable.
Therefore, Weighted Random Early Detection (WRED) will be implemented on the Business and Best
Effort traffic classes. FIFO queuing will still be applied for the other classes.
WRED is a congestion avoidance and control mechanism whereby packets will be randomly dropped
when the average class queue depth reaches a certain minimum threshold (min-threshold). As congestion
increases, packets will be randomly dropped (and with a rising drop probability) until a second threshold
(max-threshold) where packets will be dropped with a drop probability equal to the mark-probability-
denominator. Above max-threshold, packets are tail-dropped.
Figure 142 depicts the WRED algorithm.
WRED
WREDValues
Values
min = 20
min = 20
max
max==5050
prod = 1
prod = 1
Packet Drop
Probability
WRED Values
WRED Values
min = 55
min = 55
max
max==7070
prod
prod==1010
Average
Queue Size
Min1 Max1 Min2 Max2
Figure 142 - WRED algorithm
WRED will selectively instruct TCP stacks to back-off by dropping packets. Obviously, WRED has no
influence on UDP based applications (besides the fact that their packets will be dropped equally).
The average queue depth is calculated using the following formula:
• new_average = (old_average * (1-2-e) + (current_queue_depth * 2-e)
The “e” is the “exponential weighting constant”. The larger this constant, the slower the WRED
algorithm will react. The smaller this constant, the faster the WRED algorithm will react.
The exponential weighting constant can be set on a per-class basis. The min-threshold, max-threshold and
mark probability denominator can be set on a per precedence or per DSCP basis.
The mark probability denominator should always be set to 1 (100 % drop probability at max-threshold).
WRED will be applied on the Business and Best Effort traffic classes.
Again, it should be stressed that tuning QoS parameters is never a straightforward process and the results
are depending on a large number of factors, including the offered traffic load and profile, the ratio of load
to available capacity, the behavior of end-system TCP stacks in the event of packet drops, etc.

Quality of Service
Therefore, it is strongly recommended to test these settings in a testbed environment using expected
customer traffic profiles and to tune them, if required. In addition, after an initial production beta
deployment, a performance assessment phase and subsequent tuning of the QoS deployment is
recommended.
Table 32 summarises the recommendation for setting the exponential weighting constant (e). B equals the
link bandwidth in MTU sized packets (MTU = 1500 bytes).
• The reference bandwidth should be the link speed for the Best Effort traffic class.
• The reference bandwidth should be the class bandwidth for the Business traffic class. This will result
in a better quality Business class.
Link Speed B B/1 or B/10 E

in kbps
32 3 3 3
64 6 6 3
128 11 11 3
256 22 22 4
512 43 43 5
1024 86 86 6
2048 171 171 7
34684 2891 289.1 8
45210 3767 376.7 9
100000 8334 833.4 10
155000 12917 1291.7 10
622000 51834 5183.4 12
2400000 200000 20000 14
Table 32 - WRED - exponential weighting constant

Quality of Service
These tables summarize the recommendation for setting the min-threshold and max-threshold for the Best
Effort and Business traffic classes. The required values without having out-of-contract behavior are
“minTHin” and “maxTHin”.
• The reference bandwidth should be the link speed for the Best Effort traffic class.
• The reference bandwidth should be the class bandwidth for the Business traffic class.
Link Speed minTH maxTH minTH maxTH

in Kbps B in in out out
64 6 6 18 2 6
128 11 11 33 4 11
256 22 22 66 7 22
512 43 43 129 13 43
1024 86 86 258 26 86
2048 171 171 513 52 171
Table 33 – WRED - min / max threshold below 2 Mbps
Link
Speed minTH maxTH minTH maxTH
in Mbps B in in out out
34684 2891 87 290 29 87
45210 3767 110 367 37 110
100000 8334 251 834 84 251
155000 12917 388 1292 130 388
622000 51834 1556 5184 519 1556
2400000 200000 6000 20000 2000 6000
Table 34 – WRED - min / max threshold above 2 Mbps

Quality of Service
The following is a sample WRED configuration.
!
policy-map CUSTOMER_PROFILE
class REAL_TIME
set ip dscp ef
priority <bps>
class BUSINESS
set ip dscp af 41
bandwidth <bps>
random-detect dscp-based
random-detect exponential-weighting-constant <e>
random-detect dscp 34 <minTH> <maxTH> 1
class MANAGEMENT
set ip dscp 1
bandwidth 8
queue-limit 40
class RP
bandwidth 8
queue-limit 40
class BEST_EFFORT
set ip dscp 0
bandwidth <bps>
random-detect dscp-based
random-detect exponential-weighting-constant <e>
random-detect dscp 0 <minTH> <maxTH> 1
!
interface Serial0/1
bandwidth 2048
encapsulation ppp
max-reserved-bandwidth 100
service-policy output CUSTOMER_PROFILE
Figure 143 WRED Configuration

Quality of Service
Core Router Mechanisms (12816 Router)

The core QoS mechanisms are discussed in this section. This is because this section covers the QoS
mechanisms that are specific to the 12xxx router platforms. The same QoS mechanisms will be applied
on the PE router on egress to the CE router. This will be treated after the core QoS section.
Bandwidth allocation with Deficit Round Robin
Deficit Round Robin is a scheduling algorithm that provides relative amounts of bandwidth to the
different classes. Relative means that there is no hard limit but the amount is limited through “weights“ in
relation to the other configured and active queues. For a simple example we assume we have three
classes, weighted 1, 1 and 2. If there is heavy congestion and traffic present in all three queues the
proportions are 25%, 25% and 50% of the bandwidth. However if there is no traffic in the first class and
still oversubscription caused by the remaining two, the percentages are 33% and 66%, as the active
weights are 1and 2. This shows that the algorithm always distributes the bandwidth in a very efficient
way while maintaining fairness between the classes.
The actual implementation on the GSR is different from the simple example above. The basic principle of
operation is that the scheduling algorithm cycles through the configured classes and sends a certain
amount of traffic. How much is determined in the following way: Each queue has a deficit counter in
bytes, which initially is set to zero. When the queue is processed the counter is increased by the
Maximum Transmission Unit (MTU)+512*(weight-1). Packets are dequeued from the respective queue
and for every packet its number of bytes is subtracted from the deficit counter. If the counter becomes
zero or less, the queue has used up its amount and the next queue is serviced. If there are no more packets
in the queue the scheduler also moves to the next queue. If the deficit counter is negative it keeps its value
for the next cycle, and the amount of MTU+512*(weight-1) is added to it. This leads to slightly less data
that can be sent, keeping the amount of traffic sent over multiple cycles within the defined range. If the
deficit counter is greater than zero (not all of it has been used up), it will be reset to zero. This means that
a class cannot accumulate quotas in advance.
Some of the GSR line cards support a slightly modified DRR algorithm that provide for a specific low-
latency queue. This queue can run in one of two modes: strict priority or alternate priority. In strict
priority mode the low-latency queue is serviced as soon as packets arrive in it and it is serviced until it is
empty. This gives very low latency at the expense of possible unfairness towards the other queues.
Alternate priority works differently in that it services the low-latency queue every other time between the
regular queues. The low-latency queue has a weight assigned to it and can only send a certain amount of
traffic, however because the time between its turns is much shorter it will provide low latency.
The weight is the only variable parameter in DRR. It influences two things: The relative amount of
bandwidth a traffic class can use and how much traffic is sent in one turn. Using larger weights means
that the overall cycle takes longer and possibly increases latency. Also it needs to be noted that the low
latency queue in alternate priority mode is serviced more than once per cycle and thus takes more
bandwidth than other queues with the same nominal weight. How much more is a function of how many
queues are defined, e.g. with three queues total the low latency queue is serviced twice as often as the
other queues and sends twice its weight per cycle. If eight queues are defined the low latency queue is
serviced seven times more often and the “effective” weight is seven times higher.
A specific low latency queue is a key component to guarantee low delay and jitter across the backbone
network. Backbone routers have large packet buffers to accommodate bursts at high speeds, those buffers
can add tens of milliseconds of delay. Routers that do not have low latency queuing mechanisms are not
able to provide the necessary quality of service for some real-time applications.
Architecture details on Class of Service implementation on the Cisco

12000 series
The Cisco 12000 series is a fully distributed architecture that uses a crossbar matrix switch to connect the
line cards. Because of the design Class of Service features can be applied to packets queued for

Quality of Service
transmission through the fabric as well as for packets queued on outgoing interfaces. The following two
diagrams show where CAR, WRED and DRR features are used in the GSR architecture.
On the receiving or ingress line card queuing takes place on a per destination line card basis, for every
possible destination line card in the system up to 8 queues for 8 Classes of Service can be defined.
Figure 144 - 12000 ToFab DRR and WRED
On the transmit side a set of 8 output queues is available on every port. Please note that availability of features
differs across the different types of line cards for the GSR. Also note the difference between DRR and MDRR,
the former uses strict priority for the low-latency queue while the latter has the additional option of using
alternate priority.

Quality of Service
Figure 145 - 12000 FrFab MDRR and WRED
Class Queuing – MDRR

As discussed above, the 12000 architecture is distributed, hence there are 2 possible bottlenecks to
control: the egress link (FrFab queuing) and the crossbar connection from the ingress line card to the
egress line card (ToFab queuing).
There are 2 main reasons why queues can built up on the ingress line card towards the crossbar switch
fabric (ToFab queuing). This is generally a bad thing since ToFab queuing will cause head-of-line
blocking (HOL) and will not discriminate between packets that require a different CoS treatment.
• Engine 0 line cards (e.g. 4 x OC-3) have a feature called fabric backpressure. When FromFab queues
are building on the egress side of an engine 0 line card, the engine 0 line card will instruct the switch
fabric to slow down the rate of packets that it is transferring to that engine 0 line card. This could
trigger a situation where a sending line card on the ingress side of the traffic stream sees its queues
building up.
This situation can be avoided by configuring FromFab MDDR and WRED on every core link. This
will prevent FromFab queues to built up to a situation where fabric pressure will be triggered,
resulting in the undesirable ToFab queuing.
• The total bandwidth available across the crossbar switching fabric towards any line card is 48 Gbps
for the 40gb Fabric (12816), 12 Gbps for the 10 Gbps switch fabric ( 12416) and 2.88 Gbps for the
2.4 Gbps switch fabric (12008). When multiple line cards are sending packets towards a single egress
line card, the crossbar scheduler will distribute the total available bandwidth in a round-robin
fashion. For example, if each of 3 ingress line cards are sending packets towards a single egress line
card, the switch fabric scheduler will allocate 1/3 of 48 Gbps or 12 Gbps or 4 Gbps capacity to each
of the ingress line cards.

Quality of Service
It is obvious that with the current network topology and installed capacity, the crossbar switch capacity
will not become a bottleneck, therefore tofab queuing is not covered.
Typically, there will be 3 separate MDRR queues on every egress POS linecard (FrFab) within the core
network. Core network links are between PE, P. Core link speeds are:
• OC-48 (STM-16) on the Engine 5 line card (12816 Core Router)
• OC-12 (STM-4) on the Engine 3 line card (12816 Core Router)
• Gigabit Ethernet (1 G) on the Engine 5 line card (12816 Core Router)
The following queues will be configured.

• Strict low-latency queue for the Real-Time traffic class.
• Queue 0 for the Best Effort traffic class.
• Queue 1 for the Business, Routing Protocol traffic classes.
As mentioned earlier, the quantum will determine how many bytes can be served from the queue each
time the DRR scheduler visits that particular queue. The quantum for each queue is calculated as follows.
• Quantum (bytes) = MTU + (Weight – 1) * 512
Engine 3 and Engine 5 linecards allow use of MQC and hide the intricacies of MDRR from the user.

Quality of Service
Congestion Avoidance – WRED

The following are the recommended MinTH “m” and MaxTH “M” values for OC-3, OC-12, OC48 and
OC-192 POS links. These thresholds have been computed taking into account the following constraints:
• Assuming that the Best Effort (queue 0) and the Business (queue 1) traffic classes will each take up
50% of the interface bandwidth. Note that the actual MDRR weights have been computed for a much
higher guarantee for the Business traffic class (90% of remaining traffic after the low latency queue
has been served).
• Maximizing the link utilization while minimizing the mean queue depth.
• Limiting the worst case IO memory consumption for the Best Effort queue to less than 50% of the
packet memory.
• Taking into account that for the hardware based forwarding line cards (Engine 2 upwards), the
(MaxTH – MinTH) needs to be a power of 2.
Table 35 - Core WRED MinTH and MaxTH*
Physical MinTh MaxTh

Rate kbps "m" "M"
149760 94 606
599040 375 2423
2396160 1498 9690
9584640 5991 38759
*These values documented on CCO: please see:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios112/ios112p/gsr/wred_gs.htm#xtocid945524

Quality of Service
As you read the descriptions of the features above, it should quickly become apparent that using some of
these features implies that other features cannot be used or must be configured in a more-specific manner.
Below are some common examples of feature interaction.
Quality of Service MQC Examples

This is a basic sample of configuring the GSR Engine 3 and Engine 5 linecards for Quality of service
using the MQC.
Interaction of shaping and bandwidth.

The shape keyword limits a queue to a specific maximum bandwidth. The bandwidth keyword provides
a guaranteed minimum bandwidth. Thus, for any given queue, the shape rate must be greater than the
bandwidth rate. This example fails because we are trying to reserve 124M while at the same time
restricting to < 105M.
policy-map foo
class Prec_5
bandwidth 124400
shape average 104960000
router(config-if) service-policy output foo

%Shape rate cannot be less than the minimum bandwidth guarantee.
Figure 146 Shaping & CBWFQ Configuration
Reserved bandwidth and available port bandwidth interaction.

Since the bandwidth keyword guarantees a minimum amount of available bandwidth, the sum of
configured bandwidth statements must always to be less than the aggregate bandwidth of that particular
port. Applying the following policy-map (sum of bandwidths = 497M) on an OC3 (155Mb/s) interface
will result in an error.
policy-map foo
class Prec_5
bandwidth 124400
class Prec_2
bandwidth 186600
class Prec_1
bandwidth 186000

%Not enough bandwidth resources on interface to satisfy request
Figure 147 Maximum Bandwidth Reservation

Quality of Service
Priority queue and bandwidth interaction.

By default, the priority queue is always serviced if traffic is present. Because there is no predefined limit
to the amount of traffic we can dequeue from the LLQ, this makes it impossible to reserve bandwidth for
any other class.
policy-map foo
class Prec_5
priority
class Prec_2
bandwidth 186600
class Prec_1
bandwidth 186000

% Priority class cannot be configured with bandwidth
% guarantees for other classes on this interface.
% Please limit the priority class with the police configuration
% with exceed action as drop.
Figure 148 Explicit Policing on LLQ
Policing the LLQ

By applying a police statement to the LLQ, we can then calculate how much bandwidth is remaining for
assignment to other queues on that port. Take care to configure the LLQ with a conservative value, as
configuring a value below than the actual offered load will result in drops and/or additional queueing
delay in the LLQ. As portrayed earlier, the sum of all bandwidth statements plus the LLQ police rate
must be less than the available bandwidth of the interface.
policy-map foo
class Prec_5
priority
police 128000000 32000 64000 conform-action transmit exceed-action drop
class Prec_2
bandwidth 186600
class Prec_1
bandwidth 186000
Figure 149 Example Configuration for LLQ & Explicit Policing

Quality of Service
Using WRED
This example shows how to configure multiple WRED profiles for various traffic classes. WRED can
only be applied when you have defined a queue by using either the shape or bandwidth keyword.
policy-map foo
class Prec_5
bandwidth percent 5
random-detect
random-detect precedence 5
class Prec_4
bandwidth percent 30
random-detect
random-detect precedence 4 1952 4000 1
class Prec_3
random-detect
class Prec_2
random-detect
class Prec_1
random-detect
class Prec_0
random-detect
class class-default
Figure 150 WRED Configuration Example

Quality of Service
Tying it all together for E3 & E5:

This example defines the queue for precedence 5 packets as the priority queue, and limits the output of
that queue to 64 Mb/second. The remaining classes are each given an amount of reserved bandwidth.
WRED is used to avoid congestion for each queue.
Note that in the default-class, RED parameters are used on ALL traffic to insure that if anything leaks
through the previous class definitions, we'll still be doing random drop on that traffic. The statements
below in the class-default class that correspond to Prec_[0-5] are all matched by a non-default class, so in
theory they are redundant.
However, it's probably a good idea to always define a random-drop profile for ALL traffic in the default
class to avoid accidentally consuming all of the available buffers on a given linecard.
policy-map foo
class Prec_5
priority
class Prec_4
shape average 200000000 262143
random-detect
class Prec_3
random-detect
class Prec_2
random-detect
class Prec_1
random-detect
class Prec_0
random-detect
class class-default
random-detect
!
interface pos 8/2
service-policy output foo
Figure 151 Sample Configuration for E3 and E5

Quality of Service
QoS Architecture of the Cisco 7600

This section describes the QoS architecture of the Cisco 7600 router.
The diagram shows various Qos features and their application within 7600 architecture context.
Figure 152 - QoS Architecture of the Catalyst 7600
Input scheduling
Input scheduling prioritizes and schedules packets out of ingress packet queues based on several QoS
values including CoS and DSCP. However, most of Catalyst switches can deliver packets to the switching
fabric at line rate or a specified rate. This specific rate defines the maximum throughput of the switch. If
the input rate is not exceeded, input scheduling is not crucial in implementing QoS architecture.
The use of input scheduling is meant to deal with rare conditions of fabric oversubscription. Use of input
scheduling is therefore not deemed necessary in most scenarios. It will not be implemented in the Wateen
Telecom Network.
Cisco 7600 Classification and Marking

Input Classification is performed by the ingress port ASIC and the switching engine based on:
• Ingress Port or VLAN
• Destination MAC address & VLAN
• TCP/IP Header (to include any combination of Source Address, Destination Address, Source Port,
Destination Port, Protocol Type and ToS byte)
• User specified Policies
• Existing Packet Label if the port is trusted

Quality of Service
Internal DSCP
During processing of the packets, the priority of all traffic (including non-IP traffic) is represented
with an internal DSCP value in the Cisco 7600. During the classification process, QoS derives the
internal DSCP value depending on the trust state of the input interface, and the type of traffic:
• When PFC3B receives an IP packet (ip2ip, ip2mpls), it uses the input interface trust state and, if
configured, the policy-map trust command.
• For trust-cos input interface, internal DSCP is derived from received or port Layer 2 CoS values. The
following conversion table is used
7600#
7600#sh mls qos maps cos-dscp
Cos-dscp map:
cos: 0 1 2 3 4 5 6 7
------------------------------------
dscp: 0 8 16 24 32 40 48 56
7600#
• For trust-ipprec input interface, internal DSCP is derived from received IP precedence values.
The following conversion table is used:
7600#sh mls qos maps ip-prec-dscp
IpPrecedence-dscp map:
ipprec: 0 1 2 3 4 5 6 7
------------------------------------
dscp: 0 8 16 24 32 40 48 56
7600#
• For trust-dscp input interface, internal DSCP is derived from received DSCP values.
• For untrusted input interface, internal DSCP is derived from port CoS or policy map DSCP
values.
• When PFC3B receives an MPLS packet (mpls2mpls), it trusts the mpls EXP bits. The interface
trust state and the policy-map trust command have no effect.
• When PFC3B receives an MPLS packet (mpls2ip), trust depends on the type of label. However,
in all cases, the interface trust state and the policy-map trust command have no effect.
o Non-aggregate label: PFC3B trusts EXP in the topmost label.
o Aggregate label in VPN CAM: PFC3B trusts IP DSCP.
o Aggregate label not in VPN CAM: After recirculation, PFC3B trusts IP DSCP.
The type of labels for a prefix can be displayed using
sh mpls forwarding-table <prefix>
sh mpls forwarding-table vrf <name> <prefix>
sh mls cef mpls labels <label>
QoS & MPLS traffic: Trust State

When handling mpls traffic, the following needs to be taken into account
• Imposition (ip2mpls) – Untrusted
During ip2mpls imposition, for packets received on an untrusted interface, PFC3B sets the
internal dscp to zero. It then maps the internal dscp to the imposed EXP and the output CoS. It
always preserves the underlying IP ToS.

Quality of Service
• Imposition (ip2mpls) – Trust IPP/DSCP

During ip2mpls imposition, for packets received on an interface with trust IPP or trust DSCP,
PFC3B maps the IPP/DSCP to the internal dscp. It then maps the internal dscp to the imposed
EXP and the output CoS. It always preserves the underlying IP ToS.
• Swapping
During swapping, PFC3B trusts the topmost EXP and maps it to the internal dscp. During the
swap, it copies EXP from the swapped-off label to the swapped-on label. After the swap, it maps
the internal dscp to the egress frame’s CoS.
• Disposition (mpls2mpls)
During mpls2mpls disposition, PFC3B trusts the topmost EXP and maps it to the internal dscp.
By default, PFC3B discards the popped EXP and does not propagate it to the exposed EXP.
After the pop, it maps the internal dscp to the egress frame’s CoS.
• Disposition (mpls2ip) Case 1: Non-aggregate Label
PFC3B trusts EXP and maps it to the internal dscp. By default, PFC3B discards the popped EXP
and does not propagate it to the exposed IP ToS. After the pop, it maps the internal dscp to the
egress frame’s CoS.
• Disposition (mpls2ip) Case 2: Aggregate Label
PFC3B trusts the exposed IP ToS and maps it to the internal dscp. By default, PFC3B discards
the popped EXP and does not propagate it to the exposed IP ToS. After the pop, it maps the
internal dscp to the egress frame’s CoS.
Table 36 – Qos Trust state for Sup720
Packet in
Forwarding Operation Trust
Tycho
Routing ip2ip IP Interface trust; policymap trust
Imposition ip2mpls MPLS Topmost EXP
Swapping mpls2mpls MPLS Topmost EXP
Swapping +
mpls2mpls MPLS Topmost EXP
Imposition
mpls2mpls MPLS Topmost EXP
Disposition
Agg label in VPN CAM: IP DSCP
mpls2ip (no recirc) MPLS
Non-agg label: EXP
1st pass MPLS EXP
mpls2ip Agg label not in VPN CAM: IP DSCP
(recirc) nd
2 pass IP Non-agg label with output IP policy: IP
DSCP
QoS & MPLS traffic: Marking MPLS traffic

• How Marking Works
When you configure a policy map to mark traffic, PFC3B does not directly mark the packet.
Instead, it marks the internal dscp. PFC3B then maps the internal dscp to the appropriate fields
(CoS, IPP, DSCP, EXP) in the egress frame.

Quality of Service
• Marking EXP during Imposition (ip2mpls)

During ip2mpls imposition, PFC3B can mark the imposed EXP with either a ‘set’ command or a
police action.
• Marking EXP Affects IP Traffic
If you have configured a policy to mark EXP in ip2mpls traffic, PFC3B will also mark IP ToS in
ip2ip traffic. Why? The result function does not know how the internal dscp was marked – it
only knows that it has to rewrite the internal dscp into the egress frame. For egress IP frames, it
rewrites the internal dscp into the IP ToS and the output CoS. There is a workaround to avoid
this. This workaround consist on using a class-map to separate traffic and mark only the intended
traffic.
• Marking IP ToS During Imposition (ip2mpls)
During ip2mpls imposition, PFC3B always preserves IP ToS, therefore it does not support
marking IP ToS. If you configure a policy to do this, it does not mark IP ToS. It marks the
internal dscp which is mapped to the imposed EXP and the output CoS.
• Marking IP ToS During Disposition (mpls2ip)
During mpls2ip disposition, PFC3B can mark IP ToS with an output policy.

Quality of Service
• Propagating EXP during Disposition (mpls2ip)

If ‘mpls propagate-cos’ is configured for the egress interface, PFC3B maps the internal DSCP to
the exposed IP ToS and to the egress frame’s CoS. However there are some caveats:
o PFC3B propagates the popped EXP if the egress interface is known during the initial
lookup.
• Non-aggregate labels
• Aggregate labels in VPN CAM
o PFC3B cannot propagate the popped EXP if the egress interface is not known during
the initial lookup.
• For aggregate labels not in VPN CAM, propagation will not occur unless all
interfaces in the VRF have ‘mls propagate-cos’ configured.
• For explicit NULL labels, PFC3B cannot determine the egress interface,
hence, propagation does not occur.
Cisco 7600 Policing

Individual policing applies the bandwidth limit of a policer per interface. For example, an individual
policer configured to constrain ingress traffic to 32 kbps limits each applicable interface to 32 kbps on
ingress. An aggregate policer configured for the same bandwidth constraint limits the bandwidth
collectively among all interfaces. Microflow policing is available on the Catalyst 6500, and it applies
bandwidth limits to each access-control entry (ACE) of a defined policer. In Wateen Telecom’s
deployment individual policing should be considered.
Scheduling and Congestion Avoidance on LAN ports

When QoS is enabled on LAN ports, egress scheduling and congestion avoidance would automatically
take place as shown in figure below. In the Cisco 7600 the type of linecard would determine the
availability of a strict-priority queue (SP queue), number of normal queues, and the tail-drop and WRED-
drop thresholds.

Quality of Service
Frames marked
Output Port
with CoS=5
placed into SP
Queue if one is
present SP Queue
Queue 1 Data Data Data

Frames with Queue 2
other CoS values
are placed into Queue n
the other Queues
based on their
CoS value Each Queue
Drop
Threshold n
Drop
Threshold 2
Drop
Threshold 1
Within each queue, drop thresholds are used to
indicate which CoS tagged packets can be dropped
once the queue has filled beyond a certain threshold
Figure 153 Output Scheduling and Congestion Avoidance
Strict Priority Queue

A strict priority queue is always emptied first. This means that as soon as there is a packet in the strict
priority queue the packet is sent.
The strict priority queue is emptied completely, and only when it is empty, WRR or WRED queues are
checked. After each packet is transmitted from one of the WRR/WRED queues, the strict priority queue
is checked and emptied if necessary.
WRR
Weighted Round Robin (WRR) is another mechanism used in output scheduling used on the Catalyst
7600. WRR works between two or more queues. The queues that are making WRR are emptied in a
round robin fashion, and you can configure the weight for each queue.
Buffer Allocation
The transmit buffer is shared by the different queues of a port. Depending of the amount of traffic which
is expected in each of the traffic classes, and how bursty the traffic might be, the allocation of the buffer
assigned to each queue will be modified. The settings to be used in the Wateen Telecom network will be
described later in this document.
Tuning the port settings

Some of the queuing parameters cannot be defined per port. Depending on the internal architecture of
each of the linecards, the settings will need to be the same in a group of ports.

Quality of Service
LAN Port capabilities of Wateen Telecom network

The following are the capabilities of the 7600 LAN cards/ports to be used by Wateen Telecom:
• WS-X6724-SFP
7600_2#sh int capa mod 4
GigabitEthernet4/1
Dot1x: yes
Model: WS-X6724-SFP
Type: 1000BaseLH
Speed: 1000
Duplex: full
Trunk encap. type: 802.1Q,ISL
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)
Membership: static
Fast Start: yes
QOS scheduling: rx-(1q8t), tx-(1p3q8t)
CoS rewrite: yes
ToS rewrite: yes
Inline power: no
SPAN: source/destination
UDLD yes
Link Debounce: yes
Link Debounce Time: yes
Ports on ASIC: 1-12
Port-Security: yes
• Sup 720 3BXL LAN ports

7600_2#sh int capa mod 6
GigabitEthernet6/1
Dot1x: yes
Model: WS-SUP720-3BXL
Type: No Transceiver
Speed: 1000
Duplex: full
Trunk encap. type: 802.1Q,ISL
Trunk mode: on,off,desirable,nonegotiate
Channel: yes
Broadcast suppression: percentage(0-100)
Flowcontrol: rx-(off,on,desired),tx-(off,on,desired)
Membership: static
Fast Start: yes
QOS scheduling: rx-(1p1q4t), tx-(1p2q2t)
CoS rewrite: yes
ToS rewrite: yes
Inline power: no
SPAN: source/destination
UDLD yes
Link Debounce: yes
Link Debounce Time: yes
Ports on ASIC: 1-2
Port-Security: yes

Quality of Service
For each of the linecards, the different types of traffic (identified by the CoS value) will be mapped to the
different egress queues. Please note that LAN cards use WRR commands to enforce QoS policies. This
syntax is specific to LAN cards and respective capabilities of the card.
WAN port capabilities

The WAN ports using WAN cards (e.g. SIP-400) used in the cisco 7600 platform support QoS using
MQC (Modular Quality of Service CLI). This includes different classes, shaping, LLQ and marking as
described earlier.
Wateen Telecom Provisioned Quality of Service
In the Wateen Telecom network following classe of service identifiers are suggested be used to determine
which flows are queued in which queue (or are distinguished for QoS treatment):
• Realtime Class (VoIP Traffic): IP Precedence 5

• Business Class (Customers’ Premium Traffic & Network Control Traffic): IP Precedence 6, 3
• Video Class (Future Video Requirements): IP Precedence 4
• Critical Class: (In-Band Management for Access Elements, Internal to Wateen Telecom): IP
Precedence 2
• Best-Effort Class (Internet Data): 1 and 0
Table 37 – Traffic Queues and Qos Marking
Traffic Queue (Class Precedence / DSCP / Application / Traffic Type

of Service) MPLS EXP
Realtime (LLQ / 5 / EF / 5 Voice
Priority Queueing)
Realtime (LLQ / 5 / EF / 5 Video*
Priority Queueing)
Business Class 6 / CS6 / 6 Control Plane Traffic
Video class 4 / AF41/ 4 Streaming Video Traffic**
Business Class 3 / AF31 / 3 Premium Business Traffic
Critical Class 2 / CS2 / 2 InBand Management for
Access Elements
Best Effort 1 / CS1/ 1 Best Effort Class Traffic
Best Effort 0 / BE / 0 Best-Effort Class Traffic

Quality of Service
[EF=101110=46 For DSCP values details , please refer to

http://www.cisco.com/en/US/tech/tk543/tk757/technologies_tech_note09186a00800949f2.shtml]
*Video refers to interactive video (e.g. video conferencing etc.)
** In some cases CS4, while in other AF41 is used. Cisco’s latest recommendation indicates AF41 for
video conferencing while streaming video to be CS4.
Note: the Realtime queue will be a Low-Latency queue designed to offer low-latency and low-loss traffic
even under congested network conditions. LLQ is system wide, large video packets on low speed links
can cause delay and jitter for voice traffic. Ultimately it is Wateen Telecom’s choice in terms of preferred
class of service identifiers deployed on their network.
As indicated earlier, an assumption is made with following percentages:

• 40 % of the available link bandwidth will be reserved for the real time class
• 20 % of the available link bandwidth will be reserved for the “video traffic”
• 10 % of the available link bandwidth will be reserved for the business traffic
• 10% of the available link bandwidth will be reserved for the management traffic
• The remaining bw will be allocated for the best effort traffic (Internet traffic)
Wateen Telecom can adjust these according to their own requirements, once these requirements are
gathered and defined in terms of various classes of services desired.
As mentioned earlier there are several components and positions in network where QOS needs to be
defined, these are shown below:
Managed Services Scenario:
Unmanaged Services Scenario:
Unmanaged Services:
• Edge: Policing, Classification, Marking, Queueing, congestion avoidance - WRED

• Core: Queueing, WRED.
Managed Services:
• CPE: Policing, Classification, Marking, Queueing, congestion avoidance - WRED
• Edge: Queueing, WRED
• Core: Queueing, WRED.
Please note that lot of qos functionality in this case is off loaded to CPE devices.
Classifying Traffic into Classes
This section talks a little more about each of the classes of traffic and the profiles for each of them for
features such as PHB (Per-Hop Behaviour, WRED configuration etc).

Quality of Service
I. Real Time Class (VoIP Traffic)

Marking: Traffic in this class is marked with DSCP EF (or IP Precedence 5) for Voice. Traffic of other
applications can be included in this class (as shown in the example mappings). Some instead of using the
full 6bits of DSCP, SPs may use the first three bits of DSCP for historical reasons, or for easier mapping
to 3 bit MPLS EXP and/or 802.1Q CoS bits. MPLS EXP 5 is imposed to classify traffic across an MPLS
VPN core.
Link Bandwidth Allocation for Voice: Delay due to VoIP contention increases with VoIP traffic
volume as a percentage of link speed. It also increases with decrease in link bandwidth, for the same
percentage of link bandwidth usage (due to higher serialization delay). In real networks, VoIP delay will
depend several factors such as traffic mix, burstiness, link utilization, and an optimal Voice traffic
volume for a given link bandwidth is difficult to determine except via simulating the network in question,
or tests in the actual network. Many service providers try not to allocate more than 20-30% of link
bandwidth to Real-Time traffic class.
Per Hop Behavior (PHB): This traffic class must get Expedited Forwarding (EF) PHB via priority
queuing. This traffic will be served by low latency queueing.

Quality of Service
WRED: WRED is NOT recommended for this class, since packet drop target is 0% (or nearly 0%), and
typically this class carries UDP traffic which does not react to WRED drops.
Policing:
The real-time class packets shouldn’t be dropped by the service provider, except when the traffic volume
exceeds the contracted bandwidth per SLA. This traffic class therefore uses ingress policing for SLA
enforcement at QoS trust boundary. Policing is configured to drop any out-of-contract traffic (rather than
marking to a lower priority class and transmitting to avoid the possibility of delayed delivery of real-time
traffic.
“Always on” policer Vs. “Congestion aware” policer:

Cisco IOS provides two types of mechanisms to police a priority queue, as follows
Option one: Congestion-aware policer
class SP-Real-Time
priority [percent <%>|<kbps>] <burst>
class SP-Real-Time
priority [percent <%>|<kbps>] <burst>
In the above configuration, the priority queue traffic is only policed when the interface is congested.
Policing doesn’t take place if there is no congestion. This mode is initiated by specifying the percent or
the class “rate” in the priority command.
Option two: Always-on Policer:
class SP-Real-Time
priority
police <bps> bc <bytes> conform-action transmit exceed-action drop
Figure 154 Congestion aware vs Always-on Policer Configuration
The police command in the above configuration polices the priority queue permanently, even when there
is no congestion. This always-on policer is more favorable for service providers who wish to strictly
enforce the VoIP traffic volume contract. Congestion aware policer, on the other hand, can potentially
allow line rate realtime traffic if there is no other traffic (of other classes) to cause congestion.
It is recommended that Wateen Telecom uses option two (always-on-policer). In core of Wateen
Telecom, 12000 series engine 3 and engine 5 cards require second option i.e. an explicit policer must be
configured. In the absence of explicit policer on engine 3 and engine 5 lincards, all other traffic may be

Quality of Service
starved as mentioned earlier. Using always on policer allows not only for consistency everywhere in the
network it also will allow Wateen Telecom to do bandwidth provisioning in their core appropriately.
Policing Rate and Burst Calculation:

The policer burst size should be sufficient to accommodate expected traffic bursts.
Burst in bytes = N * VoIP (or Avg Real Time) packet size

where N = max number of (VoIP/Realtime) packets that could be arriving at the same time
For VoIP traffic, an upper bound could be the max number of simultaneous calls If you have set a queue
limit, then it must not be less than the policing burst
II. Business Class (Premium Business Traffic)
Marking: Traffic of this class can be marked with different ip precedence values which can then be
aggregated in the core. It is recommended that traffic with ip precedence 3, 6 and 7 be aggregated in this
class.
Link Bandwidth Allocation for Business Critical Class:

This class tries to optimize latency (and loss) for business critical applications (each enterprise decides
what applications fall into this category based on their business objectives). Service providers may
provision different fractions of link bandwidth to this class, considering their client’s requirements.
PHB: This traffic class gets Assured Forwarding (AF) PHB via CBWFQ queuing
A minimum bandwidth guarantee is assured via a “bandwidth” command.
Latency for the class is minimized by assigning proportionately more minimum bandwidth guarantee to
this class that its average traffic volume (assumed to be the case for most platforms). However it is to be
noted that only minimum bandwidth guarantee is made for this class. No guarantees for delay or jitter can
be made.
WRED: WRED may be configured for this class to avoid TCP synchronization. WRED can also be used
in this class to ensure that packets with certain ip precedence /DSCP values get dropped during
congestion before those with other ip precedence/ DSCP values (by assigning them different WRED
thresholds). This would be useful, for example, if transactional-data (DSCP CS3) packets needs to be
dropped during congestion before mission-critical data (DSCP AF31) is dropped (both belong to this
traffic class). If, however, this class is optimized to provide very low drop rates, WRED may not be
configured.
Queue limit: If queue-limit is set for a class, verify (or set) the class queue size so that, it can
accommodate the expected traffic burst for the class. However, too large a queue-limit might increase the
queue size significantly to cause excessive latency due to delay when the queue fill up during congestion.
Since the SLA for this class includes some amounts of packet loss, the excessive latency can be reduced
by reducing the queue-limit, at the cost of increasing the possibility of packet drops during congestion.

Quality of Service
If L = latency limit to be achieved in ms, and Cbw = Class bandwidth in bps, then
Queue-limit in bytes ~ L/1000 * Cbw /8
For example, if the queuing delay goal is 5ms on a router and the class minimum bandwidth guarantee is
10 Mbps, then the queue-limit in byte should be about 5/1000* 10*1024*1024/8 = 6553 bytes
Policing:
Ingress Policing is used to impose contractual traffic limits per SLA, and optionally, to mark traffic
exceeding the contractual limit with a DSCP/EXP of a lower priority traffic class such as best effort
traffic class. The policing burst size should be sufficient to accommodate expected traffic bursts in this
class.
Optimal burst size is calculated as:

Burst = Policing Rate * RTT, where RTT = Round Trip Time for the Traffic Class.
III. Critical Class (Management Traffic for Access Elements)
Marking: Traffic of this class can be marked with different ip precedence values. It is suggested that
traffic with ip precedence 2 to be aggregated in this class.

This class tries to optimize latency (and loss) for NMS applications for manageability of all access
infrastructure, including access switches, WiMAX devices, Microwave devices and any other devices
which needs to be managed in access layer.
A minimum bandwidth guarantee is assured via a “bandwidth” command.
Latency for the class: Only minimum bandwidth guarantee is made for this class. No guarantees for delay
or jitter can be made.
WRED: WRED may be configured for this class to avoid TCP synchronization (for TCP based
applications).
Queue limit: If queue-limit is set for a class, verify (or set) the class queue size so that, it can
accommodate the expected traffic burst for the class. However, too large a queue-limit might increase the
queue size significantly to cause excessive latency due to delay when the queue fill up during congestion.
Since the SLA for this class includes some amounts of packet loss, the excessive latency can be reduced
by reducing the queue-limit, at the cost of increasing the possibility of packet drops during congestion.

Quality of Service
If L = latency limit to be achieved in ms, and Cbw = Class bandwidth in bps, then
Queue-limit in bytes ~ L/1000 * Cbw /8
For example, if the queuing delay goal is 5ms on a router and the class minimum bandwidth guarantee is
10 Mbps, then the queue-limit in byte should be about 5/1000* 10*1024*1024/8 = 6553 bytes
Policing:
No ingress policing is expected be in place for this class as this is a class which carries SP infrastructure
traffic.
IV Video Class (Streaming Video Traffic)
Marking: Traffic of this class can be marked with different ip precedence values. It is suggested that
traffic with ip precedence 4 to be placed in this class.

This class tries to optimize latency (and loss) for video applications. Service providers may provision
different fractions of link bandwidth to this class, considering their client’s requirements.
When addressing the QoS needs of Streaming Video traffic, the following guidelines are recommended:
Streaming Video (whether unicast or multicast) should be marked to DSCP CS4 as designated by the
QoS Baseline.
Loss should be no more than 5 percent.
Latency should be no more than 4–5 seconds (depending on video application buffering capabilities).
There are no significant jitter requirements.
Guaranteed bandwidth (CBWFQ) requirements depend on the encoding format and rate of the video
stream. Streaming video applications have more lenient QoS requirements because they are delay-
insensitive (the video can take several seconds to cue-up) and are largely jitter-insensitive (because of
application buffering).
WRED: WRED may not be configured for this class, instead policing should be used based on video
application requirements and number of simultaneous flows .
Queue limit: Set the same way as Business class
Policing:
Ingress Policing is used to impose contractual traffic limits per SLA. No mark down is expected as the
class is strictly targeted for a single class of service.

Quality of Service
V. Best Effort Class (Internet Traffic)
This class usually has no latency, jitter, or loss commitments, and includes all traffic that are not included
in the other edge classes.
Marking: Traffic of this class is typically marked with ip precedence/ DSCP of 0 and 1 and EXP value is
set to 0 in the MPLS core.
Link Bandwidth Allocation for Best Effort Class:

Service providers may provision different fractions of link bandwidth to this class, based on their
business requirements. Please note that this class can potentially carry traffic from thousands of
applications from a large enterprise; hence assigning too less a bandwidth to this class can lead to end
user dissatisfaction.
In case of a five class implementation, a service provider may provision a fixed minimum bandwidth for
best effort class, or a “remaining percentage based” allocation. A value of 20-30% is usual in multi
service environment (voice, business-critical, data etc), although it can be lower or higher depending on
the SP’s provided services
PHB: This traffic class is assigned the class-default, and receives best effort treatment, bandwidth is
guaranteed if a bandwidth statement is assigned to this class.
WRED: WRED is configured for this class to avoid TCP synchronization. WRED can also be used to
drop certain type of best effort traffic prior to other types of best effort traffic (as mentioned in the
Business class)
Policing:
Similar to business class, except that exceeding traffic is always dropped (not re-marked to a lower
color).
Categorising Traffic at the CE Ingress
There are a number of actions that can be configured at the CE. Wateen Telecom can have three ways to
which they want to classify their traffic:
• IP address range
• TCP/UDP Port Numbers

Quality of Service
• TOS/COS Values
The most scalable method, and indeed the most commonly deployed and easiest to understand method is
to use TCP/UDP Port numbers (layer 4 information), this allows us to classify traffic into varying classes
based on the applications rather than using the IP addressing information.
The predominant drawback to using the IP addressing scheme is that it really requires the operator to
have a good knowledge of their customer network, the flows within etc, additionally as the customer
network grows and new prefixes are added the QOS policies will need to change, this will cause a
constant state of flux.
Basing QOS on the application layer means that regardless of the source/destination of the traffic the
correct QOS policy is defined based on the application (Voice, FTP, HTTP, SAP R/3, Oracle etc all use
well known ports or port ranges). This helps the SP to scale the QOS offerings and additionally makes the
per-customer policies much simpler.
At the CE, packets will be classified through the use of extended access lists (ACLs). These ACLs can
match packets on IP Source or Destination address, protocol type, and UDP/TCP port numbers.
ACLs for Real-Time and Business traffic should be agreed with the customer. VoIP typically uses UDP
ports in the range of 16384 to 32767.
!
access-list <x> permit udp any any range 16384 32767
!
An alternative for matching the VoIP traffic is to use “match ip rtp” in the class-map.
class-map match-all REAL_TIME

match ip rtp <starting_port_number> <port_range>
!
The ACL for Management traffic should match SNMP, TFTP, TELNET and any other required traffic to
and from the network management systems IP address range.
!
access-list x permit tcp any 192.168.1.0 0.0.0.255 eq telnet
access-list x permit udp any 192.168.1.0 0.0.0.255 eq snmp
access-list x permit udp any 192.168.1.0 0.0.0.255 eq tftp
!
The following is the ACL for the Best Effort traffic (100).
!
access-list x permit ip any any
!
Figure 155 Traffic Classification at CE
Voice signalling traffic will need to be classified and marked appropriately. Depending on the customer
VoIP implementation, the different possibilities are:
• RTCP: odd RTP port numbers

Quality of Service
• H.323 / H.245 standard connect: TCP 11xxx

• H.323 / H.245 fast connect: TCP 1720
• Skinny control traffic: TCP 2000-2002
• ICCP: TCP 8001-8002
• MGCP: UDP 2427, TCP 2428
Note: A list of well known port assignments can be found at:

http://www.iana.org/assignments/port-numbers
Defining mission critical data is something that the customer has to supply to Wateen Telecom, each
customer will likely have a different set of requirements for their critical data, and therefore a different
QOS service policy will likely be necessary for each customer.
Trusting Ingress Traffic at the CE layer
If Wateen Telecom trusts the traffic that is coming into the CE then there is no requirement for the CE to
have an input service policy because we are assuming that the ingress traffic is appropriately marked and
requires no further conditioning.
Untrusted Ingress Traffic at the CE Layer
If Wateen Telecom wants to apply their own policies to ingress traffic at the CE i.e where the traffic is
not trusted from the customer then there should be an input service policy at CE, this basic service policy
will simply remark the traffic so that on egress it can be appropriately queued in the correct fashion.
Following is a sample input policing policy, it basically categorizes the ingress voice and video traffic as
well as the other classes to use a particular IP precedence.
policy-map INGRESS
class VOICE
set ip precedence 5
class VIDEO
set ip precedence 4
class BUSINESS-DATA
set ip precedence 3
class CRITICAL
set ip precedence 2
class INTERNET
set ip precedence 0
The class maps mentioned above are shown below:

Quality of Service
class-map match-all VOICE

match access-group name VOICE
class-map match-all VIDEO
match access-group name VIDEO
class-map match-all CRITICAL
match access-group name CRITICAL
class-map match-all BUSINESS-DATA
match access-group name BUSINESS-DATA
class-map match-all INTERNET
match access-group name INTERNET
These in turn utilize the following Extended named access-lists:

!
ip access-list extended VOICE
permit udp any any range 16384 32767
!
ip access-list extended VIDEO
!
ip access-list extended CRITICAL
permit tcp any any eq bgp
!
ip access-list extended BUSINESS-DATA
permit tcp any any eq 3000
!
ip access-list extended INTERNET
permit ip any any
Figure 156 Traffic Marking at CE
We can see on these access-lists that the varying traffic is classified and remarked at the ingress point of
the CE based on the TCP/UDP port ranges, this could be also based on IP address ranges, existing IP
Precedence / DSCP code point values and so on. This is really upto Wateen Telecom and the customer to
define.
The egress policy (from CE towards PE) is where we can configure the Priority Queue (LLQ) and the
constraints per traffic class such as bandwidth requirements, WRED etc.
This sample assumes that there is a 10mb link between CE-PE

Quality of Service
policy-map EGRESS
class VOICE
priority 4000 # This uses 4Mb LLQ for Voice
police cir 4000 bc 1500 be 1500 conform-action transmit exceed-action drop
# # This polices the traffic to ensure
# there is no more than 40Mb
class VIDEO
bandwidth percent 20 # This uses 2Mb for Video
class CRITICAL
bandwidth percent 10 # This uses 20% of the 10mb available
random-detect # This uses WRED on the Data Class
class BUSINESS-DATA
class INTERNET
random-detect
class class-default
random-detect # run WRED on this class
Figure 157 BW reservation Per Customer Model
Note: by default an interface (on the low end CE equipment) can only have 75% of it’s bandwidth used
for QOS, this is to cover overhead etc. This can be changed by using the command “max-reserved-
bandwidth” and expressing more than 75. This limitation does not exist on the 75/7600.
Note: in this example Voice and Video is split into two classes (VOICE and VIDEO) in two separate
classes. However if it is desired to place both video and voice in the same priority class and if the
customer access link (CE-PE) is less than 2mb, it is not recommended to combine Video with Voice in
the LLQ, this is because the serialization delays caused by large video packets will cause jitter and
latency in the voice traffic. Therefore in this scenario there should be a separate class for VOICE which
uses the priority mechanism and a separate class for VIDEO.
In that case, as shown in the above example, the video class is allocated a separate queue with a
bandwidth percent 20 to allow for 2Mb of 10Mb (but not in a LLQ).
Note: the size of the LLQ and the percentage of the bandwidth specified will vary depending on the
services that Wateen Telecom will sell and the requirements of the end users. Please also note that there is
only one LLQ on the system wide basis, regardless of number classes defined and configured with
“priority” command.
Qos at the Edge Layer
In the case of an unmanged CE service, classification and marking can be achieved at the Edge (PE)
layer. It is understood in intial deployment of this service Wateen Telecom will be using both managed
and unmanaged CE services model. PE however is not the most recommended position in the network to
configure these services as these will cause additional load on the device.
The configurations for classification and marking on the edge have to be re-applied here using the Cisco
MQC in absence of managed CE device.

Quality of Service
Under managed CE scenario, at the Edge of the network we should be focused on queueing towards the
core and queuing towards the Access Layer.
As we saw earlier at the edge we can have more than three classes, however in the edge / core some of
these classes should be combined into a single queue to make QOS easier to provision in the core – most
SP’s have only 3 or 4 classes in their core. It is understood that Wateen Telecom has decided to keep all
five classes within the core as well.
For unmanaged CE, classification and marking will happen much the same way as if it was managed CE
but on the PE device, these are repeated here for convenience. These can be used on 7600.
policy-map INGRESS
class VOICE
set ip precedence 5
class VIDEO
set ip precedence 4
class BUSINESS-DATA
set ip precedence 3
class CRITICAL
set ip precedence 2
class INTERNET
set ip precedence 0
Class map mentioned above are shown below:

match access-group name VOICE
match access-group name VIDEO
match access-group name CRITICAL
match access-group name BUSINESS-DATA
match access-group name INTERNET
These in turn utilize the following Extended named access-lists as shown earlier:
!
ip access-list extended VOICE
!
ip access-list extended VIDEO
!
ip access-list extended CRITICAL
permit tcp any any eq bgp
!
ip access-list extended BUSINESS-DATA
permit tcp any any eq 3000
!
ip access-list extended INTERNET
permit ip any any

Quality of Service
Please note that above acls are given as examples only. This policy can be applied to customer facing
(PE-CE) link ingress direction.
Interface serial 0/0
Service-policy input INGRESS
Figure 158 Qos Configuration at PE (unmanaged CE)
The 7600 has many varying different ways to provision Quality Of Service based on the ingress linecard
type, egress linecard type and so on, this document focuses on QOS for Gigabit Ethernet linecards and
WAN (SIP) based linecards.
The 7600 also has some limitations for QOS, WAN based linecards can only support a single match in a
class (this match could be an ACL to provide more complexity) , additionally the following match
commands are not supported on SIP-400:
• Match Discard class (match discard-class)

• Match Input interface (match input-interface)
• Match IP RTP (match ip rtp)
• Match IPv4 and IPv6 ToS
• Match MAC address (match mac)
• Match Packet length (match packet length)
• Match Source and destination autonomous system (AS) (match as)
• Match VLAN (match vlan)
• Match protocol*
• Match class-map
• Match qos-group
• Match source-address
• Match destination-address
* In Release 12.2(18)SXE and later releases, Supervisor Engine 720 supports the match protocol ipv6 command.
Additionally, the Cisco 7600 SIP-400 does not support the following QoS classification features with
AToM:
• Matching on data-link connection identifier (DLCI) is unsupported.
• Matching on virtual LAN (VLAN) is unsupported.
• Matching on class of service (CoS) is unsupported
• Matching on input interface is unsupported.
• Matching on packet length is unsupported.
• Matching on media access control (MAC) address is unsupported.
• Matching on protocol type, including Border Gateway Protocol (BGP), is unsupported.
* In Release 12.2(18)SXF and later relases: When .1Q encapsulation is configured on the 7600-SIP-400 with GE SPA, the CoS bits
in the VLAN tag can be used to differentiate the packet priority and a QoS service policy can be applied to the interface.
Note: 7600 same when using WAN (SIP) baesd linecards, the major exception being that the 7500 can
support the “class class-default” where as the 7600 (OSM only) cannot have a class-default and therefore
Quality of Service
all possibilities have to be accounted for in the classes, the Flexwan is as per the 7500 and can have a
class default.
The following is a sample MQC policy-map that would be applied to a GigE uplink interface (egress)
towards the GSR core (PE-P) link or on a POS uplink STM-4 (oc-12) interface towards the GSR core
(PE-P).
match mpls exp <topmost> 5
match mpls exp <topmost> 3 6
match mpls exp <topmost> 1 0
policy-map edge_oc12_policy
class VOICE
priority
police 248800000 conform-action transmit exceed-action drop
class VIDEO
random-detect
random-detect precedence 4 375 2423 1 !WRED threshold per Cisco Recomm.*
class BUSINESS-DATA
random-detect
class CRITICAL
random-detect
class INTERNET
random-detect
random-detect precedence 0 175 1500 1 # randomly populated to
#differentiate and drop prec 0 first
Interface POS 2/1/1

service policy output edge_oc12_policy
Figure 159 Qos Configuration towards PE-P Link
* Please see futher discussion of WRED thresholds in the section: Queueing at

the Core layer
Exception to above policy is when LAN based cards are being used. This will be discussed below:
Following scenarios are possible with 7600:
Ingress & Egress linecards are WAN cards:

Policies as discussed above can be applied for ingress to mark/remark the traffic
Quality of Service
Policies for egress can be applied as described in example above.
Ingress line card: Gigabit Ethernet linecard, & Egress linecard: SIP-400
1. Ingress policy as described above can be applied to gigabit Ethernet linecards for
marking/remarking.
In order for Ethernet linecards to be able to classify, police and mark, qos must first be globally
enabled by using the following command:
mls qos
To verify qos is globally enabled, show mls qos command can be used.
To trust incoming traffic, the ports must be classified as trusted by using the following command
under the interface:
mls qos trust dscp
2. Egress policy will be same as described above applied to flexwan
Qos queueing features are determined by the type of ethernet linecard on 7600 series router.
While PFC on Supervisor 720 performs classification, marking, mapping, and policing functions, all
queuing and congestion avoidance policies are administered by the linecards. This inevitably leads to per-
linecard hardware-specific capabilities and syntax when it comes to configuring queuing and dropping.
While ingress queuing capabilities exist (i.e.queueing towards switch fabric), receive queues are
extremely difficult to congest, even in controlled lab environments. Ingress congestion implies that the
combined ingress rates of traffic exceed the switch fabric channel speed, and thus would need to be
queued simply to gain access to the switching fabric. On Supervisor 720, this means that a combined
ingress rate of up to 40 Gbps per slot would be required to create such an event, Therefore it is
recommended to implement egress / transmit queuing.
There are currently six main transmit queuing/dropping options for Ethernet linecards:
• 2Q2T—Indicates two standard queues, each with two configurable tail-drop thresholds.
• 1P2Q1T—Indicates one strict-priority queue and two standard queues, each with one configurable
WRED-drop threshold (however, each standard queue also has one nonconfigurable tail-drop threshold).
• 1P2Q2T—Indicates one strict-priority queue and two standard queues, each with two configurable
WRED-drop thresholds.
• 1P3Q1T—Indicates one strict-priority queue and three standard queues, each with one configurable
WRED-drop threshold (however, each standard queue also has one nonconfigurable tail-drop threshold).
• 1P3Q8T—Indicates one strict-priority queue and three standard queues, each with eight configurable
WRED-drop thresholds (however, each standard queue also has one nonconfigurable tail-drop threshold).
• 1P7Q8T—Indicates one strict-priority queue and seven standard queues, each with eight configurable
WRED-drop thresholds (on 1p7q8t ports, each standard queue also has one nonconfigurable tail-drop
threshold).
Almost all Ethernet linecards support a strict-priority queue and when supported, the switch services
traffic in the strict-priority transmit queue before servicing the standard queues. When the switch is
servicing a standard queue, after transmitting a packet, it checks for traffic in the strict-priority queue. If
the switch detects traffic in the strict-priority queue, it suspends its service of the standard queue and
completes service of all traffic in the strict-priority queue before returning to the standard queue.
The Transmit Queuing/Dropping capabilities can be determined by using the “show queueing interface”
commands.

Quality of Service
CEF256/ CEF720 Modules Receive Queue Transmit Buffer Size

Ethernet Linecard Description Structure QueueStructure
WS-X6548-GE-TX Catalyst 6000 8-port 1P1Q4T 1P2Q2T 512KB per port
GigabitEthernet
Module (with
Enhanced QoS)
WS-X6724-SFP Catalyst 6500 24- 1Q8T 1P3Q8T 1MB per port
port GigabitEthernet
SFP Module
WS-X6748-GE-TX Catalyst 6500 48- 1Q8T 1P3Q8T 1MB per port

port 10/100/1000 RJ-
45 Module
** Supervisor 720 uplink ports have following qos capabilities:

Receive Queue Structure : 1p1q4t Transmit Queue Structure : 1p2q2t
** WS-SUP32-GE-3B uplink ports have following qos capabilities :
Receive Queue Structure : 2q8t Transmit Queue Structure : 1p3q8t
In summary we have two distinct qos capabilities on transmit queue side : 1p3q8t and 1p2q2t. Each of
queuing structure is discussed below:
1P2Q2T
Qos must first be globally enabled by using “mls qos” command in config mode. Please note that the
buffer allocation for the PQ (Q3) is not configurable, but is, by default (for the 1P2Q2T queuing structure
only) set to equal the size defined for Q2.
There are two points to observe: A) there are only three queues, (one strict priority and two queues .B)
Size of PQ is set same size as Q2.
This means that In order to closely mimick our qos allocation of:
40% voice
20% video
10% business-data
10% critical
And 10% of internet, some classes will need to be aggregated.
Therefore, the queue size ratios have been changed, specifically Q1 is set to 30% and Q2 is set to 35%
(which indirectly sets Q3 to match, at 35%). Voice will be served by strict priority queue with 35%. The
internet traffic (0,1) will be mapped to Q1. Video, Business Data and Critical class will be mapped to Q2.
Q2 will be configured such that it is serviced 75% of the time, compared to Q1 which will be served 25%.
These values are chosen as examples, Wateen Telecom can modify them based on their requirements.
For example the configuration would look like following:

Quality of Service
interface range GigabitEthernet4/1

“wrr-queue queue-limit 30 35” #Sets the buffer allocations to 30%
for Q1 and 35% for Q2
#Indirectly sets PQ (Q3) size to
equal Q2 (which is set to 35%)
wrr-queue bandwidth 25 75 # Sets the WRR weights for

25:75 (Q1:Q2) bandwidth
servicing.
wrr-queue random-detect min-threshold 1 40 80 #Sets Min WRED Thresholds

for Q1T1 and Q1T2 to 40 and
80, respectively
wrr-queue random-detect max-threshold 1 80 100 #Sets Max WRED Thresholds

100, respectively
wrr-queue random-detect min-threshold 2 70 80 #Sets Min WRED Thresholds

80, respectively
wrr-queue random-detect max-threshold 2 80 100 #Sets Max WRED Thresholds
100, respectively
wrr-queue cos-map 1 1 0 #Assigns Cos value of 1 to

Q1 WRED Threshold 1
wrr-queue cos-map 1 2 1 #Assigns Best Effort to Q1
WRED Threshold 2
wrr-queue cos-map 2 1 2 4 #Assigns CoS 2,3,4 to Q2

WRED Threshold 1
wrr-queue cos-map 2 2 3 6 7 #Assigns
Network/Internetwork
Control to Q2 WRED
Threshold 2
priority-queue cos-map 1 5 #Assigns VoIP to PQ
Figure 160 Sample Qos Configuration for 1P2Q2T Linecard
1P3Q8T
Under this mode, there are three queues while one strict priority queue, this is queue number 4.

Quality of Service
interface range GigabitEthernet1/1
wrr-queue queue-limit 10 20 30 #Allocates 10% for Q1, 20%

for Q2, 30% for Q3 and
remainder 40% for Q4 (PQ)
wrr-queue bandwidth 10 30 60 #Sets the WRR weights for

10:30:60 (Q1:Q2:Q3)
bandwidth servicing,
wrr-queue random-detect 1 #Enables WRED on Q1
wrr-queue random-detect min-threshold 1 80 100 100 100 100 100 100 100
#Sets Min WRED Threshold
for Q1T1 to 80% and all
others to 100%
wrr-queue random-detect max-threshold 1 100 100 100 100 100 100 100 100
#Sets Max WRED Threshold
others to 100%
others to 100%
#Sets Max WRED Threshold
others to 100%
for Q3T1 to 50%, Q3T2 to
60%, Q3T3 to 70% Q3T4 to
80%, Q3T5 to 90% and all
others to 100%
# Sets Max WRED Threshold
for Q3T1 to 60%, Q3T2 to
70%, Q3T3 to 80% Q3T4 to
90%, Q3T5 to 100% and all
others to 100%
wrr-queue cos-map 1 1 1 #Assigns Scavenger/Bulk to

Q1 WRED Threshold 1
wrr-queue cos-map 2 1 0 #Assigns Best Effort to Q2
WRED Threshold 1
wrr-queue cos-map 3 1 4 # Assigns CoS 4 to Q3 WRED
Threshold 1
wrr-queue cos-map 3 2 2 #Assigns CoS 2 to Q3 WRED
T2
wrr-queue cos-map 3 3 3 # Mission-Critical Data
(Cos 3) to Q3 WRED T3
T4
T5
priority-queue cos-map 1 5 #Assigns VoIP to the PQ

Quality of Service
(Q4)
Figure 161 Sample Qos Configuration for 1P3Q8T Linecard
Egress on the EDGE layer (towards the CE) it is also necessary to implement a QOS policy for an end to
end qos service, however it is not needed under unmanaged CE offering. Under managed CE scenario,
this will be the same policy in egress direction towards PE-CE as it is applied in egress direction from
CE-PE.
For 7600 series routers in city and remote PoP, if the traffic is coming in on a trusted interface (managed
CE) “mls qos trust [dscp | ip-precedence | cos]” can be used to derive relative qos values and match on
ip-precedence or exp value for egress queueing.
For 7600 Series router in the core PoPs , only Ethernet cards are used, as such line card specific policies
should be used as described in the previous section. When 7600 receives MPLS packets, the PFC3BXL
always trusts EXP in the received topmost label. None of following have any effect on MPLS packets:
•Interface trust state
•Port CoS value
•Policy-map trust command
Qos should be enabled on the system with “mls qos” command.
Queueing at the Core layer
Once the core layer is reached (the GSR’s), Wateen Telecom will need to perform appropriate queuing
and scheduling in the core.
Below is a sample QOS configuration for the GSR for egress queueing for an STM-16 linecard.
Engine 3 & Engine 5 support the Cisco MQC for egress QOS.

Quality of Service

match mpls exp 5
match mpls exp 4
match mpls exp 2
match mpls exp 3 6
match mpls exp 0 1
policy-map core_oc48_policy
class REALTIME
priority
police 958400000 conform-action transmit exceed-action drop
class VIDEO
random-detect # potentially not used for Video
#traffic
random-detect precedence 4 6220 20733 1 # depends on application profile
class BUSINESS-DATA
random-detect
random-detect precedence 3 3220 10733 1 # Randomly populated to drop pre 3
#first
class CRITICAL
class INTERNET
random-detect
random-detect precedence 0 3220 10733 1 # Randomly populated to
#differentiate and drop prec 0 first
Figure 162 Sample Qos Configuration for P-P link
This shows a sample egress QOS policy for an STM-16 linecard, we can see the LLQ is policed at a
maximum of 958.4 Mb, this is approximately 40% of STM-16 linerate, the critical class uses 10% of the
bandwidth, WRED values used are values as requested by the customer, however these may still need to
be tuned by Wateen Telecom depending on the types of traffic etc in the network.
Finally there is a class called class-default which gets all other traffic types (unaccounted for by Qos
policies). Depending on any unaccounted traffic, it is possible to make bandwidth guarntees for class-
default as well. e.g. it is possible to use class class-default as a class to account for internet and any other
traffic. Given that in our case internet traffic is explicitly accounted for, class-default configuration is not
needed.
Please note that mark probability denominator as well as exponential-weighting-constant cannot be

configured on Engine 3 and Engine 5 line cards. Following error message will be encountered when
applying the service-policy on an interface.

Quality of Service
service-policy output test

% Configuration of probability denominator with value 20
% in policy test for interface GigabitEthernet1/1/1 is not supported.
% Setting it to the default value of one.
% Configuration of exponential weight not supported.
Please note that Cisco 12000 series routers by default perform qos based on Layer-3 payload
(i.e.excluding layer-2 overhead of underlying physical transport mechanism). While it is possible to have
this adjusted for some line cards and certain transport types, it is not possible to do the same for all line
cards and all interface types. The Layer-2 overhead adjustment is primarily targeted towards customer
facing ports to meet customer SLAs rather than core facing ports. This is discussed further below:
http://www.cisco.com/en/US/products/sw/iosswrel/ps1829/products_feature_guide09186a00805e8fbb.ht
ml
For Wateen Telecom it is suggested to leave qos features such as policing, shaping, and bandwidth as
layer-3 as the these are core links (not customer facing ports) and the feature is not available on all
linecards. What this implies is that while performing bandwidth allocation, overall Layer-3 capacity of
the interface should be considered rather than physical transport characteristics. E.g. STM-1 link which
has an overall capacity of 155Mbps cannot be used to make bandwidth computations, instead 149 Mbps
should be used for such computations. This number excludes SONET overhead of the traffic. Similarly
STM-4 will have overall capacity of 599Mbps and STM-16 will have overall capacity of 2396Mbps.
WRED threshold is defined as queue depth in terms of packet size. On a 12000 series router, current
version of IOS also allows defining the threshold in terms of units of times (which is subsequently
converted into packets by the system) as seen below. This is referred as “Time-Based Thresholds for
WRED” and was introduced in 12.0(28)S.
router(config-pmap-c)#random-detect precedence 3 30 ?
<1-262143> maximum threshold (in packet by default)
cells in cells
ms milliseconds
packets in packets
us microseconds
router(config-pmap-c)#random-detect precedence 3 30 ms 100 ms
STM-16 Link:
For an STM-16 (OC-48) link, minimum threshold of 6220 and maximum threshold of 20733 is suggested
per the customer requirement. It is understood that customer will like to use buffering thresholds of 30
and 100 milliseconds for MinTH and MaxTH respectively. These values are based on MTU size of 1500
bytes (worst case scenario). A better approximation can be had by using the average size of packet on
customer network. As indicated earlier, these values will need to be tuned over period of time.
Computation for different link sizes is shown below.
B= 2488Mbps/1500*8bits = 207333.3
For Min threshold: .03 * B = 6220 packets
For Max threshold: .1* B = 20733 packets
• Please note that values will be adjusted to be a number which is a power of 2 for min threshold based
on max threshold i.e. the (MaxTH – MinTH) needs to be a power of 2 as documented earlier.
1Gigabit Ethernet link:
B=1000Mbps/1500*8=83333.3
For Min threshold: .03*B = 2500 packets

Quality of Service
For Max threshold: .1*B =8333 packets
STM-4 link:
B=622Mbps/1500*8= 51833
For Min threshold:.03*B= 1555 packets
For Max threshold: .1*B =5183 packets

Traffic Engineering
Traffic Engineer Disclaimer

There have been no clear requirements provided by Wateen Telecom in terms of expected traffic patterns,
or the strategy with regards to how traffic engineering should be deployed, specifically what customer
requirements should Traffic Engineering address, it was agreed that there were no requirements to deploy
traffic engineering currently. However customer requested for design templates and technology overview
as a forward looking requirement. The traffic engineering section in this document will contain the
following sections
• MPLS Traffic Engineering Overview
• MPLS TE Fast Reoute
• Generic configuration templates for the MPLS TE
MPLS Traffic Engineering Overview

The term in general means ways of controlling traffic in an IP network. Traffic Engineering could involve
measuring traffic pattern, modeling it & then controlling it’s flow to achieve various goals. One of them
could be optimum link utilization in a network, especially for Inter PoP traffic.
Traffic Engineering creates one or more explicit paths with bandwidth assurances for each traffic trunk.
It takes into consideration the policy constraints associated with the traffic trunks, and the physical
network resources, as well as the topology of the network. This way, packets are no longer routed just
based on destination, but also based on resource availability, and policy.
The network operator must create a traffic model. Based on statistics collected from the routers, as well
as administrative policies, the network operator needs to identify the traffic trunks within the network,
and decide how these traffic trunks should be routed.
The router uses RSVP to set up Label Switching Paths (LSPs) and to reserve bandwidth at each hop along
the LSPs. During the LSP setup process, any router within the network must perform admission control
and/or preemption to ensure that resources are available to honor the reservation. After the paths are set
up, the head-end routers forward the packets belonging to traffic trunks by placing them into the
appropriate LSPs.
RSVP signaling & keepalive mechanism keeps sure that tunnel state is always detected & they stay up as
long as they’re needed & then thry’re torn down via RSVP signalling.

RSVP signaling takes care of the constraint based routing by looking for availability of the appropriate
resources on the best available paths. Before the RSVP signal goes thru, the router originating the tunnel
(aka head-end router) will run constraint based-routing algorithm (called CSPF) to see which routes
satisfy the resources for a certain tunnel setup. The outcome is a path thru which the RSVP message is
sent to see if the resources are currently available.
Then RSVP sends a PATH message via the path obtained through CSPF. The PATH message travels
from the head-end to the tail-end. This PATH message checks with each router in the path if they have
the rsources available that were requested by the tunnel. The PATH message also has new objects for
MPLS TE, which are discussed in later sections. One important new object is the LABEL_REQUEST
object. This object has information about the Layer 3 protocol being used, & in case of ATM or Frame
Relay networks, it also has a range in which labels can be allocated. This object indicates that Label
binding for this path is requested
Once the PATH message reaches the tunnel destination (aka tail-end), the tail-end generates an RSVP
RESV message which goes back to the head-end router.
The RESV message travels from the tail-end router to the head-end, in the reverse direction of the PATH
message. Every router sending the RESV message sends a LABEL object to the upstream router, so that
the upstream router can bind this as an out-going LABEL for the in-coming LABEL (which the upstream
router will send as the LABEL object in the RESV message to its upstream router).
Routers actually lock the resources requested by tunnels, when they see the RESV message. This prevents
pre-mature locking of resources during the PATH message, say if the PATH message were to fail
somewhere downstream
So in short RSVP helps signal a tunnel through a network & makes sure that the resource requirement for
the tunnel can be met.
After a tunnel is established, PATH & RESV messages keep on going thru the tunnel path & the
receiving routers use this as a keepalive mechanism for this tunnel. We tear down the tunnel, if we miss 4
consecutive PATH or RESV messages. This is because RSVP is a soft-state protocol & this is the only
way RSVP has to keep track of state.
TE tunnels forward traffic using label switching. The advantage with this is that the tunnels can be routed
independent of IGP best-path routing & this makes Source Based Routing possible, where the head-end
decides that packets need to go thru a certain tunnel because of some policy.
Traffic Engineering Components

Traffic Engineering is made up of several components, each of which is described in the following
sections.
Traffic Trunk Attributes

Traffic trunk attributes allow the network operator to describe the characteristics of traffic trunks. They
must be granular enough to account for the different types of packets traversing the network, and detailed
enough to specify the desired behaviour in failure situations. There are six traffic trunk attributes and
each is described below.
Bandwidth
This attribute specifies the amount of bandwidth the traffic trunk requires.
Path Selection Policy

This attribute gives the network operator the option to specify the order in which the head-end routers
should select explicit paths for traffic trunks. Explicit paths may be either administratively specified or
dynamically computed.
Resource class affinity

This attribute is used to allow the network operator to apply path selection policies by administratively
include or exclude network links. As will be shown later, each link on the network may be assigned a
resource class as one of the resource attributes. Resource class affinity specifies whether to include or
exclude links with resource classes in the path selection process. It takes the form of the tuple <resource
class mask, resource affinity>. The "resource class mask" attribute indicates which bits in the resource
class need to be inspected, and the "resource affinity" attribute indicates whether to explicitly include or
explicitly exclude the links.
Adaptability
This attribute indicates whether the traffic trunk should be re-optimised. The re-optimisation procedure is
discussed in a later section.
Resilience
This attribute specifies the desired behaviour under fault conditions, i.e., the path carrying the traffic
trunk no longer exists due to either network failures or preemption. TE restoration operation is discussed
in a later section.
Priority
Priority is the mechanism by which the operator controls access to resources when the resources are
under contention. It is a required function to place all traffic trunks. Another important application of the
priority mechanism is supporting multiple classes of services. We assign two types of priorities to each
traffic trunk: holding priority, and setup priority. Holding priority determines whether the traffic trunk
has the right to hold a resource reservation when other traffic trunks attempt to take away its existing
reservation. Setup priority determines whether the traffic trunk as the right to take over the resources
already reserved by other traffic trunks.
Resource Attributes
Resource attributes are used to describe the network links used for path calculations. There are three
resource attributes, each of which is described below.
Available Bandwidth
This attribute describes the amount of bandwidth available at each setup priority. Note that the available
bandwidth for the higher setup priority is always larger than that for the lower setup priority. This
attribute needs not necessarily reflect the actual available bandwidth. In some cases, the network operator
may oversubscribe a link by assigning a value which is larger than the actual bandwidth, e.g., 49.5 Mbps
for a DS-3 link.
Resource Class
This attribute indicates the resource class of a link. Recall that the trunk attribute, resource class affinity,
is used to allow the operator to administratively include or exclude links in path calculations. This
capability is achieved by matching the resource class attribute of links with resource class affinity of
traffic trunks. The resource class is a 32-bit value. The resource class affinity contains a 32-bit resource
affinity attribute and an associated 32-bit resource class mask. A link cannot be considered in the path
calculation for a traffic trunk unless the following equation holds:
(Resource Class) & (Resource Class Mask) == (Resource Affinity)

where "&" refers to a bitwise AND operation, and "==" refers to bitwise equality.
Information Distribution
Since the resource attributes are configured locally for each link, they must be distributed to the head-end
routers of traffic trunks. These resource attributes are flooded throughout the network using extensions to
the link-state intra-domain routing protocols, namely IS-IS and OSPF. The flooding takes places when (a)
link state changes, (b) resource class of a link changes (this could happen when a network administrator
reconfigures resource class of a link, or (c) the amount of available bandwidth crosses one of the pre-
configured thresholds. The frequency of flooding is bounded by the OSPF and IS-IS timers. There are
“up” thresholds and “down” thresholds. The “up” thresholds are used when a new trunk is admitted. The
“down” thresholds are used when an existing trunk goes away. Extensions to IS-IS and OSPF are
discussed in a later section.
Path Selection
Path selection for a traffic trunk takes place at the head-end routers of traffic trunks. Using extended IS-
IS/OSPF, the edge routers have knowledge of both network topology and link resources. For each traffic
trunk, the router starts from the destination of the trunk and attempts to find the shortest path toward the
source (i.e., using the shortest path first (SPF) algorithm). The SPF calculation does not consider the
links which are explicitly excluded by the resource class affinities of the trunk, as well as the links which
have insufficient bandwidth. The output of the path selection process is an explicit route consisting of a
sequence of label switching routers. This path is used as the input to the path setup procedure.
Path Setup
Path setup is initiated by the head-end routers. RSVP is the protocol which establishes the forwarding
state along the path computed in the path selection process. The head-end router sends a PATH message
for each traffic trunk it originates. The PATH message carries the explicit route computed for this traffic
trunk. As a result the PATH message always follows this explicit route. Each intermediate router along
the path performs trunk admission control after receiving the PATH message. Once the router at the end
of the path receives the PATH message, it sends a RESV message in the reverse direction towards the
head-end of the traffic trunk. As the RESV message flows toward the sender, each intermediate node
reserves bandwidth and allocates labels for the trunk. Thus when the RESV message reaches the sender,
the LSP is already established. The following Figure shows an example of the path setup procedure.
R8 R9
R3
R4
R2 Pop
R5
R1 32
49
17
R6 R7
22
Setup: Path (R1->R2->R6->R7->R4->R9)
Reply: Resv communicates Tags and

reserves bandwidth on each link
Figure 163 TE Path Setup
Trunk Admission Control

What if a router along a computed path has insufficient bandwidth to honour the resource requested in the
PATH message? Trunk admission control deals with this situation. When a router receives a PATH
message, it checks if there is enough bandwidth to honour the reservation at the set-up priority of the
traffic trunk. If there is enough bandwidth, the reservation is accepted, otherwise the path setup fails.

When the router receives the RESV message, it reserves bandwidth for the LSP. If pre-emption is
required, it must tear down existing trunks. As part of trunk admission control, the router must do local
accounting to keep track of resource utilization, and trigger IS-IS/OSPF updates when the available
resource crosses the configured thresholds
Path Maintenance
Path maintenance refers to two operations: path re-optimization and restoration. Re-optimization
periodically checks to see if here is a better path for the LSP than has previously been calculated.
Traffic Forwarding into TE Tunnel

Cisco TE Implementation provides the following methods for forwarding traffic into a TE tunnel. A
description of every method is out of the scope of this document, due to the fact, that only two (three with
TE FRR) methods are applicable for proposed Tactical TE solution.
• Dynamic Routing (IGP Autoroute)
• Policy Based Routing (PBR)
• Forwarding Adjacency
• Static Routing
• Class Based Tunnel Selection (CBT)
• AToM Tunnel Selection
MPLS TE Fast ReRoute Link Protection

A feature of traffic engineering is a facility known as Fast ReRoute (FRR). Fast ReRoute (FRR) is a link
or node protection feature, allowing for temporary bypassing of the failed link or node over a pre-
established tunnel while the head-end is rerouting the failed LSP.
FRR provides two types of protection; Link Protection and Node Protection. Link protection protects a
local link by rerouting around the failure to the next hop router in an alternate direction. Node protection
is a little more complex and reroutes around a node failure to the next-next hop that is, the next node in
the path past the node that failed. The switching time for a FRR protected link or node is designed to
approximate SONET auto-protection switching (APS) times in a layer 3 network.
The MPLS TE with link protection is documented as a forward looking feature for Wateen Telecom’s
core network (consisting of 12816 routers). Wateen Telecom is advised to use IGP fast convergence (and
BFD as a worst case failure detection mechanism). Any benefit achieved by deploying traffic engineering
needs to be weighed against the complexity of deploying, supporting and troubleshooting traffic
engineering.
As a forward looking option, it is also possible to deploy Class Based Tunnel Selection as a way to
protect voice traffic only with fast rereoute, rather than protecting all traffic. CBTS feature for 12000
series platform (with Engin 3 and Engine 5 linecard) is supported with latest IOS release of 12.0(32)SY.
This release is currently not suggested for Wateen Telecom and can be considered as a future release to
upgrade the 12816 series routers to.
Link protection means that each link that a primary TE label switch path (LSP) travels along can be
protected and a backup path initiated from the point of the link failure, independent of the head-end router
where the primary tunnel started.
The following figure illustrates this process of Fast ReRoute. A primary tunnel has been created that
traverses {R1, R2, R4, R5} with R1 being the head-end (where the tunnel was configured) and R5 being
the tail end (the tunnel destination). In our example, the link between R2 and R4 is protected by Fast
ReRoute using a secondary tunnel referred to as the backup tunnel. This backup tunnel traverses {R2, R3,
R4) where R2 is the head-end for the backup tunnel and R4 is the tail-end. In the event of a R2/R4 link
failure, R2 will detect the link is not operational, and immediately reroute the primary TE tunnel for that

link through the backup TE tunnel via {R2,R3,R4} onwards to R5. The important thing to remember
about link protection is that the traffic transmitted on the backup tunnel must always end up at the router
that is connected to the other end of the original protected link. That is, it must be the same next hop
router. During the failure the primary tunnel will be encapsulated with an additional label on the stack
representing the backup tunnel.
All primary tunnels traveling over the same protected link can be protected by the same backup tunnel.
Therefore it is conceivable that the backup path will support multiple primary tunnels during a failure.
Link protection provides a very scalable failure mechanism in contrast to path protection which is
discussed later.
Whilst link protection has kicked in and traffic is still flowing, the head-end of the primary TE tunnel has
been advised of the downstream failure and can take steps to re-establish a better path for the traffic
assuming one exists. Once original link is restored the original LSP for the primary tunnel will be re-
established if required (through the expiration of re-optimisation timers).
Backup R3
Head End Tail End

R1 R2 R4 R5
Primary Tunnel
Backup R3
Head End Tail End

R1 R2 R4 R5
Link Failure
Primary Tunnel Backup Tunnel Primary Tunnel
Figure 164 Label Switch Path with Link Protection
It is important to understand the TE tunnels are unidirectional. Therefore there must be one in each
direction. In the case of the previous example, there would be a backup tunnel running between {R4, R3,
R2} and also a primary tunnel from {R5, R4, R2, R1}
Primary Tunnels
To provide link protection two types of tunnels exist:
• Primary tunnels
• Backup Tunnels
Under normal or steady state conditions (no failure present) traffic offered to the primary tunnel will
follow the label switch path (LSP) of the primary tunnel between the head-end and the tail-end routers. In
the event of a failure the primary tunnel traffic will be diverted along the backup tunnel LSP.
For traffic to be protected by a backup tunnel with FRR it must originally be carried in a primary tunnel.
In the event of a link failure, any traffic that is not part of a tunnel will not be able to take advantage of
the FRR of a backup tunnel but instead rely on the convergence of the IGP. The following figure
illustrates this concept. All traffic that enters the tunnel at the head-end router R1 will be encapsulated
with the label provided by RSVP for the primary tunnel. Upon failure of the protected link a second label
representing the backup tunnel will be pushed onto the stack for MPLS packets holding the primary

tunnel label and traffic will continue along the backup path. In contrast, the following figure shows
packets that are not part of a tunnel, that is are either unlabelled IP packets or MPLS packets using labels
provided by LDP, will be dropped at R2 until the IGP reconverges to another path.
Traffic dropped until
IGP converges
Non
Tunnel R3
Traffic
Link Failure
Head End Tail End
R1 R5
R2 R4
Tunnel
Primary Tunnel Backup Tunnel Primary Tunnel
Traffic
Little or no traffic dropped
Figure 165 Protected Tunnel Traffic
One Hop Primary Tunnels

The fact that traffic must be in a tunnel to be protected introduces an important concept in relation to the
Wateen Telecom network known as the one hop primary tunnel.
Normally, but not necessarily, a primary tunnel will traverse many nodes. For example, a tunnel can be
extended across the core between two PE routers to provide a pipe for a particular kind of traffic such as
voice. In the case of the Wateen Telecom core, since only the link and its adjacencies need to be protected
then head-end and tail-end routers of the primary tunnel will be the physical link adjacencies. This will
result in every link in the network having a series one hop tunnels as shown in the following figure. Each
node will be a either a head-end or a tail-end of any particular primary tunnel. TE tunnels are uni-
directional therefore there will be two tunnels per link running in opposite directions. If one end of the
link is referred to as “A” and the other “B” then there will be a primary tunnel going from the A Æ B
direction and another going from the B Æ A direction.
B Í
el R3 Pr
unn Î Pr ima
T A im ry
ry n el ar Tu
a n yT nn
Pr
im Tu un el
B
a ry ne
Í lÎ
im
Pr
R2 R4
R1 R5
Primary Tunnel AÎ Primary Tunnel AÎ Primary Tunnel AÎ

Í Primary Tunnel B Í Primary Tunnel B Í Primary Tunnel B
Figure 166 One Hop Primary Tunnels
For the case of link protection, primary and backup tunnels are created as zero bandwidth tunnels.
There is no bandwidth constraint placed upon them for LSP creation.
OSPF Autoroute
By default TE tunnels are not visible to the IGP. A feature called Autoroute allows TE tunnels to be made
available to OSPF. Autoroute causes OPSF to include the primary tunnel interfaces in its enhanced SPF
calculations thereby replacing the physical next hop interfaces with a tunnel interface.

The one hop primary tunnels are created with zero bandwidth, in other words there is no constraint
imposed on the creation of the tunnel as it is only a single hop. This therefore results in a tunnel interface
that has the same metric as a physical interface. IOS will choose the tunnel interface over the physical
interface. This will result in the next hop interfaces being replaced with the primary tunnel interfaces in
the routing table.
For example, Lhe-12816-P has three physical interfaces connecting to Karachi, Islamabad and
Faisalabad. This means there will be three one hop primary tunnels corresponding to each physical
interface. OSPF will use each of these tunnels in its SPF calculations.
Primary Tunnel Labels

No additional labels are pushed onto the stack when traffic enters a primary tunnel. Since the primary
tunnel consists of one hop, the head-end router (A end of the link) will be the penultimate hop therefore
no additional label is necessary. The outgoing label for the primary tunnel is actually an implicit NULL
label.
Therefore all traffic that is offered to the primary tunnel will use the same label forwarding as if there was
no tunnel (except that the traffic is protected).
LDP over the Primary Tunnel

For label swapping and forwarding to occur over the tunnel as previously discussed, every one hope
primary tunnel must be enabled with the mpls ip command. This will form an LDP peering session over
the primary tunnel to so that next hop labels (Outgoing) can be exchanged as illustrated in the following
figure.
LDP LDP
Í
lB R3 Pr
n ne Î Pr im
ar
Tu lA im yT
ar
y
nne ar
yT un
ne
Pr
im Tu un lB
y ne
LDP Í ar lÎ LDP
im
Pr
R2 R4
R1 R5
Primary Tunnel AÎ Primary Tunnel AÎ Primary Tunnel AÎ

Í Primary Tunnel B Í Primary Tunnel B Í Primary Tunnel B
LDP LDP LDP LDP
Figure 167 LDP peering on One Hop Primary Tunnels
If an LDP session is not established over the tunnel then the outgoing label will be set to “untagged” that
will either black hole the traffic or misinterpret the exposed label (usually the VPN label) at the next hop
in the TFIB.
Backup Tunnels
As previously mentioned every link in the Wateen Telecom core can be associated with a one hop
primary tunnel. In addition, every physical interface can be protected by an associated backup tunnel. The
backup tunnel is configured on the physical link and the primary tunnel is configured to be protected by
the backup tunnel through the use of the “fast reroute” option.

In actual fact, any primary tunnel that traverses the physical interface and has the “fast reroute” option
enabled will be protected by the backup tunnel. In other words the backup tunnel can protect many
primary tunnels (one hop or otherwise) therefore provides a very scalable protection solution.
Backup tunnels are uni-directional, therefore as is the case with primary tunnels there will be two backup
tunnels for each link representing the A Æ B and B Æ A direction.
Even though each backup tunnel traverses a series links that are associated with one hop primary tunnels,
the backup tunnels themselves do not travel within a primary tunnel. They are distinct LSPs in their own
right that have been calculated independently. As with all TE tunnels, RSVP is used to distribute the
labels of a backup tunnel.
In event of a backup tunnel being brought into operation due to a link failure, the relevant outgoing label
associated with the backup will be pushed onto the label stack of the affected packets to reroute the traffic
around the fail link.
TE FRR Configuration Details

This section discusses the configurations necessary on the Cisco 12816 core routers to enable TE and
auto tunnel to provide link protection.
General Configuration
The following configuration example shows the general TE configuration for all Wateen Telecom core
routers. The first command mpls traffic-eng tunnels enables TE on a global basis. The second command
mpls traffic-eng reoptimize events enables automatic reoptimisation of TE tunnels when a link comes
into the UP state. The default reoptimisation timer is set to 3600 seconds.
The OSPF TE sub-commands associated the TE router-id with the loopback interface and also allow
flooding of TE information in to OSPF area 0.
mpls traffic-eng tunnels
mpls traffic-eng reoptimize events link-up
!
router ospf 1
mpls traffic-eng router-id Loopback0
mpls traffic-eng area 0
!
Figure 168 Sample Configuration for all TE routers
Link Configuration
The following configuration example shows the interface configuration for POS interfaces.
The carrier-delay command is also recommended as per the POS interfaces.

!
interface pos x/y/z
ip address <ip address> <subnet mask>
mpls ip
ip rsvp bandwidth
carrier-delay ms 0
!
Figure 169 TE Link Configuration
Sample Configuration
The sample configuration in the following example is shown as a template. Two tunnels would be
required for the primary and backup and associated explicit paths. The primary tunnel will need to be
configured with fast reroute and autoroute.
The example is using the details for the Lahore-Karachi link, from the LHE-12816-P perspective:

! !
interface Tunnel0
description PRIMARY PATH TO KARACHI
mpls ip
tunnel destination 10.100.0.2
tunnel mode mpls traffic-eng
tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng path-option 1 explicit name PRIMARY-C12000-Karachi
tunnel mpls traffic-eng fast-reroute
!
interface Tunnel1
description BACKUP PATH TO KARACHI
tunnel destination 10.100.0.2
tunnel mode mpls traffic-eng
no tunnel mpls traffic-eng autoroute announce
tunnel mpls traffic-eng path-option 1 explicit name BACKUP-C12000-KHI
tunnel mpls traffic-eng record-route
!
interface POS x/y/z
description LINK TO KARACHI
ip address 10.0.0.1 255.255.255.252
mpls traffic-eng backup-path Tunnel1
tag-switching ip
ip rsvp bandwidth
!
ip explicit-path name PRIMARY-KHI enable
next-address 10.0.0.2
!
ip explicit-path name BACKUP-KHI enable
exclude-address 10.0.0.2
!
Figure 170 Sample configuration- One Hop Tunnel
Auto Tunnel Feature

For 12816s prior to IOS 12.0(27)S implementing TE for Fast ReRoute Link Protection would have been a
labour intensive process of configuring primary and backup tunnels on every interface of every router in
the Wateen Telecom core. However a very useful feature called “Auto tunnel” automatically creates all
the necessary primary and backup tunnels with a fraction of the commands.
The AutoTunnel Primary and Backup feature has the following characteristics:
• Backup autotunnel - Enables a router to dynamically build backup tunnels
• Primary one-hop autotunnel - Enables a router to dynamically create one-hop primary tunnels on all
interfaces that have been configured with Multiprotocol Label Switching (MPLS) traffic engineering
(TE) tunnels
If no backup tunnels exist, the following types of backup tunnels can be created:
• Next hop (NHOP)
• Next-next hop (NNHOP)
AutoTunnel has the following benefits:

• Backup tunnels are built automatically, eliminating the need for users to preconfigure each backup
tunnel and then assign the backup tunnel to the protected interface.
• The dynamic creation of one-hop primary tunnels eliminates the need to configure an MPLS TE
tunnel with the Fast ReRoute (FRR) option for the tunnel to be protected.
• Protection is expanded; FRR does not protect IP traffic that is not using the TE tunnel or Label
Distribution Protocol (LDP) labels that are not using the TE tunnel.
NHOP backup tunnels mean that the end point of the backup tunnel is the same end point used by the
one-hop primary tunnel. Therefore traffic diverted to the backup tunnel will always end up at the same
node it would if it travelled via the primary tunnel. NHOP is essentially the B-end of the link.
Creation of Backup Auto Tunnels

To create an NHOP backup auto tunnel, explicit paths are used as follows:
An LSP is computed to the NHOP of the link. This LSP exclude the IP address of the protected link
(avoids traversing the link that is being protected)
The explicit-path name of the tunnel is <nodename>_t<xxx> where xxx matches the dynamically created
backup tunnel ID.
The interface used for ip unnumbered defaults to Loopback0. You can configure this to use a different
interface.
In the event a link along the backup LSP fails, if possible, a new path will be computed excluding the IP
address of the protected link.
Creation of Primary Auto Tunnels

Primary auto tunnels will be dynamically created as one-hop primary tunnels on all interfaces that have
been configured with MPLS traffic engineering. The tunnels are created with zero bandwidth. The
Constraint-based shortest path first (CB-SPF) algorithm is the same as the shortest path first (SPF) when
using zero bandwidth, so the routers choices are the tunnels over the physical interfaces. The tunnels
replace the physical interfaces in the routing table and because they are one-hop tunnels and the
encapsulation is label-implicit so there is no label header necessary (as the A end is the penultimate hop
of the tunnel) .
Explicit paths are used to create primary auto tunnels as follows:
• The explicit path is dynamically created.
• The explicit path includes the IP address for the interface connected to the next hop and the loopback
if the NHOP router (B end of the link)
• The explicit-path name of the tunnel is <nodename>_t<xxx> where xxx matches the dynamically
created backup tunnel ID.
• Interfaces used for ip unnumbered default to Loopback0. You can configure this to use a different
interface.
Request for Fast ReRoute protection is implicitly enabled on all primary autotunnels
Auto Tunnel Configuration

The following configuration example shows all the commands necessary to create the primary and
backup tunnels to protect all core interfaces. The first three commands define the tunnel ID pools to be
used for each category of tunnel. This is important as it allows easy identification of the purpose of the
tunnel.
The mpls traffic-eng auto-tunnel backup nhop-only command will create a backup tunnel for every
physical interface on the router. It creates an explicit path to the next hop (B-end of the link) only
excluding the subnet of the physical link. The tunnel ID will be in the 62000-62999 range.
The mpls traffic-eng auto-tunnel primary onehop command will create a one hop primary tunnel
across the physical interface. It will be automatically protected by the backup tunnel in the event the
physical interface fails. The tunnel ID will be in the 61000-61999 range. The routing table will show one
of these tunnels as the next hop for prefixes.
The mpls traffic-eng auto-tunnel primary config mpls ip is critical in the operation of the primary
tunnel. It configures the primary tunnel with the “mpls ip” command so that LDP will exchange labels
over the primary tunnel. If this command is omitted packets will be dropped.
The command mpls ldp discovery targeted-hello accept must be configured on every next-hop
terminating primary tunnels. The previous commands creates targeted LDP sessions over primary tunnels
and this command enable the receiving router to accept the targeted hellos and to establish the LDP
session.
!
!
mpls traffic-eng auto-tunnel primary tunnel-num min 61000 max 61999
mpls traffic-eng auto-tunnel backup tunnel-num min 62000 max 62999
!
mpls traffic-eng auto-tunnel backup nhop-only
mpls traffic-eng auto-tunnel primary onehop
mpls traffic-eng auto-tunnel primary config mpls ip
mpls ldp discovery targeted-hello accept
!
Figure 171 Sample AutoTunnel Configuration

Network Management & Infrastructure
Security Best Practices
Overview
Following section briefly discusses relevant network management and infrastructure related best practices
as they pertain to Wateen Telecom’s IP/MPLS network elements e.g. 7600 and 12000 series routers.
Complete network management low level design for overall Wateen Telecom’s infrastructure
including all applications and network elements is discussed in a separate low level design
document.
Similarly overall security and data center solution for Wateen Telecom is documented in a separate low
level design document.
Simple Network Management Protocol

The use of SNMP to collect management data, and in some circumstances to configure network devices,
is essential to running an IP network effectively.
The recommendation is that all devices be configured for SNMP read-only access from a select list of
NMS devices only.
The following configuration provides a baseline for an SNMP polling configuration as a starting point:
snmp-server community XXXXXXXXX RO 98

snmp-server contact NOC@wateen.com.pk
snmp-server location XXXXXXXX
snmp-server tftp-server-list 98 # Allow any NMS to TFTP to & from the device
!
access-list 98 permit 172.30.1.101 # e.g. NMS Server 1
Figure 172 Sample NMS Configuration

Network Management & Infrastructure Security Best Practices
SNMP Trap Settings

SNMP Traps allow a device to send pro-active notifications of issue via SNMP to the NMS; instead of
waiting to be polled for the date. As a bare minimum traps should be used to track major events on Cisco
routers & switches:
• Interface state changes
• Cold and warm starts
There are many other traps, which can be enabled in IOS. Their use should be carefully considered as the
intention should not be to overwhelm the NOC staff with too much information; rather provide them with
useful information. Following is an example list of traps that can be enabled.
bgp Enable BGP traps

bulkstat Enable Data-Collection-MIB Collection notifications
entity Enable SNMP entity traps
frame-relay Enable SNMP frame-relay traps
ipmulticast Enable SNMP ipmulticast traps
l2tun Enable SNMP L2 tunnel protocol traps
mpls Enable SNMP MPLS traps
msdp Enable SNMP MSDP traps
mvpn Enable Multicast Virtual Private Networks traps
ospf Enable OSPF traps
pim Enable SNMP PIM traps
pw Enable SNMP PW traps
rsvp Enable RSVP flow change traps
rtr Enable SNMP Response Time Reporter traps
snmp Enable SNMP traps
tty Enable TCP connection traps
Figure 173 Sample Trap listing
• config – notification of configuration changes. Requires the CISCO-CONFIG-MAN-MIB to be

complied on the NMS.
• envmon – notifications for temperature, fan & voltage abnormalities. Requires the CISCO-
ENVMON-MIB.
• bgp – provides notification of up/down events for BGP. Requires BGP4-MIB.
• mpls ldp – notification of LDP up/down events & abnormalities. Requires the MPLS-LDP-MIB.
• mpls vpn – notification of VRF up/down events & of exceeding label thresholds for each VPN.
Requires the MPLS-VPN-MIB. The maximum number of routes per vrf can be set in IOS and traps
sent when this threshold is exceeded.
The following commands will be required to configure SNMP trap notification on Cisco IOS devices.

snmp-server trap-authentication
snmp-server host <nms-server ip address> XXXX #Send SNMP traps to NMS server
snmp-server trap-source loopback 0 #Source all SNMP trap messages from
loopback interface
snmp-server enable traps snmp # provide linkUp, linkDown, coldstart &
warmstart traps
snmp-server enable traps config
snmp-server enable traps envmon
snmp-server enable traps bgp
snmp-server enable traps mpls ldp
snmp-server enable traps mpls vpn
Figure 174 Enabling Traps on a Router
Cisco Discovery Protocol

CDP is a media- and protocol-independent protocol that runs on all Cisco-manufactured equipment
including routers, bridges, access and communication servers, and switches. Using CDP, you can view
information about all the Cisco devices directly attached to the switch. In addition, CDP detects native
VLAN and port duplex mismatches.
Network management applications can retrieve the device type and SNMP-agent address of neighboring
Cisco devices using CDP. This allows applications to send SNMP queries to neighboring devices. CDP
allows network management applications to discover Cisco devices that are neighbors of already known
devices, in particular, neighbors running lower-layer, transparent protocols.
CDP runs on all media that support Subnetwork Access Protocol (SNAP). CDP runs over the data link
layer only. CDP does produce a small amount of overhead on the network as it uses multicast data-link
packets to exchange information between devices. CDP uses the destination MAC address of 01-00-0c-
cc-cc-cc. Cisco devices do not forward CDP packets. CDP is enabled by default on all Cisco devices and
on all interfaces except for ATM and other Non-Broadcast Multi-Access (NBMA) media, where CDP is
not supported due to lack of multicast capabilities. When new CDP information is received, old
information is discarded. The following table lists the default configuration for CDP.
Table 38 Default CDP Configuration
Feature Default Value

CDP global enable state Enabled
CDP port enable state Enabled on all ports
CDP message interval 60 seconds

CDP holdtime 180 seconds
CDP is a valuable tool for use when troubleshooting network issues. CDP output can be used to quickly
construct a visual depiction of the network as CDP neighbors can be traced from one to the next through
the use of the “show CDP neighbor” command.
From network security perspective:

• Disable CDP Server where not used - Typically CDP is used on internal networks for network
management, but can pose a risk at the edge of the network.
• From a security perspective, CDP should be disabled where it is not being used, in order to avoid
unnecessarily advertising system specific information. The following guidelines should be followed
when implementing CDP:
o CDP can be disabled on a per interface basis. Run it only where absolutely necessary,
disable it on a per interface basis everywhere else.
o Confine CDP deployment to run between devices under your control that are directly
adjacent. Since CDP is a link-level protocol, it is not transient across a network. Limit it to
run only between trusted devices.
o Disable CDP everywhere else, especially on the interfaces at the edge of the network.
To disable CDP globally on an IOS device use the command:
“no cdp run”
To disable CDP on an IOS device interface use the following interface command:
“no cdp enable”
Most service providers choose to disable CDP to provide extra security, so for Wateen Telecom the
recommendation is to leave CDP enabled during the initial installation and then to disable it
subsequently.
Logging
Logging of events on a network is an important part of any network security policy. The logging of
events assists problem troubleshooting and security investigations.
• Message Logging - Cisco routers can log information regarding configuration changes, ACL
violations, interface status changes and many other types of events. Logging messages can be
directed to several different locations:
o Console – used when modifying or testing the device while connected to the console.
Messages sent to the console are not stored by the device, and are therefore not very
valuable as security events.
o Terminal lines – enabled EXEC sessions can be configured to receive log messages on any
terminal lines. Like console logging, this type of logging is not stored by the device and
therefore is only valuable to the user on that line at that particular time.
o Memory buffer – the device can be directed to store log messages in the local memory.
Buffered memory is more useful as a security tool than console and terminal messages, but
has the disadvantage that the events are cleared out if the device is reloaded.
o SNMP traps – certain router events may be processed by the router SNMP agent and
forwarded as SNMP traps to an external SNMP host. This is a viable security logging
facility, but requires the configuration and maintenance of an SNMP system.
o Syslog – Cisco devices can be configured to forward log messages to an external Syslog
service. It is highly recommended that networks implement a logging structure based on a
Syslog infrastructure. This is a scalable solution, which provides long-term storage
capabilities and a central location for all device messages.
N.B. performing analysis on a device’s logs and messages can be difficult if the device clocks are all
running on different times. It is recommended that the NTP (Network Time Protocol) be utilised to ensure
that all devices are operating with the same time information. See later section on securing NTP.

• Log Severity Levels - When logging, it is important to capture the necessary amount of information.
The granularity of detail in logging information can also be configured to one of eight levels, as
shown below:
Table 39 Security Levels
Level Name Description

0 Emergencies Router unusable
1 Alerts Immediate action required
2 Critical Condition is critical
3 Errors Error condition
4 Warnings Warning condition
5 Notifications Normal, but important event
6 Informational Informational message
7 Debugging Debug message
The lower the level number, the higher the security level. A good level of general logging for everyday
information capture is “informational”. Additional detail can be captured with the “debug” level, but
should only be used temporarily for specific situations. Debug logging levels can become extremely
processor intensive thereby impacting system performance.
• Securing Syslog Messages - There are several security issues related to Syslog messages as detailed
below:
o Syslog is sent as clear text between the managed device and the management host (UDP
port 514).
o Syslog has no packet-level integrity checking to ensure that the packet contents have not
been altered in transit.
o There is a potential for the Syslog data to be falsified by an attacker.
o An attacker could send large amounts of false Syslog data to a management server in order
to confuse the network administrator during an attack.
When possible, the following security practices should be implemented to mitigate these issues:
o If possible Syslog traffic should be encrypted within an IPSec tunnel in order to mitigate the
chance of it being altered in transit.
o If there are no managed devices outside of the network that send Syslog messages to the
Syslog server, then ACLs should be used at the edge of the network to block external Syslog
messages from entering the network.
o If there are managed devices outside of the network, ACLs should be implemented at the
edge of the network to allow only Syslog data from the managed devices themselves to
reach the Syslog server.
The following are best practices for logging on IOS devices:

• Set Syslog Server - Cisco devices can send their log messages to a Syslog service. A Syslog service
accepts messages and stores them in a file. This form of logging is highly recommended, as it
provides protected, long-term storage for logs. The following command enables logging:
“logging <ip address of syslog server >”
• Enable Timestamp Service - Timestamps should be enabled for log messages, which will facilitate
interpretation of the messages for troubleshooting and investigating network attacks. This is enabled
using the following command:

“service timestamps log datetime show-timezone”

Please insure that the date/time is correctly set (if NTP is not configured) on the router so that the
timestamps provide the proper day/time of the log messages.
• Enable Buffered Logging - Cisco devices can store log messages in memory. The buffered data is
available only from an exec or enabled exec session, and it is cleared when the device reboots. This
form of logging is useful, even though it does not offer enough long-term protection for the logs.
Command:
“logging buffered”
• Disable Logging to the Console - Limit the severity level of messages sent to the console or disable
logging to the console. This form of logging is not persistent; the device does not store messages
printed to the console. Although useful for troubleshooting from the console port, it is possible that
excessive log messages on the console could make it impossible to manage the device, even from the
console. Console logging can be disabled using the following command:
“no logging console”
Console logging may later be enabled when required, such as during a debugging or trouble-shooting
session, or if detailed tracking is being implemented via hard logging from the console directly to a
printer.
• Set trap level – Determine the severity of messages that will generate a trap. The following
command sets the trap level:
“logging trap <level>”
• Set loopback as source of logging – If the loopback interface is not available, for example in Metro
Ethernet solutions, then a suitable interface should be used, such as the management VLAN.
“logging source-interface Loopback0”
The following example shows how to enable syslogging to a server:
no logging console
no logging monitor
!
!
service time log datetime localtime show-timezone msec
service time debug datetime localtime show-timezone msec
logging buffered 65536 # Set buffer for logs to 64kB

logging trap notifications # Send logs for level 5 and above messages
(default)
logging facility local7 # Set logging facility on remote NMS (default)
logging host <loggin server1> # Send syslog to primary Syslog server
logging host <logging server2> # Send syslog to secondary Syslog server
logging source-interface loopback 0 # Source messages from loopback
Figure 175 Log Configuration on a Router
The configuration above sends each Syslog message to two different servers for analysis. It is preferable
to send all messages to a single server to save bandwidth, and then allow other NMS platforms access by
doing an NFS mount to the syslog directory.

ICMP unreachable generation

By default, on 7600 series router the MSFC sends Internet Control Message Protocol (ICMP) unreachable
messages when a packet is denied by an access group.
With the ip unreachable command enabled (which is the default), the supervisor engine drops most of the
denied packets in hardware and sends only a small number of packets to the MSFC to be dropped (10
packets per second, maximum), which generates ICMP-unreachable messages.
To eliminate the load imposed on the MSFC CPU by the task of dropping denied packets and generating
ICMP-unreachable messages, you can enter the “no ip unreachable interface” configuration command
to disable ICMP unreachable messages, which allows all access group-denied packets to be dropped in
hardware.
From a security best practice point of view it is recommended to turn this off, since we do not want to tell
a potential attacker about the fact that an ACL did stop his traffic.
Cisco recommends as a best practice to always configure no ip unreachable on an interface that has an
ACL applied, to ensure that someone sending traffic that is denied by an ACL does not get a notification
regarding this.
Disable Unneeded Services

One of the basic rules of securing devices is to provide only those services that the network requires and
no more. Leaving unused network services enabled increases the possibility of those services being used
maliciously.
By default, Cisco routers support multiple TCP and UDP services to facilitate management and
integration into existing environments. For most installations these services are typically not required and
disabling or restricting access to them can greatly reduce overall security exposure.
Before any of these services are disabled, they should be reviewed in the context of the network to ensure
that they will not effect the current operation of the network.
The following services should be disabled if they are not being used:
• Disable Small Servers - Small services, such as echo, chargen, daytime and discard are enabled by
default on a router in IOS version < 11.3, and disabled by default in IOS version > 11.3. These
services, especially their UDP versions, can be used to launch DoS and other attacks that would
otherwise be prevented using packet filtering. As these services are rarely used they should be
disabled, especially on routers acting as firewalls and those in security-critical parts of the network.
These services are disabled by default in IOS version 12.0 and later; in earlier software can be
disabled using the following global commands:
“no service tcp-small-servers”
“no service udp-small-servers”
• Disable Finger Service - Cisco routers provide an implementation of the “finger” service, which is
used to discover the users that are logged into a device. The information it provides could be useful
to an attacker. The service is enabled by default, and if not required should be disabled using the
following commands:
“no ip finger”
“no service finger”

• Disable Configuration Auto-loading Service - Routers can be configured to load their startup config
from a network server as well as their local memory. Loading router configurations across a network
can be dangerous and should only be used in fully trusted network situations. The service is disabled
by default, but if enabled can be disabled using following commands:
no boot network <remote-url>
“no service config”
• Disable Bootp Server - Bootp is a UDP based service that allows Cisco routers to obtain copies of
IOS from other routers also running the bootp service. In reality, the service is rarely used and could
be used by an attacker to download a copy of a router’s IOS. The service is enabled by default, and
can be disabled using the following command:
“no ip bootp server”
• Disable IP Source Routing - IP source routing enables the sender of a datagram to specify the route
the datagram will take on route to its destination, and generally the route that any reply will take
when returning to the originator. Although enabled by default, source routing is rarely used and
could be used by an attacker. If a network does not require source routing, it should be disabled using
the following command:
“no ip source-route”
This will cause the router to drop any IP packet that carries a source route option.
• Disable TFTP Server - It is recommended to have a central storage location for IOS images and
restrict access to only the loopback ip address of the routers and switches on the network. It is
therefore recommended to disable the local tftp servers running on the network and to put all the
images needed for the operation of the network on a dedicated server inside the NOC with strict
ACLs that only allow tftp traffic inside, if sourced from the loopback ip addresses of the routers or
switches on the network. The following command will disable the tftp-server:
“no tftp-server”
The following interface services should be disabled if they are not being used:
• Disable IP Directed Broadcasts - An IP directed broadcast is a datagram sent to the broadcast address
of a subnet to which the sending device is not directly attached. The directed broadcast is routed
through the network as a unicast packet until it arrives at the target subnet, where it is converted into
a data-link-layer broadcast.
Directed broadcasts are rarely used for legitimate purposes, and are utilised in the “SMURF” and
other related attacks. The service is enabled in IOS versions < 12.0 and disabled in IOS versions ≥
12.0. The following interface command will cause directed broadcasts to be dropped instead turned
into data-link-layer broadcasts:
“no ip directed-broadcast”
As only the last router in the chain of routers that the packet passes through can recognise that the
packet is a directed-broadcast, the above command must be configured on every interface that could
be a target. It is not sufficient to configure only the edge routers or firewalls.
• Disable IP Redirects - ICMP redirect messages are enabled by default, and instruct an end device to
use a specific router in its path to a destination. By design, a router will send redirects only to hosts
on its local subnet, no end device will ever send a redirect, and no redirect will be sent more than one
network hop away. However, an attacker may violate these rules to launch an attack on a network. If
not required, ICMP redirect messages can be disabled using the following interface command:
“no ip redirects”
It is also possible to filter out ICMP redirect messages using ACLs on routers located at the edge of
the network. This will prevent redirect attacks launched by remote attackers.

• Disable IP Unreachable Messages - Attackers use ICMP unreachable messages to try to map a
network. These messages are enabled by default, and should be disabled on all interfaces if not
required, especially those connected to untrusted networks: Use the following interface command to
disable unreachable messages on an interface:
“no ip unreachable”
• Disable IP Mask Replies - Mask replies are disabled by default, but if enabled, the router responds to
ICMP mask requests with ICMP mask replies, which could provide an attacker with important
network information. Mask replies should be disabled on all interfaces, especially those at the edge
of a network, using the following command:
“no ip mask-reply”
• Disable Unused Router Interfaces - If an attacker has physical access to a router, any un-used
interface could be used to gain access to the router and the network. All un-used interfaces on a
router should be disabled in the configuration using the shutdown command.
Enable Protection Services
Cisco routers and switches support a number of services that can be enabled on a device to improve the
overall security of a network. Before any of these services are enabled, they should be reviewed in the
context of the network to ensure that they will not effect the current operation of the network.
• Enable TCP Keepalive Feature – Idle logged-in user sessions can be susceptible to unauthorized
access and hijacking attacks. By default, Cisco routers do not continually test whether a previously
connected TCP endpoint is still reachable. If one end of a TCP connection idles out or terminates
abnormally (for example, crashes or reloads), the opposite end of the connection may still believe the
session is available. These “orphaned” sessions use up valuable router resources. Attackers can take
advantage of this weakness to attack devices.
To mitigate this problem, Cisco routers can be configured to send periodic keepalive messages to
ensure that the remote end of a session is still available. If the remote device fails to respond to the
keepalive message, the sending router will clear the connection. This immediately frees router
resources for other more important tasks.
The following IOS commands enable the tcp-keepalive feature:
service tcp-keepalives-in
service tcp-keepalives-out
Figure 176 TCP Connection Tracking
• Core Dumps – The core dump facility allows a copy of a device’s memory image to be stored and
uploaded to another device after a crash. When a router crashes, a copy of the core memory is kept.
Before the memory is erased on reboot, the router can be set up to copy the core dump to a UNIX
server. An account (FTP, TFTP, or RCP) and sufficient disk space (equal to the amount of memory
on the router per dump) needs to be set up and allocated. The following example shows how FTP is
configured to copy the dump to a server:

ip ftp source-interface Loopback0

ip ftp username <username>
ip ftp password <password>
!
exception protocol ftp
exception dump <ipAddr>
Figure 177 Core Dump Configuration
Note the use of the loopback interface as a source interface and use this address in any system-filter
lists. It is recommended that access to the ‘username’ account above be made as secure as possible.
For example, do not send core dumps to the same FTP server as the one used to provide generic
anonymous or user FTP accounts.
Note that RCP (Remote Copy Protocol) is inherently insecure and is not recommended for use over a
public network. Also TFTP core dumps (which are the default in IOS) only support system memory
sizes up to 16 Megabytes. Generally, it is recommended that FTP core dumps be configured whatever
the situation or router hardware configuration.
Core Dumps must only be configured while troubleshooting a problem and requested by Cisco
TAC.
Enable Port Security on switches - Use the port security feature to block input to an Ethernet port
when from any of the MAC addresses specified for that port.
IOS switch command:
“switchport port-security”
Preventing Unauthorized Access

• Enable Secret - The enable secret command is used to set the password that grants enable access to
the IOS system. The enable secret command replaces the older enable password command as the
standard method of configuring the system’s privileged mode password. The new method employs a
more secure method of encryption than the older command.
All systems should be configured with an enable secret password. Additionally, since the enable
secret command simply implements an MD5 hash on the configured password, strong passwords
should be chosen to prevent dictionary attacks.
“enable secret <password>”
It is also possible to use same encryption with user passwords using the following command:
“username test secret <password>”
• Service password-encryption - With the exception of the enable secret password, all IOS device
passwords are by default stored in clear text form within the device configuration. These passwords
can be clearly seen by performing a show running-config command.
The service password-encryption command directs the IOS software to encrypt the passwords, CHAP
secrets, and similar data that are saved in its configuration file. This is useful for preventing casual
observers from reading passwords, for example, when they happen to look at the screen over an
administrator's shoulder.
The algorithm used by service password-encryption is considered relatively weak, and was not
designed to protect configuration files against serious analysis, and should not be used for this

purpose. Any Cisco configuration file that contains encrypted passwords should be treated with the
same care used for a clear text list of those same passwords. It is important to use a protected link,
when accessing the router config using the console, or when TFTPing config files. Command:
service password-encrypt
Controlling Access to the Device

• Secure Access to the Console and Aux Port - Console access requires a minimum level of security
both physically and logically. An individual that gains console access to a system will gain the
ability to recover or reset the system enable password, giving them the ability to bypass all other
security implemented on that system. Consequently it is imperative to secure access to the console.
To secure the console from access logically, a line password for the console and any available
auxiliary ports should be configured for each line using the following commands.
login con 0
login
password <password>
login aux 0
login
password <password>
Alternatively, the console lines can be protected using AAA authentication
login authentication <AAAlistName>
Figure 178 securing console & aux ports
The console may also be secured physically, by placing the router within a lockable enclosure.
If the aux port is not being used, it can be disabled using the following command:
“no exec”
As well as authenticating users accessing the network devices using AAA, access to the virtual terminal
(VTY) should also be locked down to certain addresses only. The suggestion is to restrict this to NMS
platforms as shown by the following template.
access-list 98 permit 172.30.1.101 # host1 e.g NMS Server 1

access-list 98 permit 172.30.1.102 # host2 e.g. NMS Server 2
access-list 98 permit 172.30.1.103 # host3 e.g. NMS Server 3
!
line vty 0 4
access-class 98 in
Figure 179 Limiting Access to VTYs on a Router
Wateen Telecom’s NMS Servers are given below:

Wateen NMS Server1 : <ip address>
Wateen NMS Sever2 : <ip address>
Wateen NMS Server3: <ip address>
• Use SSH for Device Access - The SSH server enables a Cisco device to receive a secure, encrypted
connection from another device running the SSH Version 1 client. This connection provides
functionality that is similar to a telnet connection except that the connection is encrypted, which
allows for secure communication over an insecure network. DES and 3DES encryption is supported.
The SSH server also supports user ID and password authentication using standard authentication
methods: local authentication, RADIUS, and TACACS+. SSH should be used in place of Telnet
wherever possible.
IOS commands:
line vty 0 4
transport input telnet ssh !permits both telnet and ssh, if possible ssh only
!should be used
ip domain-name
crypto key generate rsa
ip ssh time-out 60
ip ssh authentication-retries 3
It is understood that Wateen Telecom will like to use ssh as the only means of remote access into the
device in production (as long as ssh access is supported in IOS image). Hence above example will only
contain “transport input ssh”.
• Set Timeouts for Device Lines - By default, an administrative interface stays active (i.e. logged in)
for ten minutes after the last session activity. After that, the interface times out and logs out of the
session. It is recommended that these timers are fine-tuned to limit the amount of time that an
inactive session is left open. Command:
“exec-timeout x y”
Depending on Wateen Telecom network operations team, this value can be tuned to a lower value or
left at the default provided by Cisco IOS.
Login Banners
For both legal and administrative purposes, configuring a system-warning banner to display prior to login
is a convenient and effective way of reinforcing security and general usage policies. By clearly stating
the ownership, usage, access, and protection policies prior to a login, future potential prosecution
becomes a more solidly backed case.
Wateen Telecom is advised to configure a login banner on every device stating that access is restricted,
and with the NOC contact details. Of course, no one but authorized Wateen Telecom staff should be
accessing the devices but this is a common best practice.
banner login ^
Authorized access only
Unauthorized users will be prosecuted
This system is the property of Wateen Telecom Network

Contact noc@wateen.com.pk for any assistance
Figure 180 Login Banner Configuration
CLI Management Authentication & Logging (AAA)

It is strongly advised to utilize either TACACS+ or RADIUS to authenticate users accessing network
equipment. In addition, accounting may be used to keep a record of each command entered by a user.
It is understood that Wateen Telecom has decided not to use Cisco Secure ACS server, key component
required for implementing AAA policy.

HTTP Server
Cisco IOS has an inbuilt HTTP service to allow management from a GUI front-end. Most people manage
their routers & switches using the CLI, most service providers keep this service shutdown as part of
security practice. To confirm that service is indeed turned off use the following command in global config
mode.
“no ip http server”
NTP
Network Time Protocol (NTP) is used to synchronise the clocks of various devices across a network. The
local NTP client, which runs on the device, accepts time information from other remote time-servers and
adjusts its clock accordingly. Synchronisation of the clocks within a network is critical for the correct
interpretation of events within Syslog data. The following should be considered when deploying NTP:
• Define a trusted time source and configure all devices as part of an NTP hierarchy.
• Use proper authentication of NTP messages. Version 3 and above of NTP supports a cryptographic
authentication mechanism between peers.
• ACLs should be used to specify which network devices are allowed to synchronise with which other
network devices.
• Disable NTP messages on interfaces that are connected to untrusted interfaces (ntp disable
command). If NTP messages have been disabled on an interface of a router, the router could still
receive and process NTP messages on another interface. ACLs could also be used to drop NTP
messages.
IOS commands to enable and authenticate NTP:
ntp authentication-key 10 md5 <key>

ntp authenticate
ntp trusted-key 10
ntp source loopback 0
ntp server x.x.x.x [key 10]
ntp access-group peer 20
access-list 20 permit host x.x.x.x
access-list 20 deny any
Figure 181 NTP Configuration
Wateen Telecom will use following NTP servers.

ntp server 1: ip address
ntp server 2: ip address
Out-of-band Management

It is highly recommended that the management for all devices in the network should be performed via
console access to the devices and through a dedicated out–of-band management network. This separates
the management traffic plane from the data traffic plane on the network.
Having direct access to the console ports of the managed devices is a good practice. It allows the
operator to manage and maitain the device freely without the fear of being disconnected or loosing remote
access to the device when connecting via the in-band network.
At each remote PoP a router that acts as a terminal server should be deployed that is reachable via a
separate OOB network. For further protection, or if a permanent OOB network is not feasible, ISDN or
POTS line could be used to access the remote sites via an ISDN or a modem module in the terminal
server to provide a temporary OOB network.
It is understood that in current phase of deployment Wateen Telecom has decided not to deploy an out of
band management solution. All management of the network is expected to be inband.
ACL Log
On a 7600 series router if an access-list is configured with a log entry at the end of some lines, all traffic
matching those lines will be inspected and handled by the CPU instead of in hardware.
If there is a need to follow up what has been denied on an ACL in a border router, it is advisable not to
configure log on the ACL, but instead to poll Netflow from the router and look for entries with 0 byte.
mls rate-limiters
The 7600 architecture includes a number of rate-limiters for specific traffic that can impact the
performance of the router. These “mls rate-limiters” are processed in hardware.
Of all the rate-limiters, the following ones are recommended to enabled (in addition to the ones enabled
by default):
mls rate-limit all ttl-failure 100 10

mls rate-limit multicast ipv4 ip options 100 10
mls rate-limit unicast options 100 10
mls rate-limit unicast ip rpf-failure 100 10
Figure 182 Enabling Supervisor mls rate-limiters
Following command can be used to confirm status of mls rate-limiters:

“show mls rate-limit”
Control Plane Policing

One important step in securing a network is to limit the ability for non authorized sources to send traffic
towards the router itself, since traffic to the router normally will be sent to the CPU. This can result in the
services being affected on the router if there is a shared control-plane to the data-plane. The 7600
architecture includes complete separation between control plane and data plane. This separation has been
used in a feature called CoPP [control plane policing]. This feature gives the possibility to apply a policy-
map on the control plane. This policy-map looks like a normal QoS policy. When a policy-map is applied
to the control-plane, it will be applied to all traffic that is destined to any of the IP addresses of the router.

Guidelines
In CoPP deployment guidelines customer is recommended to explicitly identify every type of traffic.
There are three CoPP config strategies:
1. Explicitly identify all unwanted traffic, and let all others goes through the class-default (configured
with a policer to safeguard to CPU).
2. Explicitly identify all useful traffic, and harshly police the rest of traffic.
3. Combine 1 and 2, but still need class-default (configured with police) to catch the rest.
For service provider networks Cisco recommends strategy 3 since the combination of 1 & 2 is clearly the
better way to go. User already knows most/all of the useful traffic, so allow it. User already knows
previous attack traffic, so block it completely, and then allow a trickle for anything else, either
unidentified useful traffic, or un-identified attack traffic.
Having the different classes is useful to, as you may not want BGP and SNMP to have the same rates. By
identifying the useful traffic and adjusting the policed BW, you are creating a QoS profile for the control
traffic, where the really important stuff has more access to the RP.
Classes
Critical
Traffic that is crucial to the operation of the router and the network like routing protocols like BGP, ISIS,
LDP etc.
Important
Necessary and frequently used traffic that is required during day-to-day operation, like SSH, Telnet, NTP,
SNMP, igmp etc
Normal
Traffic that is expected but not essential to network operation, like ICMP echo, ping, etc
Bad
Explicitly identify bad or malicious traffic that should be dropped and denied access to the RP.
Match-all
All the remaining IP traffic destined to the RP that has not been identified.
Class-default
This is the built in class that will work as a last resort. The reason for having the Default class above as
“match all IP traffic, we will ensure that all “non IP protocols” ending up in this pre-defined class .
Identifying Undesirable Traffic

If BAD traffic cannot be identified initially the Match-all class at the end of the policy-map will catch all
non explicitly identified traffic to RP and apply the policy that will allow a small rate of traffic towards
the RP. As soon as BAD traffic is identified, it should be added to the “BAD TRAFFIC ACL” so the
BAD policy of drop everything can be applied.

Rate
Rate limiting punted traffic is recommended, care must be taken to ensure that the required rate of traffic
are well understood. The proposed configuration will not implement any rate-limiting functionality that
should have a negative impact on the convergence of routing protocols.
Requirements / dependencies for CoPP

1. "mls qos" needs to be globally enabled.
2. No CoPP support for multicast packets. However multicast packets are handled by the existing CPU
rate limiters.
3. Hardware based CoPP use QOS ACLS. QOS ACLs are complied and stored in dedicated QOS ACL
TCAM. Sometimes due to large QOS config, the system can possibly run out of TCAM space. If TCAM
capacity is exceeded, then CoPP will be done in software.
4. ARP polices are not supported by CoPP. To protect the system by ARP broadcast a useful tool is
"mls qos arp police <bps>"
5. When enabling the special case rate limiters, it must be considered that they will override the CoPP. It
is strongly recommended to disable the "CEF receive" rate limiter. Command for this is "no mls rate-
limit unicast cef receive"
6. When DFCs are installed in the system, each DFC card in addition to the PFC will send the HW
policed rates, thus if CoPP default traffic is policed to 20Mbps, and there are 2 DFCs, the SW CoPP
could receive up to 3 * 20Mbps of CoPP default traffic, that would be policed down to 20Mbps by the
SW based CoPP.
Interaction with the built-in special case rate limiters

The 7600 series router (Sup720 and Sup32) still supports the built-in “special case” rate limiters, useful
for those cases where an ACL cannot be used to classify particular scenarios, such like IP Options cases,
TTL and MTU failure cases, packets with errors and Multicast packets. When enabling the special case
rate limiters, it must be considered that they will override the CoPP policy for packets matching the rate
limiters criteria. When using CoPP, it is strongly recommended to disable the “CEF receive” rate limiter.
You can use the “no mls rate-limit unicast cef receive” command.
CoPP Configuration Template

Sample CoPP configuration is given below. Please note it should be customized according Wateen
Telecom final ip addressing and to cover all applications.

! CoPP configuration
!
mls qos
!
class-map match-all CoPP-CRITICAL
match access-group 120
class-map match-all CoPP-IMPORTANT
class-map match-all CoPP-NORMAL
class-map match-all CoPP-BAD
class-map match-all CoPP-Match-all
!
access-list 120 remark *** ACL for CoPP-Critical ***
access-list 120 remark *** LDP ***
access-list 120 permit udp 10.0.100.0 0.0.0.255 eq 646 any
access-list 120 permit udp 10.0.100.0 0.0.0.255 any eq 646
access-list 120 permit tcp 10.0.100.0 0.0.0.255 eq 646 any
access-list 120 permit tcp 10.0.100.0 0.0.0.255 any eq 646
!
access-list 120 remark *** OSPF ***
access-list 120 permit ospf 10.0.100.0 0.0.0.255 any
access-list 120 remark *** BGP ***

access-list 120 permit tcp 10.0.100.0 0.0.0.255 eq bgp any
access-list 120 permit tcp 10.0.100.0 0.0.0.255 any eq bgp
!
Access-list 121 remark *** ACL for CoPP-IMPORTANT
access-list 121 permit tcp [management subnet] any eq telnet
access-list 121 permit tcp 10.0.100.0 0.0.0.255 any eq telnet
access-list 121 permit tcp [management subnet] any eq 22

access-list 121 permit tcp 10.0.100.0 0.0.0.255 any eq 22
access-list 121 permit udp [management subnet] any eq snmp

access-list 121 permit udp 10.0.100.0 0.0.0.255 any eq ntp
access-list 121 permit udp [management subnet] any eq 1645

access-list 121 permit udp [management subnet] any eq tftp

access-list 121 permit udp [management subnet] eq tftp any

access-list 121 permit tcp [management subnet] eq 20 any
access-list 121 permit tcp [management subnet] any eq 21
access-list 121 permit tcp [management subnet] eq 21 any
access-list 121 permit tcp 10.0.100.0 0.0.0.255 any established

access-list 121 permit tcp [management subnet] any established
Access-list 122 remark *** ACL for CoPP-NORMAL

access-list 122 permit icmp any any ttl-exceeded
access-list 122 permit icmp any any port-unreachable
access-list 122 permit icmp any any echo-reply
access-list 122 permit icmp any any echo
Access-list 123 remark *** ACL for CoPP-BAD

/* add permit lines here for "known bad things */
access-list 124 remark *** ACL for CoPP-Match-all


policy-map CoPP
class CoPP-CRITICAL
police 1000000 31250 31250 conform-action transmit exceed-action
transmit
class CoPP-IMPORTANT
class CoPP-NORMAL
class CoPP-BAD
police 32000 1500 1500 conform-action drop exceed-action drop
class CoPP-Match-all
! class class-default
! police 1000000 31250 31250 conform-action transmit exceed-action
transmit
! Note until 12.2.18SXE it is not supported to configure class-default with a
! police action. In the show commands the class default will appear since
this
! class is always there.
!
control-plane
srvice-policy in CoPP
!
Figure 183 CoPP Configuration
*Note the Above ACL’s need to be changed to reflect the addressing in the Wateen Telecom network.

Conclusion
Design of IP/MPLS network for Wateen Telecom as it pertains to L3 vpns, L2vpns, internet and ip
connectivity is discussed in this document. Possible addressing scheme, IGP and BGP configuration
along with possible service offering models, are discussed in detail.
This document is part of a larger low level design effort; other documents that discuss, metro Ethernet
architecture, network security, data center architecture and network management are compiled separately.

Glossary
TBD To be delivered
Please refer to the CCO Internetworking Terms and Acronyms Guide at
http://www.cisco.com/univercd/cc/td/doc/cisintwk/ita/index.htm for additional terms.

Document Acceptance
Name Name
Title Title
Company Company
Signature Signature
Date Date
Name Name
Title Title
Company Company
Signature Signature
Date Date
Name Name
Title Title
Company Company
Signature Signature
Date Date

Corporate Headquarters European Headquarters Americas Headquarters Asia Pacific Headquarters
Cisco Systems, Inc. Cisco Systems Europe Cisco Systems, Inc. Cisco Systems Australia, Pty., Ltd
170 West Tasman Drive 11 Rue Camille Desmoulins 170 West Tasman Drive Level 9, 80 Pacific Highway
San Jose, CA 95134-1706 92782 Issy-Les-Moulineaux San Jose, CA 95134-1706 P.O. Box 469
USA Cedex 9 USA North Sydney
www.cisco.com France www.cisco.com NSW 2060 Australia
Tel: 408 526-4000 www-europe.cisco.com Tel: 408 526-7660 www.cisco.com
800 553-NETS (6387) Tel: 33 1 58 04 60 00 Fax: 408 527-0883 Tel: +61 2 8448 7100
Fax: 408 526-4100 Fax: 33 1 58 04 61 00 Fax: +61 2 9957 4350
Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on the
Cisco Web site at www.cisco.com/go/offices.
Argentina • Australia • Austria • Belgium • Brazil • Bulgaria • Canada • Chile • China • Colombia • Costa Rica • Croatia • Czech Republic Denmark • Dubai, UAE
Finland • France • Germany • Greece • Hong Kong SAR • Hungary • India • Indonesia • Ireland • Israel • Italy • Japan • Korea • Luxembourg • Malaysia • Mexico
The Netherlands • New Zealand • Norway • Peru • Philippines • Poland • Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Singapore • Slovakia • Slovenia
South Africa • Spain • Sweden • Switzerland • Taiwan • Thailand • Turkey • Ukraine • United Kingdom • United States • Venezuela • Vietnam • Zimbabwe


IP - MPLS Design - v1.5

Uploaded by

Copyright:

Available Formats

IP - MPLS Design - v1.5

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IP - MPLS Design - v1.5

Uploaded by

Copyright:

Available Formats

Cisco Systems Advanced Services

Version 1.5 DRAFT

Turn the television or radio antenna until the interference stops.

Move the equipment farther away from the television or radio.

INTELLECTUAL PROPERTY RIGHTS:

Converged Multi-Service Network - IP/MPLS Low Level Design

About This Low Level Design .................................................................................................... 21

About This Design Document.................................................................................................... 23

Document Purpose ................................................................................................................ 23

Network Overview ....................................................................................................................... 27

Network Layout: ..................................................................................................................... 28

Physical Network Design ........................................................................................................... 31

Physical Network Topology .................................................................................................. 31

October 2, 2006 Wateen Telecom 3

Logical Network Design ............................................................................................................. 54

Naming and Addressing Specifications .............................................................................. 54

MPLS Core Design ...................................................................................................................... 61

MPLS Architecture Overview................................................................................................ 61

October 2, 2006 Wateen Telecom 4

MPLS VPN Design..................................................................................................................... 111

Definition of a VPN............................................................................................................... 111

MPLS Layer 2 VPN Design ....................................................................................................... 141

AToM overview..................................................................................................................... 141

October 2, 2006 Wateen Telecom 5

Dial Services .............................................................................................................................. 154

Technology Primers ............................................................................................................ 154

October 2, 2006 Wateen Telecom 6

Internet Services ....................................................................................................................... 186

Internet Peering & Transit Service ..................................................................................... 186

October 2, 2006 Wateen Telecom 7

Quality of Service...................................................................................................................... 206

Quality of Service Overview................................................................................................ 206

October 2, 2006 Wateen Telecom 8

Traffic Engineering ................................................................................................................... 263

Traffic Engineer Disclaimer ................................................................................................ 263

Network Management & Infrastructure Security Best Practices.......................................... 276

Overview ............................................................................................................................... 276

October 2, 2006 Wateen Telecom 9

Glossary ..................................................................................................................................... 296

Document Acceptance ............................................................................................................. 297

October 2, 2006 Wateen Telecom 10

Figure 1 Network Topology Overview 28

Figure 2 Lahore topology Overview 29

Figure 3 Fiber Route 31

Figure 4 SDH Network Topology Overview 32

Figure 5 12816 Router 36

Figure 6 12816 Router Chassis/Slots 37

Figure 7 DC Power Shelf 38

Figure 8 PRP-2, Front Panel View 40

Figure 9 PRP-2 (Horizontal Orientation) 42

Figure 10 Alphanumeric Message Displays—Partial Front Panel 43

Figure 11 7609 Router 44

Figure 12 7613 Router 47

Figure 13 - Slot Allocation 48

Figure 14 Sup 720 Slot Allocation 48

Figure 15 Cisco 7609 Router Internal Airflow 50

Figure 16 Supervisor Engine 720 Front Panel Features 51

Figure 17 Supervisor Engine 32 Front Panel 51

October 2, 2006 Wateen Telecom 11