EIGRP Over The Top Routing (OTP)
EIGRP Over The Top Routing (OTP)
EIGRP Over The Top Routing (OTP)
Joe Harris
Consulting Systems Engineer
Agenda
• Overview of Current Solutions
• How OTP works
• Peering over the WAN
• Considerations
• Case Study
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Overview of Current WAN Solution
PE-CE Overview
MPLS VPN
PE1 Cloud PE2
CE1 CE2
Backdoor Link
Site 2
Site 1
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Overview of Current WAN Solution
PE-CE Issues for the Service Provider
• Service Provider must redistribute and carry Enterprise routes via MP-iBGP;
– Route flaps within sites results in BGP convergence events
– Route metric changes results in new extended communities flooded into the core
• Either EIGRP or eBGP must be run between the PE/CE
– Provider had to have trained staff on hand to manage
PE/CE Link
MPLS VPN
– Provider’s often prefer vender flexibility PE1 Core PE2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Overview of Current WAN Solution
PE-CE Issues for the Enterprise
• Managed services is required, even if not needed
– Provider often limits number of routes being redistributed
• Enterprise and Service Provider must co-support deployment
– Control of traffic flow using multiple providers is problematic
– Changing providers results in migration issues
• Service Provider route propagation impacts MPLS VPN
site to site convergence PE1 Core PE2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Overview of Current WAN Solution
PE-CE Issues with Backdoor Links
• Route redistribution adds deployment complications
– Without PE/CE support, back-door must be redistributed into a second instance of EIGRP
– With PE/CE support, use of SoO (route) tagging must be used to prevent count-to-infinity issues due
to BGP’s slower convergence and all routers between CE an Backdoor must have support for SoO
MPLS VPN
Cloud
PE1 PE2
CE1 CE2
CE2
CE2
CE1
Site 1 Site 2
C4
C3
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Problem Solution
= Control Plane
= Data Plane
Service Provider
MPLS VPN
Customer Site 1 Customer Site 2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
WAN Virtualization using OTP
Enterprise Benefits
• Single routing protocol solution
– Simple configuration and deployment for both IPv4 and IPv6
– Convergence is not depending on Service Provider
– Only the CE needs to be upgraded
• Routes are carried over the Service Provider’s network, not though it
– No artificial limitation on number of routes being exchanged between sites
– Convergence speed not impacted by BGP timers
• Works with both traditional managed and non-managed internet connections
– Compliments an L3 Any-to-Any architecture (optional hair pinning of traffic)
– Support for multiple MPLS VPN connections
– Support for connections not part of the MPLS VPN (“backdoor” links)
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
WAN Solution Overview
EIGRP OTP DMVPN / Internet MPLS VPN MPLS+DMVPN
Routing Policies EIGRP, EIGRP Stub EIGRP Stub Redistribution and route EIGRP Stub,
filtering Redistribution, filtering,
Multiple AS
Network Virtualization VRF/EVN to LISP multi- DMVPN VRF-Lite; MPLS o Multi-VRF CEs and Multi-VRF Ces and
tenancy DMVPN multiple IP VPNs DMVPN VRF-Lite
Convergence Branch Fast; Branch Fast; Branch / Hub carrier Carrier and DMVPN hub
Branch/Hub Hub – Fast Hub - Fast dependent dependent
Multicast Support Planned XE3.14 PIM Hub-n-Spoke PIM MVPN MVPN + DMVPN Hub-n-
Spoke
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
OTP – How it Works
• CE Routers have ‘private’ and ‘public’ interfaces & routers exchange information using unicast packets
– Private interfaces use addresses that are part of the Enterprise network
– Public interfaces use addresses that are part of the Service Providers network
– For OTP neighbors to form, the Public interface must also be included in the EIGRP topology
database (covered by the “network” command in IPv4)
• Packets are sourced from/to the public interface address eliminates the need for static routes
– EIGRP packets which are normally sent via multicast (Hello, Update, etc..) are sent unicast via the
public interface
– Site-to-site traffic is encapsulated using LISP and sent unicast from/to the public interface address
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
OTP – How it Works
EIGRP, LISP, and RIB – Oh My!
• EIGRP creates the LISP0 interface, and
starts sending Hello packets to remote site EIGRP
via the Public interface
Route
• Once neighborship is formed, EIGRP Updates
Default
sends and receives routes from the peer, Traffic
installing the routes into the RIB with the Inside
RIB
Public
Interface Interface
nexthop interface LISP0
Site to
• Traffic that arrives on the router destined Site
for the remote side, is first sent to LISP0 Traffic
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
OTP – Data Plane
LISP Data Encapsulation
• Why use LISP to encapsulate the data as it traverses the WAN?
• Its “stateless” tunneling, so it;
– Requires NO tunnels to configure or manage
– Is transparent to the endpoints and to the IP core
– Supports both hair-pin and site-to-site traffic
– Supports both IPv4 and IPv6 traffic
• Provides an overlay solution that enables transparent extension of network
across WAN
– IP-based for excellent transport independence
– Service provider picks optimal traffic path for site to site data
– Supports multicast and VLANs to allow for future enhancements
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
OTP – Data Plane
LISP Data Encapsulation Properties
• Path MTU needs to be considered when deploying OTP
– LISP encapsulation adds 36 bytes (20 IP + 8 UDP + 8 LISP) for IPv4
(56 bytes for IPv6)
– This could be significant for small packets (e.g., a VoIP packet)
• LISP handles packet fragmentation
– If the DF bit is set, it will generate an ICMP Destination Unreachable message
• LISP does not handle packet reassembly
– As a consequence, it is required to adjust the MTU to ensure the control plan does not
fragment
– Best practice - set the MTU is set to to 1444 (or lower) bytes.
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
OTP – Data Plane
LISP Header Format (IPv4 example)
LISP0 0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Internal Interface External Interface / |Version| IHL |Type of Service| Total Length |
/ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| | Identification |Flags| Fragment Offset |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
DATA LISPDATA OH | Time to Live | Protocol = 17 | Header Checksum |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
LISP encapsulation (36 bytes) : |
\
| Source Routing Locator
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|
• Route Reflectors
– Used to configure a router as a ‘hub’
– Forms a Hub and Spoke topology
– Configured with the command
remote-neighbors source [interface] unicast-listen lisp-encap
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Peering over the WAN
• Remote Routers
• Route Reflectors
• Redundant Remote Routers
• Redundant Route Reflectors
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Remote Routers
Point to Point Peers
• Control Plane peering is accomplished with EIGRP “neighbor” statement
– CE-1 sends unicast packets to CE-2’s public address (192.168.2.2)
– CE-2 sends unicast packets to CE-1’s public address (192.168.1.1)
Service Provider
DATA CE-1 LISPDATA
Hello Hello
DATA
CE-2
MPLS VPN
EIGRP EIGRP
AS 4453 AS 4453
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Remote Routers
Remote Peers address properties
• In order to form peers, the Public interface must be enabled for EIGRP
• For IPv4, you must include a ‘network’ statement to cover the public interface
• This does not mean the ip address of the remote peer has to match the network/mask of
the public interface interface Serial1/0
description Service Provider
• The interface is used to send packets, ip address 172.16.0.1 255.255.255.0
so the IP address of the remote peer !
router eigrp ROCKS
just has to be reachable via the WAN !
address-family ipv4 unicast auto 4453
!
topology base
exit-af-topology
neighbor 192.168.2.2 Serial1/0 remote 100 lisp-encap
network 172.16.0.0 0.0.0.255
network 10.1.0.0 0.0.255.255
exit-address-family
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Peering over the WAN
• Remote Routers
• Route Reflectors
• Redundant Remote Routers
• Redundant Route Reflectors
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
CSCuj68811:
15.4(1.16)S0.2, 15.4(1.16)S0.3
Route Reflectors 15.4(1.16)S0.4, 15.4(2.1)S
15.4(2.2)S
Point to Multi-Point – Multiple Branch Sites
• EIGRP Route-Reflectors simplifies setting up multiple branches
router eigrp ROCKS
• Chose one of the CE routers to function as address-family ipv4 unicast auto 4453
remote-neighbors source Serial 0/0 unicast-listen lisp-encap
Route Reflector (RR) network 10.0.0.0
“Hub-and-spoke” topology
Site 1
Site 3 Site 2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
CSCuj68811:
15.4(1.16)S0.2, 15.4(1.16)S0.3
Route Reflectors 15.4(1.16)S0.4, 15.4(2.1)S
15.4(2.2)S
Point to Multi-Point – Multiple Branch Sites
• Question:
In the example, if CE in Site 1 advertises a router eigrp ROCKS
address-family ipv4 unicast auto 4453
route to the Route Reflector, will the route remote-neighbors source Serial 0/0 unicast-listen lisp-encap
network 10.0.0.0
propagate to other CE routers? af-interface serial 0/0
no split-horizon
• Answer: No! exit-af-interface
RR = DP
= CP
• Configure the new CE to point to the RR
• New CE and RR exchange routes, and
RR sends new routes to other CEs Site 1
• Adding additional CE routers does not
require changes to configuration
of the Route Reflector Site 3 Site 2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 25
CSCuj68811:
15.4(1.16)S0.2, 15.4(1.16)S0.3
Route Reflectors 15.4(1.16)S0.4, 15.4(2.1)S
15.4(2.2)S
Point to Multi-Point – Any-to-Any Data
• Each CE normally shows the Route Reflector (RR) as the next hop
– Data will ‘hairpin‘ though the RR to get to other sites
– Useful for applying Policy and filtering traffic
– Will increase bandwidth requirements for the Route Reflector
• What if I want to send traffic directly
from site to site? RR = DP
= CP
Site 3 Site 2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 26
Route Reflectors
router eigrp ROCKS
3rd Party Nexthops address-family ipv4 auto 4453
af-interface Serial0/0
no next-hop-self
• Normally the Route Reflector would send the nexthop
as 0.0.0.0 which tells CE1 and CE2 to use it to reach
the destination RR
.1
• When “no next-hop-self” configured, the RR preserves
the next hop of the peer that sent it the route
• When CE1 and CE2 receives an update from the
RR, they install the route in the RIB with the
.2
supplied nexthop .3
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Peering over the WAN
• Remote Routers
• Route Reflectors
• Redundant Remote Routers
• Redundant Route Reflectors
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Redundant Remote Routers
Multiple Next Hops 10.2.0.0 [90/18600] via 192.168.1.5, LISP0
via 192.168.1.6, LISP0
• In an OTP setup, an RR can learn two or
more equal-cost paths to a site.
Site 1
• However, the RR router will only advertise RR
one of the paths to other spokes in the OTP 10.2.0.0 [90/32600] via 192.168.1.5
network.
Site 2
• Implication:
– Site to Site traffic will only be sent to one router
.5 .6
– Sites are not able to leverage multi-router
setups
Site 3
10.2.0.0/24
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Redundant Remote Routers
Multiple Next Hops
• While this isn't a route propagation problem, per se, it's still a situation that
may take you by surprise and therefore may be useful to understand
• One of the designs being implemented with OTP uses multiple paths from the
hub to reach spoke subnets. This could be two paths to the same spoke or
through two spokes (as shown on the previous slide)
• The problem is that EIGRP still uses normal distance vector rules and sends
updates based on the top topology table entry.
• Even if there are two equal cost paths, EIGRP sends updates based on the
top entry, even though there are two paths available.
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Redundant Remote Routers
Solution: Add-Path
• To avoid this situation and enable Remotes to
use all paths, configure the “add-path” option
on the RR (hub) Site 1
RR 10.2.0.0 [90/32600] via 192.168.1.5
• Add Path Support enables the Route Reflector via 192.168.1.6
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Peering over the WAN
• Remote Routers
• Route Reflectors
• Redundant Remote Routers
• Redundant Route Reflectors
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
Redundant Route Reflectors
Adding second RR
• Adding a second Route Reflector does not change the original Route
Reflector’s, configuration
• On the Remote Routers, add the new remote Site 1
neighbor configuration for the new Route Reflector
RR-1 RR-2
• Remotes do not have to be configure to connect
to all Route Reflectors
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Redundant Route Reflectors
Exchanging routes between RR’s
• If the Route Reflectors are in different sites, you may want to exchange routing
information between the Route Reflectors
• You might be tempted to setup a remote neighbor; Site 1
Site 2
RR-1 RR-2
router eigrp ROCKS
address-family ipv4 unicast auto 4453
remote-neighbors source Serial 0/0 unicast-listen lisp-encap
neighbor 192.168.2.2 Seral0/2 remote 100 lisp-encap
...
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
Redundant Route Reflectors
Support for Multiple Providers
• Support for additional Service Providers is
also possible
ISP1
• Choose a Route Reflector per Service
Provider to ensure each CE has
reachability to other sites RR-1
ISP2
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Deployment Considerations
• Route Filtering
• Backdoor Links
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Route Filtering
Limiting leaking of public routes into the LAN address-family ipv4 unicast auto 4453
neighbor 192.168.1.1 Serial 0/2 remote 100 lisp-encap
• When you setup an OTP peer, you must add a network 192.168.0.0 0.0.255.255
network 10.2.0.0 0.0.255.255
network statement covering the public interface ...
• This means the public network will show up in the .20.13 .31.14
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
Route Filtering
Limiting leaking of public routes into the LAN
• Looking on the Route Reflector we see the new peer come up..
CE1#
02:24:05: %DUAL-5-NBRCHANGE: EIGRP-IPv4 4453: Neighbor 192.168.31.14 (Serial1/0) is up: new adjacency
02:24:07: %CFC_LISP-5-ADJ_STACK: Stacking adjacency IP adj out of LISP0, addr 192.168.31.14 (incomplete)
onto other LISP adjacency IP midchain out of LISP0, addr 192.168.20.13 F0732BB8 forcing drop
• But we also see an traffic is being drop due to the LISP encapsulation failure
CE3#ping 192.168.31.14
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.31.14, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Route Filtering
Public routes subnets in the LAN can result in recursion issues
• From “show ip route” We can see the public address is recursive though another
public address
– To get to 192.168.20.0/24, the packet needs to be sent to 192.168.32.14 though the
LISP interface
– To get to 192.168.31.14, the route lookup for 192.168.0.0/24 also goes though LISP
interface
CE1#show ip route
…
D 192.168.20.0/24 [90/114980571] via 192.168.31.14, 00:00:29, LISP0
D 192.168.31.0/24 [90/114980571] via 192.168.20.13, 00:23:10, LISP0
• Peers are not effected by the LISP encap failure as EIGRP sends packets
directly to the public interface
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
Route Filtering
Solution – filter public routes from being reached via the LAN
• Best practice is to prevent the public networks from .20.13 .31.14
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Deployment Considerations
• Route Filtering
• Backdoor Links
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Deployment Considerations
Site to Site - Backdoor Links
• The use of “back-door” links for OTP does not require special handling
– Path selection determined by setting ‘delay’ on backdoor links
ISP
Headquarters
CE
CE
interface Serial0/0
delay 40000
. . .
• Use “distribute-list out” on CE’s to prevent address from leaking between sites
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Case Study
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
The Acme Corporation
Requirements:
– Fast convergence (<1s if possible)
– Direct Spoke-to-spoke traffic
– 1600+ sites across four countries
– Active/active load balancing
– Encryption across WAN
Nice to have:
– Easy provisioning
• No config changes on hubs as new sites are added
• Zero touch deployment of branch wan router (CE)
– Provider flexibility
• Multiple providers in each country
• Easy migration between providers
• No routing exchange of internal addresses
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
The Acme Corporation
France Sweden
… … … …
Corporate Backbone
MPLS MPLS
MPLS MPLS
VPN VPN
VPN VPN
… … … …
England USA
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
The Acme Corporation
• Route Exchange
RR WAN Hubs
RR
2 x ASR1000
… …
Spokes
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
The Acme Corporation
• WAN Security with GET VPN
KEY SERVER
WAN Services
2 x 3945E
MEMBER RR RR MEMBER
WAN Hubs
2 x ASR1000
… …
MEMBERS
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
The Acme Corporation
Requirements:
– Fast convergence (<1s if possible) – IGP speeds via end-to-end EIGRP solution
– Direct Spoke-to-spoke traffic – Use of no nexthop-self on RR
– 1600+ sites across four countries – Up to 500 EIGRP spokes per RR
– Active/active load balancing – Ability to add 4 additional ECMP via addpath
– Encryption across WAN – GET VPN
Nice to have:
– Easy provisioning
No config changes on hubs as new sites are added – Route Reflectors
Zero touch deployment of branch wan router (CE) – Route Reflectors
– Provider flexibility
Multiple providers in each country – Multiple neighbor configs supported
Easy migration between providers – Built into OTP
No routing exchange of internal addresses – Built into OTP
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public
Summary: What Have We Learned?
• WAN deployments are greatly simplified with OTP
• Both the Enterprise and Service Provide benefits from OTP
• EIGRP OTP supports both IPv4 and IPv6 deployments
• EIGRP’s scalability is an important factor in OTP deployment
• OTP can work over traditional WAN and LAN networks
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
For more Information on OTP
• EIGRP OTP:
– http://www.cisco.com/en/US/docs/ios-xml/ios/iproute_eigrp/configuration/xe-3s/ire-eigrp-
over-the-top.html
• Open EIGRP (IETF Draft):
– ftp://ftp.ietf.org/internet-drafts/draft-savage-eigrp-02.txt
• OTP OSPF (IETF Draft):
– http://www.ietf.org/staging/draft-white-ospf-otc-01.txt
© 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 50