BRKDCN-2035

VXLAN BGP EVPN based

Multi-Site

Max Ardica – Principal Engineer

Lukas Krattiger – Principal Engineer
Who Are the Presenters?

Max Ardica Lukas Krattiger

Principal Engineer – DCNBU Principal Engineer –DCNBU
@maxardica @ccie21921

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Session Objectives

At the end of the session, the participants should be able to:

 Articulate the value proposition of the new VXLAN Multi-
Site architecture and list several use cases for which it
should be positioned
 Understand the functionalities and specific design
considerations associated to VXLAN Multi-Site
Initial assumption:
 The audience already has a good knowledge of the VXLAN
EVPN technology and its use to deploy modern Data
Center Fabrics

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
  Data Center Interconnect (DCI) Evolution
Agenda
 VXLAN Multi-Site Introduction
• Functional Components and Use Cases
• HW/SW Support and Scalability Values
• Supported Topologies
 VXLAN Multi-Site Deep Dive
• Border Gateway Deployment Considerations
• Inter-Site BUM Traffic Handling
• Control and Data Planes
• Failure Detection on BGWs
• Connectivity to the External Layer 3 Domain
• Network Services Integration
• Legacy Site Integration
• Configuration Specifics
 Conclusions
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
 Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

cs.co/ciscolivebot#BRKDCN-2035

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Data Center
Interconnect (DCI)
Evolution
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Overlays Evolve and Spread

DC Local Overlay

End-to-End Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Single Logical Data Center

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Back Then
VXLAN for Interconnecting Networks

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
Changing the Paradigm with Overlays

DC Local Overlay

Multi-Site Overlay SS SS SS SS

S S S S S S S S

L L L L .... L L L L L .... L
Multiple Logical Data Center

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 VXLAN Evolves as the Control Plane Evolves!
Before Yesterday
Yet Another Encapsulation
 Flood & Learn (Multicast-based)
 Data-Plane only Yesterday
VXLAN for the Data Center – Intra-DC
 Control-Plane
 Active VTEP Discovery
Today
 Multicast and Unicast
VXLAN for DCI – Inter-DC
 DCI Ready
 ARP/ND caching/suppress
 Multi-Homing
 Failure Domain Isolation
 Loop Protection
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 Inter-X Connectivity
VXLAN Multi-Pod VXLAN Multi-Fabric VXLAN Multi-Site

EVPN Control- BGP EVPN EVPN Control- EVPNFabric

Control-Plane EVPNFabric
Control-Plane EVPNFabric
Control-Plane
#1 BGP EVPN EVPNFabric
Control-Plane
#2
Fabric #1 Fabric #2 #1 #2
Plane Domain 1 Plane Domain 2 Domain 1 Domain 2 Domain 1 Domain 2

Overlay Overlay Overlay Overlay Overlay Overlay

VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE VTE
P P P P P P P P P P P P P P P P P P P P P P P P

Bar Bar Bar Bar

em em em em Bar Bar Bar Bar Bar Bar Bar Bar
eta eta eta eta em em em em em em em em
l l l l etal etal etal etal etal etal etal etal

DCI DCI
Single Data-Plane – End-to-End Data-Plane Domain 1 Data-Plane Domain 2 Data-Plane Domain 1 Data-Plane Domain 2
Data-Plane Data-Plane

 Single Fabric with End-to-  Multiple Fabrics – Normalized  Multiple Fabrics with
End Encapsulation through Ethernet Integrated DCI (DCI2)
 Build Hierarchy in the  Multiple Fabrics Interconnect  Integrated DCI – Scaling
Underlay – Flatten it in the using DCI (Layer 2 and Layer 3) within and between
Overlay Fabrics

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
VXLAN Multi-Pod
VXLAN EVPN – Single Pod / Single Fabric

External Network
VTEP VTEP

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Pod 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
VXLAN EVPN – Multi-Pod

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Pod 1 Pod n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
Multi-Pod Characteristics – ”The Single”

 Single Overlay Domain – End-to-End Encapsulation

 Single Overlay Control-Plane Domain – End-to-End EVPN Updates
 Single Underlay Domain End-to-End
 Single Replication Domain for BUM
 Single VNI Administrative Domain

Building Underlay Hierarchies – Non Hierarchical Overlay

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
 Multi-Pod End-to-End Encapsulation

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP
10.1.1.1 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Unicast

Pod 1 Pod n
Baremetal Baremetal

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Multi-Pod VXLAN Tunnel Adjacencies

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP VTEP

10.1.1.1 10.1.1.4 10.2.2.7
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Switch# show nve peers

Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1
Pod 110.1.1.4 30000 03:18:06
Pod n
nve1 10.2.2.7 30000 00:12:23

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
 Multi-Pod Underlay Extension
POD1 Underlay Routing Table POD2 Underlay Routing Table

Leaf: Border: Leaf: Border: Border: Leaf: Border: Leaf:

10.1.1.1 10.1.1.101 10.2.2.1 10.2.2.101 10.2.2.101 10.2.2.1 10.1.1.101 10.1.1.1
10.1.1.2 10.1.1.102 10.2.2.2 10.2.2.102 10.2.2.102 10.2.2.2 10.1.1.102 10.1.1.2
10.1.1.3 10.2.2.3 10.2.2.3 10.1.1.3
10.1.1.4 10.2.2.4 10.2.2.4 10.1.1.4
10.1.1.5 10.2.2.5 Underlay Extension 10.2.2.5 10.1.1.5
10.1.1.6 10.2.2.6
VTEP VTEP VTEP
10.2.2.6
VTEP
10.1.1.6
10.1.1.7 10.2.2.7 10.2.2.7 10.1.1.7
Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP
10.1.1.1
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Pod 1 Pod 2

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Multi-Pod BUM Replication

Underlay Extension
VTEP VTEP VTEP VTEP

Spine Spine Spine Spine Overlay Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Pod 1 Pod 2
Baremetal

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Multi-Pod Challenges – ”The Single”

 Single Overlay Domain – End-to-End Encapsulation

• Scaling the VXLAN EVPN Network
 Single Overlay Control-Plane Domain – End-to-End EVPN Updates
• Overlay Control-Plane Update Propagation
 Single Underlay Domain End-to-End
• Network must be extended in Underlay (VTEP to VTEP reachability)
 Single Replication Domain for BUM
• One BUM flooding domain through out all connected Pods

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
VXLAN Multi-Site
Introduction
Functional Components
and Use Cases
 VXLAN Multi-Site https://tools.ietf.org/html/draft-sharma-multi-site-evpn
Functional Components
Site-External DCI
(IP Routing and Increased
Border Gateways MTU Support)
(Key Functional Components of
VXLAN Multi-Site Architecture)

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site-Internal Fabric
Site 1 (Common VXLAN and Site n
BGP-EVPN Functions)
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
VXLAN Multi-Site Characteristics

 Multiple Overlay Domains – Interconnected & Controlled

 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
 Multiple Underlay Domains - Isolated
 Multiple Replication Domains for BUM – Interconnected & Controlled
 Multiple VNI Administrative Domains – Phase 2

Underlay Isolation – Overlay Hierarchies

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
VXLAN Multi-Site
Main Use Cases

Scale-Up Model to Build a

Large Intra-DC Network

Data Center Interconnect (DCI)

Integration with Legacy Networks

(Coexistence and/or Migration)
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 VXLAN Multi-Site Inter-Site Network
Routing Table
Underlay Isolation Border Site1: Border Site2:
10.1.1.101 10.2.2.101
10.1.1.102 10.2.2.102
10.1.1.111 10.2.2.222

Multi-Site VIP Multi-Site VIP

10.1.1.111 Site-External DCI 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Border (PIP) Border (PIP) Border (PIP) Border (PIP)
10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine

Site 1 Underlay Site n Underlay

Routing Table Routing Table
Border: Leaf: Border: Leaf:
VTEP 10.1.1.101 10.1.1.1 VTEP
10.2.2.101 10.2.2.1
10.1.1.1
VTEP VTEP VTEP VTEP VTEP
10.1.1.102 10.1.1.2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
10.2.2.102 10.2.2.2
VTEP VTEP

10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3

10.1.1.4 10.2.2.4
10.1.1.5 10.2.2.5
10.1.1.6 10.2.2.6
Site 1 Site n
10.1.1.7 10.2.2.7

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 VXLAN Multi-Site
Introducing the Border Gateway
Overlay Multi-Site

Border Gateway (BGW)

- Anycast Cluster -
Multi-Site VIP Multi-Site VIP
10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Any VTEP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Multi-Site – VXLAN Tunnel Adjacencies
BG102# show nve peers
Interface Peer-IP VNI Up Time
---------- ----------- ------ ----------
nve1 Overlay
10.1.1.1 Multi-Site
30000 00:12:16
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:23

Multi-Site VIP Multi-Site VIP

10.1.1.111 10.2.2.222
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP

10.1.1.1
VTEP VTEP VTEP
10.1.1.4VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
10.2.2.7
VTEP

Leaf1-1# show nve peers

Leaf2-7# show nve peers
Interface Peer-IP VNI Up Time
Interface Peer-IP VNI Up Time
Site 1
---------- ----------- ------ ---------- Site n------
---------- ----------- ----------
nve1 10.1.1.4 30000 03:18:06
nve1 10.2.2.222 30000 00:12:25
nve1 10.1.1.111 30000 00:12:23
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
HW/SW Support and
Scalability Values
 VXLAN Multi-Site
HW/SW Support

 Minimum hardware and software requirements for Border Gateways

Item Requirement
• Cisco Nexus 9300 EX platform
• Cisco Nexus 9300 FX platform
• Cisco Nexus 9300 FX2 platform
Cisco Nexus Hardware
• Cisco Nexus 9364C platform
• Cisco Nexus 9500 platform with X9700-EX line card
• Cisco Nexus 9500 platform with X9700-FX line card

Cisco Nexus Software Cisco NX-OS Software Release 7.0(3)I7(1) or later

 The hardware and software requirements for the Site-Internal nodes of a

VXLAN BGP EVPN site remain the same as those without the EVPN Multi-Site
BGW

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
VXLAN Multi-Site
Scalability Values as of 9.2(2) Release
Multi-Site Scale
Number of Sites 10
Number of BGWs per Site 4 (Anycast), 2 (vPC)
VTEPs per Site 256

Border Gateway (BGW) Scale EX/FX/FX2 N9364C

Number of Layer-2 VNI (VLAN) 2,000
Number of Layer-3 VNI (VRF) 1,000
MAC per BGW 90,000 64,000
IPv4 Host Routes per BGW* ~530,000 ~60,000
IPv4 Network Routes per BGW* ~530,000 ~8,000
IPv6 Host Routes per BGW* ~24,000 ~7,000
IPv6 Network Routes per BGW* ~260,000 ~2,000
*The values provided in these tables focus on the scalability of one particular route scale at a time
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Supported Topologies
 BGW-to-Cloud

Layer-3
Network

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 BGWs between Spine and Super-Spine

Super-Spine Super-Spine

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 BGWs on Spine

Super-Spine Super-Spine

BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine Spine Spine
Site 1 Site 2 Site n

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 BGWs Back-to-Back

BGW BGW BGW BGW

Site 1 Site 2
Spine Spine Spine Spine

Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
VXLAN Multi-Site
Deep Dive
Border Gateway
Deployment
Considerations
VXLAN Multi-Site
Border Gateways Deployment Considerations
 Border Gateways used for two main functions: Anycast Border Gateways

Interconnecting each site to the Inter-Site network (for

BGW BGW BGW BGW

• VTEP VTEP VTEP VTEP

East-West traffic flows)

• Connecting each site to the external Layer 3 domain (for
North-South traffic flows)
• May also be used to connect endpoints and/or network
service nodes (FWs, ADCs) Site 1

 Possible deployment models:

VPC Border Gateways
• Anycast Border Gateways BGW BGW

VPC Border Gateways (from ACI release 9.2(1))

VTEP VTEP

 BGW function enablement in the VXLAN EVPN fabric:

• BGWs as leaf nodes
• BGWs as spine nodes (Border-Spines)
Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Anycast Border
Gateways
VXLAN Multi-Site
Anycast Border Gateway (1)
Anycast Border Gateway
 Up to 4 Border Gateways
 Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1)
VTEP VTEP VTEP VTEP
• Deploying at Spine – 7.0(3)I7(2)

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
VXLAN Multi-Site
Anycast Border Gateway (2)
Anycast Border Gateway
 Common Multi-Site Virtual IP (Multi-Site VIP)
Multi-Site VIP
10.1.1.111 across BGWs

BGW BGW BGW BGW

• Multi-Site VIP for communication between
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
the Border Gateways in different Sites
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• Multi-Site VIP for communication between
Border Gateways and Leaf nodes within a
Multi-Site VIP
10.1.1.111
Site
 Individual Primary IP (PIP) per BGW
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
• PIP for communication with Single-Homed
endpoints (routed only), intra- and inter-
Site
Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
VXLAN Multi-Site
Anycast Border Gateway (3)
Anycast Border Gateway
Type: 00 IP: 10.1.1.101
4 System MAC: 00:00:00:00:00:01
Ethernet Segment: 00:00:07 VNI: 30010  Per-VNI Designated Forwarder (DF) election
• Each BGW can serve as DF for a single or a set of
Layer-2 VNIs
BGW BGW BGW BGW
VTEP VTEP VTEP VTEP • DF election and assignment is automatic
DF
30010
DF
30011
DF
30012
DF
30099
 Using BGP EVPN Route Type 4 for DF election
• Operator Managed Assignment (Type: 00)
BGP EVPN • Six Octet Site Identifier (System MAC:
00:00:00:00:00:01)
• Multi-Site Discriminator (Ethernet-Segment:
RR RR 00:00:07)
Spine Spine

• Originators IP Address (PIP): 10.1.1.101

• Layer-2 VNI: 30010

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 VXLAN Multi-Site
Anycast Border Gateway (4)
External
Connectivity Anycast Border Gateway
Point-to-Point L3 Links
(Physical/Sub-Interfaces)  Single-Homed End-Points only connected with
L3 links
• Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW BGW BGW
VTEP
PIP-BGW1
VTEP
PIP-BGW2
VTEP
PIP-BGW3
VTEP
PIP-BGW4
• External routers
10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104
• No SVI support on BGW nodes
.1 .1
 Advertised and Reachable through Individual
Point-to-Point L3 Links
Primary IP Address (PIP)
Point-to-Point L3 Links
(Physical/Sub-Interfaces)
ADC ADC
• Intra-Site: Leaf nodes use PIP to reach the device
ADC ADC
connected to Border Gateways
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102 • Inter-Site: Remote Border Gateways use PIP to
reach the device connected to Border Gateways
VTEP

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
VPC Border Gateways
NXOS Release 9.2(1)
 NXOS Release
9.2(1)

Anycast BGW vs. VPC Border Gateway

Anycast Border Gateway VPC Border Gateway

• Up to 4 BGW • 2 BGW with physical VPC Peer-Link
• Shared Nothing • Small Deployments
• Simple Failure Scenarios • End-Point or Network Services
• Any Deployments Connectivity on BGW

• No End-Point or Network Services • Migration Use-Cases (Brownfield)

Connectivity on BGW • Pseudo-BGW to BGW
• Greenfield Deployments • Classic Ethernet/FabricPath to
VXLAN EVPN

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
 NXOS Release
9.2(1)

Multi-Site Border Gateway – Anycast vs. vPC

• Both Anycast and vPC Border Gateway needs to be configured with a common Multi-Site VIP address and an
individual Primary IP (PIP) address
• vPC Border Gateways share a secondary IP address to be used as vPC virtual IP (vPC VIP)

Anycast BGW vPC BGW

vPC VIP
11.11.11.11
Multi-Site VIP Multi-Site VIP
100.100.100.100 100.100.100.100

PIP1 PIP1 PIP1 PIP1

10.1.1.1 10.1.2.1 10.1.1.1 10.1.2.1

….
VTEP VTEP VTEP VTEP

BGW1 BGW4 BGW1 BGW2

Fabric Fabric

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
VXLAN Multi-Site
NXOS Release
9.2(1)

VPC Border Gateway and Transit Traffic

VPC Border Gateway
 Common Multi-Site Virtual IP (Multi-Site VIP)
across BGWs
Multi-Site VIP
10.1.1.111

• Multi-Site VIP for Inter-Site transit communication

BGW BGW
VTEP VTEP
(transit)
PIP-BGW1 PIP-BGW2
10.1.1.101 VPC VIP
10.1.1.121
10.1.1.102 • Common VPC Virtual IP (VPC VIP) across BGWs
• Used by default for communication with external
Multi-Site VIP networks
10.1.1.111
• Used for Broadcast, Unknown Unicast and
Multicast (BUM) replication
 Individual Primary IP (PIP) per BGW
• Used for communication with external networks
when “advertised-pip” is configured

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
 NXOS Release
VXLAN Multi-Site 9.2(1)

VPC Border Gateway and Locally Attached End-Points

VPC Border Gateway
 Single- or Dual-Homed End-Points
Multi-Site VIP
10.1.1.111 • Services Appliance (i.e. Firewall, ADC etc.)

BGW BGW • Physical or Virtual Servers

VTEP VTEP
• Anycast Gateway function offered to the
A nycast
endpoints
Anycast
Ga teway VPC VIP Ga teway
10.1.1.121

 Advertised and Reachable through VPC Virtual

Multi-Site VIP
10.1.1.111 IP Address (VPC VIP)
ADC Baremetal • Intra-Site: Leaf nodes use VPC VIP to reach End-
Points connected to Border Gateways
ADC EP
0000.3010.1102
192.168.10.102
0000.3010.1101
192.168.10.101
• Inter-Site: Remote Border Gateways use VPC VIP
to reach End-Points connected to Border
Gateways
• Traffic potentially traverses VPC Peer-Link
Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
 NXOS Release
VXLAN Multi-Site 9.2(1)

VPC Border Gateway and Designated BUM Forwarder

VPC Border Gateway
 VPC-based Designated Forwarder Election
 Per-Site Designated Forwarder (DF) election
BGW BGW • Using same approach as in VPC
Best Path to Rendezvous-Point or VPC Primary
VTEP VTEP
•
DF VPC VIP
10.1.1.121 Node
• Same VPC node is elected DF for all the Layer-2
VNIs

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
 NXOS Release
VPC Border Gateways 9.2(1)

Traffic between Locally Connected Endpoints across Sites

Src Dst
VXLAN
vPC VIP1 vPC VIP2 Original Packet
Header

Inter-Site
Network

vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
VTEP VTEP VTEP VTEP

BGW1 BGW2 BGW1 BGW2

Site 1 Site 2

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
 NXOS Release
VPC Border Gateways 9.2(1)

Traffic between Locally Connected Endpoints and Remote L3Out

Src Dst
VXLAN
vPC VIP1 vPC VIP2 Original Packet
Header

Inter-Site
Network

vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
VTEP VTEP VTEP VTEP

BGW1 BGW2 BGW1 BGW2

L3

Site 1 Site 2

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 NXOS Release
VPC Border Gateways 9.2(1)

Traffic between Site Connected Endpoints across Sites

Src Dst
Multi-Site Multi-Site VXLAN
Original Packet
VIP1 VIP2 Header

Inter-Site
Network

Multi-Site VIP1 Multi-Site VIP2

100.100.100.100 200.200.200.200
VTEP VTEP VTEP VTEP

BGW1 BGW2 BGW1 BGW2

VTEP VTEP

Site 1 Site 2

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
 NXOS Release
VPC Border Gateways 9.2(1)

BUM Traffic across Sites

Src Dst
VXLAN
vPC VIP1 vPC VIP2 Original Packet
Header

Inter-Site
Network

vPC VIP1 vPC VIP2

11.11.11.11 22.22.22.22
DF DF
VTEP VTEP VTEP VTEP

BGW1 BGW2 BGW1 BGW2

BUM Traffic redirected

via vPC peer-link
toward the DF

VTEP VTEP

Site 1 Site 2

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Inter-Site BUM Traffic
Handling
VXLAN Multi-Site
BUM Traffic Forwarding
Overlay Multi-Site

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
VXLAN Multi-Site
BUM Replication Modes (Multicast Intra-Site)
Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Multicast Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
VXLAN Multi-Site
BUM Replication Modes (Ingress Replication Only)

Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Ingress Replication

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
VXLAN Multi-Site
BUM Replication Modes (Mixed Mode Intra-Site)
Overlay Multi-Site

Ingress Replication

VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Overlay Site 1

Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

Ingress Replication Multicast

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Site 1 Site n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
VXLAN Multi-Site
BUM Traffic Policing
Overlay Multi-Site

Storm Control
VTEP VTEP Broadcast 0-100% VTEP VTEP

BGW BGW
Unknown Unicast 0-100% BGW BGW

Multicast 0-100%
Spine Overlay Site 1
Spine Spine Spine Spine Overlay Site n
Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BUM

Site 1 Site n
Baremetal

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Control and Data Planes
VXLAN Multi-Site
Control Plane
 VXLAN Multi-Site
Control Plane Deployment Considerations
 MP-eBGP EVPN only inter-Sites
• Next-hop behavior (VXLAN tunnel termination and reorigination) and loop protection
(as-path attribute)

 Two main options for underlay and overlay control plane deployment
1. I-E-I (Recommended)
• Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP
• Inter-Sites: eBGP for both underlay and overlay CPs
2. E-E-E*
• Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs

 Full mesh of MP-eBGP EVPN adjacencies across sites

• Recommended to deploy a couple of Route-Servers with 3 or more sites
• RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”,
IETF RFC 7947)
• RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite
*For more information on why eBGP for both underlay and overlay CP is not a good idea:
https://learningnetwork.cisco.com/blogs/community_cafe/2017/10/17/the-magic-of-super-spines-and-rfc7938-with-overlays-guest-post

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
 VXLAN Multi-Site
Underlay Control Plane

DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
 VXLAN Multi-Site
Overlay Control Plane (L3 Core)
Route Server (eBGP ”Route Reflector”)
RS

DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
 VXLAN Multi-Site
Overlay Control Plane (L3 Core, no RS)

eBGP-EVPN
DC Core
(Layer-3 Unicast)

DCI

…. ….
VTEP VTEP VTEP VTEP

Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 VXLAN Multi-Site
Overlay Control Plane
RS
L3VNI: 50001
L3VNI: 50001
Route-Target: DC Core Route-Target: 65502:50001
65501:50001
(Layer-3 Unicast)

VRF VRF
Tenant1 Tenant1
DCI

…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN
iBGP-EVPN VXLAN EVPN
iBGP-EVPN
Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 VXLAN Multi-Site
Overlay Control Plane (Site 1)
RS
L3VNI: 50001
L3VNI: 50001
Route-Target: DC Core Route-Target: 65502:50001
65501:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

VRF VRF
2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001 10.1.1.1

Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32

Tenant1 10.1.1.111
50001, 65501:50001
DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111

…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
 VXLAN Multi-Site
Overlay Control Plane (Site 2)
RS
L3VNI: 50001
L3VNI: 50001
Route-Target: DC Core Route-Target: 65502:50001
65502:50001
(Layer-3 Unicast)
Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 VRF 30010, 65502:30010

0000.3010.1101/48 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF
2 Tenant1 30020, 65502:30020
0000.3020.2101/48 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1
DCI
2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3

…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 VXLAN Multi-Site
Overlay Control Plane (DCI)
RS
L3VNI: 50001
L3VNI: 50001
Route-Target: DC Core Route-Target: 65502:50001
65501:50001
(Layer-3 Unicast)

VRF VRF
Tenant1 Tenant1
DCI

…. ….
VTEP VIP1 VTEP VTEP VIP2 VTEP
Type MAC / Length
10.1.1.111 L2VNI / RT IP / Length L3VNI / RT Next-Hop
10.2.2.222 Seq.
Fabric BGW 2 0000.3010.1101/48BGW 30010, 65599:30010 192.168.10.101/32 BGW
50001, 65599:50001 10.1.1.111 BGW

2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222

VXLAN
2
EVPN
0000.3010.1102/48 30010, 65599:30010 192.168.10.102/32
VXLAN
50001, 65599:50001
EVPN
10.2.2.222
Spine Spine Spine Spine
RR RR
L2VNI: 30010 (VLAN 10)
Site1 L2VNI: 30020 (VLAN 20)
Site2 L2VNI: 30010 (VLAN 10)
L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1)
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
VXLAN Multi-Site
Data Plane
 VXLAN Multi-Site
Overlay Data Plane
Inter-site VXLAN
Data Plane

De-capsulation and DC Core

Re-encapsulation on De-capsulation and
BGW (L2 or L3 lookup)
(Layer-3 Unicast)
Re-encapsulation on
BGW (L2 or L3 lookup)

DCI

…. ….
VTEP Multi-Site VIP1 VTEP VTEP Multi-Site VIP2 VTEP
10.1.1.111 10.2.2.222
Fabric BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine
Intra-site VXLAN
Data Plane
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host2 Host3

0000.3010.1101 0000.3020.2101 0000.3010.1102
192.168.10.101 192.168.20.101 192.168.10.102
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Multi-Site Packet Walk
(BUM)
 VXLAN Multi-Site Packet Walk
Layer 2 (BUM) – Site 1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
Traffic is replicated VTEP 30010 VTEP

intra-Site BGW11 BGW21

2
VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF

1 Host 1 sends a
VTEP 30010 VTEP

BGW12 BGW22
L2 BUM frame

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
 VXLAN Multi-Site Packet Walk
Layer 2 (DF and Split Horizon) – Site 1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP BUM Forward VTEP

VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
 VXLAN Multi-Site Packet Walk
Layer 2 (BUM) – DCI
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP

BGW11- BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

PIP Payload

BGW11- BGW22 30010 H1-MAC ALL-F H1-IP ALL-255

PIP

VTEP
DF
30010 BGW11 replicates traffic VTEP

BGW11
inter-Sites toward BGW BGW21
nodes
VTEP BUM Forward 3 VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
 VXLAN Multi-Site Packet Walk
Layer 2 (DF and Split Horizon) – DCI
Bridge
SIP DIP VXLAN SMAC DMAC SIP DIP

BGW11- BGW21 30010 H1-MAC ALL-F H1-IP ALL-255

PIP Payload

BGW11- BGW22 30010 H1-MAC ALL-F H1-IP ALL-255

PIP

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP BUM Forward VTEP

VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 VXLAN Multi-Site Packet Walk
Layer 2 (BUM) – Site 2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP VTEP
30010
Traffic is replicated
BGW11 BGW21 intra-Site

VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
4 VTEP

Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 VXLAN Multi-Site Packet Walk
Layer 2 (DF and Split Horizon) – Site 2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22

BUM Forward
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102
192.168.10.101 192.168.10.102

Drop due to Split-Horizon rule

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
 VXLAN Multi-Site Packet Walk
Layer 2 (BUM) – Site 2
Bridge

DF
VTEP 30010 VTEP

BGW11 BGW21

VTEP VTEP
VXLAN EVPN VXLAN EVPN VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

DF
VTEP 30010 VTEP

BGW12 BGW22
5
Leaf20 sends traffic
to local Host 2
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Multi-Site Packet
Walk (Bridging)
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 1 to Host 2) – Site 1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP

Leaf10 performs L2 lookup

and encapsulates toward VTEP VTEP
local BGW VIP1 address BGW11 BGW21

2
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

1 Host 1 sends traffic

VTEP VTEP

BGW12 BGW22
destined to remote Host 2

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 1 to Host 2) – DCI
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP

BGW11 performs L2 lookup

VTEP VTEP
and encapsulates toward
BGW11 remote BGW VIP2 address BGW21

VTEP
3 VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 1 to Host 2) – Site 2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP

VTEP VTEP
BGW22 performs L2 lookup
and encapsulates toward
BGW11 BGW21
destination L20 node

VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
4 VTEP

Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22
5
Leaf20 bridges traffic
to local Host 2
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 2 to Host 1) – Site 2
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP

Leaf20 performs L2 lookup

VTEP VTEP
and encapsulates toward
BGW11 BGW21 local BGW VIP2 address

VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
7 VTEP

Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22
6
Host 2 replies to
remote Host 1
Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 2 to Host 1) – DCI
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP

BGW21 performs L2
VTEP lookup and encapsulates VTEP

BGW11 toward remote BGW VIP1 BGW21

address
VTEP
8 VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP

BGW12 BGW22

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
 VXLAN Multi-Site Packet Walk
Layer 2 (Host 2 to Host 1) – Site 1
Bridge

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload
BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP

BGW12 performs L2 lookup

and encapsulates toward VTEP VTEP
destination L10 node BGW11 BGW21

9
VTEP VTEP
VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN
Leaf10 Site1 DCI Site2 Leaf20

VTEP VTEP
10 Leaf10 bridges traffic
BGW12 BGW22
toward Host 1

Baremetal Baremetal

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
Connectivity to the
External Layer 3 Domain
VXLAN Multi-Site
Connectivity to the External Layer 3 Domain

 The BGW nodes can also be used to provide Layer-3 external

connectivity to each site
 Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared
Border deployment model, aka GOLF)
• Dedicated or shared pair of WAN Edge routers across sites

 External Layer-3 network may be different from the DCI

network used for inter-site communication

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110
 VXLAN Multi-Site
Border Gateways and VRF-Lite to External Routers

Dedicated physical Separate IPv4/IPv6 routing

interfaces / sub- peering for each VRF (IGP
interfaces for each VRF External
VRF-AVRF-B VRF-C or eBGP)  Separate IPv4/IPv6 routing peering for
Connectivity each VRF established with the
Site-External

external routers on dedicated physical

Multi-Site
Overlay
interfaces/sub-interfaces
 Must use separate interfaces for
BGW BGW BGW BGW inter-site communication
No support for VXLAN encapsulated
VTEP VTEP VTEP VTEP

traffic on sub-interfaces
Site-Internal

Site 1

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111
 VXLAN Multi-Site
Border Gateway and Shared Border (aka ‘GOLF’)
External router operates like a
traditional VXLAN EVPN VTEP
(Layer 3 only)
 Single MP-BGP EVPN peering
External
VRF-AVRF-B VRF-C
established with the external routers
Connectivity
Single MP-BGP EVPN routing
to exchange routes for all the VRFs
instance to exchange routes
Routed interface extending
‘underlay’ connectivity to
for all VRFs
 VXLAN Data-Plane between the
the external routers
Multi-Site BGWs and the external routers
Site-External

Overlay
 Same spine uplinks used for all
VXLAN encapsulated traffic (North-
South and East-West)
BGW BGW BGW BGW
VXLAN Data Plane
 Required because of the use of DCI link
VTEP VTEP VTEP VTEP
between BGW and WAN tracking
Edge Router
Site-Internal

 Various northbound hand-off options

depending on specific HW support:
Site 1
VRF-Lite, MPLS-VPN, LISP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112
 VXLAN Multi-Site
Egress Traffic Optimization
172.16.1.10

External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

172.16.1.0/24  WAN Edge 1-2 eBGP-EVPN 172.16.1.0/24  WAN Edge 3-4

DC Core
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW
Less preferred
VXLAN EVPN advertisement of
172.16.1.0 because
VXLAN EVPN
Spine Spine Spine Spine
of longer AS-Path

Site1  VIP1
172.16.1.0/24 Site2  VIP2
172.16.1.0/24
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
VXLAN Multi-Site
Egress Traffic Optimization
172.16.1.10

External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

Optimized Egress Optimized Egress

Traffic Path eBGP-EVPN Traffic Path
DC Core
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
VXLAN Multi-Site
Egress Traffic Optimization – WAN Isolation
172.16.1.10

WAN Isolation
Scenario External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

eBGP-EVPN 172.16.1.0/24  WAN Edge 3-4

DC Core
(Layer-3 Unicast)
VIP1 VIP2
VTEP
10.1.1.111
VTEP 172.16.1.0/24  VIP2 VTEP
10.2.2.222
VTEP

BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1  VIP1
172.16.1.0/24 Site2  VIP2
172.16.1.0/24
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
 VXLAN Multi-Site
Ingress Traffic Optimization
192.168.10.0/24  WAN Edge 1-4
192.168.10.101/32  WAN Edge 1-2
192.168.10.102/32  WAN Edge 3-4
Host routes Deploying LISP on the
advertisement in the WAN Edge routers is a
WAN viable alternative to host
External routes advertisement
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

eBGP-EVPN
192.168.10.0/24  VIP1 DC Core 192.168.10.0/24  VIP2
192.168.10.101/32  VIP1 192.168.10.102/32  VIP2
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222 Filtering of host routes
BGW BGW BGW BGW
Host routes advertised received from remote sites.
VXLAN EVPN across sites but NOT
re-advertised toward
VXLAN EVPN Only announce local host
route information
Spine Spine Spine Spine
the local WAN Edges

Site1
192.168.10.101/32  Leaf1 Site2
192.168.10.102/32 -> Leaf3
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
VXLAN Multi-Site
Ingress Traffic Optimization
192.168.10.0/24  WAN Edge 1-4
192.168.10.101/32  WAN Edge 1-2
192.168.10.102/32  WAN Edge 3-4

External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

Optimized Ingress Optimized Ingress

Traffic Path eBGP-EVPN Traffic Path
DC Core
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
VXLAN Multi-Site
Ingress Traffic Optimization – WAN Isolation
192.168.10.0/24  WAN Edge 3-4
192.168.10.101/32  WAN Edge 1-2
192.168.10.102/32  WAN Edge 3-4
WAN Isolation
Scenario External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

eBGP-EVPN
DC Core
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
VXLAN Multi-Site
Symmetric Traffic Path and Perimeter FWs Integration
Symmetric
Ingress/Egress
Traffic Paths
External
WAN Edge 1 WAN Edge 2 Connectivity WAN Edge 3 WAN Edge 4

Active/Standby Active/Standby
FW DC Core FW
(Layer-3 Unicast)
VTEP
VIP1 VTEP VTEP
VIP2 VTEP
10.1.1.111 10.2.2.222
BGW BGW BGW BGW

VXLAN EVPN VXLAN EVPN

Spine Spine Spine Spine

Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Host1 Host3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
 VXLAN Multi-Site and Network Services Integration
WAN

Active and Standby pair deployed across Sites,

Inter-Site
Active FW Network Standby FW 
VTEP VTEP VTEP VTEP

enforcement for N-S and E-W flows

BGW BGW BGW BGW

Spine VXLAN EVPN Spine Spine VXLAN EVPN Spine

Site1 Site2
 No issues with asymmetric flows
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

 Various options possible (FW as endpoints

gateway or fabric as endpoints gateway)
Bareme Baremet Bareme
tal al tal

Active FW Standby FW

WAN
 Independent Active/Standby pairs deployed in
Active/Standby Active/Standby separate Sites
Inter-Site
Network

FW FW
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine VXLAN EVPN Spine Spine VXLAN EVPN Spine  Need to avoid the creation of asymmetric paths
VTEP
Site1
VTEP VTEP VTEP VTEP
Site2
VTEP VTEP VTEP
crossing different active FW nodes
Baremet Baremet
 Only possible for N-S flows with perimeter FWs and
host routes advertisement or with PBR
al al

Active/Standby FW Active/Standby FW

WAN

Inter-Site
Network
 Active/Active FW Cluster stretched across Sites
Split spanned ether-channel mode: not supported,
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW 

VXLAN EVPNSpine
Spine Spine VXLAN EVPNSpine scoped for 2HCY19
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
 Individual mode: supported with Cisco ASA software
for N-S and E-W flows
Active/Active FW Cluster Bareme
tal
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 121
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 122
Legacy Site Integration
VXLAN Multi-Site
Legacy Site Integration IR for BUM +
aggregated BUM Pair of Pseudo-BGWs
containment (EX/FX Switches)
BGW

VTEP VTEP VTEP VTEP

BGW BGW

Spine Spine Spine Spine Baremetal ADC

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site Legacy Site

 Coexistence and/or migration use cases

• Extend Layer-2 and Layer-3 multi-tenant connectivity across sites
 Deploy a pair of Pseudo-BGWs in the legacy site
• Simplified configuration required on Pseudo-BGWs nodes
• Still offering native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.)
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 124
 Multi-Site and Legacy Site Integration
Default Gateway Deployment – Recommended
Default Gateway
migrated to the Border
Greenfield VXLAN BGW Gateways (VXLAN EVPN
EVPN Fabric offers L2 VTEP VTEP L3 VTEP VTEP
Anycast Gateway)
and L3 services for the BGW BGW
stretched IP subnets L2

Spine Spine Spine Spine

Legacy
infrastructure offers
only L2 services
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP

L2

Distributed Anycast Greenfield Site Legacy Site

Gateway function

 Recommended approach is to migrate the default gateway from the

legacy aggregation devices to the Border Gateways (VXLAN EVPN
Anycast Gateway)
 Optimize routing between endpoints deployed across sites

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 125
 Multi-Site and Legacy Site Integration
Layer-2 Control Plane Exchange across Sites
eBGP-EVPN
MAC NH MAC NH

0000.3010.1101 Leaf1 0000.3010.1101 VIP1

0000.3010.1102 VIP2 VIP1 VIP2 0000.3010.1102 Po1

10.1.1.111 vPC Anycast 10.2.2.222 BGW
VTEP VTEP
VTEP VTEP VTEP

BGW BGW All End-Points in the

legacy site are learned as
Po1 directly connected to the
BGW
Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal

Greenfield Site Legacy Site

Host 1 Host 2
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.10.102
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 126
 Multi-Site and Legacy Site Integration
Layer-3 Control Plane Exchange across Sites
eBGP-EVPN
IP NH IP L3VNI

192.168.10.101 Leaf1 192.168.10.101 VIP1

192.168.20.101 VIP2 VIP1 VIP2 192.168.20.101 Po1

10.1.1.111 10.2.2.222 BGW
VTEP VTEP L3 VTEP VTEP

BGW BGW
L2 All End-Points in the
legacy site are learned as
Po1 directly connected to the
BGW
Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Baremetal Baremetal

Greenfield Site Legacy Site

Host 1 Host 3
0000.3010.1101 0000.3010.1102
192.168.10.101 192.168.20.101
BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127
 VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (1)
Pair of Pseudo-BGWs
Pair of Pseudo-BGWs (EX/FX Switches)
(EX/FX Switches) BGW BGW

VTEP VTEP VTEP VTEP

Legacy Site 1 Legacy Site 2

 A pair of Pseudo-BGWs inserted in each legacy site to extend Layer-2 and Layer-
3 connectivity between sites
• Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …)
 Slowly phase out the legacy networks and replace them with VXLAN EVPN fabrics

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128
 VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (2)
Convert the nodes to
Convert the nodes to full BGWs functions
full BGWs functions
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP

‘Mixed’ Site 1 ‘Mixed’ Site 2

 Introduce VXLAN EVPN spines and additional VTEPs in each site

 Convert the Pseudo-BGWs to full BGW (may require vPC support on BGWs)
 Migrate endpoints between the legacy network and the new VXLAN EVPN fabric

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129
VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (3)
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site 1 Greenfield Site 2

 Decommission the legacy networks and leave only the VXLAN EVPN
fabrics in place

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 130
VXLAN Multi-Site and Legacy Site Integration
Starting from Legacy Networks Only (4)
VTEP VTEP VTEP VTEP

BGW BGW BGW BGW

Spine Spine Spine Spine

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Greenfield Site 1 Greenfield Site 2

 Move endpoints directly connected to the vPC BGW nodes (if any) to
regular leaf nodes and migrate to the Anycast BGW model
 Anycast BGW is the recommended deployment options
 The migration can be done in a non disruptive way, one node at the time

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 131
 VXLAN Multi-Site Anycast BGW

Site 1 Setup – Multi-Site DCI Tracking vPC BGW

BGW1
interface Ethernet1/1
• For vPC BGWs, the vPC peer-link does not need description TO-DC-CORE1
to be tracked as a DCI link ip address 10.111.111.1/30 tag 12345
DC Coreevpn multisite dci-tracking
(Layer-3 Unicast)
interface Ethernet1/2
description TO-DC-CORE2
ip address 10.111.222.1/30 tag 12345
evpn multisite dci-tracking
DCI e1/1 e1/1
e1/2 e1/2

….
VTEP VTEP
BGW2

Fabric BGW1 BGW2 interface Ethernet1/1

description TO-DC-CORE1
ip address 10.222.111.1/30 tag 12345
evpn multisite dci-tracking
Spine Spine

interface Ethernet1/2
description TO-DC-CORE2
ip address 10.222.222.1/30 tag 12345
evpn multisite dci-tracking
VTEP VTEP VTEP VTEP

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 142
Conclusions
VXLAN EVPN – Multi-Site
Multi-Site Core
• Border Gateway (BGW) to Border Gateway (BGW)
reachability required
• Reachability Back-to-Back (full-mesh) or via Layer-3
transport network
• Any Routing Protocol for BG reachability No Underlay Extension
• IPv4 Unicast Transport
VTEP
(Ingress
VTEP
Replication) VTEP VTEP
• BGP full-mesh or Route-Server (eBGP ”Route Reflector”)
for Overlay Control-Plane Multi-Site Border Gateway (BGW):
• Seamless insertion into existing VXLAN EVPN Fabrics
Spine Spine Spine
(Border Gateways require Nexus 9x00-EX/-FX)
Spine Spine Spine Spine Spine

• Layer-2 and Layer-3 extension to other Sites

• BGP- or VPC-based Border Gateway (BGW) Cluster (up
to 4 nodes when using BGP)
• All Border Gateways (BGW) are representing a common
Anycast VTEP
• Failure containment through Broadcast, Unknown Unicast
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

and Layer-2 Multicast limiter (off or rate-based)

• Co-Existence with VRF-Lite for External Connectivity
• Core and Fabric link tracking
Site 1 Site n

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 156
Multi-Site Advantages – ”The Multiple”

 Multiple Overlay Domains – Interconnected & Controlled

• Scaling and Segregating VXLAN EVPN Networks
 Multiple Overlay Control-Plane Domains – Interconnected & Controlled
• Limited Overlay Control-Plane Update Propagation
 Multiple Underlay Domains - Isolated
• Isolated Underlay Domains – No need for Extension
 Multiple Replication Domains for BUM – Interconnected & Controlled
• Individual BUM flooding domain with Traffic control

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 157
Inter-X Connectivity
Multi-Pod Multi-Fabric Multi-Site
Underlay Control Plane Unified Underlay Domain Separated Underlay Domains Separated Underlay Domains

Overlay Control Plane Separated Overlay Control-Plane Domains

Overlay Data Plane Single Data-Plane Separated Data-Planes Separated Data-Planes

Unified Underlay Domain (All

BUM Replication in DCI Dependency on DCI Choice (Unicast/Multicast)
Multicast or All Ingress Replication)

ARP Flood Suppression (DCI) yes yes yes

Unknown Unicast Flood

no yes yes
Suppression (DCI)

Broadcast Suppression/Limit
no yes yes
(DCI)

Layer-2 Loop Prevention Loop mitigation (Edge Protection) VPC at Border Loop mitigation (At DCI)

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 158
Resources
• VXLAN EVPN Multi-Site Design and Deployment White Paper
https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-
739942.html

• Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide - Configuring VXLAN
EVPN Multi-Site
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-
x/vxlan/configuration/guide/b_Cisco_Nexus_9000_Series_NX-
OS_VXLAN_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-
OS_VXLAN_Configuration_Guide_7x_chapter_01100.html

• Cisco Live Online - VXLAN BGP EVPN based Multi-POD, Multi-Fabric and Multi-Site
- BRKDCN-2035
https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035&showMyInterest=false#/

• Cisco DCNM 11.1(1) - Multi-Site Domain for VXLAN BGP EVPN Fabrics
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/11_1_1/config_guide/lanfabric/b_dcnm_fabric_lan/cont
rol.html#concept_nhz_lfc_yfb

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 159
In Summary…

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 160
Cisco Webex Teams

Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 161
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations

Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 162
Continue Your Education

Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings

BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 163
Thank you