ACI Multisite Control plane and

Data plane troubleshooting

Partner DC VT Mar 2019

Roland Ducomble
CX Technical Leader – ACI Solutions Team
27th March 2019
Introduction

© 2019 Cisco and/or its affiliates. All rights reserved.

ACI Multi-Site
Overview IP Network

MP-BGP - EVPN

ACI
Multi Site
Orchestrator
(MSO)

REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’

Region ‘C’

▪ Separate ACI Fabrics with independent APIC clusters ▪ MP-BGP EVPN control plane between sites
▪ ACI MSO pushes cross-fabric configuration to multiple ▪ Data Plane VXLAN encapsulation across sites
APIC clusters providing scoping of all configuration ▪ End-to-end policy definition and enforcement
changes
© 2019 Cisco and/or its affiliates. All rights reserved.
ACI Multi-Site Network information carried across Site
(VNID for VRF or BD)
Identity information carried across site
(Class-id aka pcTag

And translation VTEP IP VNID Class-ID

Tenant
No Multicast Requirement in
Packet
Backbone, Head-End Replication
(HER) for any Layer 2 BUM
IP Network traffic)

MP-BGP - EVPN

ACI
MSO

A same VRF, BD or EPG created on two different site will have different VNID and
Class-Id as those are allocated by the local APIC cluster (not by MSO)
Need for a translation !
In ACI multisite, data packet are across site are vxlan encapsulated with the
vnid/class of the source site
The Destination site spine will have the role of translating those value
© 2019 Cisco and/or its affiliates. All rights reserved.
ACI Multi-Site
Namespace translation
Translation of Class-ID, VNID
Translation of Source IP Network (scoping of name spaces)
VTEP address

MP-BGP - EVPN

…
ACI
MSO

Site 1 Site to Site VTEP traffic (VTEPs, VNID

Site n
and Class-ID are mapped on spine)
Leaf to Leaf VTEP, Class-ID is local to the Fabric
Leaf to Leaf VTEP, Class-ID is local to the Fabric
VTEP
VNID Class-ID Tenant Packet
VTEP IP
VNID Class-ID Tenant Packet VTEP
IP VNID Class-ID Tenant Packet
IP

© 2019 Cisco and/or its affiliates. All rights reserved.

5
ACI Multipod versus Multisite
Multipod Multisite
• Controller : Single APIC controller for all • Controller : separate APIC cluster per site,
pod MSO talks to each APIC cluster
• Namespace : No need of any translation • Namespace : need to translate vnid and
(all come from same APIC cluster) pcTag across site (done on target spine)
• Control Plane : BGP EVPN used to sync • Control Plane : BGP EVPN used to sync
COOP DB COOP DB
• Unicast Data : VXLAN encapsulated • Unicast Data : VXLAN encapsulated in
• Leaf to Leaf tunnel for established flow source site VNID
• Leaf to target site spine tunnel for establish flow
(Always need to hop by Spine for Xlate)
• BUM traffic in the IPN: multicast
encapsulated in BD GIPo • BUM traffic in the ISN : unicast copy to each
target site (HREP – Head end replication)
• Control plane : PIM BiDir
© 2019 Cisco and/or its affiliates. All rights reserved. • Control plane: no need of multicast control plane
Namespace Translation
troubleshooting

© 2019 Cisco and/or its affiliates. All rights reserved.

Object Model
fvCtx fvBD fvAEPg
For every VRF (fvCtx), BD
(fvBD)or EPG (fvAEPg) that fvSiteAssociated fvSiteAssociated
fvSiteAssociated
needs to be extended.

Each APIC creates additional fvRemoteId fvRemoteId fvPeerContext fvRemoteId

object to represent the Site-Id
vrf/bd/epg on each of the Remote-Id

remote site

© 2019 Cisco and/or its affiliates. All rights reserved.

 bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ctx-L2

Example for VRF

Total Objects shown: 1

# fv.Ctx

Site1 APIC view

name : L2
annotation : orchestrator:msc
bdEnforcedEnable : no
childAction :
On site 1 APIC Under the Dn of the VRF tn-RD- descr
dn
:
: uni/tn-RD-L2/ctx-L2
L2/ctx-L2 extMngdBy : msc
knwMcastAct : permit
We have fvRemoteId for site-2 lcOwn : local
Containing the remote vrf Vnid modTs : 2018-08-27T01:42:04.727+01:00
monPolDn : uni/tn-common/monepg-default
(called for whaterver reason remote PcTag pcEnfDir : ingress fvCtx
pcEnfDirUpdated : yes
pcEnfPref : enforced
pcTag : 32770
rn : ctx-L2
This is enough to push through Object model scope : 2457600 fvSiteAssociated
seg : 2162688
translation on spine
From Local VNID 2457600 to remote VNID site2 bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ctx-L2/stAsc/site-2
2162688 Total Objects shown: 1
fvRemoteId
# fv.RemoteId
This will be pushed to APIC in site1 here siteId : 2
Site-Id
childAction :
descr : Remote-Id
dn : uni/tn-RD-L2/ctx-L2/stAsc/site-2
lcOwn : local
modTs : 2018-05-03T03:14:40.895+00:00
monPolDn : uni/tn-common/monepg-default
Similar construct will exist on APIC site2 to push name
nameAlias
:
:
reverse translation on site2 spine ownerKey :
ownerTag :
remoteCtxPcTag : 32770
remotePcTag : 2162688 - Actually this is remote vrf PcTag
© 2019 Cisco and/or its affiliates. All rights reserved.
rn : site-2
status :
uid : 15374
 bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/

Logical BD – site 1
Total Objects shown: 1

bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web | egrep

"annot|dn|seg|scope"
annotation : orchestrator:msc
Similar construct for BD dn
scope
:
:
uni/tn-RD-L2/BD-Web
2457600
seg : 15204288

fvBD
Translation of BD vnid to spine site 1
From 15204288 to 150723234
fvSiteAssociated
bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/BD-Web/stAsc/site-2
Total Objects shown: 1

# fv.RemoteId
siteId : 2
childAction :
fvRemoteId descr :
dn : uni/tn-RD-L2/BD-Web/stAsc/site-2
lcOwn : local
modTs : 2018-05-03T03:14:40.895+00:00
Site 2 BD monPolDn : uni/tn-common/monepg-default
name :
bdsol-aci36-apic1# moquery -c fvBD -f 'fv.BD.seg nameAlias :
=="15073234"' | egrep "dn|scope|seg" ownerKey :
ownerTag :
dn : uni/tn-RD-L2/BD-Web remoteCtxPcTag : any
scope : 2162688 remotePcTag : 15073234 Actually remote BD VNID
seg : 15073234 rn : site-2
status :
© 2019 Cisco and/or its affiliates. All rights reserved. uid : 15374
 bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ap-App/epg-Web/stAsc/
Total Objects shown: 1

Logical AEPg – site 1

# fv.SiteAssociated
childAction :
descr :
dn : uni/tn-RD-L2/ap-App/epg-Web/stAsc/stAsc
lcOwn : local
modTs : 2018-05-03T03:14:39.572+00:00
In case of EPG we have the pcTag translation monPolDn : uni/tn-common/monepg-default
name : msc-local
nameAlias :
ownerKey :
fvAEPg ownerTag
rn
:
: stAsc
siteId : 1
status :
uid : 15374

fvSiteAssociated

bdsol-aci35-apic1# moquery -d uni/tn-RD-L2/ap-App/epg-

Web/stAsc/site-2
Total Objects shown: 1

fvPeerContext fvRemoteId # fv.RemoteId

siteId : 2
childAction :
descr :
dn : uni/tn-RD-L2/ap-App/epg-Web/stAsc/site-2

Site 2 EPG
lcOwn : local
modTs : 2018-05-03T03:14:40.895+00:00
monPolDn : uni/tn-common/monepg-default
name :
nameAlias :
bdsol-aci36-apic1# moquery -c fvAEPg -f 'fv.AEPg.pcTag ownerKey :
=="49155"' | egrep "dn|scope|pcTag" ownerTag :
dn : uni/tn-RD-L2/ap-App/epg-Web remoteCtxPcTag : any
pcTag : 49155 remotePcTag : 49155
rn : site-2
scope © 2019 Cisco and/or its affiliates.
: 2162688
All rights reserved. status :
uid : 15374
 Process involved for vnidmap/sclass/site-etep
on Spine only

This runs on
This run on sup-lc. Call
sup. Listen sdk hal api to
for events Sdk Programs the
Dcimgr Dcimc program site-
from PE and etep, Hal hardware
send it Dcimc vnidmap
on Lc sclassmap

DCImgr is the NXOS process

On spine that will program the
Translation based on the objectmodel

This is what you need to check first (easiest to check ☺)

© 2019 Cisco and/or its affiliates. All rights reserved.
Dcimgr/dcimc/sdkTraces for sclass/vnid map
• Dcimgr (on sup)
• show dcimgr internal event-history events

And log file :

pod35-spine1# ls -al /var/sysmgr/tmp_logs/dcimgr.log
-rw-rw-rw- 1 root root 3162338 May 2 16:37 /var/sysmgr/tmp_logs/dcimgr.log
pod35-spine1#

• HAL CLI :
module-2# show platform internal hal objects dci ?
all Dump All HAL objects
remotesite Remotesite or wan instance
remotesiteetep Unicast etep that belongs to this remotesite
remotevrfvnid Vrf for remotesite object
sclassmap Sclass mapping for remotesite vrf
vnidmap Vnid mapping for remotesite object
© 2019 Cisco and/or its affiliates. All rights reserved.
 DCI mgr – xlate

Vnid translate (vrf and bd) pcTag (sclass) translate

pod36-spine1# show dcimgr repo vnid-maps detail pod36-spine1# show dcimgr repo sclass-maps detail
-------------------------------------------------------------- ----------------------------------------------------------
Remote | Local Remote | Local
site Vrf Bd | Vrf Bd Rel-state site Vrf PcTag | Vrf PcTag Rel-state
-------------------------------------------------------------- ----------------------------------------------------------
1 2981888 | 2293760 [formed] 1 2981888 49153 | 2293760 49153 [formed]
0x2d8000 | 0x230000 0x2d8000 0xc001 | 0x230000 0xc001
-------------------------------------------------------------- -------------------------------------------------------------------
1 2981888 16678778 | 2293760 16154554 [formed] 1 2981888 49154 | 2293760 49155 [formed]
0x2d8000 0xfe7f7a | 0x230000 0xf67fba 0x2d8000 0xc002 | 0x230000 0xc003
-------------------------------------------------------------- -------------------------------------------------------------------
1 3014656 | 2457600 [formed] 1 2981888 16387 | 2293760 16386 [formed]
0x2e0000 | 0x258000 0x2d8000 0x4003 | 0x230000 0x4002
-------------------------------------------------------------------
1 3014656 49153 | 2457600 49153 [formed]
0x2e0000 0xc001 | 0x258000 0xc001
-------------------------------------------------------------------
1 3014656 16387 | 2457600 32772 [formed]
0x2e0000 0x4003 | 0x258000 0x8004

© 2019 Cisco and/or its affiliates. All rights reserved.

Unicast Control Plane
BGP route exchange detail

© 2019 Cisco and/or its affiliates. All rights reserved.

 ACI Multi-Site
Inter-Site MP-BGP EVPN Control Plane

S3-S4 Table S5-S8 Table

▪ MP-BGP EVPN used to communicate
EP1 Leaf 1 EP2 Leaf 4
Endpoint (EP) information across MP-BGP EVPN
EP2 DP-ETEP B EP1 DP-ETEP A
Sites
EP3 Leaf 4 EP4 Leaf 6
MP-iBGP or MP-EBGP peering supported
across sites IP
Network
Remote host route entries (EVPN Type-2)
are associated to the remote site Anycast DP-ETEP A DP-ETEP B
S2 S3 S4 S5 S6 S7 S8
DP-ETEP address S1

▪ Automatic filtering of endpoint COOP COOP

information across Sites ACI
Multi-Zone
Host routes are exchanged only if there is
EP2 EP4
a cross-site contract requiring EP1 EP3
Site 1 Site 2
communication between endpoints
Define and push inter-site policy
EP1 EP2
© 2019 Cisco and/or its affiliates. All rights reserved.
EPG
C EPG
19
19
MP-BGP in ACI summary
• So for we use BGP in infra (vrf overlay-1) for many reasons:
• Intra Pod : VPNv4 AF for L3 out routes only
• Multipod : VPNv4 AF for L3 out routes across Pod and l2vpn evpn type2
for End point synchro across Pod
• GOLF : l2vpn evpn (type2 and type5) for L3 subnet between GOLF and
ACI (routes in VRF)
• Multisite : L2vpn evpn for end point synchro across site

© 2019 Cisco and/or its affiliates. All rights reserved.

 Route-Target and Route Distinguisher in ACI
bdsol-aci32-leaf1# show bgp process vrf RD-BGP:RD

• For VPNv4 : ..
VRF RD : 10.0.88.95:6
Export RT list:
• RD is typically the PTEP of the origination:vrf_id 132:2654211
Import RT list:
• RT is typically the BGP_ASN:VRF_VNID 132:2654211

• In single Pod, or multipod, every switches have the same

RT and RD in the VRF so the route-target import/export
are done automatically
• And can be seen in show bgp process vrf XXX

• In multisite, RT will be different across site for the same vrf, so we must configure cross
route-target import/export for correct BGP path exchange to happen
• This is done using bgp EVI (EVPN instance) (show bgp internal evi XXXX) . Similar to pcTag
and VNID translation this is also pushed using object model
© 2019 Cisco and/or its affiliates. All rights reserved.
• Note that here we need route-target per BD (not per VRF)
BGP VNI
• Route Exchange issues can be seen either in the source or on the
remote site.
• Check if the BGP MOs are created for VNIs/RTs and RDs correctly. These
MOs are created only on spines in every site. These MOs are created
when the VRF/BD/EPGs are stretched or the contracts are created at EPG
level
• Following shows mapping of BGP VNIDs and what routes are
requested from COOP and why they are used:

© 2019 Cisco and/or its affiliates. All rights reserved.

BGP Route Target for a BD
Site 1 spine Import RT Site 2 spine Import RT
# bgp.RttEntry # bgp.RttEntry
rtt : route-target:as2-nn4:136:49676223 rtt : route-target:as2-nn4:135:33128354
childAction : childAction :
dn : sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan- dn : sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-2457600]-
3014656]-bd-[vxlan-16351138]-epg-[unknown]/rtp-import/ent-route- bd-[vxlan-16121791]-epg-[unknown]/rtp-import/ent-route-target:as2-
target:as2-nn4:136:49676223 nn4:135:33128354
lcOwn : local lcOwn : local
modTs : 2018-04-11T04:28:21.600+00:00 modTs : 2018-04-11T04:28:16.142+00:00
rn : ent-route-target:as2-nn4:136:49676223 rn : ent-route-target:as2-nn4:135:33128354
status : status :

Site 1 spine export RT Site 2 spine export RT

# bgp.RttEntry # bgp.RttEntry
rtt : route-target:as2-nn4:135:33128354 rtt : route-target:as2-nn4:136:49676223
childAction : childAction :
dn : sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan- dn : sys/bgp/inst/encapgroupevi-1/vni-bd-vrf-[vxlan-2457600]-
3014656]-bd-[vxlan-16351138]-epg-[unknown]/rtp-export/ent-route- bd-[vxlan-16121791]-epg-[unknown]/rtp-export/ent-route-target:as2-
target:as2-nn4:135:33128354 nn4:136:49676223
lcOwn : local lcOwn : local
modTs : 2018-04-11T04:28:21.600+00:00 modTs : 2018-04-11T04:28:16.142+00:00
rn : ent-route-target:as2-nn4:135:33128354 rn : ent-route-target:as2-nn4:136:49676223
status : status :

© 2019 Cisco and/or its affiliates. All rights reserved.

BGP EVI check (NXOS) – BD on site 1 ++++++++++++++++++++++++++++++++++++++++++

Use show bgp internal evi xx to verify

BGP VNI Information for vni_16351138
L2VNI ID : 16351138 (vni_16351138)
RD : 1:33128354
RD and RT exp/import (where xx is BD VNID) VRF Vnid : 3014656
Prefixes (local/total) : 2/2
(kind of similar to show bgp process for GOLF) VNID registered with COOP
Enabled
: Yes
: Yes
Delete pending : 0
Stale : No
pod35-spine1# show bgp internal evi 16351138 Import pending : 0
Import in progress : 0
.. Encap : VxLAN
************************************************* Topo Id : 16351138
BGP L2VPN/EVPN RD Information for 1:33128354 VTEP IP : 0.0.0.0
L2VNI ID : 16351138 (vni_16351138) VTEP VPC IP : 0.0.0.0
#Prefixes Local/BRIB : 2 / 2 Active Export RTs : 1
#Paths L3VPN->EVPN/EVPN->L3VPN : 0 / 0 Active Export RT list : 135:33128354
************************************************* Config Export RTs : 1
============================================== Export RT cfg list: 135:33128354(refcount:1
BGP Configured VNI Information: Export RT chg/chg-pending : 0/0
VNI ID (Index) : 16351138 (0) Active Import RTs : 1
RD : 1:33128354 Active Import RT list : 136:49676223
Export RTs : 1 Config Import RTs : 1
Export RT cfg list: 135:33128354(refcount:1 Import RT cfg list: 136:49676223(refcount:1
Import RTs : 1 Import RT chg/chg-pending : 0/0
Import RT cfg list: 136:49676223(refcount:1 IMET Reg/Unreg from L2RIB : 1/0
Topo Id : 16351138 MAC Reg/Unreg from L2RIB : 1/0
VTEP IP : 0.0.0.0 MAC IP Reg/Unreg from L2RIB : 1/0
VTEP VPC IP : 0.0.0.0 IP-only Reg/Unreg from L2RIB : 0/0
Enabled : Yes SMAD Reg/Unreg from L2RIB : 1/0
Delete Pending : No IMET Add/Del from L2RIB : 0/0
RD/Import RT/Export RT : Yes/Yes/Yes MAC Add/Del from L2RIB : 3/2
Type : 3 MAC IP Add/Del from L2RIB : 3/2
Usage : 2 SMAD Add/Del from L2RIB : 0/0
L2 stretch enabled : 1 IMET Dnld/Wdraw to L2RIB : 0/0
VRF Vnid : 3014656 IMET Dnld/Wdraw to L2RIB failures : 0/0
Refcount : 00000003 MAC Dnld/Wdraw to L2RIB : 0/0
Encap : VxLAN MAC Dnld/Wdraw to L2RIB failures : 0/0
SMAD Dnld/Wdraw to L2RIB : 0/0
============================================== SMAD Dnld/Wdraw to L2RIB failures : 0/0
++++++++++++++++++++++++++++++++++++++++++ MAC-IP/SMAD Msite-RD routes : 2

Note the EVI number if the BD VNID we are looking for

© 2019 Cisco and/or its affiliates. All rights reserved.
MAC-IP WAN-RD routes : 0
MAC-IP network host routes : 0
Type : 3
Unicast forwarding across site

© 2019 Cisco and/or its affiliates. All rights reserved.

 Overview
1. Unicast TX proxy/(local to remote site)
Leaf has not learned the remote site ep. Leaf sends the traffic to local spine proxy. Local spine looks up the route.
The route for remote site Ep is programmed with next hop of remote site’s ETEP. Dipo is re-written with remote site
ETEP. Sipo is re-written with local site ETEP

2. Unicast TX (local to remote site)

Leaf has learned the remote site ep against remote site ETEP. Leaf sends the traffic to remote site ETEP. Local site
spine will intercept this packet and re-write the sipo with Local site ETEP

3. Unicast RX (remote to local site)

Incoming traffic destined to the local site’s unicast ETEP goes through vnid and sclass translations. The receiving spine
looks up the route for destination ep and sends the traffic to correct leaf.

1. Proxy – Spine COOP lookup 3. In all case Rx on spine does vnid/sclass translation
SIP and DIP outer rewritten IP Network

1 3
MP-BGP - EVPN

2
2. Known EP – EPM lookup on leaf
© 2019 Cisco and/or its affiliates. All rights reserved.
DIP outer set on lef
SIP outer Rewritten when passing by ingress spine
LAB Stretched VRF ACI Multi-Site

APIC Site 1 APIC Site 2

Pod35 Pod36
Tenant IPA

BD GW
Aci-35-interconnect 172.16.2.254/24 VRF DC:DC1
BD GW
1/51-52 172.16.4.254/24 BD2 172.16.2.54/24
172.16.1.254/24 1/49-50 Route
BD1 172.16.1.54/24
Web-EPG2
172.16.3.254/24 172.16.[1-4].0/24
Web-EPG1
Route 2/5-6 2/5-6
172.16.[1-4].0/24
aci35-spine1
aci36-spine1
C1 C1 C2 C2

BD3 172.16.3.254/24 BD4 172.16.4.254/24

2/1-2 2/1-2
App-EPG1 App-EPG2

1/49-50 1/49-50

aci35-leaf1 aci36-leaf1
Test :
172.16.3.2 to 172.16.2.2

Web-EPG1 172.16.1.1/24 vlan-101 Web-EPG2 172.16.2.2/24 VM

App-EPG1 172.16.3.2/24 VM App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

 Control Plane EP –in site 2 to COOP in site 1
BGP EVPN type2
For 172.16.2.2 Control plane to reach 172.16.2.2
Pod35 Pod36 BD GW
Aci-35-interconnect 172.16.2.254/24
Note that dataplane to 172.16.2.2
BD GW
1/51-52 172.16.4.254/24
172.16.1.254/24 1/49-50 Route
172.16.3.254/24
Route
172.16.[1-4].0/24
2/5-6 2/5-6
172.16.[1-4].0/24
(site1 to site2 is translated in site2
aci35-spine1
aci36-spine1 spine
BGP to COOP 2/1-2 2/1-2
172.16.2.2 to DP-ETEP Site 2
Leaf to spine COOP

1/49-50 1/49-50 Epm local Learn

aci35-leaf1 aci36-leaf1

Web-EPG1 172.16.1.1/24 vlan-101 Web-EPG2 172.16.2.2/24 VM

App-EPG1 172.16.3.2/24 VM App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

 Control plane EP in Site 2 Local COOP site 2
Publisher id is the local leaf in site2
Local EPM pod36-spine1# show coop internal info ip-db key 2457600 172.16.2.2

IP address : 172.16.2.2
Vrf : 2457600
pod36-leaf1# show system internal epm endpoint ip 172.16.2.2 Flags : 0
EP bd vnid : 16220082
MAC : 0050.56b1.4403 ::: Num IPs : 1 EP mac : 00:50:56:B1:44:03
IP# 0 : 172.16.2.2 ::: IP# 0 flags : Publisher Id : 10.1.48.64
Vlan id : 21 ::: Vlan vnid : 8194 ::: VRF name : DC:DC1 Record timestamp : 05 02 2018 02:29:12 339899902
BD vnid : 16220082 ::: VRF vnid : 2457600 Publish timestamp : 05 02 2018 02:29:12 340145880
Phy If : 0x1a001000 ::: Tunnel If : 0 Seq No: 0
Interface : Ethernet1/2 Remote publish timestamp: 01 01 1970 00:00:00 0
Flags : 0x80004c04 ::: sclass : 32771 ::: Ref count : 5 URIB Tunnel Info
EP Create Timestamp : 04/19/2018 07:03:23.999543 Num tunnels : 1
EP Update Timestamp : 05/02/2018 02:33:29.507208 Tunnel address : 10.1.48.64
EP Flags : local|IP|MAC|sclass|timer| Tunnel ref count : 1
:::: Remote COOP entry site 1
Extract BGP table site 2 Publisher id is the spine DP TEP in site 2
pod36-spine1# show bgp l2vpn evpn vrf overlay-1 | egrep "Route Dis|172.16.2.2\]” pod35-spine1# show coop internal info ip-db | egrep -A
Route Distinguisher: 1:49774514 (L2VNI 16220082) 15 -B 1 "172.16.2.2$"
*>l[2]:[0]:[16220082]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272 ------------------------------
Route Distinguisher: 10.10.35.102:136 (L2VNI 1) IP address : 172.16.2.2
*>l[2]:[0]:[16220082]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272 Vrf : 3014656
Flags : 0x4
EP bd vnid : 15925206
Extract BGP table site 1 EP mac : 00:50:56:B1:44:03
Publisher Id : 10.10.35.102
pod35-spine1# show bgp l2vpn evpn vrf overlay-1 | egrep "Route Dis|172.16.2.2\]" Record timestamp : 01 01 1970 00:00:00 0
Publish timestamp : 01 01 1970 00:00:00 0
Route Distinguisher: 1:49774514 Seq No: 0
*>e[2]:[0]:[0]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272 Remote publish timestamp: 04 24 2018 05:05:34 611613733
Route Distinguisher: 1:32702422 (L2VNI 15925206) URIB Tunnel Info
*>e[2]:[0]:[15925206]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272 Num tunnels : 1
© 2019 Cisco and/or its affiliates. All rights reserved.
Route Distinguisher: 10.10.35.101:135 (L2VNI 1) Tunnel address : 10.10.35.102
*>e[2]:[0]:[15925206]:[48]:[0050.56b1.4403]:[32]:[172.16.2.2]/272 Tunnel ref count : 1
 DCI Mgr on spine pod 36 (site 2)
Remote Site pod36-spine1# moquery -c dciAnycastExtn
DP-ETEP and Total Objects shown: 2

Mcast ETEP(dcimgr and Object model) # dci.AnycastExtn

etep : 10.10.35.101/32
childAction :
dn : sys/inst-overlay-1/remoteSite-1/anycastExtn-[10.10.35.101/32]
is_ucast : yes
pod36-spine1# show dcimgr repo eteps
lcOwn : local
modTs : 2018-03-30T05:50:34.562+00:00
Remote site=1 :
rn : anycastExtn-[10.10.35.101/32]
Rem Etep=10.10.35.101/32, is_ucast=yes
status :
Rem Etep=10.10.35.121/32, is_ucast=no
pod36-spine1#
# dci.AnycastExtn
etep : 10.10.35.121/32
childAction :
dn : sys/inst-overlay-1/remoteSite-1/anycastExtn-[10.10.35.121/32]
is_ucast : no
lcOwn : local
modTs : 2018-03-30T05:50:34.562+00:00
rn : anycastExtn-[10.10.35.121/32]
status :

© 2019 Cisco and/or its affiliates. All rights reserved.

 DCI Mgr on spine pod 36 (site 2) – VNID MAP

Aci-35-interconnect
DCI mgr vnid map 1/49-50
1/51-52

pod35-spine1# show dcimgr repo vnid-maps 2/5-6 2/5-6

-------------------------------------------------------------- aci35-spine1
Remote | Local aci36-spine1
site Vrf Bd | Vrf Bd Rel-state
--------------------------------------------------------------
1 3014656 | 2457600 [formed] 2/1-2 2/1-2
1 3014656 16056263 | 2457600 16121790 [formed]
1 3014656 16351138 | 2457600 16121791 [formed]
1 3014656 15925206 | 2457600 16220082 [formed]
1 3014656 16056262 | 2457600 15794151 [formed]
1/49-50 1/49-50

aci35-leaf1 aci36-leaf1

Translation for packet to 172.16.2.2

Web-EPG1 172.16.1.1/24 vlan-101 Web-EPG2 172.16.2.2/24 VM
App-EPG2 172.16.4.1/24 vlan-104
Here packet received on site2 From site1 App-EPG1 172.16.3.2/24 VM

(l2 case)with BD VNID 159250206 will be Xlated to 16220082

(L3 case) with
© 2019 Cisco and/or VRF VNID
its affiliates. 3014656 will be xlated to 2457600
All rights reserved.
 DCI Mgr on spine pod 36 (site 2) – SCLASS MAP

DCI mgr vnid map

pod36-spine1# show dcimgr repo sclass-maps
----------------------------------------------------------
Remote | Local
site Vrf PcTag | Vrf PcTag Rel-state
----------------------------------------------------------
1 3014656 49153 | 2457600 49153 [formed]
1 3014656 16387 | 2457600 32772 [formed]
1 3014656 16388 | 2457600 16387 [formed]
1 3014656 32770 | 2457600 16390 [formed]
1 3014656 32772 | 2457600 32771 [formed]

© 2019 Cisco and/or its affiliates. All rights reserved.

ivxlan header review

Note in Outer L4 header you can

Get :
VNID (BD or VRF)
Sclass (src sclass) as part of
Nounce field (last 4 nibble):

Ex :
hom_elam_in_l4v_tn.tn_nonce_info: 0x188002

Sclass of Rx frame is 0x8002

© 2019 Cisco and/or its affiliates. All rights reserved.

Data path – known EP Site 1 to Site 2 (Known
unicast on ingress leaf)
Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep)
Outer L4 : site1 vrf VNID – Sclass App-EPG1
Pod35 Inner : 172.16.3.2 to 172.16.2.2 Pod36
Aci-35-interconnect
1/51-52
1/49-50

Spine just RW Outer Sip 2/5-6 2/5-6

aci35-spine1 Outer Dst IP is my DP-etep  DCI-rx
 Translate Vnid and sclass
aci36-spine1
Send to pod36-leaf1 tep per coop

2/1-2 2/1-2
Outer : pod35-leaf1 PTEP to 10.10.35.102 (site2 dp-etep) Outer : 10.10.35.101 (site1 DP-ETEP) to pod36-leaf1 PTEP
Outer L4 : site1 vrf VNID – Sclas App-EPG1 Outer L4 : site2 vrf VNID – Sclass App-EPG1 Translated
Inner : 172.16.3.2 to 172.16.2.2 Inner : 172.16.3.2 to 172.16.2.2

Epm entry for Dest 1/49-50 1/49-50

going to tunnel to EPM learning for 172.16.3.2 to tunnel to site1 DP-ETEP
aci35-leaf1
aci36-spine DP-ETEP aci36-leaf1

Web-EPG1 172.16.1.1/24 vlan-101 Web-EPG2 172.16.2.2/24 VM

App-EPG1 172.16.3.2/24 VM App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

Ingress Leaf Known EP
Pod35 Pod36
Aci-35-interconnect
1/49-50 1/51-52
pod35-leaf1# show system internal epm endpoint ip 172.16.2.2
2/5-6 2/5-6
aci35-spine1
aci36-spine1 MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 172.16.2.2 ::: IP# 0 flags :
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : DC:DC1
2/1-2 2/1-2
BD vnid : 0 ::: VRF vnid : 3014656
Phy If : 0 ::: Tunnel If : 0x18010007
Interface : Tunnel7
1/49-50 1/49-50 Flags : 0x80004400 ::: sclass : 32772 ::: Ref count : 3
aci35-leaf1 EP Create Timestamp : 04/24/2018 05:05:32.831665
aci36-leaf1
EP Update Timestamp : 04/25/2018 04:58:50.374323
EP Flags : IP|sclass|timer|
::::

pod35-leaf1# show interface tunnel 7

Tunnel7 is up
MTU 9000 bytes, BW 0 Kbit
Transport protocol is in VRF "overlay-1"
Tunnel protocol/transport is ivxlan
Tunnel source 10.0.112.64/32 (lo0)
Tunnel destination 10.10.35.102/32
Last clearing of "show interface" counters never
Tx
0 packets output, 1 minute output rate 0 packets/sec
Rx
© 2019 Cisco and/or its affiliates. All rights reserved. 0 packets input, 1 minute input rate 0 packets/sec
ELAM Ingress LC Spine Site 1 – EP known
module-2# debug platform internal roc elam asic 0
module-2(DBG-elam)# trigger init in-select 14 out-select 1
module-2(DBG-elam-insel15)# set inner ipv4 src_ip 172.16.3.2 dst_ip 172.16.2.2

#########################HOMEWOOD ELAM REPORT START#########################

Dumping report for asic type 8 inst 0 slice 0 a_to_d 1 insel 15 outsel 1
Pod35 Pod36
LUA captured data with :
Aci-35-interconnect
SRCID: 20
1/49-50 1/51-52
*** Parsed Outer l2 vector
2/5-6 2/5-6 hom_elam_in_l2v_da_sa_qtag0.qtag0_vlan: 0x2
aci35-spine1
aci36-spine1 *** Parsed Outer l3 vector
hom_elam_in_l3v_ipv4.da: 0xA0A2366 - 10.0.35.102 (Dp-ETEP site2)
2/1-2 2/1-2 hom_elam_in_l3v_ipv4.sa: 0xA007040 - 10.0.112.64 (leaf1 pod35 PTEP)
*** Parsed Outer l4 vector
hom_elam_in_l4v_tn.tn_seg_id: 0x2E0000 - 3014656
1/49-50 1/49-50
hom_elam_in_l4v_tn.tn_nonce_info: 0x8002
aci35-leaf1 aci36-leaf1
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_rx: 0x0
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_tx: 0x0

module-2(DBG-elam-insel15)# show platform internal hal l2 port gpd | egrep "Eth2/1|==|IfId|Uc|Xla"

============================================================================================================================= ===============
Uc Uc | Reprogram | | Rep |
I PC Pc L | R I R D R U U X | L Xla Ovx N NI Vif RwV Ing Egr | V R | PROF H
IfId Ifname P Cfg MbrID As AP Sl Sp Ss Ovec S | P P P S P Sp Sp C M L | 3 Idx Idx L3 L3 Tid Tid Lbl Lbl | S V | ID I
============================================================================================================================= ================
1a080000 Eth2/1 0 9a 28 0 11 0 10 20 20 1 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 D -f3 D-61 100 0 0 0 4 0

pod35-spine1# show lldp neighbors | egrep "Eth2/1"

pod35-leaf1 Eth2/1 120 BR Eth1/49
pod35-spine1#

© 2019 Cisco and/or its affiliates. All rights reserved.

ELAM Ingress LC Spine Site 2 – Proxy
module-2# debug platform internal roc elam asic 0
module-2(DBG-elam)# trigger reset
module-2(DBG-elam)# trigger init in-select 15 out-select 1
module-2(DBG-elam-insel15)# set inner ipv4 src_ip 172.16.3.2 dst_ip 172.16.2.2

#########################HOMEWOOD ELAM REPORT START#########################

Pod35 Pod36 Dumping report for asic type 8 inst 0 slice 0 a_to_d 1 insel 15 outsel 1
Aci-35-interconnect LUA captured data with :
1/51-52 SRCID: 0
1/49-50
*** Parsed Outer l2 vector
2/5-6 2/5-6
aci35-spine1 hom_elam_in_l2v_da_sa_qtag0.qtag0_vlan: 0x4
aci36-spine1
*** Parsed Outer l3 vector
hom_elam_in_l3v_ipv6_da_only.da: 0x000000000000000000A0A2366 – 10.10.35.102 (site2 – DP-ETEP)
2/1-2 2/1-2 hom_elam_in_l3v_ipv6_da_only.sa: 0xA0A2365 - 10.10.35.101 (site1 – DP-ETEP)
*** Parsed Outer l4 vector
hom_elam_in_l4v_tn.tn_nonce_info: 0x188002 - Rx sclass is 0x8002 = 16387
1/49-50 1/49-50 hom_elam_in_l4v_tn.tn_seg_id: 0x2E0000 - 3014656 (vnid before rewrite)
aci35-leaf1 aci36-leaf1

hom_elam_out_sidebnd_no_spare_vec.ovector_idx: 0x78 (useless internal port to FC)

hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_rx: 0x1
hom_lua_latch_results_vec.lua4_1.lux_ispine_dci_tx: 0x0

hom_lurw_vec.info.ifabric_spine.vnid: 0x258000 - Vnid after rewrite = 2457600

hom_lurw_vec.info.ifabric_spine.sclass: 0x4006 - rewritten Sclass is 16390

======== lux_fwd_mode = 0x09516040

LUX_FWD_MODE: ISPINE_LC bit is set ingress LC
LUX_FWD_MODE: ISPINE_DCI bit is set
..

pod36-spine1# show dcimgr repo sclass-maps | egrep "3014656.*16387"

1 3014656 16387 | 2457600 32772 [formed]
© 2019 Cisco and/or
pod36-spine1# show its affiliates.
dcimgr All rights
repo reserved. | egrep 3014656
vnid-maps
1 3014656 | 2457600 [formed]
 Data path – unknown EP on leaf Site 1 to Site 2 -
Proxy Only differences is that ingress
Outer : 10.10.35.101 (site1 DP-ETEP to 10.10.35.102 (site2 dp-etep) leaf does send to Local Proxy
Outer L4 : site1 vrf VNID – Sclass App-EPG1 spine (like in single site pod
Pod35 Inner : 172.16.3.2 to 172.16.2.2 Pod36 case). Ingress spine does Rw
Aci-35-interconnect
1/51-52 Outer IP
1/49-50

Spine just RW Outer Sip 2/5-6 2/5-6

And Outer Dest IP Outer Src IP is my DP-etep  DCI-rx
aci35-spine1
 Translate Vnid and sclass
aci36-spine1
Send to pod36-leaf1 tep per coop

2/1-2 2/1-2
Outer : pod35-leaf1 PTEP to 10.0.88.66 (site1 Outer : 10.10.35.101 (site1 DP-ETEP) to pod36-leaf1 PTEP
anycast proxy) Outer L4 : site2 vrf VNID – Sclass App-EPG1 Translated
Outer L4 : site1 vrf VNID – Sclas App-EPG1 Inner : 172.16.3.2 to 172.16.2.2
Inner : 172.16.3.2 to 172.16.2.2
No EPM entry relying1/49-50
on 1/49-50
BD subnet route to EPM learning for 172.16.3.2 to tunnel to site1 DP-ETEP
SPine aci35-leaf1 aci36-leaf1

Web-EPG1 172.16.1.1/24 vlan-101 Web-EPG2 172.16.2.2/24 VM

App-EPG1 172.16.3.2/24 VM App-EPG2 172.16.4.1/24 vlan-104

© 2019 Cisco and/or its affiliates. All rights reserved.

Policy enforcement

© 2019 Cisco and/or its affiliates. All rights reserved.

Sclass Translation

Pod35 Pod36
Aci-35-interconnect
DCI mgr translation DCI mgr translation
Vrf vnid 2457600 -> 3014656 1/49-50
1/51-52
Vrf vnid 3014656 -> 2457600
Sclass 32771 -> 32772 Sclass 32770 -> 16390
2/5-6 2/5-6
aci35-spine1
aci36-spine1
Policy Enforcement
- Ingress leaf derives sclass and vnid based on local EPM
2/1-2 2/1-2
- If Remote EPM is populated – Enforce Policy (as usual)

- Transmit to Remote Spine Site

1/49-50 1/49-50
- Remote spine site translate sclass and VNId
- - sent it to Dest leaf
aci35-leaf1 aci36-leaf1

- Dest leaf learn remote EP entry in translated sclass

- Enforce policy if not done on ingress

App-EPG1 172.16.3.2/24 VM Web-EPG2 172.16.2.2/24 VM

Sclass 32770 in VRF vnid 3014656 Sclass 32771 in vrf VNID 2457600
© 2019 Cisco and/or its affiliates. All rights reserved.
Policy from Site 2 (172.16.2.2) to Site 1 (172.16.3.2)
pod35-leaf1# show system internal epm endpoint ip 172.16.3.2
Packet Data in IPN MAC : 0050.56b1.4b52 ::: Num IPs : 1
IP# 0 : 172.16.3.2 ::: IP# 0 flags :
VNID 2457600 sclass 32771 BD vnid : 16351138 ::: VRF vnid : 3014656
Phy If : 0x1a001000 ::: Tunnel If : 0
Flags : 0x80004c04 ::: sclass : 32770 ::: Ref count : 5 - Local EP learn with 32770

Aci-35-interconnect pod35-leaf1# show system internal epm endpoint ip 172.16.2.2

DCI mgr translation MAC : 0000.0000.0000 ::: Num IPs : 1
Vrf vnid 2457600 -> 3014656 1/49-50
1/51-52 IP# 0 : 172.16.2.2 ::: IP# 0 flags :
Sclass 32771 -> 32772 BD vnid : 0 ::: VRF vnid : 3014656
Phy If : 0 ::: Tunnel If : 0x18010007
2/5-6 2/5-6
Interface : Tunnel7
aci35-spine1 Flags : 0x80004400 ::: sclass : 32772 ::: Ref count : 3 - We learn Remote EP with translated Sclass
aci36-spine1
pod35-leaf1# show zoning-rule| egrep "32770.*3014656"
4123 32770 32772 10 enabled 3014656 permit fully_qual(7)
4124 32772 32770 10 enabled 3014656 permit fully_qual(7)
2/1-2 2/1-2 pod35-leaf1# show system internal policy-mgr stats | egrep "3014656.*32770“
Rule(4123)DN(sys/actrl/scope-3014656/rule-3014656-s-32770-d-32772-f-10) , Pkts: 495659 RevPkts: 0
Rule(4124)DN(sys/actrl/scope-3014656/rule-3014656-s-32772-d-32770-f-10) , Pkts: 6 RevPkts: 0
Packet Data in site1
VNID 3014656 sclass 32772 Packet enforcement is mostly done in Ingress (if XR remote EP is learn), in egress otherwise

pod35-spine1# show dcimgr repo sclass-maps | egrep

1/49-50 1/49-50
"Remote|Vrf|32771"
Remote | Local
aci35-leaf1 aci36-leaf1 site Vrf PcTag | Vrf PcTag Rel-state
2 2457600 32771 | 3014656 32772 [formed]

pod36-leaf1# show system internal epm endpoint ip 172.16.2.2

MAC : 0050.56b1.4403 ::: Num IPs : 1

IP# 0 : 172.16.2.2 ::: IP# 0 flags :
Vlan id : 21 ::: Vlan vnid : 8194 ::: VRF name : DC:DC1
BD vnid : 16220082 ::: VRF vnid : 2457600
App-EPG1 172.16.3.2/24 VM Web-EPG2 172.16.2.2/24 VM
Phy If : 0x1a001000 ::: Tunnel If
If :: 00
Sclass 32770 in VRF vnid 3014656 Sclass 32771 in vrf VNID 2457600 Interface : Ethernet1/2
Flags : 0x80004c04 ::: sclass : 32771 ::: Ref count : 5
© 2019 Cisco and/or its affiliates. All rights reserved.
Multicast Multisite

© 2019 Cisco and/or its affiliates. All rights reserved.

Overview - Layer 2 BUM traffic across Sites
• TX (local to remote site)
• GIPo (BUM) traffic sourced from the local site is Head-end replicated (HREP)
to each remote site from the Spine. DIPo is rewritten to a unicast address
called as Multicast HREP TEP IP (also called Multicast DP-TEP IP) of the
remote site. SIPo is rewritten with the Unicast ETEP IP

• RX (remote to local site)

• Incoming traffic destined to the local site’s Multicast HREP TEP IP gets
translated, derives the local site’s BD-GIPo, and follows the regular GIPo
lookup path from there

© 2019 Cisco and/or its affiliates. All rights reserved.

Multi-Site
Stretched BD with L2 Broadcast Extension

ACI Multi-Site

L2 flooding

Use Case Properties

APIC Site 1 APIC Site 2 ▪ Active/Active deployment with inter-site Layer 2
extension
Tenant IPA
▪ Objects stretched across sites:
VRF Stone-IPA • Tenant ID
BD1/Subnet1 • VRF context
Web-EPG
• BD/Subnet
• Provider and Consumer EPGs
• Policy between EPGs
C1
▪ L2 flooding enabled at the BD level
BD2/Subnet2
• L2 BUM traffic forwarded over head-end
App-EPG replicated VXLAN tunnels

© 2019 Cisco and/or its affiliates. All rights reserved.

Use case – lab VRF RD-L2:L2 ACI Multi-Site

L2 flooding

Pod35 Pod36 APIC Site 1 APIC Site 2

Aci-35-interconnect

BD GW 1/49-50
1/51-52 BD GW Tenant RD-L2
10.1.1.254/24 10.1.1.254/24
10.2.2.254/24 2/5-6 2/5-6 10.2.2.254/24 VRF L2
aci35-spine1 BD1/10.1.1.254/24
aci36-spine1
Web-EPG

2/1-2 2/1-2 Test

BD2/10.2.2.254/24
App-EPG

1/49-50 1/49-50

aci35-leaf1 aci36-leaf1

Vm 10.1.1.35 Vm 10.1.1.36

© 2019 Cisco and/or its affiliates. All rights reserved.

Config Check
admin@bdsol-aci35-apic1:~> moquery -d uni/tn-RD-L2/BD-Web
Total Objects shown: 1

# fv.BD
name : Web
OptimizeWanBandwidth : yes
arpFlood : yes
bcastP : 225.0.216.80
• BD must be set with intersite childAction
configIssues
:
:

BUM allow flag descr

dn
epClear
:
:
:
uni/tn-RD-L2/BD-Web
no
epMoveDetectMode :
extMngdBy : msc
intersiteBumTrafficAllow : yes
intersiteL2Stretch : yes
ipLearning : yes
lcOwn : local
limitIpLearnToSubnets : yes
llAddr : ::
mac : 00:22:BD:F8:19:FF
mcastAllow : no
modTs : 2018-05-03T03:14:39.650+00:00
monPolDn : uni/tn-common/monepg-default
mtu : inherit
multiDstPktAct : bd-flood
nameAlias :
ownerKey :
ownerTag :
pcTag : 32770
rn : BD-Web
scope : 2457600
seg : 15204288
status :
type : regular
uid : 15374
unicastRoute : yes
unkMacUcastAct : flood
unkMcastAct : flood
© 2019 Cisco and/or its affiliates. All rights reserved.
vmac : not-applicable
 Config check
• Multicast HREP TEP IP per Site
• Tunnel to each Remote site’s Multicast HREP TEP

pod35-spine1# show ip interface vrf overlay-1 | egrep -A 1 mcast-hrep

loopback14, Interface status: protocol-up/link-up/admin-up, iod: 120, mode: mcast-hrep, vrf_vnid: 16777199
IP address: 10.10.35.121, IP subnet: 10.10.35.121/32

pod35-spine1# show interface tunnel 5

Tunnel5 is up
MTU 9000 bytes, BW 9 Kbit
Transport protocol is in VRF "overlay-1"
Tunnel protocol/transport ivxlan
Tunnel source 10.0.112.65, destination 10.10.35.122
© 2019 Cisco and/or its affiliates. All rights reserved.
 Control Plane interaction
• ISIS
• For the stretched BDs (with intersiteBUMTrafficAllow), based on HREP-TEP configuration,
ISIS adds the Remote site’s HREP Tunnel If to the BD-GIPO of the Stretched BD.
• BD-GIPOs are striped across the Multisite-capable Spines – meaning HREP Tunnel If is
added to BD-GIPo only on one of the Multi-site capable Spines in a site
• Unlike Multi-pod, no IGMP joins are sent out towards IPN, since native multicast is not
used for forwarding BUM traffic across the sites
pod35-spine1# show isis internal mcast routes gipo | egrep -A 6 "225.0.216.80"
GIPo: 225.0.216.80 [LOCAL] One spine per site
OIF List:
Ethernet2/1.35
Should have Tunnel
Ethernet2/2.36
Tunnel5
Interface as BD GIPo
OIL

© 2019 Cisco and/or its affiliates. All rights reserved.

Use case – lab VRF RD-L2:L2 ACI Multi-Site

L2 flooding

Pod35 Pod36 APIC Site 1 APIC Site 2

Aci-35-interconnect

BD GW 1/49-50
1/51-52 BD GW Tenant RD-L2
10.1.1.254/24 10.1.1.254/24
10.2.2.254/24 2/5-6 2/5-6 10.2.2.254/24 VRF L2
aci35-spine1 BD1/10.1.1.254/24
aci36-spine1
Web-EPG

2/1-2 2/1-2 Test

BD2/10.2.2.254/24
App-EPG

1/49-50 1/49-50

aci35-leaf1 aci36-leaf1

Lab Test ARP broadcast

Vm 10.1.1.35 Vm 10.1.1.36
From 10.1.1.35 to 10.1.1.32
© 2019 Cisco and/or its affiliates. All rights reserved.
Use case – lab VRF RD-L2:L2
Outer : 10.10.35.101 to 10.10.35.122 (site2
dci-mcast) VNID 0xE7FFC0
Pod35 Pod36
Aci-35-interconnect
Inner : arp from 10.1.1.25 to 10.1.1.32

BD GW 1/49-50
1/51-52 BD GW
10.1.1.254/24 10.1.1.254/24
10.2.2.254/24 2/5-6 2/5-6 10.2.2.254/24
aci35-spine1
aci36-spine1

2/1-2 2/1-2 Outer : 10.10.35.101 to 225.0.191.0 + ftag

(site2 gipo) VNID 0xe5ffd2
Outer : pod35-leaf1 PTEP to GIPo (site1
Inner : arp from 10.1.1.25 to 10.1.1.32
225.0.216.90 +FTAG) VNID 0xE7FFC0
Inner : arp from 10.1.1.25
1/49-50 to 10.1.1.32 1/49-50

aci35-leaf1 aci36-leaf1

Lab Test ARP broadcast

Vm 10.1.1.35 Vm 10.1.1.36
From 10.1.1.35 to 10.1.1.32
© 2019 Cisco and/or its affiliates. All rights reserved.
GIPo route on line card site 1 spine

module-2# show forwarding multicast route group 225.0.216.80 vrf all

(*, 225.0.216.80/32), RPF Interface: NULL, flags: Dc

Received Packets: 0 Bytes: 0
Number of Outgoing Interfaces: 3
Outgoing Interface List Index: 15
Ethernet2/1.35 Outgoing Packets:0 Bytes:0
Ethernet2/2.36 Outgoing Packets:0 Bytes:0
Tunnel5 Outgoing Packets:0 Bytes:0

© 2019 Cisco and/or its affiliates. All rights reserved.

L3 out and Multisite

© 2019 Cisco and/or its affiliates. All rights reserved.

Use case 3 – lab VRF RD-L2:L2

Pod35 Pod36
Aci-35-interconnect
BD GW BD GW
10.1.1.254/24 1/51-52 10.1.1.254/24
1/49-50
10.2.2.254/24 10.2.2.254/24
2/5-6 2/5-6
aci35-spine1
aci36-spine1

2/1-2 2/1-2

1/49-50 1/49-50

aci35-leaf1 aci36-leaf1

OSPF OSPF

Vm 10.1.1.35 Vm 10.1.1.36
Lo1 - 10.30.1.1/24 Lo1 10.30.2.1/24

© 2019 Cisco and/or its affiliates. All rights reserved.

 L3 out – lab VRF RD-L2:L2
• Working Session :

• in EPG web : 10.1.1.35 to 10.1.1.36

Pod35 Pod36 • Local L3 out :

Aci-35-interconnect 10.1.1.35 to 10.30.1.1
BD GW BD GW •

10.1.1.254/24 1/51-52 10.1.1.254/24

1/49-50
10.2.2.254/24 • 10.1.1.36 to 10.30.2.1
10.2.2.254/24
2/5-6 2/5-6 • Non working connection (expected)
aci35-spine1
aci36-spine1 • 10.1.1.35 to 10.30.2.1
• 10.1.1.36 to 10.30.1.1
2/1-2 2/1-2
• Might or not be Working Direction (return from L3 out):
• 10.30.2.1 can reach 10.1.1.35
• 10.30.1.1 can reach 10.1.1.36
• Non working direction (from VM to L3 remote):
1/49-50 1/49-50
• 10.1.1.35 to 10.30.2.1
aci35-leaf1 aci36-leaf1
• 10.1.1.36 to 10.30.1.1
OSPF OSPF

Vm 10.1.1.35
Lo1 10.30.2.1/24
Vm 10.1.1.36 In summary: we do not support traffic
Lo1 - 10.30.1.1/24
From EPG in siteX to L3 out in siteY
© 2019 Cisco and/or its affiliates. All rights reserved. (planned for 4.2)
 Why EP to remote L3 out do not work
• No VPNv4 route exchange across multisite BGP session
• No l2vpn evpn type 5 neither
• Site 2 never got route from Site 1 – L3 out

Only l2vpn evpn capa nego with Peer on the intersite)

pod35-spine1# show bgp l2vpn evpn neigh 10.10.35.112 vrf overlay-1 | egrep -A 1
"capabili"

Additional Paths capability: advertised received

Additional Paths Capability Parameters:
Send capability advertised to Peer for AF:
L2VPN EVPN
Receive capability advertised to Peer for AF:
L2VPN EVPN

No l2vpn evpn type5 for subnet neither is advert

pod35-spine1# show bgp l2vpn evpn neigh 10.10.35.112 adver vrf overlay-1 | egrep
"10.30"
pod35-spine1#

© 2019 Cisco and/or its affiliates. All rights reserved.

Traffic return from L3 out
• Will work if no subnet configured on l3 out epg (or 0.0.0.0/0)
• Will not work if any subnet is configured under the l3 out with ext
subnet for external EPG

• See MSC release notes :

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/aci
_multi-site/sw/1x/release_notes/Cisco_ACI_Multi-
Site_RN_112.html
• NOTE: The subnet in the L3extInstP must be the same for all inter-
related sites (and variable length network masks are not supported).
© 2019 Cisco and/or its affiliates. All rights reserved.
Thanks – Q&A

© 2019 Cisco and/or its affiliates. All rights reserved.