BRKDCN-2958 DC Operation and Maintenance Best Practice

Data Center Operations
Maintenance and Migration

Best Practices
Anis Edavalath, Principal Architect, Cisco CX
BRKDCN-2958
Anis Edavalath
• 11 years with Cisco Advanced Services
• Enterprise Campus and Datacenter across different verticals
• Worked 10 years with BU engineering groups in Security ,
switching, datacenter and Network Management products
• Design and deployment of Next Gen Data center architecture
enterprise and cloud customers
• AS team lead for ACI, VxLAN, Tetration, SDA (uniform policy)
• Worked with major telecom vendors and Cloud providers prior
to Cisco
• CCIE Datacenter # 48152
Contributor: Arvind Durai, Cisco Cx
BRKDCN-2958 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here
4 Enter messages/questions in the Webex space
Webex spaces will be moderated

until February 24, 2023.
Course Objective and Goal
• To help Data Center operations and engineering staff understand the
change management best practice to maintain a datacenter
environment or migrate a legacy environment to next gen Cisco
Nexus data center network deployment.
• Attendees should leave the session with a firm understanding of

• Baseline - Fabric best practices
• Operational Best Practices
• Features and Tools used to manage DC fabric
• Migration - Features and Tools (controllers)
• Migration Methodology
• Baseline
• VPC, VxLAN & ACI Refresher
• Operational Best Practices
• Change Management best practices
• Features and Tools

Graceful insertion and removal
Agenda
•
• Fabric controllers and Nexus Dashboard
• Migration Methodology
• Five key Use cases
• Change Window Best Practices
DC Baseline Refresher
vPC Feature Overview For Your
Reference
vPC Peer
vPC Terminology Keepalive Link
vPC vPC Domain

Peer
Peer Link
S2
S1
Orphan Port CFS
vPC Member
Port
Failure Scenario
vPC
• If both peers are active, then Secondary vPC peer will
disable all vPCs to avoid Dual-Active.
Orphan
Device • Data will automatically forward down remaining active
S3 port channel ports.
• Loss of in-flight packets will depend on deployment of

vPC best practice.
VXLAN Overview Layer 2 overlay on top of your Layer 3 underlay
SPINE • Each VXLAN Segment is identified by a unique

24-bit segment ID called a VXLAN Network
Identifier (VNI)
L3 UNDERLAY
Unicast/Multicast • Only hosts on the same VNI are allowed to
Routing communicate with each other
LEAF
• Original L2 packet is encapsulated with VXLAN
VTEP A
HOST1
VTEP B header in a UDP->IP->Ethernet
HOST2
MAC H1
MAC H2
VLAN 10 → VNI1000
VLAN 10 → VNI1000 Overcome 4094 VLAN Scale Limitation
DMAC SMAC Original
DMAC SMAC Original • VLANs use a 10-bit VLAN ID
H2 H1 L2 Data
H2 H1 L2 Data
Better utilization of available network paths

Outer Outer
VxLAN
UDP
VNI DMAC SMAC
Original
L2
• No need of Spanning Tree (blocks paths)
MAC SIP/DIP 1000 H2 H1
Port Data • Utilize L3 underlay network (ECMP, Link Agg,…)
VTEP A or VTEP B in deployment will be a pair, and this Multi-Tenant with virtualization
pair will provide host redundancy for Layer 2 via VPC. • Isolation of network traffic by a tenant and
reusability of networking taxonomy for tenancy
VPC is still NEEDED and VTEP will represent the
VPC pair!
VxLAN Refresher With BGP EVPN Address Learning
Host C
MAC-C GARP for IP C
IP-C Target MAC: MAC-C
Broadcast, Unicast and Target IP: IP-C
Multicast traffic can use BGP EVPN
MAC-A 10000 VTEP-1-IP
either Multicast group or update 5
IP-A VRF FOO VTEP-1-IP
Ingress replication of MAC-C: VNI
traffic- not covered 10001 MAC-B 10000 VTEP-2-IP
IP-C: VNI 6 IP-B VRF FOO VTEP-2-IP
VTEP-
10001
3
VTEP-3-IP
Nexthop: VTEP-3-
VTEP-3-IP MAC BGP EVPN
BGP EVPN update
Host B
update MAC-B: VNI MAC-B
MAC-A: VNI 10000 IP-B
10000 IP-B: VNI
IP-A: VNI 2 20000
10000 4 Nexthop:
VTEP-1 VTEP-
VTEP-2-IP
Nexthop:
2 3
VTEP-1-IP
VTEP-1-IP VTEP-2-IP GARP for IP B
Host A 1 VTEP-1-MAC VTEP-2-MAC Target MAC: MAC-B
MAC-A Target IP: IP-B
IP-A MAC-B 10000 VTEP-2-IP
Hosts’ Setup
IP-B VRF FOO VTEP-2-IP MAC-A 10000 VTEP-1-IP
GARP for IP A Vlan 10: VNI IP-A VRF FOO VTEP-1-IP
Target MAC: MAC-A MAC-C 10000 VTEP-3-IP
10000
Target IP: IP-A MAC-C 10000 VTEP-3-IP
IP-C VRF FOO VTEP-3-IP VRF FOO: VNI
20000 IP-C VRF FOO VTEP-3-IP
SDN ‘with’ FCAPS ‘and’ Automation
Application Centric Programmable
Infrastructure EVPN Fabric
Turnkey integrated solution with security, centralized

management, compliance and scale Modern NX-OS with enhanced NX-APIs
Automated application centric-policy model with DevOps toolset used for Network Management
embedded security (Puppet, Chef, Ansible etc.)
Custom Script based Operations and Workflows

Broad and deep ecosystem
Fault
Configuration External
Tools NDFC
Integrated
Accounting
Toolset
Performance
Security
Application Network Profiles (ANP) & ACI:
how it works?
F/W WEB ADC APP DB

ADC
SLA
QoS
APPLICATION STORAGE
CONNECTIVIT SECURITY
Security QOS L4..7 AND
Y POLICY POLICIES
Classification SERVICES COMPUTE
APP PROFILE
HYPERVISOR HYPERVISOR HYPERVISOR
Operational Best Practices
NX-OS High Availability
Process Modularity
SNMP, XML, CLI Management

Layer-2 Protocols Layer-3 Protocols Storage Protocols Other Services
• Independent memory-protected VLAN mgr UDLD OSPF GLBP
Sysmgr, PSS & MTS

VSANs
STP CDP BGP HSRP Zoning
restart-able processes IGMP snp

LACP
802.1X
CTS
EIGRP
PIM
VRRP
SNMP
FCIP
FSPF
IVR
… …
• Service Restart-ability
Future Services
Protocol Stack (IPv4 / IPv6 / L2) Possibilities
• Stateful Restart with Persistent Storage Interface Management
Service (PSS)
Chassis Management
Chip/Driver Infrastructure
• Checkpoints states to PSS
Kernel
• Recover states from PSS
upon restart.
• Stateful Restart with Graceful Restart
• Recover states based on information from
other services and/or network.
• Mainly Routing Protocols
• Stateless Restart
• Fresh start, no trace of former instantiation.
In-Service Software Upgrade
Nexus# install all nxos bootflash:nxos.9.2.3.bin
Upgrade and reboot
Initiate stateful failover
Upgrade and reboot
Upgrade I/O modules
Active Standby
Release
Release Release
Release
7.0(3)I7(4)
OSPF
OSPF
9.2(3)
7.0(3)I7(4) 9.2(3)
BGP
BGP
PIM
etc.
PIM
etc.
HA Manager HA Manager
Linux Kernel Linux Kernel
Nexus Data Plane
Best Practice:
Release
Release VPCs should be distributed.
7.0(3)I7(4)
9.2(3)
I/O Module Images
NX-OS High Availability
Supervisor Switchover
• Stateful Switchover (SSO) Active Sup

• Active-backup supervisors synchronized at all times
Backup Sup
• Routing Protocols: → PSS Stateful Restart
→ NSF Graceful Restart failover
LC - NSF
• Other components: → PSS Stateful Restart
• Triggers:
• HA Policy Initiated – e.g. 3 component crashes → SSO
• User Initiated – system switchover
• ISSU initiated SSO
Defect Impact
TAC: You’ve encountered defect CSCxy12345. Belay my last.

It’s operationally impacting and, I’m sorry to say, We have a SMU
there’s no workaround. You’ll need to upgrade. for that.
You: Fine. Let’s just get it fixed. What?

Bill, start up a war room. Gesundheit.
! John, get our AS NCE on the phone.
Sally, schedule testers in two hours.
Where’s my $#@! coffee?
Sally: You know how Boss gets when we call him at 2 AM...
Software Patching in NX-OS
Who’s familiar with Software Maintenance Updates (SMU)?
Overview Benefits
• Software Patching is Platform Independent • Reduce time to resolution in your network.

• Available on Nexus 9000 (6.1(2)I2) • SMUs in NX-OS build upon years of
• FCS NX-OS 7.2 (5/6/7k) experience in IOS XR.
• Fully supported with ISSU • Simplify customer operations for defect
resolution and code qualification.
• Better utilize the software HA capabilities
of NX-OS.
• Provide a common cross-platform
experience (N9K/N7K/N6K/N5K).
SMU Lifecycle – CLI
SMU SMU
SMU
Repository
Switch# install add …
Switch# install remove …
SMU Committed Copy to Device

Memory: Process:
Any Nexus
Memory: Process:
Switch# install commit … Chassis

show install active Switch# install activate …
show install committed
Supporting
show install inactive
Patching
SMU Removed
show install packages
Memory: Process: SMU Applied
show install pkg-info … Memory: Process:
Switch# install deactivate … Switch# install commit …

SMU Committed
Memory: Process:
Patching Highlights
• Patching is for operationally impacting bugs

SMU Types
without a workaround.
• Restart: Restarts affected
• Cannot patch to next release.
process
• Process restarted in all
• Patching is done in default/admin VDC and
VDCs where running.
applies to all VDCs.
• ISSU SMU:
• Patching is not available per-VDC.
• Dual Sup -> ISSU
• Single Sup -> Reload
• ISSU will work with all, or a subset of patches
applied.
• You don’t need to apply all patches.
• Some SMUs may only have a single fix,

others may have multiple packaged.
Stateful Switchover Mode
SSO-Aware and SSO-Compliant Applications
SSO-Compliant Applications SSO-Aware Applications

Redundancy
Facility Forwarding Information Base
Routing Protocols
IEEE 802.1x
NetFlow
Checkpointing PAgP / LACP
Cisco Discovery Protocol
Facility …and more
…and more
Active Supervisor
Standby Hot Supervisor
SSO-Compliant Applications SSO-Aware Applications

Forwarding Information Base
Routing Protocols
Checkpointing IEEE 802.1x
NetFlow
Facility PAgP / LACP
Cisco Discovery Protocol
…and more
…and more Redundancy
Facility
Routing Protocol Redundancy With NSF
(Graceful Restart)
Active Supervisor Engine Slot 1 Standby Supervisor Engine Slot 2
EIGRP RIB OSPF RIB ARP Table EIGRP RIB OSPF RIB ARP Table
Prefix Next Hop Prefix Next Hop IP MAC Prefix Next Hop Prefix Next Hop IP MAC
10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee3 - - - - - -

2
10.1.0.0 10.1.1.1 192.168.55. 192.168.55. - - - - - -
0 1 10.1.1.2 adbb32:d34e4
10.20.0.0 10.1.1.1 3 - - - - - -
192.168.32. 192.168.32.
0 1 10.20.1. aa25cc:ddeee
1 8
FIB Table
SSO FIB Table
Prefix Next HOP Redundancy Facility Prefix Next HOP
10.1.1.1 aabbcc:ddee32 10.1.1.1 aabbcc:ddee32
10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43
192.168.0. aa25cc:ddeee8 Checkpoint Facility 192.168.0. aa25cc:ddeee8

0 0
Routing Protocol Redundancy With NSF (Graceful Restart)

EIGRP RIB OSPF RIB ARP Table EIGRP RIB OSPF RIB ARP Table
10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32 - - - - - -
10.1.0.0 10.1.1.1 192.168.55. 192.168.55. 10.1.1.2 adbb32:d34e43 - - - - - -

0 1
10.20.0.0 10.1.1.1 10.20.1.1 aa25cc:ddeee8 - - - - - -
192.168.32. 192.168.32.
0 1
FIB Table
SSO FIB Table
10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43
192.168.0. aa25cc:ddeee8 Checkpoint Facility 192.168.0.0 aa25cc:ddeee8

0
Routing Protocol Redundancy With NSF
(Graceful Restart)
Standby Supervisor Engine Slot 2
EIGRP RIB OSPF RIB ARP Table
Prefix Next Hop Prefix Next Hop IP MAC
10.0.0.0
- -
10.1.1.1 - 192.168.0 -192.168.0.1 -10.1.1.1 aabbcc:ddee32
-
-
10.1.0.0 -
10.1.1.1 192.168.55.0
- 192.168.55.1
- -10.1.1.2 adbb32:d34e43
-
-
10.20.0.0 -
10.1.1.1 192.168.32.0
- 192.168.32.1
- 10.20.1.1
- aa25cc:ddeee8
-
FIB Table
Prefix Next HOP
10.1.1.1 aabbcc:ddee32
10.1.1.2 adbb32:d34e43
192.168.0.0 aa25cc:ddeee8
GR/NSF Signaling per protocol
Synchronization per protocol
Routing Protocol Redundancy With NSR
(Stateful Restart)
BGP RIB OSPF RIB ARP Table BGP RIB OSPF RIB ARP Table
10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32 10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32
10.1.0.0 10.1.1.1 192.168.55.0 192.168.55.1 10.1.1.2 adbb32:d34e43 10.1.0.0 10.1.1.1 192.168.55.0 192.168.55.1 10.1.1.2 adbb32:d34e43
10.20.0.0 10.1.1.1 192.168.32.0 192.168.32.1 10.20.1.1 aa25cc:ddeee8 10.20.0.0 10.1.1.1 192.168.32.0 192.168.32.1 10.20.1.1 aa25cc:ddeee8
FIB Table
SSO FIB Table
10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43
192.168.0.0 aa25cc:ddeee8 Checkpoint Facility 192.168.0.0 aa25cc:ddeee8
Routing Protocol Redundancy With NSR
(Stateful Restart)
Standby Supervisor Engine Slot 2
BGP RIB OSPF RIB ARP Table
Prefix Next Hop Prefix Next Hop IP MAC
10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32
10.1.0.0 10.1.1.1 192.168.55.0 192.168.55.1 10.1.1.2 adbb32:d34e43
10.20.0.0 10.1.1.1 192.168.32.0 192.168.32.1 10.20.1.1 aa25cc:ddeee8
FIB Table
Prefix Next HOP
10.1.1.1 aabbcc:ddee32
10.1.1.2 adbb32:d34e43
192.168.0.0 aa25cc:ddeee8
No additional signaling required to maintain topology
vPC Best Practice
• vPC Domain ID’s
✓ Use a unique vPC domain ID within a contiguous L2 domain to avoid MAC overlap.
• vPC Peer Link

✓ Should be point-to-point connection & dedicated links.
• vPC Peer Keepalive Link

✓ Dedicate a control plane in a dual-supervisor environment. Use a management switch.
• vPC peer-gateway
✓ Acts as active gateway for frames addressed to peer switch. Avoid Peer Link forwarding.
• Use vPC peer-switch

✓ Optimizes BPDU processing, single logical L2 entity
• Distribute port-channel member interfaces across line cards within the same chassis.
• Create a map for oversubscription aligned to current and future demand.

✓ Deployment practice – 20:1 at access and 2:1 at Core.
vPC Configuration Best Practices
• vPC object tracking, tracks both peer-link and
uplinks in a list of Boolean OR S4 S5
• Object Tracking triggered when the track object
goes down
• Suspends the vPCs on the impaired device.
• Traffic forwarded over the remaining vPC peer.
! Track the vpc peer link
track 1 interface port-channel11 line-protocol
! Track the uplinks
track 2
track 3
interface Ethernet1/1 line-protocol
interface Ethernet1/2 line-protocol S1 S2
! Combine all tracked objects into one.
! “OR” means if ALL objects are down, this object will go down
track 10 list boolean OR
object 1
object 2
object 3
! If object 10 goes down on the primary vPC peer,

S3
! system will switch over to other vPC peer and disable all local vPCs
vpc domain 1
track 10
Vxlan Considerations and Best Practices
• Use the Nexus Dashboard NDFC App for Vxlan configuration and management
• Vxlan EVPN preferred over Flood and learn
• Design Hierarchical Vxlan fabric to accommodate the scale out
• Ingress replication , Underlay multicast for BUM traffic handling
• Summarize external routes on border leaf or Advertise default routes on a per tenant
basis
• Advertise only the LPM prefix routes of internal public layer 3 subnets out of border
leaf switches
• Include all local loopbacks in underlay routing to make troubleshooting easy
• Same Vlan Per L2 VNI and consistent naming for vlan VNI pair
Operational Best Practices
Loop Mitigation Settings
MO Naming Convention
• Develop and plan the MO(Managed Objects) Naming • Enable MCP ( Miscabling Protocol) per vlan
Convention according to Organizations best Practice
• To find out and disable loops caused by misbehaving
external L2
Tags and Aliases • Rogue EP Control preferred over Endpoint loop protection
• Workaround to Rename Objects
• Objects can be grouped to make query easier
• Tags/Aliases have no functional impact- Where as Labels
have
BD Level Configuration
• Do Not enable Unicast routing when gateway is not BD SVI
• Limit IP Learning to Subnet
• L2 Unknown unicast set to Flood
• ARP Flooding enabled
AAA Fallback to Local Auth Fabric Wide Configuration
• Fallback domain should be set to local to avoid lockout • IP Aging Policy
• Disable Remote EP Learning – On Border Leaf
• Enforce Subnet Check
Change Management
NX-OS
Graceful Insertion
and Removal
Nexus Graceful Removal
router bgp 33
Discontinue advertisement of all prefixes.
isolate
router eigrp 1
isolate Advertises maximum metrics for all K-values.
router ospf 1
isolate max-metric router-lsa
router isis 1
isolate
set-overload-bit
Nexus feature
Graceful Insertion N9372(config)# no system
mode maintenance
Following configuration
will be applied:
• Move the switch from Maintenance mode to
Normal mode. router bgp 33
• Control plane maintained throughout no isolate
isolation of the switch.
router eigrp 1
• Protocols advertise routes only after it is
installed in hardware.
no isolate
router ospf 1
no isolate
router isis 1
no isolate
VPC Shutdown Feature Configure
PKA
Secondary
This feature allows customer to manually “isolate” a switch
Primary
from vPC domain. This is a vPC configuration option. Vlan 1-100
Pre-VPC Shutdown VPC Shutdown Behavior Vlan 1-100

Vlan 1-100
• No “shutdown” • Local switch isolated from remote.

command. • Cannot exit shutdown without
• Manual Shutdown manual intervention. De-configure
Required • When exiting, PKA, PL, and vPCs PKA
• Down vPCs will be re-initialized; vPC domain
Secondary
Primary
Vlan 1-100
• Down Peer Link brought to normal state.
• vPC Members
• Etc.
Vlan 1-100 Vlan 1-100
Graceful Insertion and Removal
OSPF:
max-metric router-lsa
Isolate for
Change Window
VPC:
shutdown
feature ospf
feature vpc Scripting takes time.
It’d be nice to automate
this…
Change window begins.
vPC vPC
system mode maintenance
One command!
Pre-change System Snapshot
Change window complete.
vPC vPC
no system mode maintenance
One command!
Post-change System Snapshot
Configuration Profiles
• Maintenance-mode profile is applied when entering GIR mode,
• Normal-mode profile is applied when GIR mode is exited.
Automatic Profiles Manual Profiles
• Generated by default • User created profile for maintenance-
• Parses configuration to determine mode and normal-mode
changes going into and out of GIR • Flexible selection of protocols for
• Changes based on base protocol isolation
configuration settings.
• Use: maintenance windows and
• Use: Maintenance Windows isolation during troubleshooting using
preconfigured scripts.
Enabling Graceful Insertion and Removal
Automatic Profile Generation
N7K-1-Core# show system mode
System Mode : Normal
N7K-1-Core# config
Enter configuration commands, one per line. End with
CNTL/Z.
N7K-1-Core(config)# system mode maintenance
BGP is not enabled, nothing to be done
EIGRP is not enabled, nothing to be done

Generating maintenance-mode profile
OSPF is up..... will be shutdown Progressing...................Done.
OSPF TAG = 100, VRF = default
config terminal System mode operation completed successfully
router ospf 100
shutdown N7K-1-Core# show system mode
end System Mode : Maintenance
N7K-1-Core#
OSPFv3 is not enabled, nothing to be done
ISIS is not enabled, nothing to be done
vPC is not enabled, nothing to be done
Interfaces will be shutdown

Do you want to continue (y/n)? [n] y
Enabling Graceful Insertion and Removal
Custom Profile Generation
config-profile maintenance-mode type admin config-profile normal-mode type admin
router bgp 65001 router bgp 65001
isolate no isolate
sleep instance 1 10 sleep instance 1 10
router ospf 100 router ospf 100
isolate no isolate
sleep instance 3 20 sleep instance 3 20
vpc domain 20 vpc domain 20
shutdown no shutdown
system interface shutdown exclude fex-fabric no system interface shutdown
• By default, GIR Mode will automatically generate profiles.

• CLI to disable automatic profile generation: dont-generate-profile
• If you enter GIR mode with automatic profile, it will overwrite your custom profile.
Graceful Insertion and Removal Mode for
Unplanned Outages
system mode maintenance on-reload reset-reason reason
HW_ERROR-Hardware error,
SVC_FAILURE-Critical service failure,
KERN_FAILURE-Kernel panic,
WDOG_TIMEOUT-Watchdog timeout,
FATAL_ERROR-Fatal error,
MANUAL_RELOAD---Manual reload,
MATCH_ANY-Any of the above reasons,
ANY_OTHER-Any reload reason not specified above.
Nexus GIR Snapshots
• Used before and after a GIR mode to compare pre/post change operation.
• Snapshots are automatically generated when entering GIR mode.
switch# snapshot create snap1 For testing

Executing show interface... Done
Executing show bgp sessions vrf all... Done
Executing show ip eigrp topology summary... Done
Executing show vpc... Done
Executing show ip ospf vrf all... Done
Feature 'ospfv3' not enabled, skipping...
Snapshot 'snap1' created
Switch#
Nexus GIR Snapshots Comparison
Nexus# sh snapshots compare before_maintenance after_maintenance switch# show snapshots compare snapshot1 snapshot2 ipv4routes
================================================================================ metric snapshot1 snapshot2 changed
Feature Tag before_maintenance after_maintenance # of routes 33 3 *
================================================================================ # of adjacencies 10 4 *
[bgp]
-------------------------------------------------------------------------------- Prefix Changed Attribute
------ -----------------
[neighbor-id:100.120.1.221] 23.0.0.0/8 not in snapshot2
connectionsdropped 2 **3** 10.10.10.1/32 not in snapshot2
lastflap P1DT21H5M12S **P1DT21H25M47S** 21.1.2.3/8 adjacency index has changed from 29 (snapshot1) to 38
lastread P1DT21H25M12S **PT0S** (snapshot2)
lastwrite P1DT21H25M14S **PT0S**
state Established **Idle**
localport 52737 **0**
{+-}
remoteport 179 **0**
notificationssent 2 **3**
<...>
Tools manage DC
• Controller (APIC / NDFC)
• Nexus Insights
Health Score
Aggregated View
Fabric Topology
View
Aggregation of system-wide health, including pod health scores, tenant
health scores, system fault counts domain and type and the APIC cluster
health state.
Capacity Dashboard
View the Capacity of Data Center Fabric -ACI
Troubleshoot a Flow
Use Inbuilt Visibility Engine
Faults
Eligible Path
Drops
Troubleshoot a Flow
Use Inbuilt Visibility Engine
Fabric Real Time

Security Traffic
Policies Capture
Maintenance Upgrade #1
Upgrade APIC
Maintenance Upgrade #2
Create Groups
Cisco Nexus Dashboard Fabric Controller With Nexus Dashboard
Cisco Nexus Fabric Controller
Part of Nexus Dashboard
Dashboard
(NDFC)
Simple to automate, simple to consume
1. Single plane to build and
manage multisite fabrics
2. NDFC is an application that
you can invoke in single
Insights Fabric controller
Fabric discovery
installation for SAN and for
the fabric
Fabric controller
Orchestrator SAN controller
Fabric discovery 3. IPAM integration for VXLAN
EVPN fabrics
4. Micro Services Scaled mode
Data broker SAN controller – active/ active, increased
scale for
Consume all services in one place managed/monitored objects
5. Manages IOS-XE and IOS-
XR platforms
Private cloud | Third-party apps Public cloud
| 6. Granular RBAC applicable to
Multisite fabric management
NDFC Topology View Detailed Views
• Dynamic Arrangement
• Multi-Fabric/Overlay
• Arrange by Tier
• Core, Ag, Access
Leaf, Spine etc..
• Metadata Tags
• Device Pop-Over
Capacity Dashboard
View the Capacity of non ACI Fabric
Nexus
Dashboard
Efficient Operations at the DC (AIOPs)-
Nexus Dashboard
Note: Application insights using AppD integration with nexus insight upgrade is not covered in this session
Unified Datacenter Controller views
Nexus #1 Benchmarking
Dashboard
#2 Upgrade planning
• Day 0 , Day 1 and Day 2

• Software Image management
Orchestrator Insights Broker
• Migrations
#3 Pre and Post Change window

baselining
Old environments that have DCNM can be application
APIC NDFC in Nexus Dashboard as an Application called Nexus
Dashboard Fabric Controller
Verify Fabric Communication & relationship
Analyse the Anomalies
Pre-Change Analysis using Nexus Dashboard
Analyzes and Reveals the impacts of intended configuration changes.
Base Assurance
Snapshot Analysis
Results
(The current configuration)
Pre-change Assurance
Snapshot Analysis
Results
(The current configuration)

+
(The intended changes) ✓ Pre Change Analysis – Reveals the impact of the changes.
✓ Operations can also have custom snapshots
Other features: Anomaly detection and correlation (software, config or hardware), Upgrade pre-check and post-check
across multiple fabrics, data plane dependency mapping and micro burst detection
Create and Run a Pre-Change Analysis
Pre-Change Analysis is under the “Change Management” Option in the main
panel. Users can create, edit, clone or delete a pre-change analysis.
Network Compliances Use Cases
Network Compliances
Regulatory Business IT Governance

Compliance Requirements Operational
EPGs in
SecureVRF_PCI EPGs in VDI tenant
• Golden Configuration
must always talk to
tenant must be DHCP service in
• Naming Convention
segmented tenant common
Network Compliances Use Cases
View Compliance Analysis Status
Compliance is a top-level option in
the main panel.
On the Compliance page, the
summary of the compliance
analysis of the site is shown for the
selected time window.
Upgrade Assist
Overview Upgrade Assist
• The upgrade assist feature helps to identify

• What issues are affecting nodes with the current software versions
running on them
• What issues will no longer be seen post software upgrade
• This is done based off of metadata and affects both

ACI and NDFC
How to do an upgrade analysis Reference
• To start an upgrade analysis, select
Change management
analysis -> Firmware upgrade analysis
• Select the site you wish to upgrade
1. Select the target firmware version for your site
2. Click select nodes to select the nodes you wish to analyze
3. Check all the boxes against the nodes you wish to analyze
4. Click save to start the analysis
Click Save to start the

analysis
How to do an upgrade analysis
Reference
• Wait for the results to populate
• Double click any of the listed analysis results to open and see detailed results
• The pre-upgrade analysis specifies validation checks done before

the software upgrade along with an anomaly and advisory forecast to specify what issues are
seen prior to the upgrade and will no longer be seen after the upgrade
Reference
The areas checked for during pre-upgrade analysis for ‘ACI are listed’ below
1 Critical faults No Critical Faults from System -> Faults
2 OOB management IP Ensure static OOB Management IP is configured
3 vPC nodes Configure vPC for the listed leaf nodes to avoid traffic loss during the reboot of leaf nodes.
4 Route reflectors Configure vPC for the listed leaf nodes to avoid traffic loss during the reboot of leaf nodes.
5 NTP status Configure NTP to avoid any issues in DB sync between nodes, SSL certificate check, etc.
6 Infra VLAN id Check if configured infra VLAN ID are same across nodes
7 Fabric Recovery Enabled Check if Fabric Recovery is in progress
8 CIMC compatibility Check if running recommended CIMC version
9 Version compatibility Check version is compatible or multi-hop upgrade is needed
10 Target firmware check Check if image is copied and available in device
11 SNMPv3 auth compatibility Check SNMPv3 authorization and/or privacy
12 Remote leaf compatibility Check if remote leaf is not supported in the target version.
13 Multi-Tier compatibility Check if Remote leaf is not supported in the target version
14 Bootflash storage Check for space in bootflash folder to download image
15 Spine redundancy Check if each pod upgrades spine nodes with at least two separate groups to avoid traffic loss. Spines should not be in maintenance group.
16 Hardware compatibility Check if hardware is compatible with version

17 Maintenance groups Check if maintenance groups were created in pre 4.0 release
18 APIC Cluster Status Check if APIC Cluster status is fully fit for all APIC nodes
Reference
The areas checked for during pre-upgrade analysis for NXOS are listed below
# Validation Step Description
1 validate po summary Check if all port-channel members are in (P) state
2 validate vpcs Check if vpc status is “up”
3 validate vpc role Check if local vPC role is secondary
4 validate sticky bit Check if local switch vPC sticky bit is False
5 validate hsrp state Check if "hsrp mgo state" is Active/Standby
6 validate mods Check if module in ok/active/standby state and diag pass
7 validate ospf Check if OSPF is in FULL FULL/DR state
8 validate bgp Check if BGP session is in Up State
9 validate free space Check if bootflash on active/standby supervisor is greater than threshold
10 validate logging nvram Check for Severity 1, 2 or 3 messages
11 validate logging log Check for Severity 1, 2 or 3 messages
12 validate mod exceptions Check for non-user initiated resets
13 validate redundancy Check if redundancy status is "Active with HA standby" for EOR
14 validate reset reason Check if module was reset due to reasons other than those initiated by user
15 validate system reset reason Check if module was reset due to reasons other than those initiated by user
16 validate diag module Checks if previously initiated diag had shown failure
17 validate flash ramdisk Check if all filesystems are equal to or below integer percentage
18 validate console mgmt Check if console register bits are RTS|DTR|DSR
19 validate environment Check if all modules are in ok state and backup power present
20 validate arp table Check if certain number of ARP'S are in INCOMPLETE
21 validate cores Check if core files are present
22 validate device connectivity Check if we are able to reach the device with PolicyGateway/NDFC/SIM
Migration Best Practices
Datacenter Migration – Strategy and Approach
• Architecture Governance
Source Architecture 1 • Target Architecture definition
• Scale Considerations
Spanning • Impact Analysis
Fabric Path • Prechecks and Validations
Tree Analysis &
Planning
Vxlan ACI
Post Migration Application 2 • Nexus Dashboard or Cisco Secure
6 monitoring and Dependency
• APM monitoring Workload (flow metrics or ADM)
support Mapping
• Issue resolution and • Open-Source automation
SLA • Business Process
• Detailed Health • Server to port Mapping
analysis • Automated policy generation
Destination Architecture Migration Migration

5 Execution Strategy 3
Vxlan
• Change Management Best Parallel Vs Hybrid
practices Network Centric migration
ACI • SW and HW Platform best Documenting Application based migration
practices APM during Migration
• Go – No Go strategy alignment
• Validations before , during and
after migration 4
• Reporting
Runbook, Validation , Rollback
Discover Hidden Interdependencies
Top-down approach
Discovery
• What do I need to migrate? Business Process

− Understand installed asset base (applications, server, storage,
network)
− Understand interdependencies Application Application

▪ Understand business and operational constraints
Bottom-up approach
− Core business processes
− Operational processes Application Components
− Application criticality (prioritize)
− Current DR capability
Databases Application Servers Web Servers
− Available downtime window for migration
• Understand facilities requirements and constraints Si

Si
− Power, cooling and rack space

Servers, Networks, SAN, Storage Arrays
− Regulatory compliance constraints
Datacenter Migration Scenarios and Considerations
• Application workload visibility options
• Baseline connectivity Considerations
• Gateway Considerations
• Site Based considerations – for scalability
• Constructs based considerations
Visibility Considerations and Best Practices
for Migration
Application dependency
Application flow Map
Mapping 1. Selecting Application for Migration
2. benchmark digital experience
3. review the following during Migrations

a) Application performance monitoring
b) Latency considerations
c) Avoiding Suboptimal routing

conditions
4. Application Performance Monitoring

After the migration
Application Dependency Mapping Application Flow Mapping
Network Considerations for Datacenter Migration
1 Migration Build a parallel vxlan fabric
Planning
Establish L2 connection Existing/Brownfield
Datacenter New Vxlan EVPN Datcenter
between legacy and new
vxlan fabric
Establish Dedicated L3
Border Leaf
interconnect Between 2 Core
fabrics
Border Leaf
2 Layer 2 Dedicated Leaf for L2 Core
Consideratio Connection using double

n sided VPC
STP Root bridge placement

Distribution Back to back
in the fabric vPC
3 Layer 3 Non VPC Border Leaf
Consideratio switches for existing
vPC Peer Link

n Connections to the core
No summarization on border
routers during Migration
Distribution
4 Overlapping Vlan translation on the
Vlans Migration Leaf
Gateway Migrations from Legacy to EVPN environment
X
FHRP approach for
Gateway in CE
DCI
L3
DAG
192.168.1.1
L2
Gateway Consideration
New Architecture
Legacy Architecture • Pre migration
• 80% migration Preferred

• Post migration
VPC VPC
DAG DAG DAG DAG
………………....
192.168.1.1 192.168.1.1 192.168.1.1 192.168.1.1
Align Gateway and Migrate Workload

Migration practices for Scaling out – use case with ACI
Datacenter 1 L3/ ingress
replication
Datacenter 1 1. Scale Consideration in Data Center 1;

Pod 1
Site B options –
• Multi Pod – same mgmt domain
VM VM VM
• Multi site – separate mgmt.
Single Fabric to Multisite scaling
domain
2. Multi-Pod Consideration
Single Fabric to Multi Pod
VM VM VM • Extension of APIC cluster- no

new policy consideration
L3 /
Mcast
• Special emphasis on the
BUM
underlay BUM (Mcast support)
Nexus Dashboard
Orchestrator
3. Multi-site consideration for policy –
• Creating new policies Vs importing
Pod 2 existing policies
Hardware consideration and service
Note: EVPN VxLAN only •
block localization for multisite
supports multisite using Border • Controller based redundancy for
multisite
VM VM VM
Gateway
ACI Network Centric Deployment
Network configuration
WAN/
Corp-L3out Internet
• VRF CORP …. vrf
Tenant: Example-Corp
configuration VRF: Corp
• Interface VLAN 100

192.168.10.0/24), BD: 192.168.10.0/24 BD: 192.168.20.0/24 BD: 192.168.30.0/24
GW: 192.168.10.1 GW: 192.168.20.1 GW: 192.168.30.1
VIP 192.168.10.1, Advertise Ext: Yes Advertise Ext: Yes Advertise Ext: No
VRF corp
• Trunk the switch ports App Prof: App1

EPG: Web
with respective vlans vDS: Corp-VDS01
EPG: App
vDS: Corp-VDS01
EPG: DB
Path: 101/1/1-4
Vlan: dynamic Vlan: dynamic Vlan: 100
• VMware port Group

Assignment vDS
• Routing Configuration Portgroup: Portgroup: Physical Servers

Corp:App1:Web Corp:App1:App
for subnets
VM
ACI Migration Example
Layer 3 Routing
Static, OSPF, BGP
APIC
1.1.1.12/30
1.1.1.0/30
Blue Tenant
and Context
Policies
Migration
BD Blue_1 BD Blue_2
Vlan 10,11 10.10.11.1/24
L2_
Out EPG EPG
L2_ L3Out
blue_1 Out
blue_2
Layer 2 vPC Trunk
VLAN 10 ▪ Step 1: Layer 3 and Layer 2 bridge between current and

new fabric
(10.10.10.0/24) ▪ Step 2: Choose networks to move to ACI moved the end
points
▪ VLAN 10 maps to BD Blue_1
VLAN 11 ▪ VLAN 11 maps to BD Blue_2
Access
(10.10.11.0/24) ▪ Classic Devices are still the Default Gateway Tag 2101 Tag 2102
& L4-7 services (FW/LB)
.101 .102 ▪ Flooding enabled on ACI BDs during migration
(compatibility with STP)
▪ Step3 : Once migration completed, Default Gateway ACI
BDs
▪ Step 4 : Scale to subnet migration
▪ Step 5: Move the service block / Layer 3 uplink to new
Tag could be VLAN ID or VNID.
fabric BRKDCN-2958 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Interconnectivity using VXLAN EVPN Border Gateway
• Border Gateway ( BGW) provides
interconnectivity and translation between
multiple Vxlan Sites
• vPC Border Gateway is positioned as Vxlan

replacement for traditional DCI
• Benefits: VTEP VTEP VTEP VTEP
✓ Connects to layer 2 domains or 2 BG BG BG BG

domains to EVPN fabric with fault W W W W
isolation
✓ Simple to deploy compare to old DCI
technology and uses EVPN concept
✓ Supports endpoints connected to
BGW – cost effective for smaller
fabrics Vx Dedicated anycast BGW
• Key technical features
• Fault Containment: BGW

provides EVPN Multicast Storm
Control
• Transport Agnostic - Vxlan
tunnel built over any IP Legacy Network Legacy Network
connectivity
• Multihoming and Multipath Load
sharing
Legacy Network to VXLAN EVPN Migration
using Border Gateway
• Introduce a pair of BGW to Legacy sites
✓ Back to back vPC provide multipath connectivity Layer 3 Inter site Network
✓ No STP loops as Double sided vpc provides a single link
• Bring up vPC BGW Underlay network MP BGP EVPN Peering

✓ Route peering between BGW and first Hop layer 3 Anycast
Anycast
devices in the intersite network. Gateway
VTEP VTEP Gateways Up VTEP VTEP
Down
✓ eBGP is recommended as the Underlay protocol BGW BGW BGW BGW
• Configure vPC BGW overlay network

✓ MP-BGP as the overlay Control Plane between BGW HSRP
HSRP
Gateways
nides in two sites.
Gateways
Down Up
✓ Full mesh eBGP or route servers in external network
depending on size of network
• Configure L2 extension across sites

Legacy Network Legacy Network
• Migrate HSRP Gateway on Distribution to Anycast
Gateway on BGW
Transition Legacy Network to VXLAN EVPN
using Border Gateway – Final Step
• Build a parallel Nexus 9000 Hub and Spoke Evpn Layer 3 Inter site Network
Vxlan Fabric
✓ Workload migration commences at this point.
Vxlan Overlay
Site 1 VTEP VTEP Site 2 VTEP VTEP
BGW BGW
Spine Spine Spine

Spine
VTEP VTEP VTEP VTEP

VTEP VTEP VTEP VTEP
Legacy Network to VXLAN EVPN Migration using
Border Gateway
Layer 3 Inter site Network
MpBGp EVPN Overlay
VTEP VTEP
VTEP VTEP
Spine Spine Spine

Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Migrated EVPN Vxlan Site 1

Migrated EVPN Vxlan Site 2
Change Management
Maintenance Windows – Golden Rules
• Change Review Board
• Schedule when environment

will be least impacted.
• Software Staging
• Verify out of band.
• Test! After and before.
Traditional vPC Environment Change
Change Best Practice and Window
Primary Secondary
Core Isolation
1. Graceful L3 Protocol Isolation
vPC
2. Layer 2 Isolation
• VPC
3. Interface Isolation
Using GIR Mode Steps 1-3 could be achieved prescriptively.
Access Isolation
• VPC
1. Fex-fabric (include/exclude)
2. Dual-attached FEX Procedure * Recommended
Fex Using GIR Mode Steps 1-2 could be achieved prescriptively.
NOTE: Maintenance mode consideration should be based on Fex-
fabric connectivity.
If change window is for software upgrade or spot fix, consider ISSU or SMU feasibility.
L3 Environment
Change Best Practice and Window
Core Isolation
1. Graceful L3 Protocol Isolation
Using GIR Mode Steps 1-2 could be achieved
prescriptively.
Access Isolation
Layer 3 1. L3 Protocol isolation
• vPC
1. Fex-fabric (include/exclude)
2. Dual-attached FEX Procedure * Recommended
Using GIR Mode, prescriptive isolation is possible.
If change window is for software upgrade or spot fix, consider ISSU or SMU feasibility.
Summary
Putting it all Together
▪ What to use? GIR Mode? Patching? ISSU? All of them?
Situation Critical Bug Hardware New

Option
Fix & PSIRT Upgrade Features
ISSU ✓ X ✓
GIR + Cold Boot ✓ X ✓
GIR + Disruptive
✓ X ✓
Installer
SMU Restart ✓ X X
GIR + SMU ISSU ✓ X X
GIR X ✓ X
Datacenter Operations –key take ways
• Understand the features and relations to best practices
• Utilize Software hardware best practice in deployment of the Data center
• Change Management features
• Isolation (GIR)
• Tools used to manage DC environment
• Day 0,1,2 use controllers based on automation
• Assurance to manage DC use Nexus Insights
Datacenter Migrations –key take ways
1. Verify environment conforms to data center networking best
practices, and leverage DC controllers
2. Isolate Node to minimize the disruption - leverage features like
GIR for change window planning
3. Leverage the Migration methodology and use cases to customize
your transformation
Complete your Session Survey
• Please complete your session survey
after each session. Your feedback
is important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (open from Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events Mobile App or
by logging in to the Session Catalog and clicking the
"Attendee Dashboard” at
https://www.ciscolive.com/emea/learn/sessions/session-catalog.html
Pay for Learning with Cisco
Cisco learning and certifications

Learning Credits
(CLCs) are prepaid training
vouchers redeemed directly
From technology training and team development to Cisco certifications and learning with Cisco.
plans, let us help you empower your business and career. www.cisco.com/go/certs
Learn Train Certify

Cisco U. Cisco Training Bootcamps Cisco Certifications and Specialist
IT learning hub that guides teams Intensive team & individual automation Certifications
and learners toward their goals and technology training programs Award-winning certification
program empowers students
Cisco Digital Learning Cisco Learning Partner Program and IT Professionals to advance
Subscription-based product, technology, Authorized training partners supporting their technical careers
and certification training Cisco technology and career certifications
Cisco Guided Study Groups
Cisco Modeling Labs Cisco Instructor-led and 180-day certification prep program
Network simulation platform for design, with learning and support
Virtual Instructor-led training
testing, and troubleshooting Accelerated curriculum of product,
technology, and certification courses Cisco Continuing
Cisco Learning Network Education Program
Resource community portal for Recertification training options
certifications and learning for Cisco certified individuals
Here at the event? Visit us at The Learning and Certifications lounge at the World of Solutions
Continue
Agenda Your Education
Visit the Cisco Showcase for related demos.
Book your one-on-one Meet the Engineer meeting.
Attend any of the related sessions at the DevNet,

Capture the Flag, and Walk-in Labs zones.
Visit the On-Demand Library for more sessions

at ciscolive.com/on-demand.
Thank you

BRKDCN-2958 DC Operation and Maintenance Best Practice

Uploaded by

Copyright:

Available Formats

BRKDCN-2958 DC Operation and Maintenance Best Practice

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN-2958 DC Operation and Maintenance Best Practice

Uploaded by

Copyright:

Available Formats

Data Center Operations

Maintenance and Migration

Anis Edavalath, Principal Architect, Cisco CX

Contributor: Arvind Durai, Cisco Cx

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

• Attendees should leave the session with a firm understanding of

• Features and Tools

• Fabric controllers and Nexus Dashboard

• Change Window Best Practices

vPC vPC Domain

• Loss of in-flight packets will depend on deployment of

SPINE • Each VXLAN Segment is identified by a unique

Better utilization of available network paths

Turnkey integrated solution with security, centralized

Custom Script based Operations and Workflows

F/W WEB ADC APP DB

HYPERVISOR HYPERVISOR HYPERVISOR

SNMP, XML, CLI Management

Sysmgr, PSS & MTS

restart-able processes IGMP snp

• Stateful Restart with Persistent Storage Interface Management

Nexus Data Plane

• Stateful Switchover (SSO) Active Sup

TAC: You’ve encountered defect CSCxy12345. Belay my last.

You: Fine. Let’s just get it fixed. What?

• Software Patching is Platform Independent • Reduce time to resolution in your network.

SMU Committed Copy to Device

Switch# install commit … Chassis

Switch# install deactivate … Switch# install commit …

• Patching is for operationally impacting bugs

• Some SMUs may only have a single fix,

SSO-Compliant Applications SSO-Aware Applications

SSO-Compliant Applications SSO-Aware Applications

10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee3 - - - - - -

Prefix Next HOP Redundancy Facility Prefix Next HOP

10.1.1.1 aabbcc:ddee32 10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43

192.168.0. aa25cc:ddeee8 Checkpoint Facility 192.168.0. aa25cc:ddeee8

Active Supervisor Engine Slot 1 Standby Supervisor Engine Slot 2

10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32 - - - - - -

10.1.0.0 10.1.1.1 192.168.55. 192.168.55. 10.1.1.2 adbb32:d34e43 - - - - - -

Prefix Next HOP Redundancy Facility Prefix Next HOP

10.1.1.1 aabbcc:ddee32 10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43

192.168.0. aa25cc:ddeee8 Checkpoint Facility 192.168.0.0 aa25cc:ddeee8

Prefix Next Hop Prefix Next Hop IP MAC

Prefix Next HOP

GR/NSF Signaling per protocol

Synchronization per protocol

Prefix Next HOP Redundancy Facility Prefix Next HOP

10.1.1.1 aabbcc:ddee32 10.1.1.1 aabbcc:ddee32

10.1.1.2 adbb32:d34e43 10.1.1.2 adbb32:d34e43

192.168.0.0 aa25cc:ddeee8 Checkpoint Facility 192.168.0.0 aa25cc:ddeee8

Prefix Next Hop Prefix Next Hop IP MAC

10.0.0.0 10.1.1.1 192.168.0 192.168.0.1 10.1.1.1 aabbcc:ddee32

10.1.0.0 10.1.1.1 192.168.55.0 192.168.55.1 10.1.1.2 adbb32:d34e43

10.20.0.0 10.1.1.1 192.168.32.0 192.168.32.1 10.20.1.1 aa25cc:ddeee8

Prefix Next HOP