Cisco ACI

Fabric Setup Process

www.lumoscloud.com
learning@lumosconsultinginc.com
Agenda

 APIC Installation
 Fabric Configuration
 New Features
APIC Installation
 Fabric Configuration Process
 Assemble Spine and Leafs (more can be added
later)
 Configure first APIC
 Fabric is discovered automatically
 Commission each spine and leaf
 Add rest of APIC cluster
 Configure fabric policies
 Configure router reflectors
 Assemble Spine and Leafs
1-6 Spines
(per pod)
1-6 connections from
leaf to spines (based
on leaf uplinks)
Differing spine
models supported
for transitions
A mix of 40 and
 In general, uplink symmetry is preferred (same number of links from leaf to
each spine) 100 GE Ethernet
 Uplink asymmetry tolerated in failure scenarios (link down, etc.) uplinks supported
 Only leafs plug into spines (unless it’s a multi-site or multi-pod, but not
hosts, firewalls, or load balancers plug into spines
APIC Hardware Ports

1. Serial Console Port

2. CIMC 4. VIC (SFP+ or 10GBase-T)
3. OOB Management 5. Inband Ports to Leafs
Initial APIC config via UCS console access

 Connect via console port or CIMC

 Change default BIOS password
 When the screen displays Press <F2> Setup, press F2
 In the Setup Utility, choose the Security tab, change admin password
Where to connect APIC?
APIC fabric interfaces to Leafs (dual-homed)

APIC

UCS CIMC interface APIC OOB management interfaces (dual-homed)

CIMC Network Configuration in GUI

Dedicated
On first boot, APIC console presents initial
setup options
Make this 3967

After first time setup, APIC UI is accessible via URL: https://<APIC-mgmt-IP>

 APIC and Fabric Node Connections
Switch nodes will have:
1. Inband access to Infra & management
VRF’s
2. management Port (OOB)
3. Console port

APIC will have:

1. 2 attached to fabric for data
APIC APIC APIC 2. 2 for management (OOB)
3. 1 console ethernet port (can be only
OOB Management Network used for direct laptop hookup)
4. CIMC/IPMI ports
 Connect leafs to spines to build fabric
 Connect each APIC to two leafs (PortChannel auto-config’d)
 Connect OOB Management Network – APIC and switch node dedicated management ports
 Ready for APIC config and fabric bring-up
Fabric Initialization & Maintenance
4 Fabric will self
3 Spine switch discovers assemble
attached Leaf via LLDP,
requests TEP address
and boot file via DHCP

2
Leaf switch discovers
attached APIC via LLDP,
requests TEP address 6
and boot file via DHCP
APIC Cluster will form
APIC APIC Cluster APIC APIC when members discovery
each other via Appliance
APIC bootstrap configuration 5 Vector (AV)
1 All nodes in the same APIC cluster
1) APIC Cluster Configuration should contain same bootstrap
2) Fabric Name information if they are intended to
3) TEP Address space (Infra-VRF) form a cluster
4) …
Fabric Initialization & Maintenance
Node Identity Policy
POST: https://192.168.10.1/api/node/mo/uni/controller.xml

<fabricNodeIdentPol>
<fabricNodeIdentP serial=”TNAX234ZA"
 Assigns ID/Name to switches based on serial number name="leaf1" nodeId=”101"/>
<fabricNodeIdentP serial=” JNAX234ZZ"
name="leaf2" nodeId=”102"/>
 Controls which switches can join the fabric <fabricNodeIdentP serial=“KLAX234ZZ”
name="spine1" nodeId=”103"/>
</fabricNodeIdentPol>
 Allows zero touch provisioning of switches
Fabric Upgrades
“A-Spines” “B-Spines”

“A-Leafs” “B-Leafs”

“A-APICs” APIC APIC APIC “B-APICs”

 APIC firmware has three categories:  APIC and switch node image management controlled
 APIC controller images via APIC policies
 Switch images (leafs/spines)  Policies control which images should be on which
groupings of devices, when the images should be
 Catalog images (model info, capabilities, etc.) upgraded/downgraded
 Also control the upgrade process, automatic,
manual step by step, …
 Management Networks
Switch nodes will have:
1. Inband access to Infra & management
VRF’s
Infrastructure VRF 2. management Port (OOB)
3. Console port

APIC will have:

1. 2 attached to fabric for data
2. 2 for management (OOB)
3. 1 console ethernet port (can be only
APIC APIC APIC used for direct laptop hookup)
OOB Management 4. CIMC/IPMI ports
Network

 Infra VRF – Used for inband APIC to switch node communication, non routable outside the fabric
 Inband Management Network – ‘tenant’ VRF created for inband access to switch nodes
 OOB Management Network – ‘tenant’ VRF (mgmt) for APIC and switch node dedicated management ports
OOB Configuration
 Access ACI Devices Inband and Out-of-band (OOB)
• Configuring default gateway for OOB network allows accessibility to ACI devices from
external devices; APIC Initial Setup dialog asks for one OOB IP per APIC

Spine/Leaf
OOB port
(oobmgmt)

UCS CIMC APIC OOB

interface management
interfaces
(dual-homed)
Out-of-Band Management Configuration
Network Time Protocol
NTP Configuration Steps
Time synchronization is critical to fabric operations
 Set system date & time via CIMC BIOS (F2)
 Can be done when BIOS password is changed
 Set time in UTC
 Create Date and Time Policy
 Step 1: Identity
 Step 2: Specify the NTP servers to be used in this policy
 Configure the Date/time Format
 Configure the Fabric Pod Policy to use your Date and Time Policy
 Create a POD Policy Group
 Assign New Policy Group as the DEFAULT "Fabric Policy Group”
Create Date and Time Policy
FABRIC > FABRIC POLICIES > Policies > Pod > Date and Time
Configure the DateTime Format
FABRIC > FABRIC POLICIES > Policies > Pod > Date and Time
Configure the Pod Policy
FABRIC > FABRIC POLICIES > Pods> Policy Groups
 ACI 2.x (Congo) Features
ACI 2.x Supports “multi-
pod” topologies
 Pods can have
policies (Date and
Time, SNMP, ISIS)
 Policies are grouped
into Policy Groups
 Policy Groups are
applied to profiles
 Each pod can have its
own set of policies
Profiles for Pod Policies is
new to ACI 2.0
ACI Policy Types
Two policy types under fabric:
 Fabric Policies configure interfaces
that connect spine and leaf switches
 Access Policies configure external-
facing interfaces (i.e. servers, etc.)
Infrastructure admin categorizes
servers based on their requirements
 Prepares configuration templates for
servers (i.e. active-standby teaming,
PortChannels, vPCs)
 Policy groups bundle port settings
 ACI 3.x (Ebro) Features
ACI 3.x Supports:
 Multisite Management
 Kubernetes integration
 GUI Enhancements
 Enhanced security
 ACI Virtual Edge
 Multicloud Extensions
 Virtual Pod
 Remote Leaf Hover over port to display

 AWS Extension status & configuration

 CloudSec
 GiR
 PBR
 ACI Multisite Multi-Site
Consistent Policy across sites

Single Point of Orchestration

Fault Isolation

Scale

Site A
Site C

Site D
Site B
VM VM VM

VM VM VM

VM VM VM

VM VM VM

Geographically Dispersed Active/Standby Data Centers Stretch VRF, EPG, BD Up to One sec
Active/Active Data Centers Disaster Recovery Across Sites with VXLAN Latency