FCoE Bootcamp

FCOE
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 1
Agenda
 The Evolution of the Data Center
 Introduction to FCoE
 Standards Defined
 Nexus and the Unified Fabric
 Nexus 5000
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2
The Evolution of the
Data Center
Data Center Access Layer Trends
Multi-Core CPU architectures allowing

bigger and multiple workloads on the same
machine
Server virtualization driving the need for

more I/O bandwidth per server
Growing need for network storage driving

the demand for higher network bandwidth
to the server
Increasing adoption of Blades in data

centers.
10G LOM on server Motherboard
Next-Gen Switch Design Goals
•Consolidate LAN & SAN
infrasctucture
•Standards based solution
•Reduce total cost of
ownership
•End-to-end data center •Build with superior
architecture performance in mind
•Operational consistency •Support low latency
across platforms applications (e.g. HPC,
clustered app’s)
•Enable Virtualization •Scale to 40G and

•Address increase in 100G in future
server processing •Increase feature
power velocity
Cisco Nexus Family
 Complete data center class switching portfolio
 Consistent data center operating system across all platforms
 Infrastructure scalability, transport flexibility and operational
manageability Nexus 7000
(Modular Switch
Platform)
Nexus 1000V
(Virtual Switch) Nexus 4000
Nexus 2000 Nexus 5000
(Blade Switch)
2008 (Fabric (Fixed Config
1K
2008
Extender) Switch)
1K
Cisco Nexus 1000V
x86
Cisco Nexus 1000V
x86
NX-OS Data Center Operating System
Data Center Network Manager

Before I/O Consolidation
 Parallel LAN/SAN Infrastructure
LAN SAN A SAN B
 Inefficient use of Network Infrastructure
 5+ connections per server – higher
adapter and cabling costs
• Adds downstream port costs;
cap-ex and op-ex
• Each connection adds additional points of
failure in the fabric
 Multiple switching modules in Blade

Chassis
 Longer lead time for server provisioning
 Multiple fault domains – complex
Server with Blade Chassis diagnostics
NICs and HBAs with I/O Modules
 Management complexity
Ethernet FC
I/O Consolidation
 Reduction of server adapters
LAN SAN A SAN B
 Simplification of access layer and
cabling
 Gateway free implementation – fits in
installed base of existing LAN and
SAN
Nexus 5000 Nexus 5000
 Lower Total Cost of Ownership
 Fewer Cables
 Investment Protection (LANs and
SANs)
Blade Chassis
with Nexus  Consistent Operational Model
4000
Server with
CNAs
Data Center Bridging Ethernet Fibre Channel (FC)

and FCoE
Adapter Evolution:
Consolidation Network Adapter
Operating System View
Evolution of 10G Ethernet Physical Media
Role of Transport in Enabling these Technologies!
Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s
10Mb 100Mb 1Gb 10Gb
UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

SFP Fiber SFP+ Cu (BER better than 10 -18 )
SFP+ Fiber
Cat 6/7
Technology Cable Distance Power Transceiver
(each side) Latency (link)
SFP+ CU Twinax 7m ~0.1W ~0.1μs
Copper
SFP+ USR MM OM2 10m 1W ~0.1μs
Ultra short reach MM OM3 100m
SFP+ SR MM 62.5 μm 26-33m 1W ~0.1μs
Short reach MM 50 μm 66-300m
SFP+ LR SMF G.652 10km 0.5W
Long range
10GBASE-T Cat6 55m ~8W 2.5μs
Cat6a/7 100m ~8W 2.5μs
Cat6a/7 30m ~4W 1.5μs
Introduction to FCoE
What is Fibre Channel over Ethernet?
 From a Fibre Channel standpoint it’s

FC connectivity over a new type of cable called… an Ethernet
cloud
 From an Ethernet standpoints it’s

Yet another ULP (Upper Layer Protocol) to be transported
FCoE is an extension of Fibre Channel

onto a Lossless Ethernet fabric
Unified Fabric Overview
Fibre Channel over Ethernet (FCoE)
FCoE Benefits
•• Mapping
Mapping of of FC
FC Frames
Frames over
over • Fewer Cables
Ethernet
Ethernet •• Both
Both block
block I/O
I/O &
& Ethernet
Ethernet
•• Enables
Enables FC
FC to
to Run
Run traffic
traffic co-exist
co-exist on
on same
same
on
on aa Lossless
Lossless cable
cable
Ethernet
Ethernet Network
Network
• Fewer adapters needed
• Overall less power
Ethernet • Interoperates with
existing SAN’s
Fibre •• Management
Management SAN’s
SAN’s
Channel remains
remains constant
constant
Traffic
• No Gateway
05/26/20
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15 15
FCoE Enablers
 10Gbps Ethernet
 Lossless Ethernet
Matches the lossless behavior guaranteed in FC by B2B credits
 Ethernet jumbo frames
Normal ethernet frame, ethertype = FCoE

Same as a physical FC frame
Ethernet
Header
Header
Header
FCoE
CRC
EOF
FC Payload
FCS
FC
Control information: version, ordered sets (SOF,

EOF)
Unified I/O
Fibre Channel over Ethernet (FCoE)
FCoE is managed like FC at initiator, target, and switch level
Easy to Completely based on the

Understand FC model
Same Same host-to-switch and switch-to-

Operational Model switch behavior as FC
FCoE is
FibreTechniques
Same Channel of e.g. in order delivery,
Traffic Management FSPF load balancing
Same Management WWNs, FC-IDs, hard/soft

and Security Models zoning, DNS, RSCN
Network Stack Comparison
SCSI SCSI SCSI SCSI SCSI
iSCSI FCP FCP FCP
FC FC FC
FCIP Less Overhead

than FCIP, iSCSI
TCP TCP
IP IP FCoE
Ethernet Ethernet Ethernet
PHYSICAL WIRE
SCSI iSCSI FCIP FCoE FC
FCoE Frame Format
Destination MAC Address
Source MAC Address
(IEEE 802.1Q Tag)
ET = FCoE Ver Reserved
Reserved
Reserved
Reserved SOF
Encapsulated FC Frame (with CRC)
EOF Reserved
FCS
FCoE Standards
Defined
A larger picture
 IEEE 802
• Evolution of Ethernet (10 GE, 40 GE, 100 GE, copper and fiber)
• Evolution of switching (Priority Flow Control, Enhanced Transmission,
Congestion Management, Data Center Bridging eXchange)
 INCITS/T11
• Evolution of Fibre Channel (FC-BB-5)
• FCoE (Fibre Channel over Ethernet)
 IETF
• Layer 2 Multi-Path
•TRILL (Transparent Interconnection of Lots of Links)
DCE versus DCB
 DCE is an old Cisco marketing term
 Cisco is now using the term DCB
The term IEEE uses
 Cisco supports the DCB standard activity

By implementing products that are DCB compliant
 CIN-DCBX – Cisco, Intel, Nuova Data Center Bridging Exchange protocol, pre-
standard
 CEE-DCBX – Converged Enhanced Ethernet Data Center Bridging Exchange
protocol, which is standards base
What’s FC-BB-5
 FC-BB-5 covers the majority of the FC features, using

Ethernet
 From an Ethernet perspective, FC-BB-5 is
Ethernet control plane referred to as FIP (Fibre Channel over
Ethernet Initiation Protocol)
discover and build virtual paths between end points
Ethernet data plane providing FCoE forwarding
including both FC control plane and FC data plane (FCF)
FC-BB-6
 It is an active working group of T11 that will discuss the

future of FCoE or FCoE v2.0
 It is just started, 18 months to have a standard
Approximate target spring 2011
 You can track it on

http://www.fcoe.com
Protocol Organization
FCoE itself … FIP (FCoE initiation

 Is the data plane protocol protocol)
 It is used to carry most of the  It is the control plane protocol
FC frames and all the SCSI traffic
 It is used to discover the FC entities
connected to an Ethernet cloud
 It is used to login to and logout from the
FC fabric
The two protocols have:

• Two different Ethertypes
• Two different frame formats
What’s NOT FC-BB-5
 FC-BB-5 doesn’t deal with how lossless is realized in

Ethernet
no Priority Flow Control, Bandwidth Management, etc.
 FC-BB-5 doesn’t deal with management functions
IEEE DCB standards status
DCB technologies allow Ethernet to be lossless and to
manage bandwidth allocation of SAN and LAN flows
Feature / Standard Standards Status
IEEE 802.1Qbb
PAR approved
Priority Flow Control (PFC)
Enable multiple traffic types to share a common 1.0 published
Ethernet link without interfering with each other
IEEE 802.1Qaz
PAR approved
Bandwidth Management (ETS)
Enable consistent management of QoS at the 1.0 published
network level by providing consistent scheduling
Data Center Bridging Exchange

Protocol (DCBX)
This is part of IEEE 802.1Qaz
Management protocol for enhanced Ethernet
capabilities
Data Center Ethernet:
PFC & Bandwidth Management
CoS based
Priority Flow Control Bandwidth Management
Transmit Queues Receive Buffers
Ethernet Link Offered Traffic 10 GE Realized Traffic Utilization
Zero Zero
Zero
3G/s 3G/s 2G/s 3G/s HPC Traffic 2G/s
One One 3G/s
Two Two
3G/s Storage Traffic 3G/s
Three STOP PAUSE Three Eight 3G/s 3G/s 3G/s 3G/s
Virtual
Four Four
Four Lanes
Five Five 3G/s LAN Traffic 5G/s
3G/s 4G/s 6G/s
Six Six
Six 4G/s
Seven
Seven Seven
t1 t2 t3 t1 t2 t3
• Enables lossless behavior • Enables Intelligent sharing of

for each class of service bandwidth between traffic classes
• PAUSE sent per virtual lane control of bandwidth
when buffers limit exceeded • 802.1Qaz Enhanced Transmission
05/26/20
C97-485980-00 © 2008 Cisco Systems, Inc. All rights reserved.
Nuova Systems Inc.
Cisco Confidential
29 29
DCBX Overview
Auto-negotiation of capability and configuration
Priority Flow Control capability and associated CoS values
Allows one link peer to push config to other link peer

Link partners can choose supported features and willingness to accept
Discovers FCoE Capabilities

Responsible for Logical Link Up/Down signaling of
Ethernet and FC
DCBX negotiation failures will result in:
vfc not coming up
Per-priority-pause not enabled on CoS values with PFC configuration
http://download.intel.com/technology/eedc/dcb_cep_spec.pdf
http://www.ieee802.org/1/files/public/docs2008/
FCoE control plane
FIP: FCoE Initialization Protocol
 FCoE VLAN discovery
Automatic discovery of FCoE VLANs
 Device discovery
ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual
Links
VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for
VE_Port to VE_Port Virtual Links
The protocol verifies the Lossless Ethernet network supports the required Max
FCoE Size
 Virtual Link instantiation
Builds on the existing Fibre Channel Login process, adding the Negotiation of the
MAC address to use
Fabric Provided MAC Address (FPMA), or
Server Provided MAC Address (SPMA)
 Virtual Links maintenance
Timer based
Server Provided MAC Fabric Provided MAC
Addresses Addresses
Adapter uses burned-in or configured MAC address assigned for each FC_ID:
MAC address: Consistent with the Fibre Channel model
Multiple FC-MAPs may be supported
Consistent with the Ethernet
model One per SAN
No table needed for Encapsulation
FCF needs a table to map between
MAC addresses and FC_IDs Multiple MACs may be needed for NPIV
FC-MAP FC-ID
MAC (0E-FC-00) 7.8.9
Address 24 24
bits bits
Burned in or Configured FC-MAP FC-ID
(0E-FC-00) 7.8.9
48 48
bits bits
Cisco Nexus 5000 uses FPMA
Initial Login Flow ladder
ENode FCoE Switch
VLAN VLAN
Discovery Discovery
Solicitat
ion FIP:
FCF FCF FCoE
Discovery Advertisement Discovery Initialization
Protocol
FLOGI/FDISC FLOGI/FDISC Accept
FC Command FCOE
FC Command responses Protocol
FCoE data plane
ENode: Simplified Model
 ENode (FCoE Node): a Fibre Channel HBA implemented within an
Ethernet NIC aka CNA (Converged Network Adapter)
 FCoE LEP : The data forwarding component that handles FC
frame encapsulation/decapsulation
 FCoE Controller is the functional entity that performs the FIP and
instantiates VN_Port/FCoE_LEP pairs.
FC Node
FCoE_Controller FCoE_Controller
FCoE_LEP FCoE_LEP
Enet Enet
port port
FCoE Switch: Simplified Model
 FCF (Fibre Channel Forwarder), the forwarding entity

inside an FCoE switch
FC
port
FCoE Switch
FC
FCF port
FCoE_Controller
FC
FCoE_LEP
port
Ethernet Bridge FC
port
Eth Eth Eth Eth Eth Eth Eth Eth

port port port port port port port port
FCoE Network
Topology
FCoE: Initial Deployment
SAN A SAN B
10GE
Backbone
VF_Ports Nexus 5000 (FCF)
VN_Ports
10GE
4/8 Gbps FC
FCoE: Adding Blade Servers
SAN A SAN B
10GE
Backbone
VF_Ports
10GE
VN_Ports 4/8 Gbps FC
FCoE: Adding Native FCoE Storage
SAN A SAN B
10GE
Backbone
VN_Ports
VF_Ports
10GE
FCoE: Adding VE_ports
SAN A
SAN B
10GE
Backbone
VE_Ports
VF_Ports
10GE
Nexus Topologies
The Unified Data Center Architecture
Core: L3 boundary to the DC network. Functional
point for route summarization, the injection of default
L3 NEXUS 7000
routes and termination of segmented virtual
transport networks
Service Service Aggregation: Typical L3/L2 boundary. DC

L3 Appliances NEXUS 7000 - Modules aggregation point for uplink and DC services offering
VPC key features: VPC, VDC, 10GE density and 1st point
L2 Catalyst
6500
of migration to 40GE and 100GE
Access: Classic network layer providing non-

NEXUS 7000 - Unified blocking paths to servers & IP storage devices
L2 NEXUS 5000
VPC Compute through VPC. It leverages Distributed Access Fabric
System Model (DAF) to centralize config & mgmt and ease
horizontal cabling demands related to 1G and 10GE
A B server environments
NEXUS 2000 Virtual Access: A virtual layer of network intelligence

vL2 NEXUS 1000v
offering access layer-like controls to extend
VM VM VM VM
VM VM VM
VM VM VM
VM VM
VM VM traditional visibility, flexibility and mgmt into virtual
VM VM
VM VM VM VM
server environments. Virtual network switches bring
VM VM VM
VM VM VM VM VM
VM VM
VM VM VM VM VM
access layer switching capabilities to virtual servers

VM VM VM VM VM
VM VM VM VM VM
VM VM VM VM VM
VM VM VM
VM VM VM
VM VM
VM VM without burden of topology control plane protocols.
Virtual Adapters provide granular control over virtual
and physical server IO resources
POD
POD
Rack 1 Rack 2 Rack 3 Rack 1 Rack x
Fitting the pieces together…
DC Core
Gigabit Ethernet
Nexus 7000 WAN 10 Gigabit Ethernet
10GbE Core
IP+MPLS WAN 4, 8Gb Fibre Channel
Agg Router 10 Gigabit FCoE/DCE
DC Aggregation
SAN A/B
Nexus 7000 MDS 9500
Catalyst 6500 10GbE Agg
10GbE VSS Agg Storage
Catalyst 6500 Services
DC Services DC Services
DC Access
Catalyst 6500 Catalyst 49xx CBS 3100 Nexus 7000 Nexus 5K|2K UCS blade MDS 9500
End-of-Row Rack | MDS 9100 End-of-Row Top of Rack or Storage
Blade Nexus 4K
Nexus 1000V VN-Link
C97-485980-00
1GbE Server Access
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Confidential
1GbE,10GbE Server Access Storage 53
Nexus 5000 and
FC Connectivity
Switch Mode
 Nexus 5000 FC module can be ISL’ed to another FC switch (E_port)
 Zoning, DPVM, etc. are enforced on the Nexus 5000
 Domain manager, FSPF, zone server, fabric login server, name
server run on Nexus 5000
 Require a domain ID for every VSAN
 Interop mode considerations when connecting to non-Cisco FC
switches
 Note: Nexus 5000 supports direct connectivity to FC initiator (server

HBAs) and targets (storage arrays)
N-Port Virtualization (NPV) mode
 Nexus 5000 FC module can work in NPV mode
Server-facing ports are regular F ports
Uplinks toward SAN core fabric are NP ports
 Nexus 5000 switches assign FCIDs to attached devices
First byte in FCID received from core SAN switch
 One VSAN per uplink on Nexus 5000 (will change in future)
No trunking or channelling of NP ports
 Zoning, DPVM, etc. are not enforced on the Nexus 5000
 Domain manager, FSPF, zone server, fabric login server, name server
They do not run on Nexus 5000
 No local switching
All traffic routed via the core SAN switches
N-Port Virtualization (NPV): An Overview
NPV-Core Switch (MDS or 3rd party switch with NPIV support)
FC
F-port
5
AN
NP-port VS N 10
VS A Can have multiple
uplinks – one VSAN per uplink
Two uplinks can be in the same VSAN
No port channel or trunking
F-ports
Host
Host
N-ports Host Nexus 5000 to SAN Fabric A & B
Assign FCIDs to servers – no domain to configure!
Servers log in (FLOGI) locally
Working with
Nexus 2148
(Optional)
Nexus 2000 Fabric Extender
Virtual Chassis
Nexus 2000 Fabric Extender
1GE Connectivity
Fabric Extender Static Pinning
Uplink Modes
 Fabric Extender associates (pins)
a server side (1GE) port with an
uplink (10GE) port
 Server ports are either individually
pinned to specific uplinks (static
pinning) or all interfaces pinned to Server Interface
a single logical port channel goes down
 Behaviour on FEX uplink failure
depends on the configuration Port Channel
 Static Pinning – Server ports
pinned to the specific uplink are
brought down with the failure of
the pinned uplink
 Port Channel – Server traffic is
shifted to remaining uplinks based
on port channel hash
Server Interface
stays active

FCoE Bootcamp

Uploaded by

Copyright:

Available Formats

FCoE Bootcamp

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

FCoE Bootcamp

Uploaded by

Copyright:

Available Formats

FCOE

Multi-Core CPU architectures allowing

Server virtualization driving the need for

Growing need for network storage driving

Increasing adoption of Blades in data

10G LOM on server Motherboard

•Enable Virtualization •Scale to 40G and

NX-OS Data Center Operating System

Data Center Network Manager

 Multiple switching modules in Blade

Data Center Bridging Ethernet Fibre Channel (FC)

10Mb 100Mb 1Gb 10Gb

UTP Cat 3 UTP Cat 5 UTP Cat 5 X2

 From a Fibre Channel standpoint it’s

 From an Ethernet standpoints it’s

FCoE is an extension of Fibre Channel

Normal ethernet frame, ethertype = FCoE

Control information: version, ordered sets (SOF,

Easy to Completely based on the

Same Same host-to-switch and switch-to-

Same Management WWNs, FC-IDs, hard/soft

SCSI SCSI SCSI SCSI SCSI

iSCSI FCP FCP FCP

FCIP Less Overhead

Ethernet Ethernet Ethernet

Destination MAC Address

Source MAC Address

(IEEE 802.1Q Tag)

ET = FCoE Ver Reserved

Encapsulated FC Frame (with CRC)

 Cisco supports the DCB standard activity

 FC-BB-5 covers the majority of the FC features, using

 It is an active working group of T11 that will discuss the

 You can track it on

FCoE itself … FIP (FCoE initiation

The two protocols have:

 FC-BB-5 doesn’t deal with how lossless is realized in

 FC-BB-5 doesn’t deal with management functions

Data Center Bridging Exchange

• Enables lossless behavior • Enables Intelligent sharing of

Allows one link peer to push config to other link peer

Discovers FCoE Capabilities

FLOGI/FDISC FLOGI/FDISC Accept

 FCF (Fibre Channel Forwarder), the forwarding entity

Eth Eth Eth Eth Eth Eth Eth Eth

VF_Ports Nexus 5000 (FCF)

Service Service Aggregation: Typical L3/L2 boundary. DC

Access: Classic network layer providing non-

NEXUS 2000 Virtual Access: A virtual layer of network intelligence

access layer switching capabilities to virtual servers

 Note: Nexus 5000 supports direct connectivity to FC initiator (server

NPV-Core Switch (MDS or 3rd party switch with NPIV support)

Servers log in (FLOGI) locally

You might also like