Cisco Catalyst 6500/6000 Switch High CPU Utilization

Table of Contents
Catalyst 6500/6000 Switch High CPU Utilization............................................................................................1
Document ID: 63992................................................................................................................................1
Introduction..........................................................................................................................................................1
Prerequisites.........................................................................................................................................................1
Requirements..........................................................................................................................................1
Components Used...................................................................................................................................2
Conventions............................................................................................................................................2
Difference Between CatOS and Cisco IOS System Software.............................................................................2
Understand CPU Utilization on Catalyst 6500/6000 Switches............................................................................2
Situations and Features That Trigger Traffic to Go to Software.........................................................................3
Packets That Are Destined to the Switch................................................................................................3
Packets and Conditions That Require Special Processing......................................................................3
ACLBased Features..............................................................................................................................4
NetFlowBased Features........................................................................................................................5
Multicast Traffic.....................................................................................................................................5
Other Features.........................................................................................................................................6
IPv6 Situations........................................................................................................................................6
Common Causes and Solutions for High CPU Utilization Issues.......................................................................6
IP Unreachables......................................................................................................................................6
Use of CEF FIB Table Space in the Flow Cache Table.........................................................................6
Optimized ACL Logging........................................................................................................................8
Rate Limit of Packets to the CPU...........................................................................................................8
Physical Merger of VLANs Due to Incorrect Cabling...........................................................................9
Check CPU Utilization........................................................................................................................................9
Utilities and Tools to Determine the Traffic That Is Punted to the CPU...........................................................10
Cisco IOS System Software..................................................................................................................10
CatOS System Software........................................................................................................................12
Recommendations..............................................................................................................................................13
NetPro Discussion Forums Featured Conversations......................................................................................14
Related Information...........................................................................................................................................14

Catalyst 6500/6000 Switch High CPU Utilization

Document ID: 63992
Introduction
Prerequisites
Requirements
Components Used
Conventions
Difference Between CatOS and Cisco IOS System Software
Understand CPU Utilization on Catalyst 6500/6000 Switches
Situations and Features That Trigger Traffic to Go to Software
Packets That Are Destined to the Switch
Packets and Conditions That Require Special Processing
ACLBased Features
NetFlowBased Features
Multicast Traffic
Other Features
IPv6 Situations
Common Causes and Solutions for High CPU Utilization Issues
IP Unreachables
Use of CEF FIB Table Space in the Flow Cache Table
Optimized ACL Logging
Rate Limit of Packets to the CPU
Physical Merger of VLANs Due to Incorrect Cabling
Check CPU Utilization
Utilities and Tools to Determine the Traffic That Is Punted to the CPU
Cisco IOS System Software
CatOS System Software
Recommendations
NetPro Discussion Forums Featured Conversations
Related Information

Introduction
This document describes causes of high CPU utilization on Cisco Catalyst 6500/6000 series switches. Like
Cisco routers, switches use the show processes cpu command in order to show CPU utilization for the switch
Supervisor Engine processor. However, due to the differences in architecture and forwarding mechanisms
between Cisco routers and switches, the typical output of the show processes cpu command differs
significantly. The meaning of the output differs, too. This document clarifies these differences. The document
describes use of the CPU on the switches and how to interpret the show processes cpu command output.
Note: In this document, the words "switch" and "switches" refer to the Catalyst 6500/6000 switches.

Prerequisites
Requirements
There are no specific requirements for this document.

Cisco Catalyst 6500/6000 Switch High CPU Utilization

Components Used
The information in this document is based on the software and hardware versions for:
Catalyst 6500/6000 switches
The information in this document was created from the devices in a specific lab environment. All of the
devices used in this document started with a cleared (default) configuration. If your network is live, make sure
that you understand the potential impact of any command.

Conventions
For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Difference Between CatOS and Cisco IOS System Software

Catalyst OS (CatOS) on the Supervisor Engine and Cisco IOS Software on the Multilayer Switch
Feature Card (MSFC) (Hybrid): You can use a CatOS image as the system software to run the Supervisor
Engine on Catalyst 6500/6000 switches. If the optional MSFC is installed, a separate Cisco IOS Software
image is used to run the MSFC.
Cisco IOS Software on both the Supervisor Engine and MSFC (Native): You can use a single Cisco IOS
Software image as the system software to run both the Supervisor Engine and MSFC on Catalyst 6500/6000
switches.
Note: For more information, refer to Comparison of the Cisco Catalyst and Cisco IOS Operating Systems for
the Cisco Catalyst 6500 Series Switch.

Understand CPU Utilization on Catalyst 6500/6000 Switches

Cisco softwarebased routers use software in order to process and route packets. CPU utilization on a Cisco
router tends to increase as the router performs more packet processing and routing. Therefore, the show
processes cpu command can provide a fairly accurate indication of the traffic processing load on the router.
Catalyst 6500/6000 switches do not use the CPU in the same way. These switches make forwarding decisions
in hardware, not in software. Therefore, when the switches make the forwarding or switching decision for
most frames that pass through the switch, the process does not involve the Supervisor Engine CPU.
On Catalyst 6500/6000 switches, there are two CPUs. One CPU is the Supervisor Engine CPU, which is
called the Network Management Processor (NMP) or the Switch Processor (SP). The other CPU is the Layer
3 (L3) routing engine CPU, which is called the MSFC or the Route Processor (RP).
The SP CPU performs functions that include:
Assists in MAC address learning and aging
Note: MAC address learning is also called path setup.
Runs protocols and processes that provide network control
Examples include Spanning Tree Protocol (STP), Cisco Discovery Protocol (CDP), VLAN Trunk
Protocol (VTP), Dynamic Trunking Protocol (DTP), and Port Aggregation Protocol (PAgP).
Cisco Catalyst 6500/6000 Switch High CPU Utilization

 Handles network management traffic that is destined to the CPU of the switch
Examples include Telnet, HTTP, and Simple Network Management Protocol (SNMP) traffic.
The RP CPU performs functions that include:
Builds and updates the L3 routing and Address Resolution Protocol (ARP) tables
Generates the Cisco Express Forwarding (CEF) Forwarding Information Base (FIB) and adjacency
tables, and downloads the tables into the Policy Feature Card (PFC)
Handles network management traffic that is destined to the RP
Examples include Telnet, HTTP, and SNMP traffic.

Situations and Features That Trigger Traffic to Go to

Software
Packets That Are Destined to the Switch
Any packet that is destined to the switch goes to software. Such packets include:
Control packets
Control packets are received for STP, CDP, VTP, Hot Standby Router Protocol (HSRP), PAgP, Link
Aggregation Control Protocol (LACP), and UniDirectional Link Detection (UDLD).
Routing protocol updates
Examples of these protocols are Routing Information Protocol (RIP), Enhanced Interior Gateway
Routing Protocol (EIGRP), Border Gateway Protocol (BGP), and Open Shortest Path First Protocol
(OSPF Protocol).
SNMP traffic that is destined to the switch
Telnet and Secure Shell Protocol (SSH) traffic to the switch
ARP responses to ARP requests

Packets and Conditions That Require Special Processing

This list provides specific packet types and conditions that force packets to be handled in software:
Packets with IP options, an expired Time to Live (TTL), or nonAdvanced Research Projects Agency
(ARPA) encapsulation
Packets with special handling, such as tunneling
IP fragmentation
Packets that require Internet Control Message Protocol (ICMP) messages from the RP or SP
Maximum transmission unit (MTU) check failure
Packets with IP errors, which include IP checksum and length errors
Adjacency same interface
Packets that fail the Reverse Path Forwarding (RPF) checkrpffailure
Glean/receive
Glean refers to packets that require ARP resolution, and receive refers to packets that fall in the
receive case.

Cisco Catalyst 6500/6000 Switch High CPU Utilization

 Internetwork Packet Exchange (IPX) traffic that is softwareswitched on the Supervisor Engine 720
in both Cisco IOS Software and CatOS
IPX traffic is also softwareswitched on the Supervisor Engine 2/Cisco IOS Software, but the traffic
is hardwareswitched on the Supervisor Engine 2/CatOS. IPX traffic is hardwareswitched on the
Supervisor Engine 1A for both operating systems.
AppleTalk traffic
Hardware resources full conditions
These resources include FIB, contentaddressable memory (CAM), and ternary CAM (TCAM).

ACLBased Features
Access control list (ACL)denied traffic with the ICMP unreachables feature turned on
Note: This is the default.
Some ACLdenied packets are leaked to the MSFC if IP unreachables are enabled. Packets that
require ICMP unreachables are leaked at a userconfigurable rate. By default, the rate is 500 packets
per second (pps).
IPX filtering on the basis of unsupported parameters, such as source host
On the Supervisor Engine 720, the process of L3 IPX traffic is always in software.
Access control entries (ACEs) that require logging, with the log keyword
This applies to ACL log and VLAN ACL (VACL) log features. ACEs in the same ACL that do not
require logging still process in hardware. The Supervisor Engine 720 with PFC3 supports the rate
limit of packets that are redirected to the MSFC for ACL and VACL logging. The Supervisor Engine
2 supports the rate limit of packets that are redirected to the MSFC for VACL logging. Support for
ACL logging on the Supervisor Engine 2 is scheduled for the Cisco IOS Software Release 12.2S
branch.
Policyrouted traffic, with use of match length, set ip precedence, or other unsupported parameters
The set interface parameter has support in software. However, the set interface null 0 parameter is
an exception. This traffic is handled in hardware on the Supervisor Engine 2 with PFC2 and the
Supervisor Engine 720 with PFC3.
NonIP and nonIPX router ACLs (RACLs)
NonIP RACLs apply to all Supervisor Engines. The nonIPX RACLs apply to the Supervisor
Engine 1a with PFC and the Supervisor Engine 2 with PFC2 only.
Broadcast traffic that is denied in an RACL
Traffic that is denied in a unicast RPF (uRPF) check, ACL ACE
This uRPF check applies to the Supervisor Engine 2 with PFC2 and Supervisor Engine 720 with
PFC3.
Authentication proxy
Traffic that is subject to authentication proxy can be ratelimited on the Supervisor Engine 720.
Cisco IOS Software IP Security (IPSec)
Traffic that is subject to Cisco IOS encryption can be ratelimited on the Supervisor Engine 720.

Cisco Catalyst 6500/6000 Switch High CPU Utilization

NetFlowBased Features
The NetFlowbased features that this section describes apply to the Supervisor Engine 2 and Supervisor
Engine 720 only.
NetFlowbased features always need to see the first packet of a flow in software. Once the first
packet of the flow reaches software, subsequent packets for the same flow are hardwareswitched.
This flow arrangement applies to reflexive ACLs, Web Cache Communication Protocol (WCCP), and
Cisco IOS Server Load Balancing (SLB).
Note: On the Supervisor Engine 1, reflexive ACLs rely on dynamic TCAM entries to create hardware
shortcuts for a particular flow. The principle is the same: the first packet of a flow goes to software.
Subsequent packets for that flow are hardwareswitched.
With the TCP Intercept feature, the threeway handshake and session close are handled in software.
The rest of the traffic is handled in hardware.
Note: Synchronize (SYN), SYN acknowledge (SYN ACK), and ACK packets comprise the
threeway handshake. Session close occurs with finish (FIN) or reset (RST).
With Network Address Translation (NAT), traffic is handled in this way:
On the Supervisor Engine 720:
Traffic that requires NAT is handled in hardware after the initial translation. Translation of
the first packet of a flow occurs in software, and subsequent packets for that flow are
hardwareswitched. For TCP packets, a hardware shortcut is created in the NetFlow table
after completion of the TCP threeway handshake.
On the Supervisor Engine 2 and Supervisor Engine 1:
All traffic that requires NAT is softwareswitched.
Contextbased Access Control (CBAC) uses NetFlow shortcuts in order to classify traffic that
requires inspection. Then, CBAC sends only this traffic to software. CBAC is a softwareonly
feature; traffic that is subject to inspection is not hardwareswitched.
Note: Traffic that is subject to inspection can be ratelimited on the Supervisor Engine 720.

Multicast Traffic
Protocol Independent Multicast (PIM) snooping
Internet Group Management Protocol (IGMP) snooping (TTL = 1)
This traffic is indeed destined to the router.
Multicast Listener Discovery (MLD) snooping (TTL = 1)
This traffic is indeed destined to the router.
FIB miss
Multicast packets for registration that have direct connection to the multicast source
These multicast packets are tunneled to the rendezvous point.
IP version 6 (IPv6) multicast

Cisco Catalyst 6500/6000 Switch High CPU Utilization

Other Features
NetworkBased Application Recognition (NBAR)
ARP Inspection, with CatOS only
Port Security, with CatOS only
DHCP snooping

IPv6 Situations
Packets with a hopbyhop option header
Packets with the same destination IPv6 address as that of routers
Packets that fail the scope enforcement check
Packets that exceed the MTU of the output link
Packets with a TTL that is less than or equal to 1
Packets with an input VLAN that equals the output VLAN
IPv6 uRPF
Software performs this uRPF for all packets.
IPv6 reflexive ACLs
Software handles these reflexive ACLs.
6to4 prefixes for IPv6 IntraSite Automatic Tunnel Addressing Protocol (ISATAP) tunnels
Software handles this tunneling. All other traffic that enters an ISATAP tunnel is hardwareswitched.

Common Causes and Solutions for High CPU Utilization

Issues
IP Unreachables
When an access group denies a packet, the MSFC sends ICMP unreachable messages. This action occurs by
default.
With the default enablement of the ip unreachables command, the Supervisor Engine drops most of the
denied packets in hardware. Then, the Supervisor Engine sends only a small number of packets, a maximum
of 10 pps, to the MSFC for drop. This action generates ICMPunreachable messages.
The drop of denied packets and generation of ICMPunreachable messages imposes a load on the MSFC
CPU. In order to eliminate the load, you can issue the no ip unreachables interface configuration command.
This command disables ICMPunreachable messages, which allows the drop in hardware of all access
groupdenied packets.
ICMPunreachable messages are not sent if a VACL denies a packet.

Use of CEF FIB Table Space in the Flow Cache Table

The Supervisor Engine 1 has a Flow Cache Table that supports 128,000 entries. However, on the basis of the
efficiency of the hashing algorithm, these entries range from 32,000 to 120,000. On the Supervisor Engine 2,
the FIB table is generated and programmed into the PFC. The table holds as many as 256,000 entries. The
Supervisor Engine 720 with PFC3BXL supports up to 1,000,000 entries. Once this space is exceeded, the
Cisco Catalyst 6500/6000 Switch High CPU Utilization

packets become switched in software. This can cause high CPU utilization on the RP. In order to check the
number of routes in the CEF FIB table, use these commands:
Router# show processes cpu
CPU utilization for five seconds: 99.26%
one minute: 100.00%
five minutes: 100.00%
PID

1
2
3
4
5

Runtime(ms)

0
2
0
0
653

Invoked

0
245
1
1
11737

uSecs
5Sec
1Min
5Min

0
0.74%
0.00%
0.00%
1000
0.00%
0.00%
0.00%
0
0.00%
0.00%
0.00%
0
0.00%
0.00%
0.00%
1000
0.00%
0.00%
0.00%

TTY

2
2
2
2
2

Process

Kernel and Idle

Flash MIB Updat
L2L3IntHdlr
L2L3PatchRev
SynDi

1000
8000
501000
0

0
0
0
2

L3Aging
NetFlow
Fib
Fib_bg_task

! Output suppressed.
26
27
28
29

10576
47432
6758259
0

615970
51696
1060831
1

0.00%
0.02%
96.62%
0.00%

0.00%
0.00%
96.00%
0.00%

0.00%
0.00%
96.00%
0.00%

! Output suppressed.

CATOS% show mls cef

Total L3 packets switched:
Total L3 octets switched:
Total route entries:
IP route entries:
IPX route entries:
IPM route entries:
IP load sharing entries:
IPX load sharing entries:
Forwarding entries:
Bridge entries:
Drop entries:

124893998234
53019378962495
112579
112578
1
0
295
0
112521
56
2

IOS% show ip cef summary

IP Distributed CEF with switching (Table Version 86771423), flags=0x0
112564 routes, 1 reresolve, 0 unresolved (0 old, 0 new)
112567 leaves, 6888 nodes, 21156688 bytes, 86771426
inserts, 86658859
invalidations
295 load sharing elements, 96760 bytes, 112359 references
universal perdestination load sharing algorithm, id 8ADDA64A
2 CEF resets, 2306608 revisions of existing leaves
refcounts: 1981829 leaf, 1763584 node
! You see these messages if the TCAM space is exceeded:
%MLSCEFSP7FIB_EXCEPTION: FIB TCAM exception, Some entries will be software switched
%MLSCEFSP7END_FIB_EXCEPTION: FIB TCAM exception cleared, all CEF entries will be
hardware switched

On the Supervisor Engine 2, the number of FIB entries reduces to half if you have configured RPF check on
the interfaces. This configuration can lead to the software switch of more packets and, consequently, high
CPU utilization.
For additional information about TCAM utilization and optimization, refer to this document:
Cisco Catalyst 6500/6000 Switch High CPU Utilization

 Understanding ACL on Catalyst 6500 Series Switches

Optimized ACL Logging

Optimized ACL Logging (OAL) provides hardware support for ACL logging. Unless you configure OAL, the
process of packets that require logging takes place completely in software on the MSFC3. OAL permits or
drops packets in hardware on the PFC3. OAL uses an optimized routine to send information to the MSFC3 in
order to generate the logging messages.
Note: For information about OAL, refer to the Optimized ACL Logging on Supervisor Engine 720 section of
the document Understanding Cisco IOS ACL Support.

Rate Limit of Packets to the CPU

On the Supervisor Engine 720, rate limiters can control the rate at which packets can go to software. This rate
control helps prevent denialofservice attacks. You can also use a few of these rate limiters on the
Supervisor Engine 2:
Router# show mls ratelimit
Rate Limiter Type
Status

MCAST NON RPF

Off
MCAST DFLT ADJ
On
MCAST DIRECT CON
Off
ACL BRIDGED IN
Off
ACL BRIDGED OUT
Off
IP FEATURES
Off
ACL VACL LOG
On
CEF RECEIVE
Off
CEF GLEAN
Off
MCAST PARTIAL SC
On
IP RPF FAILURE
On
TTL FAILURE
Off
ICMP UNREAC. NOROUTE
On
ICMP UNREAC. ACLDROP
On
ICMP REDIRECT
Off
MTU FAILURE
Off
LAYER_2 PDU
Off
LAYER_2 PT
Off
IP ERRORS
On
CAPTURE PKT
Off
MCAST IGMP
Off

Packets/s

100000

2000

100000
500

500
500

500

Burst

100

100
10

10
10

10

Router(config)# mls ratelimit ?

all
Rate Limiting for both Unicast and Multicast packets
layer2
layer2 protocol cases
multicast Rate limiting for Multicast packets
unicast
Rate limiting for Unicast packets

Here is an example:
Router(config)# mls ratelimit layer2 l2pt 3000

In order to ratelimit all CEFpunted packets to the MSFC, issue the command that is in this example:
Router(config)# mls ip cef ratelimit 50000

Cisco Catalyst 6500/6000 Switch High CPU Utilization

Physical Merger of VLANs Due to Incorrect Cabling

High CPU utilization also can result from the merge together of two or more VLANs due to improper cabling.
Also, if STP is disabled on those ports where the VLAN merger happens, high CPU utilization can occur.
In order to resolve this problem, identify the cabling errors and correct them. If your requirement allows, you
can also enable STP on those ports.

Check CPU Utilization

If the CPU utilization is high, issue the show processes cpu command first. The output shows you the CPU
utilization on the switch as well as the CPU consumption by each process.
Router# show processes cpu
CPU utilization for five seconds: 57%/48%; one minute: 56%; five minutes: 48%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
1
0
5
0 0.00% 0.00% 0.00%
0 Chunk Manager
2
12
18062
0 0.00% 0.00% 0.00%
0 Load Meter
4
164532
13717
11994 0.00% 0.21% 0.17%
0 Check heaps
5
0
1
0 0.00% 0.00% 0.00%
0 Pool Manager
! Output suppressed.
172
173
174

0
243912
68

9
2171455
463

0
112
146

0.00%
9.25%
0.00%

0.00%
8.11%
0.00%

0.00%
7.39%
0.00%

0 RPC aapi_rp
0 SNMP ENGINE
0 RPC pmmp

! Output suppressed.

In this output, the total CPU utilization is 57 percent and the interrupt CPU utilization is 48 percent. Here,
these percentages appear in boldface text. The interrupt switch of traffic by the CPU causes the interrupt CPU
utilization. The command output lists the processes that cause the difference between the two utilizations. In
this case, the cause is the SNMP process.
On the Supervisor Engine that runs CatOS, the output looks like this:
Switch> (enable) show processes cpu

CPU utilization for five seconds: 99.72%

one minute: 100.00%
five minutes: 100.00%
PID

1
2
3
4

Runtime(ms)

0
2
0
0

Invoked

0
261
1
1

uSecs
5Sec
1Min
5Min

0
0.28%
0.00%
0.00%
1000
0.00%
0.00%
0.00%
0
0.00%
0.00%
0.00%
0
0.00%
0.00%
0.00%

TTY

2
2
2
2

Process

Kernel and Idle

Flash MIB Updat
L2L3IntHdlr
L2L3PatchRev

! Output suppressed.
61
62
63

727295
18185410
845683

172025
3712736
91691

18000
106000
105000

0.82%
22.22%
0.92%

Cisco Catalyst 6500/6000 Switch High CPU Utilization

0.00%
21.84%
0.00%

0.00% 2
21.96% 2
0.00% 2

SptTimer
SptBpduRx
SptBpduTx

In this output, the first process is Kernel and Idle, which shows idle CPU utilization. This process is
normally high, unless some other processes consume CPU cycles. In this example, the SptBpduRx process
causes high CPU utilization.
If the CPU utilization is high due to one of these processes, you can troubleshoot and determine why this
process runs high. But, if the CPU is high due to traffic being punted to the CPU, you need to determine why
the traffic is being punted. This determination can help you identify what the traffic is.

Utilities and Tools to Determine the Traffic That Is Punted

to the CPU
This section identifies some utilities and tools that can help you look at this traffic.

Cisco IOS System Software

In Cisco IOS Software, the Switch processor on the Supervisor Engine is referred to as the SP, and the MSFC
is called the RP.
The show interface command gives basic information on the state of the interface and the traffic rate on the
interface. The command also provides error counters.
Router# show interface gigabitethernet 4/1
GigabitEthernet4/1 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is 000a.42d1.7580 (bia 000a.42d1.7580)
Internet address is 100.100.100.2/24
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Halfduplex, 100Mb/s
input flowcontrol is off, output flowcontrol is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 5/75/1/24075 (size/max/drops/flushes); Total output drops: 2
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 7609000 bits/sec, 14859 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
L2 Switched: ucast: 0 pkt, 184954624 bytes mcast: 1 pkt, 500 bytes
L3 in Switched: ucast: 2889916 pkt, 0 bytes mcast: 0 pkt, 0 bytes mcast
L3 out Switched: ucast: 0 pkt, 0 bytes mcast: 0 pkt, 0 bytes
2982871 packets input, 190904816 bytes, 0 no buffer
Received 9 broadcasts, 0 runts, 0 giants, 0 throttles
1 input errors, 1 CRC, 0 frame, 28 overrun, 0 ignored
0 input packets with dribble condition detected
1256 packets output, 124317 bytes, 0 underruns
2 output errors, 1 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out

In this output, you can see that the incoming traffic is L3switched instead of Layer 2 (L2)switched. This
indicates that the traffic is being punted to the CPU.

Cisco Catalyst 6500/6000 Switch High CPU Utilization

The show processes cpu command tells you whether these packets are regular traffic packets or control
packets.
Router# show processes cpu | exclude 0.00
CPU utilization for five seconds: 91%/50%; one minute: 89%; five minutes: 47%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
5
881160
79142
11133 0.49% 0.19% 0.16%
0 Check heaps
98
121064
3020704
40 40.53% 38.67% 20.59%
0 IP Input
245
209336
894828
233 0.08% 0.05% 0.02%
0 IFCOM Msg Hdlr

If the packets are processswitched, you see that the IP Input process runs high. Issue this command in
order to see these packets:
show buffers inputinterface
Router# show buffers inputinterface gigabitethernet 4/1 packet
Buffer information for Small buffer at 0x437874D4
data_area 0x8060F04, refcount 1, next 0x5006D400, flags 0x280
linktype 7 (IP), enctype 1 (ARPA), encsize 14, rxtype 1
if_input 0x505BC20C (GigabitEthernet4/1), if_output 0x0 (None)
inputtime 00:00:00.000 (elapsed never)
outputtime 00:00:00.000 (elapsed never), oqnumber 65535
datagramstart 0x8060F7A, datagramsize 60, maximum size 308
mac_start 0x8060F7A, addr_start 0x8060F7A, info_start 0x0
network_start 0x8060F88, transport_start 0x8060F9C, caller_pc 0x403519B4
source: 100.100.100.1, destination: 100.100.100.2, id: 0x0000, ttl: 63,
TOS: 0 prot: 17, source port 63, destination port 63
08060F70:
08060F80:
08060F90:
08060FA0:
08060FB0:

00000000
3F11EAF3
001A261F
0C0D0E0F

000A 42D17580
11110800 4500002E 00000000
64646401 64646402 003F003F
00010203 04050607 08090A0B
101164

..BQu.
........E.......
?.jsddd.ddd..?.?
..&.............
......d

If the traffic is interrupt switched, you cannot see those packets with the show buffers inputinterface
command. In order to see the packets that are punted to the RP for interrupt switching, you can perform a
Switched Port Analyzer (SPAN) capture of the RP port.
Note: Refer to this document for additional information about interruptswitched versus processswitched
CPU utilization:
High CPU Utilization due to Interrupts section of the document Troubleshooting High CPU
Utilization on Cisco Routers
SPAN RPInband and SPInband
A SPAN for the RP or SP port in Cisco IOS Software is available in Cisco IOS Software Release 12.1(19)E
and later.
This is the command syntax:
test monitor session 166 add {rpinband | spinband} [rx | tx | both]

Here is an example on an RP console:

Router# monitor session 1 source interface fast 3/3

Cisco Catalyst 6500/6000 Switch High CPU Utilization

! Use any interface that is administratively shut down.

Router# monitor session 1 destination interface 3/2

Now, go to the SP console. Here is an example:

Routersp# test monitor session 1 add rpinband rx
Router# show monitor
Session 1

Type : Local Session

Source Ports :
Both : Fa3/3
Destination Ports : Fa3/2
SP console:
Routersp# test monitor session 1 show
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

Here is an example on an SP console:

Routersp# test monitor session 1 show
Ingress Source Ports: 3/3 15/1
Egress Source Ports: 3/3
Ingress Source Vlans: <empty>
Egress Source Vlans: <empty>
Filter Vlans: <empty>
Destination Ports: 3/2

CatOS System Software

For switches that run CatOS system software, the Supervisor Engine runs CatOS and the MSFC runs Cisco
IOS Software.
If you issue the show mac command, you can see the number of frames that are punted to the MSFC. Port
15/1 is the Supervisor Engine connection to the MSFC.
Note: The port is 16/1 for Supervisor Engines in slot 2.
Console> (enable) show mac 15/1
Port
RcvUnicast
RcvMulticast
RcvBroadcast

15/1
193576
0
1
Port
XmitUnicast
XmitMulticast
XmitBroadcast

15/1
3
0
0
Port
RcvOctet
XmitOctet

15/1
18583370
0
MAC

DelyExced MTUExced

InDiscard OutDiscard

Cisco Catalyst 6500/6000 Switch High CPU Utilization


15/1
0

0
0

A quick increase in this number indicates that packets are punted to the MSFC, which causes high CPU
utilization. You can then look at the packets in these ways:
SPAN MSFC port 15/1 or 16/1
SPAN sc0
SPAN MSFC Port 15/1 or 16/1
Set up a SPAN session in which the source is the MSFC port 15/1 (or 16/1) and the destination is an Ethernet
port.
Here is an example:
Console> (enable) set span 15/1 5/10
Console> (enable) show span
Destination
:
Admin Source
:
Oper Source
:
Direction
:
Incoming Packets:
Learning
:
Multicast
:
Filter
:
Status
:

Port 5/10
Port 15/1
None
transmit/receive
disabled
enabled
enabled

active

If you collect a sniffer trace on port 5/10, the sniffer trace shows packets that transmit to and from the
MSFC. Configure the SPAN session as tx in order to capture packets that are only destined to the MSFC, and
not from the MSFC.
SPAN sc0
Set up a SPAN session with the sc0 interface as the source in order to capture frames that go to the Supervisor
Engine CPU.
Console> (enable) set span ?
disable
Disable port monitoring
sc0
Set span on interface sc0
<mod/port>
Source module and port numbers
<vlan>
Source VLAN numbers

Note: For Optical Services Modules (OSMs), you cannot perform a SPAN capture of traffic.

Recommendations
The Supervisor Engine CPU utilization does not reflect the hardware forwarding performance of the switch.
Still, you must baseline and monitor the Supervisor Engine CPU utilization.
1. Baseline the Supervisor Engine CPU utilization for the switch in a steadystate network with normal
traffic patterns and load.
Note which processes generate the highest CPU utilization.
Cisco Catalyst 6500/6000 Switch High CPU Utilization

2. When you troubleshoot CPU utilization, consider these questions:

Which processes generate the highest utilization? Are these processes different from your
baseline?
Is the CPU consistently elevated, over the baseline? Or are there spikes of high utilization,
then a return to the baseline levels?
Are there Topology Change Notifications (TCNs) in the network?
Note: Flapping ports or host ports with STP PortFast disabled cause TCNs.
Is there excessive broadcast or multicast traffic in the management subnets/VLAN?
Is there excessive management traffic, such as SNMP polling, on the switch?
3. If possible, isolate the management VLAN from the VLANs with user data traffic, particularly heavy
broadcast traffic.
Examples of this type of traffic include IPX RIP/Service Advertising Protocol (SAP), AppleTalk, and
other broadcast traffic. Such traffic can impact the Supervisor Engine CPU utilization and, in extreme
cases, can interfere with the normal operation of the switch.
4. If the CPU runs high due to the punt of traffic to the RP, determine what that traffic is and why the
traffic is punted.
In order to make this determination, use the utilities that the Utilities and Tools to Determine the
Traffic That Is Punted to the CPU section describes.

NetPro Discussion Forums Featured Conversations

Networking Professionals Connection is a forum for networking professionals to share questions, suggestions,
and information about networking solutions, products, and technologies. The featured links are some of the
most recent conversations available in this technology.
NetPro Discussion Forums Featured Conversations for LAN
Network Infrastructure: LAN Routing and Switching
Network Infrastructure: Getting Started with LANs

Related Information
Common CatOS Error Messages on Catalyst 6000/6500 Series Switches
Common Error Messages on Catalyst 6500/6000 Series Switches Running Cisco IOS Software
Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches Running
Cisco IOS System Software
Unicast Flooding in Switched Campus Networks
Cisco Catalyst 6500 Series Switches Product Support
LAN Product Support Pages
LAN Switching Support Page
Technical Support & Documentation Cisco Systems

All contents are Copyright 19922005 Cisco Systems, Inc. All rights reserved. Important Notices and Privacy Statement.

Updated: Jul 05, 2005

Cisco Catalyst 6500/6000 Switch High CPU Utilization

Document ID: 63992