IPSEC Troubleshooting

IPSec Troubleshooting: Problem Scenarios Part 1
In this part I will be discussing the following problem scenarios----
• IKE SA not established

• IPSec SA’s not established
• MTU/Fragmentation Issues
Problem Scenario 1: No IKE SAs
If we are unable to establish IPSec tunnel from Branch location to Hub location
Check for Routing. Ping the Branch (using HUB’s IKE endpoint)
HUB# ping ip 40.10.1.1 source 30.3.1.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 40.10.1.1, timeout is 2 seconds:
Packet sent with a source address of 30.3.1.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/4 ms
Check for IKE SA
HUB# sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id slot status
30.3.1.1 40.10.1.1 MM_NO_STATE 0 0 ACTIVE (deleted)

Use IKE Debugs to troubleshoot [ debug crypto isakmp ]
Problem Scenario 1a:
No IKE SAs
ISAKMP:(0):Checking ISAKMP transform 1 against priority 10 policy
ISAKMP: encryption 3DES-CBC
ISAKMP: hash SHA
ISAKMP: default group 2
ISAKMP: auth pre-share
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 7200
ISAKMP:(0):Encryption algorithm offered does not match policy!
ISAKMP:(0):atts are not acceptable. Next payload is 0
ISAKMP:(0):Checking ISAKMP transform 1 against priority 65535 policy
ISAKMP: encryption 3DES-CBC
ISAKMP: hash SHA
ISAKMP: default group 2
ISAKMP: auth pre-share
ISAKMP: life type in seconds
ISAKMP: life duration (basic) of 7200
ISAKMP:(0):Encryption algorithm offered does not match policy!ISAKMP:(0):atts are
not acceptable. Next payload is 0
ISAKMP:(0):no offers accepted!ISAKMP:(0): phase 1 SA policy not acceptable! (local
30.3.1.1 remote 40.10.1.1)
Check the IKE Policies
HUB# sh crypto isakmp policy
Global IKE policy

Protection suite of priority 10
encryption algorithm: AES - Advanced Encryption Standard (128 bit
keys).
hash algorithm: Secure Hash Standard
authentication method: Pre-Shared Key
Diffie-Hellman group: #2 (1024 bit)
lifetime: 7200 seconds, no volume limit
Default protection suite
encryption algorithm: DES - Data Encryption Standard (56 bit keys).
So once we change the encryption algorithm at spoke side to aes, phase 1 will come
up.
Problem Scenario 1b:
No IKE SAs
ISAKMP:(1017): sending packet to 40.10.1.1 my_port 500 peer_port 500 (R)
MM_KEY_EXCH
ISAKMP:(1017):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
ISAKMP:(1017):Old State = IKE_R_MM3 New State = IKE_R_MM4
ISAKMP (0:1017): received packet from 40.10.1.1 dport 500 sport 500 Global (R)
MM_KEY_EXCH
ISAKMP: reserved not zero on ID payload!
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 40.10.1.1 failed its sanity
check or is malformed
It means we have a mismatch in pre-shared key, on correcting it our IKE SA should
come up.
HUB# sh cry isa sa
dst src state conn-id slot status
30.3.1.1 40.10.1.1 QM_IDLE 1019 0 ACTIVE
Problem Scenario 2: No IPSec SAs
If you notice that there is no traffic is being received through the IPSec tunnel
IKE SAs exist, but no IPSec SAs
Check for IPSEC SA (look for inbound and outbound SPI’s)
HUB# sh crypto ipsec sa peer 40.10.1.1
interface: GigabitEthernet0/1
Crypto map tag: CMAP, local addr 30.3.1.1
protected vrf: (none)

local ident (addr/mask/prot/port): (3.1.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (4.1.1.0/255.255.255.0/0/0)
current_peer 40.10.1.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 0
local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1

path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/1
current outbound spi: 0x0(0)
inbound esp sas:

inbound ah sas:
outbound esp sas:

outbound ah sas:
HUB#
Use IPSec Debugs to troubleshoot [ debug crypto ipsec ]
Problem Scenario 2a: No IPSec SAs
ISAKMP (0:1022): received packet from 40.10.1.1 dport 500 sport 500 Global (R)
QM_IDLE
ISAKMP:(1022): processing SA payload. message ID = -549695704
ISAKMP:(1022):Checking IPSec proposal 1
ISAKMP: transform 1, ESP_3DES
ISAKMP: attributes in transform:
ISAKMP: encaps is 1 (Tunnel)
ISAKMP: SA life type in seconds
ISAKMP: SA life duration (basic) of 1800
ISAKMP: SA life type in kilobytes
ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0
ISAKMP: authenticator is HMAC-SHA
ISAKMP:(1022):atts are acceptable.
IPSEC(validate_proposal_request): proposal part #1,
(key eng. msg.) INBOUND local= 30.3.1.1, remote= 40.10.1.1,
local_proxy= 3.1.1.0/255.255.255.0/0/0 (type=4),
remote_proxy= 4.1.1.0/255.255.255.0/0/0 (type=4),
protocol= ESP, transform= NONE (Tunnel),
lifedur= 0s and 0kb,
spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
Crypto mapdb : proxy_match
src addr : 3.1.1.0
dst addr : 4.1.1.0
protocol :0
src port :0
dst port :0
IPSEC(ipsec_process_proposal): transform proposal not supported for identity:
{esp-3des esp-sha-hmac }
ISAKMP:(1022): IPSec policy invalidated proposal with error 256ISAKMP:(1022):
phase 2 SA policy not acceptable! (local 30.3.1.1 remote 40.10.1.1)
Check the IPSec Transform Sets
HUB# sh cry ips transform-set
Transform set TS: { esp-aes esp-sha-hmac }
will negotiate = { Tunnel, },
On Correcting encryption algorithm in tranform-set , tunnel should come up.

Problem Scenario 2b: No IPSec SAs
Check the Crypto ACLs
HUB# sh access-list SPOKE-10-ACL
Extended IP access list SPOKE10-ACL
10 permit ip 3.1.1.0 0.0.0.255 5.1.1.0 0.0.0.255
HUB#
On Correcting crypto access-list , tunnel should come up.
Problem Scenario 3: Anti-Replay Issues
If you notice that some of the applications are losing intermittent traffic, or that Voice
quality through tunnel is bad.
Check if the IPSec SA is showing anti-replay drops
HUB# sh cry ips sa peer 40.10.1.1 detail


#pkts no sa (send) 0, #pkts invalid sa (rcv) 0
#pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
#pkts invalid prot (recv) 0, #pkts verify failed: 0
#pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
##pkts replay failed (rcv): 1000 #pkts internal err (send): 0, #pkts internal err (recv)
0

current outbound spi: 0xC37422AA(3279168170)
inbound esp sas:

spi: 0x135E76B1(324957873)
transform: esp-3des esp-sha-hmac ,
in use settings ={Tunnel, }
conn id: 41, flow_id: SW:41, crypto map: CMAP
sa timing: remaining key lifetime (k/sec): (4419198/860)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
Default IPSec Anti-Replay window is 64
Packets received outside the window are dropped
Re-ordering of packets could happen due to QoS on the encrypting router (Spoke) or
in the Transit Network
In current Cisco IOS versions, the Anti-Replay window can be increased up to 1024, or
diabled altogether
crypto ipsec security-association window-size
crypto ipsec security-association replay disable
Not recommended to disable anti-replay; first try to fix the QoS issue in the network
or encrypting router; give better QoS to Voice traffic, or use crypto LLQ; then try to
increase the anti-replay window size.
IPSec Troubleshooting: Problem Scenarios Part 2
In this part 2 will be discussing the following problem scenarios----
• Routing Issues (Reverse Route Injection)

• DPD
• Anti-Replay
Problem Scenario 1:
Routing Issues
User complains there is no traffic received through the IPSec tunnel. On further
checking you find that IKE and IPSec SAs exist, but no end-end traffic; spoke shows
its encrypting traffic however no decrpyt.
Check for IPSec SA on Hub Site (look for inbound and outbound SPIs, encr/decr
counts)
HUB# sh crypto session remote 40.10.1.1 detail
Crypto session current status
Code: C - IKE Configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal, X - IKE Extended Authentication
Interface: GigabitEthernet0/1
Profile: SPOKE10-PROF
Uptime: 00:01:49
Session status: UP-ACTIVE
Peer: 40.10.1.1 port 500 fvrf: (none) ivrf: (none)
Phase1_id: 40.10.1.1
Desc: (none)
IKE SA: local 30.3.1.1/500 remote 40.10.1.1/500 Active
Capabilities:D connid:1029 lifetime:01:58:10
IPSEC FLOW: permit ip 3.1.1.0/255.255.255.0 4.1.1.0/255.255.255.0
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 9949 drop 60 life (KB/Sec) 4483560/1690
Outbound: #pkts enc'ed 0 drop 0 life (KB/Sec) 4485046/1690
HUB# sh crypto ipsec sa peer 40.10.1.1


current outbound spi: 0xF6278D63(4129787235)
inbound esp sas:

spi: 0x16C58DD4(382045652)
IV size: 8 bytes
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:

spi: 0xF6278D63(4129787235)
IV size: 8 bytes
Status: ACTIVE
outbound ah sas:
outbound pcp sas:

Check the Routes (for the Spoke protected networks)
HUB# sh ip route 4.1.1.1
% Network not in table
HUB#
HUB# sh ip cef 4.1.1.1
0.0.0.0/0, version 80, epoch 0, attached, default route handler
0 packets, 0 bytes
via 0.0.0.0, 0 dependencies
valid no route adjacencyHUB#
Check the Crypto Map for Reverse-Route Injection. This is needed for the Hub to
inject a route for the Spoke protected subnets into its local routing table. The route is
created when the IPSec SA is established. Since 12.4T, for this route to be created
(based on the Crypto ACL) before the IPSec SA is established (so that the router can
initiate the tunnel), we need the “reverse-route static” configuration.
In the VRF-Aware IPSec scenario, it is better to use the “reverse-route remote-peer
<next-hop-gateway>” configuration under the crypto map.
Old Crytpo Map was----
crypto map CMAP 10 ipsec-isakmp
set peer 40.10.1.1
set transform-set TS
set isakmp-profile SPOKE10-PROF
match address SPOKE10-ACL
Lets add reverse route---
crypto map CMAP 10 ipsec-isakmp
set peer 40.10.1.1
set transform-set TS
set isakmp-profile SPOKE10-PROF
match address SPOKE10-ACL
reverse-route <static>
IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and
peer 40.10.1.1
IPSEC(rte_mgr): VPN Route Event create SA based on crypto ACL in real time for
40.10.1.1IPSEC(rte_mgr): VPN Route Refcount 1 GigabitEthernet0/1
IPSEC(rte_mgr): VPN Route Added 4.1.1.0 255.255.255.0 via 0.0.0.0 in IP DEFAULT
TABLE with tag 0 distance 1
HUB# sh ip route 4.1.1.1
Routing entry for 4.1.1.0/24
Known via "static", distance 1, metric 0
Redistributing via ospf 1
Advertised by ospf 1 subnets
Routing Descriptor Blocks:
* 40.10.1.1
Route metric is 0, traffic share count is 1
HUB# sh ip cef 4.1.1.1
4.1.1.0/24, version 83, epoch 0, cached adjacency 30.3.1.2
0 packets, 0 bytes
via 40.10.1.1, 0 dependencies, recursive
next hop 30.3.1.2, GigabitEthernet0/1 via 40.0.0.0/8
valid cached adjacency
Complete Crypto Map
HUB# show crypto map
Crypto Map "CMAP" 10 ipsec-isakmp
Peer = 40.10.1.1
ISAKMP Profile: SPOKE10-PROF
Extended IP access list SPOKE10-ACL
access-list SPOKE10-ACL permit ip 3.1.1.0 0.0.0.255 4.1.1.0 0.0.0.255
Current peer: 40.10.1.1
Security association lifetime: 4608000 kilobytes/1800 seconds
PFS (Y/N): N
Transform sets={
TS,
}
Reverse Route Injection Enabled
Interfaces using crypto map CMAP:
GigabitEthernet0/1
Problem Scenario 2:
DPD
This is a scenario where HUB keeps sending encrypted traffic, but it is not receiving
any encrypted traffic from Spoke. IKE and IPSec SAs are up.
Please perform the following steps------
Check if the Spoke is reachable (ping tunnel endpoint address)
HUB# ping 40.10.10.1
Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 40.10.10.1, timeout is 2 seconds:
…..
Success rate is 0 percent (0/5)
No replies.
Check if Dead peer Detection is turned on
HUB# sh cry isa sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal
X - IKE Extended Authentication
psk - Preshared key, rsig - RSA signature
renc - RSA encryption
C-id Local Remote I-VRF Status Encr Hash Auth DH

Lifetime Cap.
1035 30.3.1.1 40.10.1.1 ACTIVE 3des sha psk 2
01:59:45
Engine-id:Conn-id = SW:35
So Under Cap, its not listing anything, hence it is disabled.
Check the IPSec SA
HUB#sh cry ips sa


current outbound spi: 0x8BDBBA86(2346433158)
inbound esp sas:

spi: 0x67C89AAD(1741200045)
We notice very few decaps, lets go ahead and configure DPD.
Configure DPD
crypto isakmp keepalive 60 10
crypto isakmp keepalive 60 periodic
If DPD had been configured earlier, then you would have seen following-----
HUB# sh cry isakmp peer de
Peer: 40.10.1.1 Port: 500 Local: 30.3.1.1
Phase1 id: 40.10.1.1
flags:
NAS Port: 0 (Normal) DPD information, struct 0x6727E0E8:
Last_received: 237, dpd threshold (elapsed) 0
my_last_seq_num: 0x5B72ECCC, peers_last_seq_num: 0x0
sent_and_waiting: TRUE
IKE SAs: 1 IPSec SA bundles: 1
last_locker: 0x62FE32FC, last_last_locker: 0x0
last_unlocker: 0x0, last_last_unlocker: 0x0
HUB# sh cry isa sa detail
Codes: C - IKE configuration mode, D - Dead Peer Detection

K - Keepalives, N - NAT-traversal
X - IKE Extended Authentication
psk - Preshared key, rsig - RSA signature
renc - RSA encryption
C-id Local Remote I-VRF Status Encr Hash Auth DH Lifetime Cap.
1037 30.3.1.1 40.10.1.1 ACTIVE 3des sha psk 2 01:59:43 D

Engine-id:Conn-id = SW:37
PS: It shows D under Cap now.
ISAKMP:(1036):DPD incrementing error counter (4/5)
ISAKMP: set new node 1992211651 to QM_IDLE
ISAKMP:(1036):Sending NOTIFY DPD/R_U_THERE protocol 1
spi 1718567840, message ID = 1992211651
ISAKMP:(1036): seq. no 0x5B72ECCD
ISAKMP:(1036): sending packet to 40.10.1.1 my_port 500 peer_port 500 (R)
QM_IDLE
ISAKMP:(1036):Sending an IKE IPv4 Packet.
ISAKMP:(1036):purging node 1992211651
ISAKMP:(1036):Input = IKE_MESG_FROM_TIMER, IKE_TIMER_PEERS_ALIVE
ISAKMP:(1036):Old State = IKE_P1_COMPLETE New State = IKE_P1_COMPLETE
ISAKMP:(1036):DPD incrementing error counter (5/5)

ISAKMP:(1036):peer 40.10.1.1 not responding!
ISAKMP:(1036):peer does not do paranoid keepalives.
ISAKMP:(1036):deleting SA reason "P1 errcounter exceeded (PEERS_ALIVE_TIMER)"

state (R) QM_IDLE (peer 40.10.1.1)
It is always better to use DPD instead of Periodic Keepalives. DPD works well in
conjunction with IPSec HA – geographically distributed peers (multiple ‘set peer’
under crypto map), or HSRP adjacent peers (peer to VIP address).
Problem Scenario 3:
Anti-Replay Issues
Users complain that application is losing intermittent traffic, or that Voice quality
through tunnel is bad. Check if the IPSec SA is showing anti-replay drops
HUB# sh cry ips sa peer 40.10.1.1 detail
#pkts no sa (send) 0, #pkts invalid sa (rcv) 0
#pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
#pkts invalid prot (recv) 0, #pkts verify failed: 0
#pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
#pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
##pkts replay failed (rcv): 1000
#pkts internal err (send): 0, #pkts internal err (recv) 0

current outbound spi: 0xC37422AA(3279168170)
inbound esp sas:

spi: 0x135E76B1(324957873)
IV size: 8 bytes
Status: ACTIVE
So we can see there is a good number of packets replay failed in above show
commands.
By Default IPSec Anti-Replay window size is 64. Hence, Packets received outside the
window will be dropped. Normally Re-ordering of packets could happen due to QoS
on the encrypting router (Spoke) or in the Transit Network.
In current Cisco IOS versions, the Anti-Replay window can be increased up to 1024, or
diabled altogether
crypto ipsec security-association window-size < Size>
crypto ipsec security-association replay disable

It is not recommended to disable anti-replay. Hence first try to fix the QoS issue in the
network or encrypting router; give better QoS to Voice traffic, or use crypto LLQ; then
try to increase the anti-replay window size by above mentioned command.

IPSEC Troubleshooting

Uploaded by

Copyright:

Available Formats

IPSEC Troubleshooting

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

IPSEC Troubleshooting

Uploaded by

Copyright:

Available Formats

IPSec Troubleshooting: Problem Scenarios Part 1

In this part I will be discussing the following problem scenarios----

• IKE SA not established

Type escape sequence to abort.

IPv6 Crypto ISAKMP SA

HUB# sh crypto isakmp policy

Global IKE policy

protected vrf: (none)

local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1

inbound esp sas:

outbound esp sas:

On Correcting encryption algorithm in tranform-set , tunnel should come up.

HUB# sh cry ips sa peer 40.10.1.1 detail

protected vrf: (none)

local crypto endpt.: 30.3.1.1, remote crypto endpt.: 40.10.1.1

inbound esp sas:

crypto ipsec security-association window-size

crypto ipsec security-association replay disable