BRKCCT-1593 Troubleshooting in The UCCX

BRKCCT-1593
Troubleshooting in the UCCX

Solution
Arunabh Bhattacharjee, Software Engineer, Customer Success

Customer Care Business Unit
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKCCT-1593
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What’s most important is ..
Questions ?
Evidence …
Answers !!
BRKCCT-1593 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Agenda
• Problem Isolation
• System Monitoring Tools – CLI, CUIC & RTMT
• UCCX Engine - CUCM Interaction
• Cisco Finesse
• Live Data
• Database and Reporting
• HA Failover
• Chat and Email
Problem Isolation
Troubleshooting Understand the Problem Statement
“Configuration was done a year ago, and
now we are facing ..”
“Agents are unable to login.”
“Appadmin is not responding.”
“Agents are not receiving calls.”
“Database Replication Down !”
Asking The Right Questions
• WHEN? WHAT Changed?
• Configuration and Script Changes

• Recent Upgrades
• Firewalls, Browsers, ESXi, CUCM, Gateways, MS Exchange
• Patches (COP files)
• Failovers
• Could unveil hidden DB replication issues, network issues
• Contact Volume Increases, performance issues
• Holidays, Certain Time of Day, etc…
Asking The Right Questions
• WHO is affected by the Problem?

• Are other Sites/Locations affected by the same issue?
• WHAT ELSE in the environment is affected?
• ISOLATION – Breaking the problem down, pinning it to a single problem
suspect.
• Identify the EVIDENCE needed to CONFIRM the root cause.
Evidence - System Monitoring Tools
RTMT – Real Time Monitoring Tool
• Java based plug-in for UCCX. Installable on Windows or Linux
RTMT – System Summary Memory | CPU | Disk
RTMT – Performance Charts CCX Engine JVM Heap
RTMT – Performance Charts CCX Database Stats
RTMT – Alert Central Catch Problems Before Users do !
RTMT – Log Central Several Ways to View Logs
RTMT – Scheduled Trace Collection
Intermittent Issue?
System Monitoring Tools –
Command Line
Command Line Tools – Basic Health Check
• CLI: show status
• CLI: utils service list
• CLI: utils diagnose test
• CLI: show network cluster
• CLI: show myself
• CLI: show process load [cpu|memory] [cont|num XX]
• CLI: utils uccx ?
• CLI: show uccx ?
• CLI: set uccx ?
Command Line Tools – Tailing Logs
• Currently active logs can be tailed at real-time
• CLI: show open files regexp [MIVR|MADM|MCVD|Desktop-webservices]
• CLI: file tail activelog <relative path from /common/log/taos-log location>
• file tail activelog <relative path from /common/log/taos-log location>
CLI: file tail activelog /uccx/log/MIVR/Cisco001MIVR186.log regexp SS_TEL
System Monitoring Tools - CUIC
Utilization Monitoring – License and Call Handling
• CUIC provides stock reports for this
• License Utilization Hourly report
• Application Performance Analysis report
• Application Summary Report
• Aborted and Rejected Report
License Utilization Hourly Grid and Chart Views
License Utilization Hourly Grid and Chart Views
Call Failures – Aborted and Rejected Reports
Aborted Contacts
• Exception occurred in the script workflow – ABORT!
Rejected Contacts
• System reached maximum capacity – REJECT!
• Cause Codes :
• Reject - TRIGGER_MAX_SESSION
• Reject - REMOTE_TIMEOUT
• Reject - CHANNELS BUSY
• Aborted – Too many transfer failures
Reports give helpful contact details and
• Aborted – Max Steps Executed (1000) exact timestamps!
UCCX Engine and CUCM Interaction
UCCX and CUCM Integration
PSTN
Router/GW
UCCX
Communications Manager
Cluster
Finesse Agent Desktops
SIP (5060)
SIP/H323/MGCP
(5060/1720/2427)
LDAP (389 | 3268)
CTI/JTAPI (2748)
AD/LDAP Recording
AXL (443 | 8443)
CTI (12028)
SCCP | SIP (2000 | 5060)
UCCX Call flow – PSTN call flow
6 2 1
PSTN
4 3 6 Router/GW
23 5 5 4 5
Agent 1. Inbound PSTN Call to VGW.
Stations
UCCX 2. VoIP leg to CUCM.
CallManager 3. CUCM routes call to RP registered by UCCX.

Supervisor Cluster 4. UCCX responds with REDIRECT request to
Stations available CTI Port.
5. UCCX triggers application, instantiates and
executes script. Call is answered upon Accept
Agent ICD Extension: 6000 step executing.
CTI Route Point 8222
CTI Ports: 9988-9990 6. UCCX provides scripted queuing treatment or
identifies an available resource and CONSULT
TRANSFER the call to the agents device.
Setting up the Logging Levels
Under Engine UCCX Engine logs (MIVR): There are multiple logging
facilities. Each facility is a sub-component.
• SS_CM
• SS_RM
• SS_RMCM
• ICD_CTI
• SS_TEL
• JTAPI plugin logs: JTAPI logging is not enabled by default.

Enable it upto Debugging level
JTAPI logs: Information about the interactions with CUCM
There is an easier way!
Logging Profiles. Specific logging facilities for specific problem descriptions.
• CCX Serviceability > Trace > Profile
UCCX Engine in Partial Service?
Manager Manager? OR Subsystem Manager? Click to Expand !
UCCX Engine in Partial Service?
• If CM Telephony Subsystem is in Partial Service, check the Port Monitor
Tool
• If Application Manager, then check for an INVALID Application

Appadmin > Tools > Real Time Reporting > Applications
Analyze the UCCX Engine logs
Call States - Reference
STATE LOG DESCRIPTION
RECEIVED Call.received() Call was received at a configured CTI Route Point
ASSOCIATED Call.associated() An available CTI port was assigned.
ACCEPTED Call.accepted() Call was routed to available CTI port
REJECTED Call.rejected() Call failed to be routed to a port
ATTRIBUTED Call.attributed() An application task was invoked to handle this call
ANSWERED Call.answered() Call was answered by the application (Accept Step)
ABANDONED Call.abandoned() Call was dropped by the caller or CTI port during consult
DISCONNECTED Call.disconnected() Call was terminated by the application
TRANSFERRED Call.transferred() Call was successfully transferred by the application
ABORTED Call.aborted() Call was terminated prematurely
//Call is offered to the CTI RP. cgn=1001 and cn=8222
2872713: Mar 28 09:15:31.264 IST %MIVR-SS_TEL-7-UNK:Route Connection=[8222::1/(P1-abhi_1)
GCID=(1,7057)->ACTIVE]->OFFERED, reason=1, Event= (P1-abhi_1) 7057/1 CallCtlConnOfferedEv
8222::1 [#827] Cause:100 CallCtlCause:100 CiscoCause:0 FeatReason:12, cause=100,
metacode=129, isMaster=true
//CTI Port selected

2872720: Mar 28 09:15:31.266 IST %MIVR-SS_TEL-7-UNK:Route Connection: [8222::1/(P1-
abhi_1) GCID=(1,7057)->ACTIVE]->OFFERED, CTI Port selected:
TP[id=1,implId=9989,state=IN_USE]
//Call disconnected at the CTI RP since it’s REDIRECTED

2872734: Mar 28 09:15:31.299 IST %MIVR-SS_TEL-7-UNK:RP[num=8222], conn=[8222::1/(P1-
abhi_1) GCID=(1,7057)->ACTIVE]->DISCONNECTED, event=(P1-abhi_1) 7057/1
CallCtlConnDisconnectedEv 8222::1 [#842] Cause:100 CallCtlCause:210 CiscoCause:0
FeatReason:6, cause=CAUSE_NORMAL[100], meta=META_CALL_REMOVING_PARTY[131]
//The call for the trigger 8222 is accepted at CTI port 9989
2872739: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:CallID: 13, MediaID: 7057/1 Accepting
call for CTI Route Point: 8222 on CTI Port: 9989, ciscoCause=0
//Call is accepted and the call is ringing at the CTI port
2872740: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:Call.accepted()
JTAPICallContact[id=13,implId=7057/1,state=STATE_ACCEPTED_IDX,inbound=true,App
name=app_new,task=null,session=19000000014,seq
num=0,cn=8222,dn=8222,cgn=1001,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=null
,ocn=8222,route=RP[num=8222],OrigProtocolCallRef=0000000000001B9101685E5300000000
,DestProtocolCallRef=null,TP=9989]
2872741: Mar 28 09:15:31.329 IST %MIVR-SS_TEL-7-UNK:CallID:13 MediaId:7057/1,
TerminalConnection to Terminal: renji_9989 is RINGING, [9989::1/(P1-abhi_1)
GCID=(1,7057)->ACTIVE]->ALERTING
//TaskID created: 37000000025
2872744: Mar 28 09:15:31.330 IST %MIVR-SS_TEL-7-UNK:CallID:13 MediaId:7057/1
Task:37000000025 associated with Task ID: 37000000025
//Agent selected and the state is changed to RESERVED
2872829: Mar 28 09:15:32.657 IST %MIVR-SS_CM-7-UNK:RmCm contact 16784273[7057/1]
(12) .addAssociatedResource(abhi)
2872830: Mar 28 09:15:32.657 IST %MIVR-SS_RM-7-UNK:Rsrc: abhi New State:RESERVED
Old State:AVAILABLE Reason code:0
//Consult transfer starts
Task:37000000025, transfer(6000, 12000, ACKNOWLEDGED)
//Consult call is offered at the agent's phone
2872869: Mar 28 09:15:32.690 IST %MIVR-SS_TEL-7-UNK:OrigCall=CallID:13
MediaId:7057/1 Task:37000000025, ConsultEvent= (P1-abhi_1) 7058/1
CallCtlConnOfferedEv 6000::4 [#881] Cause:100 CallCtlCause:100 CiscoCause:0
FeatReason:12
//Consult call ringing on agent's extension
Task:37000000025, ConsultCallObserver RINGING
//Agent answers the call
Task:37000000025, ConsultCallObserver TALKING - CallCtlConnEstablished,
[6000::1/(P1-abhi_1) GCID=(1,7058)->ACTIVE]->ESTABLISHED
Common call failure scenario-1
Partition and CSS issue: it is important to note that the original calling number should be
able to reach BOTH THE CTI ROUTE POINT AND THE CTI PORTS.
In the MIVR logs, the following error message is seen:
302849: Dec 02 23:14:25.566 PST %MIVR-SS_TEL-3-ROUTE_FAILED:Route failed: All
Call
ids=JTAPICallContact[id=10,implId=1481/1,state=STATE_RECEIVED_IDX,inbound=true
,App name=sample_1,task=null,session=1000000011,seq
num=0,cn=7420,dn=7420,cgn=1009,ani=null,dnis=null,clid=null,atype=DIRECT,lrd=n
ull,ocn=7420,route=RP[num=7420],TP=null,List of Active
Connections=[7420::1/(P1-sydney_cti_1) GCID=(1,1481)->ACTIVE]-
>OFFERED,Extension=7410,Exception=com.cisco.jtapi.InvalidPartyExceptionImpl:
Attempt to redirect to an unknown destination,Failure reason= call will be
rejected, CTIERR_REDIRECT_CALL_UNKNOWN_DESTINATION=0x8ccc0034::Attempt to
redirect to an unknown
destination,Contact.Reject.reason=TRIGGER_FAIL,(SelectRouteTime,ObtainingIdleC
hannelTime,RedirectTime=0,0,16)
99016: Dec 01 09:22:53.644 IST 99066: Dec 01 09:22:54.039 IST %JTAPI- 99067: Dec 01 09:22:54.211 IST %JTAPI-PROTOCOL-
%JTAPI-PROTOCOL-7-UNK:(P1- PROTOCOL-7-UNK:(P1-10.106.112.196) 7-UNK:(P1-10.106.112.196) received Event:
10.106.112.196) received [MIVR_SS_TEL_TPG_ROUTE_EXE-44-24- com.cisco.cti.protocol.NewCallEvent {
Event: ROUTE_CALL_EV] sending: eventSequence = 397
com.cisco.cti.protocol.NewCall com.cisco.cti.protocol.CallRedirectReq lineCallManagerID = 2
Event { uest { lineID = 64
eventSequence sequenceNumber = 148 lineHandleSpecified = true
= 395 lineCallManagerID = 2 deviceName = CCG1_8000
lineCallManagerID lineID = 30 callCallManagerID = 1
= 2 callLegCallManagerID = 1 callLegID = 32362289
lineID callLegID = globalCallManagerID = 1
= 30 32362287 callID = 1464
lineHandleSpecified redirectAddress = 8000 callingParty = 1009
= true unconditional = true calledParty = 8000
deviceName callingSearchSpace = 0 originalCalledParty = 4000
= ACD calledAddressOption = 0 state = 2
callCallManagerID redirectReason = 0 origin = 1
= 1 preferredOriginalCalledParty = 8000 userData = null
callLegID modifiedCallingParty = null auxiliaryData = 1
= 32362287 farEndpointCallManagerID = 1
globalCallManagerID farEndpointType = 1
= 1 reason = 6
callID callingPartyName =
= 1464 calledPartyName =
callingParty
= 1009
calledParty
= 4000
originalCalledParty
= 4000
The call from the CTI Route point to the CTI
Port is a REDIRECT, therefore the original
calling party should have access to both the
route points and ports.
Remember.. Select
Resource
Timeout < RNA
Steps in Script on IPCC Line Max Calls (2)
(Interruptible) Busy Trigger (1)
MoH 2 Second
Timeout before
(Non-Multicast) Accept (Script)
CUCM
Avoid Providers
Shared Lines Unsupported Local, CTI
Actions Ports, RPs in
separate DPs
Troubleshooting Finesse
Cisco Finesse
Finesse is the only available

Desktop in UCCX 11.6(1) release
-Browser-based
-Co-resident with CCX
- Open REST APIs
- Customizable desktop
- Cisco and third party gadgets
- Integrated administration
- E.164 support
Finesse health
• Finesse depends on the following services for its normal functioning :
• Cisco Unified CCX Engine Service
• Cisco Unified CCX Notification Service
• Cisco Finesse Tomcat (Tomcat Service exclusive to Finesse)
Checking Finesse health
Service status on the GUI shows
‘STARTED’ – does this mean Finesse is
usable?
NO. Service state : STARTED = Finesse
Tomcat status only
For actual status of Finesse:

http://<UCCX IP>:8082/finesse/api/SystemInfo
https://<UCCX IP>:8445/finesse/api/SystemInfo
Slowness? Check Finesse Health
https://host.domain.com:8445/finesse-dp/rest/DiagnosticPortal/GetPerformanceInformation
• Finesse diagnostic portal:

• Tomcat Web Server stats
• CTI Connection stats
• Agent and call stats
Finesse Issues – Further Analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for Cisco Finesse, Cisco
Unified CCX Engine, Cisco Unified CCX Notification Service
Finesse Logs Notification Service

Engine Logs
Logs
• Desktop- • MIVR logs, • Needs to be
webservices, • LIB_AXL: Axlclient explicitly enabled
client-logs, realm logs, for login from CLI.
log file for issues issues • For specific login
related to Finesse issues, state
change issues on
Finesse.
Failure Scenario – Finesse Monitoring Error
Problem: Monitoring Fails to Start
CCX Engine Receives Request to Silent Monitor agent106

CCX Engine log:
%MIVR-SS_RM-7-UNK:Processing msg: CTIMgrTPCCReqMsg
Socket:Socket[addr=127.0.0.1,port=35064,localport=12028] invokeID:3233 Msg Type =
SUPERVISOR_CALL_REQ Details = length=57 type=SUPERVISOR_CALL_REQ,invokeId=3233
AgentConnectionCallID = -1, SupervisorConnectionCallID = -1, AgentConnectionDeviceIDType = 0,
SupervisorConnectionDeviceIDType = 0, SupervisoryAction = SUPERVISOR_MONITOR,
AgentConnectionDeviceID = 2002, SupervisorConnectionDeviceID = 2001, AgentID = null, AgentDevice
= 2002, SupervisorDevice = 2001
CCX Engine detects BiB Is Enabled, Uses JTAPI to Begin Monitoring Call Setup
CCX Engine log:
%MIVR-SS_RM-3-Initiating silent monitor request to JTAPI server...:'Initiating silent monitor

request to JTAPI server...‘
%MIVR-SS_RM-7-UNK:processCallCtlConnInitiatedEv featureReason:1257684:
%MIVR-SS_RM-7-UNK:RIMgrAddressCallObserver: CallCtlConnInitiatedEv received for call 16779233
[2017/1] and agent agent1
JTAPI Shows the Supervisor’s Monitoring Call State Go Dialtone > Dialing > Disconnected with Error:
JTAPI log:
%JTAPI-CTI-7-UNK:(P2-
rmcm){Line:SEP88755651168C:2001:NewYork:(1,100)|Call:[GCID=(2017/1),CID=19750048]}
CallStateChanged (V2) [state=DISCONNECTED cause=UNALLOCATEDNUMBER destType=UNKNOWN destCM=0
fwdDest=: SecStat=1 unCg=2001 unCd= unOrigCd= unLRP= cg=2001 …….
CCX Engine Reports this Failure back to Finesse Server

CCX Engine log:
%MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

type=CONTROL_FAILURE_CONF,invokeId=3233,failureCode=CF_INVALID_DESTINATION,errorCode=0,
text=Error from Supervisor Monitor request. Description: ICDJtapiCallControlChannel (monitor);
nested exception is:
Troubleshooting High Availability
High Availability
• LAN: Both nodes share AXL, Unified CM Telephony and RMCM Provider list
configuration
HA over LAN (HAoLAN) HA over WAN (HAoWAN)
CM Publisher CM Subscriber CM Publisher CM Subscriber
WAN
Primary CCX Secondary CCX Primary CCX Secondary CCX
• WAN: Both nodes have unique AXL, Unified CM Telephony and RMCM
Provider list configuration
Failover Issues – Further Analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for
• Cisco Unified CCX Cluster View Daemon (MCVD) logs
Engine Failure
Engine logs – MIVR logs
Database Failure
Online logs – CCX Database
Event Viewer logs
Application logs
System logs
Finesse logs
High Availability
Scenario 1: Lost Connectivity
Secondary CVD detects missing keep-alives from Primary CVD
%MCVD-CVD-5-HEARTBEAT_MISSING_HEARTBEAT:
CVD does not receive heartbeat from node for a long period: nodeId=1,dt=2049
Primary CCX Server Secondary CCX Server .....
CVD M ? ? S ? M CVD CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10245
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash:
Engine ! Engine state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash:
state=Convergence State,nodeInfo=Node id=1 ip=192.168.12.5
Database Database
Reporting Master Election initiated on Secondary CVD

Reporting %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
Finesse Finesse 7631: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-
UNK:JavaService66:
Cisco Unified CCX Engine on node 1 change master from true to false
7632: Apr 28 07:45:13.986 CEST %MCVD-CLUSTER_MGR-7-UNK:Post Master
• Primary lost connectivity Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1
• Master Election initiated Secondary CVD elects Secondary as Master

%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
node 2 change master from false to true
• Master Election completed %MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,
name=Cisco Unified CCX Engine, node=2
%MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
• Failover completed CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
High Availability
Scenario 2: Partitioned Network – ISLAND MODE
Primary CVD detects missing Heartbeats
Primary CCX Server Secondary CCX Server and assumes Primary has failed
CVD M M S ? M CVD CVD does not receive heartbeat from node for a long period: nodeId=2,dt=10197
%MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node
Engine ! Engine crash:
state=Convergence State,nodeInfo=Node id=2 ip=192.168.13.5
Database Database
Secondary CVD detects missing Heartbeats
Reporting Reporting and assumes Primary has failed
CVD does not receive heartbeat from node for a long period: nodeId=1,dt=10242
Finesse Finesse %MCVD-CVD-4-HEARTBEAT_SUSPECT_NODE_CRASH:CVD suspects node crash:
state=Heartbeat State,nodeInfo=Node id=1 ip=192.168.12.5
Secondary CVD performs Master Election

• WAN network failure CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
%MCVD-CLUSTER_MGR-7-UNK:JavaService66:
Cisco Unified CCX Engine on node 1 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master
• Missing Heartbeats detected Event: MASTER_DROPPED, name=Cisco Unified CCX Engine, node=1
Secondary CVD elects Secondary as Master

• Master Election performed %MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine on
node 2 change master from false to true
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_ELECTED,
• Failover completed %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
High Availability
Recovery from Network partition
Primary CVD detects Secondary and keeps Master
%MCVD-CVD-7-UNK:Split after network partition is detected,new nodeId=2
Primary CCX Server Secondary CCX Server %MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
because primaryEngineComputerName=UC85CCXPRI
CVD M M M ? S CVD %MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine conditional-Keep-
LocalMaster
from localServiceCisco Unified CCX Engine on node 1
Engine Engine
!
Database Database Secondary CVD detects Primary and drops Master
%MCVD-CVD-7-UNK:Split after network partition is detected, new nodeId=1
Reporting Reporting %MCVD-CVD-7-UNK:Engine bestCandidate runs on nodeId=1
because primaryEngineComputerName=UC85CCXPRI
%MCVD-CVD-7-UNK:Master Cisco Unified CCX Engine DropLocalMaster from
Finesse Finesse localServiceCisco Unified CCX Engine on node 2, conditional=true
Secondary CVD performs Master election and drops Master

• WAN network restored %MCVD-CLUSTER_MGR-7-UNK:Post Convergence Event:
CONVERGENCE_STARTED, name=Cisco Unified CCX Engine
%MCVD-CLUSTER_MGR-7-UNK:JavaService167: Cisco Unified CCX Engine
on
• CVDs detect dual Masters node 2 change master from true to false
%MCVD-CLUSTER_MGR-7-UNK:Post Master Event: MASTER_DROPPED,
• Master Election performed
Secondary CVD completes Master election
• Failover completed CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine
Finesse Failover Behavior before 11.6
Scenario CCX HA Behavior Finesse Server Finesse Client Recovery
HA Behavior Behavior
CCX Engine Failure Failover to HA node Failover to HA node Failover to HA node Finesse follows
Engine mastership
CCX Notification No Failover Finesse goes Out of Sessions closed Finesse unavailable
Service Failure Service until Notification
Service comes online
Finesse Tomcat No Failover Finesse goes Out of Sessions closed Finesse unavailable
Failure Service until Tomcat Service
comes online
Finesse Service OOS No Failover Finesse goes Out of Sessions closed Finesse unavailable
Service until issue fixed
Island Mode Both HA nodes Both Finesse servers Clients connect to Clients reconnect to
become Master In Service either Master node
Finesse Failover Enhancements (11.6)
Finesse Failover Enhancements (11.6)
Unified CCX HA
Failure Scenario Behavior Cisco Finesse - Node 1 Cisco Finesse - Node 2
Engine Mastership Return to IN_SERVICE as Return to IN_SERVICE as
Unified CCX Engine fails over from changes from master soon as it connects to the soon as it connects to the
master to Non-Master node to non-master. new master engine. new master engine.
Finesse on Node 2 will

Engine mastership is continue to be
Finesse Service Out of Service not affected OUT_OF_SERVICE in IN_SERVICE
Engine mastership is Finesse on Node 2 will

CCX Notification Service Failure not affected OUT_OF_SERVICE continue to be IN_SERVICE
Finesse on Node 1 will Finesse goes Out Of Service

continue to and will return
be IN_SERVICE and will be to IN_SERVICE as soon as it
Both HA nodes connected to Engine on connects to the engine on
Island Mode become Master Node 1 Node 2 which is also the
master
Troubleshooting Live Data
Live Data – How it works
Live Reports on Cisco Finesse
Key Flow Points
• Publishes every 3 CUIC Live Data

Engine seconds into JMS Streaming Source
(ONLINE)
SocketIO • Receives through JMS all the

messages
Consumer • Creates Rooms.
SocketIO • Dispatches the updates to

subscribed Web Clients
Dispatcher (Browsers)
Logging required
• Engine logs (MIVR) – ICD_RTDM, SS_RMCM, upto Xdebugging5
• SocketIO Service logs – DEBUG mode
UCCX Engine as Live Data Source
• Engine drafts JSON updates to JMS (ActiveMQ) every 3 seconds
558501: Jan 26 18:57:26.008 EST %MIVR-ICD_RTDM-7-UNK:LiveData:

Sending JMS message :
[{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceI
d":"agent1","resourceName":"agent1","resourceState":2,"durationInSt
ateMillis":148093,"nHandledContacts":0,"nPresentedContacts":0,"avgT
alkDuration":0,"longestTalkDuration":0,"avgHoldDuration":0,"longest
HoldDurati……… }}]
UCCX Engine as Live Data Source
• For Event based Rooms, Engine sends empty heartbeats
558509: Jan 26 18:57:26.274 EST %MIVR-SS_RMCM-7-UNK:HeartBeatAction : Sending
JMS message : [] for destinations [topic://ChatAgentStats,
topic://ChatQueueStatistics, topic://AgentCSQStats,
topic://AgentStateDetailStats]
558510: Jan 26 18:57:26.279 EST %MIVR-SS_RMCM-7-UNK:HeartBeatAction : Sending

JMS message : [] SUCCESS
SocketIO debugs
• The Socket IO Server receives messages from the JMS bus for the Topic.
• Here agent=agent1, JMS topic=ResourceIAQStats.

• Room name=ResourceIAQStats*agent1, Status=RECEIVED,
0000001076: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

%CCBU_Camel (camel-3) thread #3 - JmsConsumer[ResourceIAQStats]-6-MessageProducer:
%[message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":
{"resourceId":"agent1","resourceName":
"agent1","resourceState":3,.................
..
room_name=ResourceIAQStats*agent1][room_prefix=ResourceIAQStats][status=RECEIVED][topic=jm
s:topic:ResourceIAQStats]: Event Detail Trace
SocketIO debugs
• Socket IO Server processes the messages and maps them to the appropriate
rooms.
• Room name=ResourceIAQStats*agent1, Status=ENQUEUED
0000005789: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:
%CCBU_pool-19-thread-1-6-MessageDispatcher:
%[client_count=-
1][message={"id":"agent1","operation":"UPDATE","ResourceIAQStats":
{"resourceId":"agent1","resourceName":"agent1",".....
..
room_name=ResourceIAQStats*agent1][socket_io_server_type=WS/WSS][status=ENQUEU
ED]: Event Detail Trace
SocketIO debugs
• SocketIO message dispatcher sending the update to clients
• The Message Dispatcher receives the discrete room named messages and transmits
them to all the clients that subscribe to the room.
• Operation = UPDATE, Status=SENT
• 0000005791: 10.106.87.133: Jan 26 2016 18:57:27.008 +0530:

%CCBU_pool-19-thread-1-6-MessageDispatcher:
%[client_count=2][message={"id":"agent1","operation":"UPDATE","ResourceIAQSt
ats":{"resourceId":"agent1","resourceName":"agent1",...................
.....
[room_name=ResourceIAQStats*agent1][socket_io_server_type=WSS][status=SENT]:
Event Detail Trace
SocketIO debugs
• 4. On the Browser Side logs (F12) we see the following:
Message received on
socket::{"id":"agent1","operation":"UPDATE","ResourceIAQStats":{"resourceId"
:"agent1","resourceName":"agent1","resourceState":3,"durationInStateMillis":
264720,"nHandledContacts":6,"nPresentedContacts":8,"avgTalkDuration":1165394
,"longestTalkDuration":2266049,"avgHoldDuration":0,"longestHoldDuration":0,"
avgHandleDuration":0,"avgWorkDuration":0,"totalTalkTime":6992368,"totalHoldT
ime":0,"maxReadyTime":10199923,"avgReadyTime":1156099,
The above message can also be tallied with Finesse side Client logs to check browser updates coming
in.
Live Data – Checklist
NTP • Synchronized
• Resolution Forward and Reverse
DNS • Diagnostic test : utils diagnose test
OVA Profile • Resources, NIC = VMXNET3 adapter

• Sizing – Under Limit, Agents per Team!
Sizing
Hostname • Lowercase
• Mismatch, Unsupported Algorithms
Certificates
Remember.. Browser
Versions
Browser
Field Note
Cache
Awareness
(Incognito)
Disable
Supported
Compatibility
Sizing Limits
mode
UCCX Database and Reporting
UCCX Databases
CCX Application Configuration Datastore
Historical Datastore UCCX DB

CCX DB
Repository Datastore
IBM Informix
Platform DB
Finesse CUIC
CM DB
UCOS
RHEL
UCCX Replication Types
Publisher Subscriber
DB Replication States
utils dbreplication status – Provides the status of Platform DB replication.
CUIC and Finesse share the platform DB.
utils uccx dbreplication status – Provides the status of the UCCX DB replication.
All the CCX historical, configuration and repository data is stored on this DB.
CLI: utils network connectivity <hostname of secondary>
CLI: utils ntp status
CLI: utils diagnose test
CCX Database Replication
• View from CCX Admin or CLI
• Connected / Active state shows healthy replication
• Dropped / Timed Out status indicates replication issue
Tools-> Datastore Control Center->

Replication Servers
admin:show uccx dbreplication servers

This operation may take a few minutes to complete. Please wait...
SERVER ID STATE STATUS QUEUE CONNECTION CHANGED

-----------------------------------------------------------------------
g_uc115ccxpri_uccx 1 Active Local 0
g_uc115ccxsec_uccx 2 Active Connected 0 May 9 18:07:55
CLI Command: Command successful.
show uccx dbreplication servers admin:
Command to show the DB Replication Files
CLI: utils uccx dbreplication dump configfiles
File Purpose
/usr/local/cm/db/informix/etc/sqlhosts.uccx Entries for Informix SQLHOSTS for the UCCX Instance
/etc/hosts Entries for the hosts in the cluster

/home/informix/.rhosts Entries for replicate hosts
Database Repair - Checklist
CCX Serviceability - UI Command Line Interface

Driven Driven
Reset Replication Reset Replication
Enable/Disable
CDS/HDS forcedatasync
Synchronize Data dbserver integrity
Database Replication Reset
• Replication can be broken if Subscriber is unavailable for too long and send
queues buffer is exceeded Alert Raised!
• Typically 3-4 days (*can vary with load) DBReplicationStopped
• Alert will be raised
Tools-> Datastore Control Center->

Replication Servers
Issuing a Reset Replication causes the following to

occur:
1. Remove database replication
utils uccx dbreplication teardown
2. Setup database replication
utils uccx dbreplication setup
3. Perform data repair process
utils uccx dbreplication repair all
Disable/Enable CDS/HDS
Tools -> Datastore Control Center ->
Replication Servers
Replication Servers
Node 1 Node 2
CDS
CDS
HDS HDS
RDS RDS
Replication Servers
Node 1 Disable CDS/HDS Node 2

1. Tear down HDS replication.
CDS 2. Deactivate CDS on node 2. CDS
HDS HDS
RDS RDS
Replication Servers
Node 1 Node 2
CDS
CDS
HDS HDS
Enable CDS/HDS
RDS RDS
1. Setup HDS replication.
2. Merge HDS data.
3. Copy node 1 CDS to node 2.
4. Activate node 2 CDS.
Common Scenario : Configuration Issue
Making Config Changes – Error occurred while saving
• Both DBs must be AVAILABLE!
• WORKAROUND?
• Temporarily Disable CDS on Subscriber
CCX Administration (MADM) Logs:
%MADM-ADM_CFG-7-UNK:ICDServlet :: Exception occurred
The SUBSCRIBER node of the HA is not available
%MADM-ADM_CFG-3-ADM_EXCEPTION:Unknown ADM Exception:
CCX Serviceability Admin -> Tools -> Datastore Exception=java.lang.RuntimeException: The SUBSCRIBER node of the HA is not available
Control Center -> Replication Servers
• Disable CDS and HDS
• Enable after connectivity to Subscriber is

restored
• Enable CDS and HDS
Only for when Subscriber is down.
If Publisher is down, no configuration
changes are possible.
Synchronize Data - GUI
Datastores
CDS CDS
HDS HDS
RDS RDS
Synchronize Data - GUI
Datastores
CDS •Merges a single datastore. CDS

•CDS not eligible (no ER).
HDS HDS
RDS RDS
Force Database Sync – CLI Command
Local DB Remote DB
CDS CDS
HDS HDS
RDS RDS
Local DB Remote DB
CDS
Overwrite Target = Local! CDS
HDS HDS
RDS RDS
Local DB Remote DB
CDS
Overwrite Target = Local! CDS
HDS HDS
RDS RDS
Cluster Reboot + Replication Reset

required!
Database Integrity
utils uccx database dbserver integrity
• Uses informix “oncheck” to check database for inconsistencies
• System Catalog Tables
• Chunk and extents
• Root reserved pages
• Output logged. CLI: file view activelog /uccx/cli/DbServerIntegrity_xxxx.txt
• Repairs NULL data entries in Config Tables

• Output logged. CLI: file view activelog /uccx/cli/DbConfigIntegrity_xxxx.txt
Reporting Review - Datasources
CUIC Data Sources:
Publisher CUIC uses

Subscriber’s db_cra
Subscriber’s CUIC
Also uses Subscriber’s
db_cra
Database Issues – Further Analysis
• From Real-Time Monitoring Tool (RTMT) collect logs for
• Cisco Unified CCX Database
DB
online.uccx.log file for
issues related to database
service
DB REPLICATION
uccx_repl_output_util.log
file for issues related to
database replication or at a
minimum gather these logs
for TAC
Reporting Review – User Accounts
• uccxhruser - ”Historical Reporting User”
• CUIC, Wallboards, 3rd Party Reporting Software, Workforce Management
uccxwallboard
Wallboard User limited to
RtCSQsSummary and
RtICDStatistics tables
uccxworkforce: Workforce User used by Calabrio WFO.

Large queries on historical tables.
uccxhruser : Historical Reporting User for
CUIC co-resident and custom report access to
all historical tables.
Scenario: Reporting and Database Performance
• Historical Reports slow to execute..
• Overall high server CPU utilization..
• Sluggishness in the system..
• Appadmin is slow..
Looking into processes

You’ll find ‘uccxoninit’ as the top process!
Reporting and Database Performance
• Review the configured CUIC reports and their frequency
• Check the historical tables size
• If tables have millions of rows, review purge config:
Default of 90 months can be reduced
Individual table purge is new in version 11
• DBPerf Script:
2 parameters - total duration and interval at which to get data
• DBPerf log contains:

• Timestamps,
• Full CPU snapshot: top,
• Users: onstat –u,
• Top Session information: onstat –g ses 0,
• Memory: onstat –g mem,
• Disk: onstat –d,
• Locks: onstat- k,
Troubleshooting Chat and Email
Chat and Email Topology
Voice Gateway
UCM
UCCX
SocialMiner
End User
Finesse Client
Internet DMZ MS Exchange Corporate
From Customer to Agent
Cloud Mail 3
UCCX
2 4
Social Miner
Customer chat 5
Finesse Client
Intranet
DMZ Exchange
Exchange-Finesse Email Architecture with Social Miner – note that the exchange can be in the DMZ too.
IMAP HTTPS HTTP/S XMPP XMPP HTTP/REST Traffic Proxied via Finesse Tomcat
(Agent) (BOSH) Server to Server
Interacts w/ Private REST APIs Communication
Hosted in SocialMiner Shared BOSH Connection
Hosted in UCCX Cisco Unified
Cisco Finesse Tomcat

CCX
Notification
Service
UCCX
OpenAjax Hub
Cisco Tomcat
& Other
Services
SocialMiner
XMPP Service
SocialMiner
Cisco Tomcat
& Other
Services
Reply Template JSP loaded via SocialMiner Tomcat Chat XMPP Service
Gadget to Gadget Communication via HTML5 Interacts w/ Private REST APIs Customer Website
HTTP/REST Traffic Proxied via Finesse Tomcat Exchange Server

Gadget to Gadget Communication
Reply Template JSP HTTP/REST© 2018
APICisco
Traffic
and/or its affiliates. All rights reserved. Cisco Public 102
Office 365 and Gmail Support If the feed is good, it shows green
• Only Socks 5 protocol is supported Else, a red X is seen
Hovering over the X will show the
• Configure Socks Proxy in SM and enable it. reason for the state
• There is a Office365 email feed status in SocialMiner:
Object Missing
Object Mismatch
Object Normal
Not Configured: Email Server not configured

Not Applicable: Cisco Finesse not active
SocialMiner incompatible
Email CSQ’s not created
Finesse Agent Email Message Flow
MS Exchange SocialMiner UCCX Finesse Client
Email
Fetch Emails
Notify New Contact
Notify New Contact
Load Email UI
Get Email Body and Attachments
Reserve Contact
Fetch Complete Email
Load Complete Email
Upload/Download
Upload/Download Attachments / Reply
Attachments / Reply
Email Reply Contact Handled
Contact Handled
Email & Chat Fault Tolerance
SocialMiner
• SocialMiner does not support High Availability (HA)

• Fault Tolerance is provided via UCCX HA
• SocialMiner configured to communicate
with both UCCX nodes
UCCX UCCX
• When new contact arrives, both CCX nodes notified
• Master node queues the request M S
• Secondary node ignores request
• If UCCX nodes failover

• Chat: existing sessions terminated, new contacts queued to HA node
• Email: on desk emails requeued, new contacts queued to HA node
Reinject Email Contacts
UCCX SocialMiner Configuration Page
Reinject Emails will send the emails back to

the Agent once error corrected
Email auto routed to Agent

When Agent goes to retrieve email,
network connection between SocialMiner and
Exchange broke
Email contact becomes parked in SocialMiner
Finesse Agent Email – Failure Scenario 1
• Denotes email deleted on Exchange Server
• Verify that there are no auto-routing rules that moved the email
• Verify CSQ alias folder has not been deleted or moved
Finesse Agent Email – Failure Scenario 2
• Denotes a connectivity problem with SocialMiner Tomcat Service
• Verify SocialMiner is reachable

• Verify status of Tomcat Service
• Restart Tomcat Service
Finesse Web Chat Message Flow
SocialMiner Finesse Client
Web Server UCCX
Chat Request
New Contact Notification
Contact Dialog Event
Open Reply Template

Agent Joined
Reply Template
End Session
Session Complete
Session Complete
Contact Dialog Event
Failure Scenario – Non-Voice Gadget
• Cisco Unified CCX Webservices or Non-Voice Subsystem offline
• Agent not configured but gadget displayed
• Controls not enabled – SocialMiner unreachable
SocialMiner with UCCX - Checklist
Network VM Settings Certificates Firewall, Proxy

• NTP • ESXi – LRO • Hostnames in • Open ALL
Synchronized. Disabled Lowercase required ports
• DNS - Forward • OVA Profile • SSL CCX and • Refer Port
and Reverse • NIC (VMXNET3) SocialMiner trust Utilization Guide
resolved relationship
• Network (tomcat-trust)
Latency < 250
ms
• utils network
host
<IP/Hostname>
SocialMiner Logs
Runtime Logs
• The Main SocialMiner Engine (Runtime)
• RTMT – SocialMiner Runtime
• CCBU-runtime.*.startup.log, Error-runtime.*.startup.log
API Logs
• Main API logs for the SocialMiner REST API
• RTMT – CCP API Logs
• CCBU-ccpapi.*.startup.log, Error-ccpapi.*.startup.log
Public App Logs

• Public interfacing API (Incoming Chat, Facebook, Twitter poll)
• RTMT – CCP PublicApps Logs
• CCBU-ccppublicapps.*.startup.log, Error-ccppublicapps.*.startup.log
What’s most important is ..
Questions ?
Evidence …
Answers !!
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKCCT-1593
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs : LABCCT-2011
• Tech Circle
• Meet the Engineer 1:1 meetings
• Related sessions
Call to action !
• UCCX solution health check (Once a month)
• UCCX EFTs (Early Field Trials) – Spark!
• UCCX Field Notices on Cisco.com

https://www.cisco.com/cisco/support/notifications.html
• Build your own checklists!
Thank you

BRKCCT-1593 Troubleshooting in The UCCX

Uploaded by

Copyright:

Available Formats

BRKCCT-1593 Troubleshooting in The UCCX

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKCCT-1593 Troubleshooting in The UCCX

Uploaded by

Copyright:

Available Formats

BRKCCT-1593

Troubleshooting in the UCCX

Arunabh Bhattacharjee, Software Engineer, Customer Success

• WHEN? WHAT Changed?

• Configuration and Script Changes

• WHO is affected by the Problem?

CLI: file tail activelog /uccx/log/MIVR/Cisco001MIVR186.log regexp SS_TEL

CallManager 3. CUCM routes call to RP registered by UCCX.

• JTAPI plugin logs: JTAPI logging is not enabled by default.

Manager Manager? OR Subsystem Manager? Click to Expand !

• If Application Manager, then check for an INVALID Application

ASSOCIATED Call.associated() An available CTI port was assigned.

ACCEPTED Call.accepted() Call was routed to available CTI port

REJECTED Call.rejected() Call failed to be routed to a port

ATTRIBUTED Call.attributed() An application task was invoked to handle this call

ANSWERED Call.answered() Call was answered by the application (Accept Step)

DISCONNECTED Call.disconnected() Call was terminated by the application

TRANSFERRED Call.transferred() Call was successfully transferred by the application

ABORTED Call.aborted() Call was terminated prematurely

//CTI Port selected

//Call disconnected at the CTI RP since it’s REDIRECTED

Finesse is the only available

• Cisco Unified CCX Engine Service

• Cisco Unified CCX Notification Service

• Cisco Finesse Tomcat (Tomcat Service exclusive to Finesse)

For actual status of Finesse:

• Finesse diagnostic portal:

Finesse Logs Notification Service

CCX Engine Receives Request to Silent Monitor agent106

%MIVR-SS_RM-3-Initiating silent monitor request to JTAPI server...:'Initiating silent monitor

CCX Engine Reports this Failure back to Finesse Server

%MIVR-ICD_CTI-7-UNK:OutboundMessageprocessor : sending msg : { length=-1

Primary CCX Secondary CCX Primary CCX Secondary CCX

Reporting Master Election initiated on Secondary CVD

• Master Election initiated Secondary CVD elects Secondary as Master

• Failover completed CONVERGENCE_COMPLETED, name=Cisco Unified CCX Engine

Secondary CVD performs Master Election

Secondary CVD elects Secondary as Master

Secondary CVD performs Master election and drops Master

Finesse on Node 2 will

Engine mastership is Finesse on Node 2 will

Finesse on Node 1 will Finesse goes Out Of Service

• Publishes every 3 CUIC Live Data

SocketIO • Receives through JMS all the

SocketIO • Dispatches the updates to

• SocketIO Service logs – DEBUG mode

558501: Jan 26 18:57:26.008 EST %MIVR-ICD_RTDM-7-UNK:LiveData:

558510: Jan 26 18:57:26.279 EST %MIVR-SS_RMCM-7-UNK:HeartBeatAction : Sending

• Here agent=agent1, JMS topic=ResourceIAQStats.

0000001076: 10.106.87.133: Jan 26 2016 18:57:27.007 +0530:

• 0000005791: 10.106.87.133: Jan 26 2016 18:57:27.008 +0530:

OVA Profile • Resources, NIC = VMXNET3 adapter

CCX Application Configuration Datastore

Historical Datastore UCCX DB

CLI: utils network connectivity <hostname of secondary>

CLI: utils ntp status

CLI: utils diagnose test

Tools-> Datastore Control Center->

admin:show uccx dbreplication servers