Troubleshooter 43028
Troubleshooter 43028
Troubleshooter 43028
Version 5 Release 4
Troubleshooting Guide
GC27-2507-00
Tivoli IBM Tivoli NetView for z/OS
®
Version 5 Release 4
Troubleshooting Guide
GC27-2507-00
Note
Before using this information and the product it supports, read the information in “Notices” on page 661.
This edition applies to version 5, release 4 of IBM Tivoli NetView for z/OS (product number 5697-ENV) and to all
subsequent versions, releases, and modifications until otherwise indicated in new editions.
This edition replaces LY43-0093.
© Copyright International Business Machines Corporation 1997, 2009.
US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract
with IBM Corp.
Contents
Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
iv Troubleshooting Guide
Using NetView Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
Locating the Trace When MODE=INT Is Specified . . . . . . . . . . . . . . . . . . . . . 100
Printing the Trace When MODE=EXT Is Specified . . . . . . . . . . . . . . . . . . . . . 103
Describing NetView Trace Records Displayed with the TRACE Command’s MONOPER Keyword . . . . . 105
Trace Record Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
IP Services Trace Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
Status Monitor Internal Trace Records . . . . . . . . . . . . . . . . . . . . . . . . . 135
Security Authorization Facility Trace Record . . . . . . . . . . . . . . . . . . . . . . . 137
SNA Topology Manager NetView Trace Entries . . . . . . . . . . . . . . . . . . . . . . 144
First Failure Data Capture Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Program-to-Program Interface (PPI) Trace Facility . . . . . . . . . . . . . . . . . . . . . . 148
Understanding the PPI Trace Anchor Block and the PPI Trace Table . . . . . . . . . . . . . . . 148
Understanding the Program-to-Program Interface Trace Record. . . . . . . . . . . . . . . . . 149
Locating the Program-to-Program Interface (PPI) Trace Table . . . . . . . . . . . . . . . . . 151
Locating the Oldest Program-to-Program Interface Trace Record . . . . . . . . . . . . . . . . 152
Generalized Trace Facility (GTF) Output Files . . . . . . . . . . . . . . . . . . . . . . 152
Chapter 10. Graphic Monitor Facility Host Subsystem Problem Worksheet . . . . . . 169
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Contents v
System Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
GMFHS Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
RODM Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
RODM Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Problem Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Abend problems (processor exception/trap/fault) . . . . . . . . . . . . . . . . . . . . . 170
Message Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Loop Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Wait Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Incorrect Output Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Documentation Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console
and GMFHS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
Alert and Alert History Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Alerts Are Not Listed in the Event Viewer at the NetView Management Console Workstation . . . . . . 177
Alerts Are Not Listed in the Hardware Monitor Alerts History Panel . . . . . . . . . . . . . . . 178
Alerts Do Not Change Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Command Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Cannot Initiate an IP Session Using NETCONV . . . . . . . . . . . . . . . . . . . . . . 180
Cannot Initiate an LU 6.2 Session Using NETCONV . . . . . . . . . . . . . . . . . . . . 181
Command Results Are Unexpected from Network Management Gateways . . . . . . . . . . . . . 182
Commands Failed to Run Because of COS Gateway Errors . . . . . . . . . . . . . . . . . . 182
Commands Failed to Run Because of OST Errors . . . . . . . . . . . . . . . . . . . . . 182
Commands Failed to Run Because of PPI Gateway Errors . . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of RODM Attribute Errors . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of Service Point Errors . . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of Time-out Errors . . . . . . . . . . . . . . . . . . . . 183
Commands Failed with Message IHS2069W, Command Exit Not Installed . . . . . . . . . . . . . 184
GMFHS Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Errors Are Received During GMFHS Configuration Initialization . . . . . . . . . . . . . . . . 185
GMFHS Status Solicitation Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Instrumentation (z/OS-based only) Problems . . . . . . . . . . . . . . . . . . . . . . . . 186
Events Are Not Received from z/OS Instrumentation . . . . . . . . . . . . . . . . . . . . 186
Component or Connection Status is not Properly Reflected on the Topology Console . . . . . . . . . 186
Status Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Resource Status Is Incorrect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
The Resource Exists but the Status Is Not Updated . . . . . . . . . . . . . . . . . . . . . 188
Multiple Init_Accept Flows Received . . . . . . . . . . . . . . . . . . . . . . . . . 188
Status Update Performance Decreases . . . . . . . . . . . . . . . . . . . . . . . . . 189
Status Changes to Resources Are Not Reflected in Views . . . . . . . . . . . . . . . . . . . 189
Topology Console Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
Unable to Connect to the Topology Server from the Topology Console . . . . . . . . . . . . . . 190
Topology Console Hangs During Sign-on . . . . . . . . . . . . . . . . . . . . . . . . 190
Topology Console Hangs When Accessing a View . . . . . . . . . . . . . . . . . . . . . 190
There Is a Duplicate GMFHS Resource on the Topology Console . . . . . . . . . . . . . . . . 191
Problems Occur with Minimized Windows . . . . . . . . . . . . . . . . . . . . . . . 191
Property Changes Are Lost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Topology Server Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
Server Does not Start and setup_env.cmd Is not Found . . . . . . . . . . . . . . . . . . . 192
Setup_env.cmd Is Found but BINDIR Is not Set . . . . . . . . . . . . . . . . . . . . . . 192
| Topology Server Starts but Then Closes (Windows). . . . . . . . . . . . . . . . . . . . . 192
Topology Server Starts but Then Closes (All Platforms) . . . . . . . . . . . . . . . . . . . 193
Topology Server Does Not Complete Initialization on AIX . . . . . . . . . . . . . . . . . . 193
Server Windows Disappear on AIX Platform . . . . . . . . . . . . . . . . . . . . . . . 193
Incorrect Timestamps If the Topology Server is on the Windows Platform . . . . . . . . . . . . . 193
View Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
Expected Configuration or More Detail View Does Not Exist . . . . . . . . . . . . . . . . . 196
Resource Icon Is Missing from View . . . . . . . . . . . . . . . . . . . . . . . . . . 197
vi Troubleshooting Guide
Tree View List Is Incorrect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
View Layout Is Incorrect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
Unable to Open View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
Unable to Monitor Views of Your Network . . . . . . . . . . . . . . . . . . . . . . . 200
View Does Not Show Correct Connectivity . . . . . . . . . . . . . . . . . . . . . . . 201
View Does Not Contain Resource. . . . . . . . . . . . . . . . . . . . . . . . . . . 202
Multiple Correlated Aggregate Objects Contain the Same Object . . . . . . . . . . . . . . . . 204
Real Resource Is Not Shown as a Member of a Correlated Aggregate Object . . . . . . . . . . . . 205
Information Displayed for Correlated Aggregate Object Changes . . . . . . . . . . . . . . . . 206
Cannot Navigate Between Correlated Aggregate Object and Contained Resources . . . . . . . . . . 207
Pop-up Menu in the Business Tree Is Not Displayed on AIX Platform . . . . . . . . . . . . . . 207
Preview Image Partially Painted in View Properties Notebook . . . . . . . . . . . . . . . . . 207
The Topology Display Subsystem View Is Not Complete . . . . . . . . . . . . . . . . . . . 208
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS . . . . . 209
Diagnostic Tools for the NetView Management Console . . . . . . . . . . . . . . . . . . . . 209
Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Access to Online Help without the Console . . . . . . . . . . . . . . . . . . . . . . . 209
Topology Console Environment Information Window . . . . . . . . . . . . . . . . . . . . 210
Message Help for the Topology Server . . . . . . . . . . . . . . . . . . . . . . . . . 210
Diagnostic Tools for GMFHS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
GMFHS Message Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
GMFHS Output Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Console Log Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
GMFHS Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Starting and Stopping the GMFHS Trace . . . . . . . . . . . . . . . . . . . . . . . . 214
Viewing the GMFHS Trace Online . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Using the GMFHS Internal Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
IPC Task Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Event Manager Task Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Contents vii
Abend 9C5 with Reason Code 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Abend 9C5 with Reason Code 33 . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
RODM Abends When SNA Topology Manager Is Starting . . . . . . . . . . . . . . . . . . . 237
User Application Looping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
Asynchronous Method Looping . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
User API Does Not Return from EKGWAIT . . . . . . . . . . . . . . . . . . . . . . . . 239
Incorrect Output is in the EKGPRINT Data Set . . . . . . . . . . . . . . . . . . . . . . . 239
RODM Fails to Complete Checkpoint Processing . . . . . . . . . . . . . . . . . . . . . . 239
Abnormal Reaction from RODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
Slow Response from RODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager. . . 307
Problems During Initialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Wrong Autotask Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Insufficient Storage for Topology Manager Initialization . . . . . . . . . . . . . . . . . . . 309
Error Reading Initialization File FLBSYSD . . . . . . . . . . . . . . . . . . . . . . . . 309
Error Reading Customization Table FLBOSIDS, FLBSRT, or FLBEXV . . . . . . . . . . . . . . . 309
Cannot Connect to VTAM CMIP Services . . . . . . . . . . . . . . . . . . . . . . . . 311
Cannot Connect to RODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
FLBTOPO Task Abends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Abend During Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Abend After Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
User Abend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Abend Error Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Chapter 18. Diagnostic Tools for the SNA Topology Manager . . . . . . . . . . . . 363
SNA Topology Manager Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
SNA Topology Manager Log Entries. . . . . . . . . . . . . . . . . . . . . . . . . . . 364
SNA Topology Manager Log Record Formats. . . . . . . . . . . . . . . . . . . . . . . 365
System Interface Log Entries-Major Code 22 . . . . . . . . . . . . . . . . . . . . . . . 367
SNA Topology Manager Log Entries—Major Code 78 . . . . . . . . . . . . . . . . . . . . 373
Common Log Entries-Major Code 79 . . . . . . . . . . . . . . . . . . . . . . . . . 399
SNA Topology Manager Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
Contents ix
External Tracing (GTF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
Tracing Internally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Trace Events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
Recovery from Trace Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
TOPOSNA LISTxxxx Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . 427
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager
Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
Routing Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Improving INITTOPO Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
RODM Errors - FLC070E and FLC076E . . . . . . . . . . . . . . . . . . . . . . . . . . 436
RODM Errors - Return Code 12 and Reason Code 122. . . . . . . . . . . . . . . . . . . . . 437
RODM Error 8/45077 Occurs after Reissuing a GETTOPO TMERES Command . . . . . . . . . . . . 437
Issuing Commands That Contain Case-Sensitive Text . . . . . . . . . . . . . . . . . . . . . 438
Command Support Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
GETTOPO Command Failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
Tracing GETTOPO Command Processing . . . . . . . . . . . . . . . . . . . . . . . . 438
GMFHS Is Unavailable During GETTOPO Command Processing . . . . . . . . . . . . . . . . 439
| Failures in the IP Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
Command Failures in the SNA Environment . . . . . . . . . . . . . . . . . . . . . . . 440
Failures Caused by Improperly Installed IP Agent on AIX Using LU 6.2 . . . . . . . . . . . . . . 440
Failures After Opening a New Map . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Object Status Update Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Missing IP Objects from NetView Management Console Views . . . . . . . . . . . . . . . . . . 443
Extra IP Objects in NetView Management Console Views. . . . . . . . . . . . . . . . . . . . 443
Aggregate Object Contains Identical Real Objects . . . . . . . . . . . . . . . . . . . . . . 443
x Troubleshooting Guide
AON Automation Routines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
EZLEFAIL Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 452
EZLERECV Routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 467
Event/Automation Service Abends . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
Event/Automation Service Task Might be Suspended . . . . . . . . . . . . . . . . . . . . . 468
START, STOP, or RECYCLE Commands Do Not Function Properly . . . . . . . . . . . . . . . . 469
A Service Does Not Complete Initialization . . . . . . . . . . . . . . . . . . . . . . . . 469
Event/Automation Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . 469
Alert Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
Confirmed Alert Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . 470
Message Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
Confirmed Message Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . 472
Event Receiver Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
Trap-to-Alert Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . 473
Alert-to-Trap Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . 473
Unwanted Services are Starting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
Alerts Are Not Forwarded to the Expected Event Server . . . . . . . . . . . . . . . . . . . . 474
Alerts Are Not Converted to the Expected IBM Tivoli Enterprise Console Events . . . . . . . . . . . . 477
An Alert Is Continuously Forwarded . . . . . . . . . . . . . . . . . . . . . . . . . . 477
An Alert Is Incorrectly Cached . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
| Messages Are Not Forwarded to the Expected Event Server . . . . . . . . . . . . . . . . . . . 479
Messages Are Not Converted to IBM Tivoli Enterprise Console Events . . . . . . . . . . . . . . . 481
A Message Is Incorrectly Cached . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
IBM Tivoli Enterprise Console Events Are Not Forwarded to the Hardware Monitor . . . . . . . . . . 482
IBM Tivoli Enterprise Console Events Are Not Converted to Alerts . . . . . . . . . . . . . . . . 483
| No Reply from an Event Server to which a Tivoli Enterprise Console Event Was Sent . . . . . . . . . . 484
| Negative Response from an Event Server to which a Tivoli Enterprise Console Event Was Sent . . . . . . . 485
SNMP Traps Are Not Forwarded to the Hardware Monitor . . . . . . . . . . . . . . . . . . . 485
SNMP Traps Are Not Converted to Alerts . . . . . . . . . . . . . . . . . . . . . . . . . 486
Recycling the NetView PPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
Recycling the Event Receiver for IP Connectivity Problems . . . . . . . . . . . . . . . . . . . 487
Recycling the Trap-to-Alert Service for IP Connectivity Problems . . . . . . . . . . . . . . . . . 487
Contents xi
Using Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
DISPLAY STATUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 489
DISPLAY QSTATS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491
Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
The IP Trace Option . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492
NCCF GENALERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
RPCINFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495
Using the TestMode Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
Looping the Alert or Message Adapter to the Event Receiver . . . . . . . . . . . . . . . . . . 496
Looping the Alert-to-Trap Service to the Trap-to-Alert Service . . . . . . . . . . . . . . . . . . 497
Chapter 27. Troubleshooting and Initial Diagnosis for the NetView Web Application 505
Web Application Cannot Be Started . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
Web Pages Are Not Displaying On a Browser . . . . . . . . . . . . . . . . . . . . . . . 506
Events Are Not Displayed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
Unexpected Signon Panel or Browser Session Timeout. . . . . . . . . . . . . . . . . . . . . 507
| Problems Viewing NetView Web Services Gateway Data . . . . . . . . . . . . . . . . . . . . 508
Problems Viewing OMEGAMON XE Mainframe Network Data . . . . . . . . . . . . . . . . . 508
CT_Get Request Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
Performance Data Cannot Be Viewed . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Open Incident Button Not Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
3270 Console Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Task Assistant and Task Buttons Do Not Work . . . . . . . . . . . . . . . . . . . . . . . 510
Part 10. Diagnosing Tivoli NetView for z/OS Enterprise Management Agent
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
Chapter 28. Tivoli NetView for z/OS Enterprise Management Agent Worksheet . . . . 513
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
System-Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Problem Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Abend problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 514
Processor Traps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
Message Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 515
Loop, Hang, or Lockup Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . 516
Wait Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 516
Incorrect Output Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
| Verifying TCP/IP Stack or DVIPA Information . . . . . . . . . . . . . . . . . . . . . . 517
Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 517
Documentation Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 518
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS
Enterprise Management Agent . . . . . . . . . . . . . . . . . . . . . . . . . 519
The NetView agent is not displayed in the Navigator view . . . . . . . . . . . . . . . . . . . 520
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management
Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 529
Using NetView online message and command help for the NetView agent . . . . . . . . . . . . . . 529
Using the CNMTRACE function for NetView host components of the NetView agent function . . . . . . . 529
Using the NACTL command to troubleshoot the NetView agent . . . . . . . . . . . . . . . . . 531
Using the DISPPI command to troubleshoot a PPI connection between NetView and the NetView agent . . . . 531
Troubleshooting Data Spaces for a given data collector . . . . . . . . . . . . . . . . . . . . 531
Problem determination for a NetView agent . . . . . . . . . . . . . . . . . . . . . . . . 532
Problem determination flow for the NetView agent. . . . . . . . . . . . . . . . . . . . . 532
Determining if the problem was caused by the NetView agent . . . . . . . . . . . . . . . . . 533
NetView agent communication layer messages and tracing . . . . . . . . . . . . . . . . . . 538
| Using the KDC_DEBUG environment variable . . . . . . . . . . . . . . . . . . . . . . 539
Setting up RAS1 tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
Understanding and using RAS1 logs . . . . . . . . . . . . . . . . . . . . . . . . . 546
Capturing z/OS logs to send to software support . . . . . . . . . . . . . . . . . . . . . 546
Contents xiii
RECFMS Header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
RECFMS 00 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
Text Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
Detail Qualifier Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
Name List Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 613
Null Vector . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 614
RECFMS 01 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
RECFMS 02 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
RECFMS 03 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615
RECFMS 04 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 617
Loop Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Loop Errors and Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Loop Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 618
Workstation Response Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Host Batch Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 619
Message Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 620
Basic and Extended Statistical Counters . . . . . . . . . . . . . . . . . . . . . . . . 620
RECFMS 05 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
Release Level Data (RECFMS 05) . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
IBM System/38. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
IBM 3104 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 621
RECFMS 05, 3174 Configuration Information . . . . . . . . . . . . . . . . . . . . . . . 622
RECFMS 05 for the 3174: RPQ, Patch, and DFT Information . . . . . . . . . . . . . . . . . . 634
Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 661
Programming Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 662
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 665
Intended audience
This publication is for system programmers, network programmers, and operators
who need more information than the help panels provide. It presents formats and
procedures for people who diagnose, document, and report software and hardware
problems.
Publications
This section lists publications in the IBM Tivoli NetView for z/OS library and
related documents. It also describes how to access Tivoli publications online and
how to order Tivoli publications.
Related publications
You can find additional product information on the NetView for z/OS Web site:
http://www.ibm.com/software/tivoli/products/netview-zos/
For information about the NetView Bridge function, see Tivoli NetView for OS/390
Bridge Implementation, SC31-8238-03 (available only in the V1R4 library).
http://publib.boulder.ibm.com/tividd/glossary/tivoliglossarymst.htm
The IBM Terminology Web site consolidates the terminology from IBM product
libraries in one convenient location. You can access the Terminology Web site at the
following Web address:
http://www.ibm.com/software/globalization/terminology/
For a list of NetView for z/OS terms and definitions, refer to the IBM Terminology
Web site. The following terms are used in this library:
NetView
For the following products:
| v Tivoli NetView for z/OS version 5 release 4
v Tivoli NetView for z/OS version 5 release 3
v Tivoli NetView for z/OS version 5 release 2
v Tivoli NetView for z/OS version 5 release 1
v Tivoli NetView for OS/390® version 1 release 4
MVS For z/OS operating systems
MVS element
For the BCP element of the z/OS operating system
CNMCMD
For the CNMCMD member and the members that are included in it using
the %INCLUDE statement
Unless otherwise indicated, references to programs indicate the latest version and
release of the programs. If only a version is indicated, the reference is to all
releases within that version.
You can use LookAt from the following locations to find IBM message
explanations for z/OS elements and features, z/VM®, VSE/ESA, and Clusters for
AIX® and Linux® systems:
v The Internet. You can access IBM message explanations directly from the LookAt
Web site at http://www.ibm.com/systems/z/os/zos/bkserv/lookat/ .
v Your z/OS TSO/E host system. You can install code on your z/OS or z/OS.e
system to access IBM message explanations, using LookAt from a TSO/E
command line (for example, TSO/E prompt, ISPF, or z/OS UNIX® System
Services running OMVS).
v Your Microsoft® Windows® workstation. You can install LookAt directly from the
z/OS Collection (SK3T-4269) or the z/OS and Software Products DVD Collection
(SK3T-4271) and use it from the resulting Windows graphical user interface
(GUI). The command prompt (also known as the DOS command line) version
can still be used from the directory in which you install the Windows version of
LookAt.
v Your wireless handheld device. You can use the LookAt Mobile Edition from
http://www.ibm.com/systems/z/os/zos/bkserv/lookat/lookatm.html with a
handheld device that has wireless access and an Internet browser.
You can obtain code to install LookAt on your host system or Microsoft Windows
workstation from the following locations:
v A CD in the z/OS Collection (SK3T-4269).
v The z/OS and Software Products DVD Collection (SK3T-4271).
xx Troubleshooting Guide
v The LookAt Web site. Click Download and then select the platform, release,
collection, and location that you want. More information is available in the
LOOKAT.ME files that is available during the download process.
IBM posts publications for this and all other Tivoli products, as they become
available and whenever they are updated, to the Tivoli Information Center Web
site at http://publib.boulder.ibm.com/infocenter/tivihelp/v3r1/index.jsp.
Note: If you print PDF documents on other than letter-sized paper, set the option
in the File → Print window that enables Adobe® Reader to print letter-sized
pages on your local paper.
Ordering publications
You can order many Tivoli publications online at
http://www.elink.ibmlink.ibm.com/publications/servlet/pbi.wss
Accessibility
Accessibility features help users with a physical disability, such as restricted
mobility or limited vision, to use software products successfully. Standard shortcut
and accelerator keys are used by the product and are documented by the operating
system. Refer to the documentation provided by your operating system for more
information.
For additional information, see the Accessibility appendix in the User’s Guide:
NetView.
Downloads
Clients and agents, NetView product demonstrations, and several free NetView
applications can be downloaded from the NetView for z/OS support Web site:
In the ″IBM Tivoli for NetView for z/OS support″ pane, click Download to go to a
page where you can search for or select downloads.
Typeface conventions
This publication uses the following typeface conventions:
Bold
When using the Windows command line, replace $variable with %variable% for
environment variables and replace each forward slash (/) with a backslash (\) in
directory paths. The names of environment variables are not always the same in
the Windows and UNIX environments. For example, %TEMP% in Windows
environments is equivalent to $TMPDIR in UNIX environments.
Note: If you are using the bash shell on a Windows system, you can use the UNIX
conventions.
Syntax diagrams
Read syntax diagrams from left-to-right, top-to-bottom, following the horizontal
line (the main path). This section describes how syntax elements are shown in
syntax diagrams.
Symbols
The following symbols are used in syntax diagrams:
Marks the beginning of the command syntax.
Indicates that the command syntax is continued.
| Marks the beginning and end of a fragment or part of the command
syntax.
Parameters
The following types of parameters are used in syntax diagrams:
Required Required parameters are shown on the main path.
Optional Optional parameters are shown below the main path.
Default Default parameters are shown above the main path. In parameter
descriptions, default parameters are underlined.
USER user_id
password
When an operand can have more than one value, the values typically are enclosed
in parentheses and separated by commas. For a single value, the parentheses
typically can be omitted. For more information, see “Multiple operands or values”
on page xxv.
When examples of commands are shown, commas are also used to indicate the
absence of a positional operand. For example, the second comma indicates that an
optional operand is not being used:
COMMAND_NAME opt_variable_1,,opt_variable_3
You do not need to specify the trailing positional commas. Trailing positional and
non-positional commas either are ignored or cause a command to be rejected.
Restrictions for each command state whether trailing commas cause the command
to be rejected.
Abbreviations
Command and keyword abbreviations are listed in synonym tables after each
command description.
Syntax examples
This section show examples for the different uses of syntax elements.
REQUIRED_KEYWORD required_variable
REQUIRED_OPERAND_OR_VALUE_1
REQUIRED_OPERAND_OR_VALUE_2
Optional syntax elements: Optional keywords and variables are shown below the
main syntax line. You can choose not to code optional keywords and variables.
OPTIONAL_OPERAND
OPTIONAL_OPERAND_OR_VALUE_1
OPTIONAL_OPERAND_OR_VALUE_2
Default keywords and values: Default keywords and values are shown above the
main syntax line in one of the following ways:
v A default keyword is shown only above the main syntax line. You can specify
this keyword or allow it to default. The following syntax example shows the
default keyword KEYWORD1 above the main syntax line and the rest of the
optional keywords below the main syntax line.
v If an operand has a default value, the operand is shown both above and below
the main syntax line. A value below the main syntax line indicates that if you
specify the operand, you must also specify either the default value or another
value shown. If you do not specify the operand, the default value above the
main syntax line is used. The following syntax example shows the default values
for operand OPTION=* above and below the main syntax line.
,KEYWORD1 ,OPTION=*
COMMAND_NAME
,KEYWORD2 ,OPTION= *
,KEYWORD3 VALUE1
,KEYWORD4 VALUE2
KEYWORD=( value_n )
,
REPEATABLE_OPERAND_OR_VALUE_1
REPEATABLE_OPERAND_OR_VALUE_2
REPEATABLE_OPERAND_OR_VALUE_3
Syntax that is longer than one line: If a diagram is longer than one line, each line
that is to be continued ends with a single arrowhead and the following line begins
with a single arrowhead.
Syntax fragments: Some syntax diagrams contain syntax fragments, which are
used for lengthy, complex, or repeated sections of syntax. Syntax fragments follow
the main diagram. Each syntax fragment name is mixed case and is shown in the
main diagram and in the heading of the fragment. The following syntax example
shows a syntax diagram with two fragments that are identified as Fragment1 and
Fragment2.
COMMAND_NAME Fragment1
Fragment2
Fragment1
Fragment2
Finding Solutions
If you have the IBM Information/Access program, you can search the RETAIN®
database, on the basis of a keyword string, to find similar problems and their
solutions.
The representative uses the information from the worksheet to form a keyword
string and search a database containing symptoms and resolutions for problems.
This database also contains information on problems currently under investigation.
The representative might ask you for additional information to produce other
keywords that can help locate and solve the problem.
If the search does not produce a solution, the representative verifies that you have
the necessary information to discuss the problem with a specialist. Your call is then
placed in a queue accessed by IBM Software Support specialists.
An IBM Software Support specialist helps you refine keyword strings and conducts
additional searches of the database.
In general, a IBM Software Support representative can solve most problems, but in
cases when no solution is found, the problem is sent to an IBM Software Support
specialist. If the specialist cannot find a solution, and if the problem is a new one,
the specialist can enter an authorized program analysis report (APAR) into the
RETAIN database. An APAR is a request for a correction in the program.
The APAR and other types of documentation enable the program specialist to
examine the problem in greater detail and develop a solution. If this solution is a
coding change, it is put into a program temporary fix (PTF) and sent to you. All
Using Keywords
Each problem type has an associated keyword. The keyword is used as a general
identifier and to search the RETAIN database. If you have access to the RETAIN
database, you can search it. Otherwise, you can provide IBM Software Support
with the keyword and they will do the search. In searching the RETAIN database,
you can determine whether:
v Your particular problem has already been reported
v There is a bypass for your problem
v Your problem has been solved
v A solution already exists for your problem
An accurate and precise search string produces usable results. A string search
contains the following:
v The keyword that represents your problem type
v The level of the NetView product and, if applicable, the load level of the
NetView management console you are using
v Additional symptoms of the problem
If you proceed through all seven classifications and cannot match your problem to
any of those described, see “Documenting Any Problem” on page 19.
4 Troubleshooting Guide
Table 1 describes how to develop a keyword string:
Table 1. How to Develop a Keyword String
Use This Type of Example of a
Type of Keyword Description or Value Keyword to... Keyword String
Component The component Find all reported xxxxxxxxx
identification identification number problems with the 5697ENV00
for the operating NetView product or
system. The one of its
component ID components.
number for Tivoli
NetView for z/OS
V5R4 operating
under z/OS is
5697ENV00.
Failure v ABEND Refine your search to MSGDSIxxxx, where
v DOC just that type of xxxx is the message
v INCORROUT failure for NetView number (for example,
v LOOP or one of its 172I).
v MSG components.
v PERFM
v WAIT
Symptom Details about the Refine your search BNJyyyyy identifies
failure. gradually (combining the name of the
the symptom NetView module that
keywords in various issued the abend.
ways) so that you
receive all problem
descriptions that
might match your
problem.
Dependency Program or device Help reduce the APPN
dependent keywords number of problem
that define the descriptions you
environment in need to examine.
which the problem
occurred.
For example, if there is an abend in a DSI NetView module, enter the following
keyword string:
5697ENV00 ABENDnnn DSIyyyyy
Where:
5697ENV00
Component ID for the program
ABEND
Type of problem
nnn Abend code number
DSIyyyyy
NetView module that issued the abend failure message.
6 Troubleshooting Guide
Can you No
classify the
problem
Yes
Gather
information for
your problem
Gather
information for
all problems
Yes
Build search
string and look
for a match
Do
your
symptoms No
match a known
problem
Yes
Yes
If this is a valid
problem, submit APAR
documentation and
await APAR resolution
Apply fix
Figure 1. The Diagnostic Path for Classifying, Documenting, and Reporting Problems
When you decide what the problem keyword is, you can use it to develop a
keyword string. See Chapter 1, “Diagnosing Problems,” on page 3 for information
about the keyword string.
In the following topics, the symptoms are described for ABEND, DOC,
INCORROUT, LOOP, MSG, PERFM, and WAIT problems.
Identifying Symptoms
The symptoms described in this section can apply to any NetView component.
Note: For problem determination, keep the internal NetView trace active or use
the default size of 4000.
ABEND
The ABEND symptoms apply to NetView, GMFHS, RODM, SNA topology
manager, and Event/Automation Service (E/AS).
If, after reading about abends, you categorize your problem as an abend, see
“Documenting ABEND Problems” on page 24.
NetView
Choose the ABEND keyword when one or more of the following symptoms occur:
v An MVS ABEND message is displayed at the system operator console. The
message that contains the abend code is found in the system console log.
v Message DSI172I is displayed.
GMFHS
For GMFHS, choose the ABEND keyword when the following messages are
written to the system console:
GMFHS IS DUMPING FOR TASK taskname, COMPLETION CODE = completioncode
GMFHS SDUMP FOR TASK taskname COMPLETED, RETURN CODE = returncode,
REASON CODE =reasoncode
where
taskname Name of the GMFHS task that caused the abend
completioncode Abend completion code
returncode SDUMP return code
reasoncode SDUMP reason code
RODM
If RODM, one of its components, or an application fails, RODM writes a return
code and reason code to the RODM log. The return code and reason code might
also be returned to your application. You might not see an external symptom of
the failure (unless the return code with reason code is returned to the application
to signal the failure).
For RODM, choose the ABEND keyword when one or more of the following
symptoms occur:
v An MVS ABEND message for the RODM address space is displayed at the
system operator console.
v One of the following RODM messages is received:
– EKG5010E
– EKG1981E
– EKG1982E
– EKG1983E
– EKG1984I
– EKG1985I
– EKG1986I
– EKG1987E
– EKG1988E
– EKG1989E
– EKG1996E
v The user application receives a return code of 12 and a reason code of either 20
or 194.
v The RODM log contains a type 7 log record.
10 Troubleshooting Guide
For information about: See:
RODM return code and reason code Chapter 14, “Troubleshooting and Initial
combinations Diagnosis for RODM,” on page 227
The contents of the RODM log Chapter 15, “Diagnostic Tools for RODM,”
on page 243
The following is an example of the abend message that is generated if the SNA
topology manager abends:
DSI819I NETVIEW IS DUMPING FOR TASK FLBTOPO.
COMPLETION CODE= X'hhhhhh', DOMAIN=domainid.
DOC
Choose the DOC keyword when one or more of the following symptoms is true
for the documentation or online help panels:
v They contain incomplete or inaccurate information about installation, operation,
customization, messages, or diagnosis.
v They are inconsistent in describing the use of a program function.
INCORROUT
Choose the INCORROUT keyword when you receive one of the following
symptoms:
v You receive unexpected output such as a garbled message, and the problem does
not seem to be a loop.
v When displaying the view, the resource information contains strange or garbled
characters.
v The view displayed does not show a resource that is part of your network.
If you suspect that the SNA topology manager component is producing incorrect
output, verify that all required functions are working. For example, if the status of
an object is not being updated, verify that the following is true:
v NetView management console is active and communicating with your
mainframe server system.
v GMFHS is active and processing data in the RODM data cache.
v RODM is active and processing requests.
v The SNA topology manager is active, storing any received information in the
RODM data cache, and monitoring the required agent nodes.
v The agent nodes are sending the correct SNA topology information.
LOOP
Choose the LOOP keyword when one or more of the following symptoms occur:
v Part of the program repeats itself as seen in a system or NetView trace. A
repeating program is indicated when the same message or set of messages is
being repeatedly displayed or logged.
v The same message or set of messages is being repeatedly displayed on the
workstation.
v A command has not completed after the expected time period, and the processor
is used more frequently than usual.
v There is high processor use, console (operator terminal) lockout, or high channel
activity to a NetView database.
12 Troubleshooting Guide
v System commands are not accepted after issuing a RODM subsystem command
or a NetView RODM component command.
v The TASKUTIL, TASKMON, or TASKURPT command display shows increased
processor use by a particular NetView task that cannot be explained.
The SNA topology manager has a command (TOPOSNA QUERYDEF) that queries
local settings and does not require a significant amount of time to process. You can
use this command to determine whether the manager task is looping.
Note: Consider the current workload on the SNA topology manager. Sometimes,
the manager has to process a large amount of incoming data; therefore,
increased processor usage is not necessarily a sign of a loop. A loop is
probably occurring if the increased usage is sustained for an excessive
period of time.
If you have categorized your problem as a loop problem, see “Documenting LOOP
Problems” on page 33.
MSG
A problem can cause a message to be displayed at the system console or at an
operator terminal. Choose the MSG keyword when one or more of the following
symptoms occur:
v The message received is not the expected response or indicates an error
condition.
Use the HELP command for an online explanation of a message. For example, for
more information about RODM message EKG3100E, enter the following:
HELP EKG3100E
When you are using NetView management console and a problem originates at the
mainframe server, a message is displayed at the system console or at the
workstation.
Each message issued by the NetView program is displayed in the form xxxn...ny,
where:
xxx Is a prefix identifier, such as DSI, BNJ, AAU.
Notes:
v If the message associated with your problem does not have a prefix of
AAU, BNH, BNJ, CNM, DFI, DSI, DUI, DWO, EZL, FKV, FKX, FLB,
FLC, and IHS, the problem is probably not with the NetView program.
v IHS and EGV prefixed messages are issued from the NetView
management console for programmable workstations.
v EKG prefixed messages are from RODM.
v FLB prefixed messages are from the SNA topology manager.
n...n Is a message number. The message number is component-unique. For
informational messages displayed at the workstation, the prefix and
message number might not be displayed. However, for Presentation
Manager type messages, online help is available by pressing the F1 key.
y Is a suffix defining the type. This suffix is not displayed for VIO pop-up
messages. The types are as follows:
I Is an information message
A Signals that an action must be taken
D Signals that a decision is required immediately
14 Troubleshooting Guide
W Is a warning message
E Indicates an error condition
S Indicates a severe error condition
DUI 1611 W
product suffix
prefix
message number
PERFM
Choose the PERFM keyword if performance is not as expected. Performance
problems can occur because one or more of the following conditions exist:
v NetView commands (including VTAM commands and system commands
entered from a terminal logged on to the NetView program) take an excessive
time to complete.
v NetView performance characteristics are below expectations.
v System response is slow.
v CPU initialization is increased.
Use the TASKUTIL or TASKMON command to measure CPU utilization.
v A large number of status updates are being forwarded to the graphic data
server.
v Resource definitions at the mainframe server or workstation, or both, are not
correct.
v Communication errors exist between the mainframe server and the workstation.
If all available disk storage is used, the workstation cannot perform the specified
request and you receive an error message. You can receive either of the following:
v A message stating that no disk storage is available.
This message includes the cause of the error, the time the error occurred, and
instructions on how to increase your storage space.
v A message stating that a resource cannot be allocated.
This message is received when the specified maximum number of resource
definitions is reached. Determine whether the number of resource definition
specifications can be increased.
The SNA topology manager at times has to process a large amount of incoming
data. There can be periods of peak activity where the performance of the topology
manager is degraded. This is usually a temporary condition, depending on the
frequency and amount of data being received from the agent nodes.
16 Troubleshooting Guide
WAIT
Use the WAIT keyword when processing stops for a NetView task with no
abnormal completion (ABEND) codes and no increased processor use. For
example, use WAIT if you enter a NetView command and receive no response, but
the processor and all other jobs start and end normally.
Choose the WAIT keyword when one or more of the following symptoms occur:
v The operator cannot enter commands or communicate with the NetView
program, and the system does not seem to be in a loop. Using several
TASKUTIL commands from another task should not show an increase in the
CPU time for the operator task in question.
v There is no response to commands.
v The workstation does not respond to keyboard or mouse commands, and the
system does not seem to be in a loop.
v There is no response from a graphical workstation.
v RODM has not ended abnormally.
v The SNA topology manager related views at the graphical workstation are not
receiving the expected updates.
v There is no excessive processor use.
v The processor and all other jobs are starting and ending normally.
A message from the NetView program that indicates you are waiting for some
data, such as one of the following messages, is not necessarily evidence of a
problem:
BNJ913I HARDWARE MONITOR WAITING FOR DATA,
ENTER 'NPDA END' TO CANCEL REQUEST
Workstation specifics:
The SNA topology manager has a command (TOPOSNA QUERYDEF) that queries
local settings, and does not require a significant amount of time to process. You
can use this command to determine whether the manager task is suspended.
1. Issue the TASKUTIL command for the SNA topology manager autotask
(FLBTOPO). If the results indicate little or no processor usage by the task, the
task might be suspended.
2. Issue the TOPOSNA QUERYDEF command to determine whether the task is
suspended.
3. If the response to the command is received, the manager task is not suspended.
v If you are experiencing slow response time to local commands (such as
QUERYDEF), the rest of your system might be overloaded.
If you have categorized your problem as a wait problem, see “Documenting WAIT
Problems” on page 37.
18 Troubleshooting Guide
Chapter 3. Documenting and Reporting Problems
IBM Software Support is the first point of contact for NetView customers who
need help with a program problem after installation is complete. Contact the local
marketing systems engineer for assistance on problems encountered during
installation.
h The NetView component ID number, FMID, release number, and RSU level
Record any recently applied NetView maintenance, such as PTFs or APAR fixes.
Use the DISPMOD command to obtain PTF levels online. The PTF level is in the
dump, following the module identifier.
Research the scenario leading to the problem, including the commands entered
before the problem occurred. You can obtain this information in the network log
from the operator that has the problem.
Record the commands exactly as they were entered. Consider the following:
v What was the first indication of the problem?
v What were you trying to do?
v What should have happened?
v What actually did happen?
v Has the function worked before?
v Can you re-create the problem?
Unless otherwise requested, keep the internal NetView trace active at all times
with SIZE=4000 or more. If you have specified MODE=EXT, see “Description of
NetView Trace Records (MODE=EXT)” on page 103. If you have specified
MODE=GTF, see “Generalized Trace Facility (GTF) Output Files” on page 152.
Locate and save a copy of the network log that includes entries recorded before
and during the problem. The network log is a sequential record of operator station
activity, including commands entered and messages received. For automation
command execution problems, the CNM493I parameter on the DEFAULTS or
OVERRIDE command has been set to YES, so that indications of automation are
included in the network log.
Locate and save a copy of the system log that was generated from the time before
and during the error. The system log is the data set that stores job-related
information, operational data, descriptions of unusual occurrences, commands, and
messages.
h CNMSTYLE
Locate and save a copy of the file (and any member that it includes) used to start
the NetView program.
h The output from the status monitor preprocessor job (if applicable)
The output from the status monitor preprocessor job contains status monitor
preprocessor messages.
Save the application trace log for traces created by the graphic service facility.
20 Troubleshooting Guide
While RODM is running:
1. Use the MODIFY command to move all the RODM log records into the log file.
2. Use the RODM log formatter to format the log file and print it.
Locate and save a copy of the customization file (EKGCUST) used to start RODM.
If your problem is related to a failure in accessing data, dump the RODM address
space. Otherwise, most RODM diagnosis is accomplished without a dump of the
RODM address space.
Notes:
v The NetView HLL API service routines maintain an eight-entry, continuously
wrapping trace area. This 48-byte area is referred to as the first failure data
capture area (FFDCA). Its name is HLBFFDCA and it is located in the DSIPHLB
control block (for PL/I), and the DSICHLB control block (for C).
v You can print the contents of this trace area during job execution by including
the appropriate PL/I or C print statements in your service routines. If a failure
occurs, this area identifies the server support API module that was running at
the time of the failure.
v The remote interactive debugger (RID) can be used to trace all high-level
language (HLL) calls and their results. RID can be used on any SNA topology
manager task and command processor, except for the LOGOFF command
processors.
Locate and save an unformatted user address space dump if you are using a
user-written program that uses RODM API or a PPI task.
Locate and save a copy of the files used to load the GMFHS data model and
resource definition files into the RODM data cache. These files are not necessary if
you have not modified the GMFHS data model and you are not creating any
user-defined objects in the RODM data cache. These files are documented in the
IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s
Guide.
Locate and save a copy of the GTF or the GMFHS printed trace data sets that
include entries from the time before and during the problem. These procedures are
documented in Chapter 12, “Diagnostic Tools for NetView Management Console
and GMFHS,” on page 209.
Use the diagnosis procedures described in the VTAM library to gather information
about problems with VTAM CMIP services.
h The SNA topology data model and resource definition files (if applicable)
Locate and save a copy of the files used to load the SNA topology data model and
resource definition files into the RODM data cache. You do not have to provide
these files if you have not modified the SNA topology data model or the
definitions of the objects created by these files.
h The initialization files used to start SNA topology manager (if applicable)
Locate and save a copy of the initialization file FLBSYSD used to start the SNA
topology manager.
Locate and save a copy of the following customization files containing the tables
used to customize the mapping of OSI status to display status, the solved status
for resources created in RODM, and the resources in exception views:
FLBOSIDS
OSI-Display Status
FLBSRT
Status Resolution
FLBEXV
Exception View
Locate and save a copy of the GTF trace data set that includes SNA topology
manager trace entries from the time before and during the problem. You will
probably have to create the problem again to obtain this trace information. The
topology manager traces are usually not active because some of the trace categories
can significantly affect performance.
Use the TOPOSNA TRACE command to enable all trace categories; then, create the
problem again. When creating a problem again, ensure that all the information that
is provided is obtained from the same occurrence of the problem.
22 Troubleshooting Guide
The information you collect about a problem helps you create a keyword string.
You might find it easier to keep track of the information you gather if you record it
on one of the following worksheets:
v Chapter 4, “NetView Program Problem Worksheet,” on page 45
v Chapter 10, “Graphic Monitor Facility Host Subsystem Problem Worksheet,” on
page 169
v Chapter 9, “NetView Management Console Problem Worksheet,” on page 165
v Chapter 13, “RODM Problem Worksheet,” on page 221
v Chapter 16, “SNA Topology Manager Problem Worksheet,” on page 301
v Chapter 19, “MultiSystem Manager Worksheet,” on page 431
v Chapter 21, “AON Problem Worksheet,” on page 447
v Chapter 23, “Event/Automation Service Problem Worksheet,” on page 459
v Chapter 28, “Tivoli NetView for z/OS Enterprise Management Agent
Worksheet,” on page 513
All applicable information in the list under “Documenting Any Problem” on page
19.
If a dump occurs, save the dump data set (such as SYS1.DUMP) onto a tape or
cartridge. The terms dump data set and unformatted dump refer to the unchanged
data set of the dump. The unformatted dump is the data required by IBM Software
Support.
If there is a RODM problem and a dump does not occur, use MVS commands to
create a dump of the RODM address and a dump of the user application.
You can view or print the dump data set, without altering it, by using an
interactive dump viewing utility such as the Interactive Problem Control System
(IPCS).
Note: You can use a formatting utility on the unformatted dump to create a
formatted file for printing out the dump. The formatted files contain printer
control characters, making these files unusable by IBM Software Support;
therefore, keep a copy of the original source.
h The program status word (PSW) at the time of the abend failure
In the dump, find the instruction to which the address in the program status word
(PSW) points. To help you locate the PSW instruction in a dump, use a dump
formatting utility to find the work area labeled RTM2WA SUMMARY. The
summary shows the completion code, the registers at the time of the error, and the
PSW.
Note: Use the VERBX CNMIPCS SUMMARY command or the IPCS STATUS
command to view the dump data set.
In most cases, the instruction address in the PSW points to the instruction in error.
If the last word of the PSW has the high order (far left) bit on, the address is the
remaining 31 bits of that word. If the high order bit is off in the last word of the
PSW, the address is the last 3 bytes of the word.
For system ABEND 0C4 with a reason code 10 (segment translation exceptions) or
11 (page translation exceptions), the PSW points to the instruction that failed. After
finding the PSW address in the dump, record the name of the module containing
24 Troubleshooting Guide
the PSW address by scanning from the right margin of the dump printout
backward to the module name. You can then record the program temporary fix
(PTF) level and entry point of the module.
For any type of NetView problem, keep a copy of the unformatted dump that is
used by the IBM Software Support representative to create an authorized program
analysis report (APAR). Also, save the network log and the MVS system log to use
for reference.
If the module begins with a DSI, BNJ, AAU, DUI, DWO, EKG, or CNM prefix, it is
a NetView module. If the module does not begin with one of these prefixes, the
problem is probably not in NetView.
h The contents of the general registers at the time of the abend failure
Note: The procedure used to locate abend information is different for the SNA
topology manager component. This procedure is described in “Dump of an
FLBTOPO ABEND” on page 30.
All dumps taken by the NetView program contain a title. The title format is: xxxx
ABEND IN NETVIEW, ERRMOD=yyyy, RCYMOD=DSIMSX, DOMAIN=zzzz.
Where:
xxxx Specifies a system or user abend code.
ERRMOD=yyyy
Specifies the name of the load module that abended.
The NetView program uses the CSVQUERY macro to try to determine the
actual name of the module where the error occurred. Failing that, it uses
the program module name in the SDWA. The ERRMOD field is blank if
the SDWA is blank.
If you receive:
DUMP BY DSIMSX, NO SDWA PROVIDED
If REGF was found to contain zeros, the following example is the keyword string
used to perform a database search for this abend:
5697ENV00 ABEND0C1 DSIEND
Where:
5697ENV00 Specifies the NetView component ID number
ABEND Specifies the type of problem
26 Troubleshooting Guide
0C1 Specifies the abend code
DSIEND Specifies the module containing the ABEND
If the abend is from a failure, the keyword is ABENDxxx, where xxx is the abend
code in hexadecimal (such as ABEND0C4, ABEND604 or ABEND13E).
Typical user abends are documented by ABENDUxxxx, where xxxx is the abend
code in decimal. Refer to the NetView abend codes. A NetView user abend can be
caused by circumstances in the system. If you have no information on how to
recover from or prevent the abend, continue gathering documentation.
To determine the task where an abend occurred in NetView, locate the task name
and associated MVS TCB.
Notes:
v You can use the IPCS verb exit to determine the task name and associated MVS
TCB. The information is in the task vector block (TVB). Each task within the
NetView program is represented by a TVB. With the exception of the autotask
TVBs, all TVBs are built at initialization and are in contiguous storage.
v The first TVB in the chain is pointed to by the main vector table (MVT). The
MVT is the main control block in NetView. You can find the MVT in the dump
in one of the following ways:
– Use the contents directory entry (CDE) for DSIMNTEX. This CDE is under
the TCB for the main task. When looking at the formatted TCBs, the main
task TCB, DSIMNT, is the TCB with the formatted CDE for the subtask
module DSIMNT. You can scan the CDE entries for DSIMNTEX.
– Use the following commands to search the dump for the module name
DSIMTM:
- VERBX CNMIPCS IPCS 'FIND'
- VERBX CNMIPCS 'DISPMOD'
- VERBX CNMIPCS 'SUM'
The characters MVT follow the module name. The next word contains the
address of the MVT.
Notes:
v A NetView internal trace is always necessary for these out-of-storage conditions.
Run the NetView internal trace with its default options.
v Message DSI124I indicates that the NetView program is running out of storage.
You can automate this message to create a console dump of the NetView
program before it abends.
v Message BNH160I indicates storage loss or problems with the storage accounting
in NetView for global storage.
v Messages BNH161I—BNH163I give you warnings about storage shortages before
they occur. Message DSI124I indicates a more severe storage condition. If you
see DSI124I messages, you might be ignoring important early warning messages
or have disabled storage limits using the DEFAULTS or OVERRIDE commands.
The SMF record 38 subtype 2 data can help you review the storage history for
NetView. You can use the NetView TASKURPT sample to retrieve this data, or
use TASKURPT as a guide to writing other SMF reports.
If your system is running successfully and the NetView program begins to run out
of storage, you will probably receive multiple abends. Save the dump data set
from the first abend. In the system log, look for a message that indicates that a
partial dump was created. If only a partial dump was created, you might need to
re-create the problem with a larger dump data set to get a complete dump. A
complete dump is usually required by IBM Software Support to solve
short-of-storage problems.
If your storage calculations are correct, but you still have short-of-storage
problems, run the NetView internal trace and gather the following:
v Storage calculations from IBM Tivoli NetView for z/OS Tuning Guide for your
operating system.
v The network log after several TASKUTIL or TASKMON, RESOURCE, and
SESSMDIS command invocations.
v The first dump of the abend, or a console dump if you are getting messages
indicating an out-of-storage condition. Dump the entire NetView address space.
Out-of-Storage Dump:
28 Troubleshooting Guide
Look for queues in an out-of-storage dump. Use IPCS to obtain a list of these
queues:
VERBX CNMIPCS 'QUE(ALL)'
v Each task has a public and private message queue that can build up if you do
not process the message event control block (ECB). This might happen if you
have AUTOWRAP turned off. You can use the IPCS RUNCHAIN command to
find out how many elements exist. The message queues point to an internal
function request (IFR) that contains a normal BUFHDR and are chained together
at X'18'. To determine how many messages are queued, run the entire chain.
The NetView program monitors the message queue, counts the number of
messages, and places the count in the public queue. You can determine the
number of messages on the public queue by checking the TVB X'CC' for 4 bytes.
The TVB characters begin with X'F2'.
The operator station task (OST), NetView-to-NetView task (NNT), and the
primary POI task (PPT) have three public and three private queues, as follows:
– The normal public and private message queues are chained off the TVB, TVB
X'24', and X'28'.
– The high private and public message queues are chained off the TVB, TVB
X'DC', and X'D4'.
– The low private and public message queues are chained off the TVB, TVB
X'D8', and X'D0'.
The optional task (OPT) has one public and one private message queue. The
normal public and private message queues are chained off the TVB, TVB X'24',
and X'28'.
The data services task (DST) has one public message queue, one private message
queue, and two internal queues. The normal public and private message queues
are chained off the TVB, TVB X'24', and X'28'. For DSTs, you can check two
internal queues to determine if they are backed up and possibly using more
storage. You can find the internal queues in the TID control block. The TID is
pointed to by TIB X'70' and begins with F4.
In the TID, check queues X'AF0' and X'AF8'. These two queues point to a
NetView buffer that has a BUFHDR. The chain pointer is at X'18' in the
BUFHDR. These two queues represent requests to be processed by the DST. For
example, the BNJDSERV task represents hardware events to be recorded to
VSAM. The amount of storage used for these buffers is variable, but 400 bytes is
the average size.
If the hardware monitor TID queues are backed up, you might receive abend 778
in VTAM because of a shortage in CSA SP229. This happens because VTAM
expands the buffers into the CSA SP229 subpool.
v Coding the RATE statement allows you to detect an excessive rate of hardware
events and set a blocking filter when the excessive rate is detected.
v In the session monitor, check a VSAM record queue to determine whether it is
backed up. Find the TVB for AAUTSKLP and the TIB pointer at X'8'. The TIB
X'6C' points to the AAUTSCT control block. Scan the AAUTSCT, looking for the
name AAUTSTRR. The word that follows AAUTSTRR is the address of the
AAUTSTRR. The AAUTSTRR X'24' is a pointer to the AAUTSTAT control block.
The AAUTSTAT X'84' is a count of the number of records waiting to be written
to the session monitor database. The amount of storage used by each of these
requests varies, but an average size is 400 bytes.
The procedure used to obtain information about the location of an abend for the
SNA topology manager is different from NetView dumps described previously.
Using the sample dump data shown in Figure 4 as an illustration, use the
following procedure to collect information for an SNA topology manager abend:
1. Locate the address of the failed instruction in the PSW.
Use the existing NetView procedures to locate the value of the PSW and the
address of the instruction that failed.
The contents of the second word of the PSW (not shown in the sample data) is
X'8E29E870'. Ignoring the high-order bit gives the address of the failed
instruction: X'0E29E870' (1).
2. Find the name of the failing module.
After finding the PSW address in the dump, scan forward from that location
until you find the module name: FLBTRFE (2) in the right margin of the
dump printout. You can identify it as a module name because it has a
compilation date and possibly a PTF level (COMPILED ON...).
3. Record the compilation data of the module
After recording the name of the module, you can also record the compilation
data of the module: August 11, 2005 at 2:45 p.m. (3).
4. Obtain and record the level of the module
30 Troubleshooting Guide
Obtain the PTF level of the module: TIVNV54 (4). TIVNV54 indicates that
this is the base V5R4 version of this module to which no PTFs have been
applied.
Notes:
v Remember that character data can be split when the dump is printed or
browsed. This is especially important when visually searching for
character data, such as COMPILED ON when trying to locate module names.
Of course, this is not a concern when using the IPCS tool (FIND
command).
v If the module does not begin with an FLB prefix, the problem is
probably not with the SNA topology manager.
v The module name and other module information is at the end of the
module for the SNA topology manager.
All applicable information in the list under “Documenting Any Problem” on page
19.
h The order number and revision level of the manual or the number of the
online help panel
Identify the order number and revision level of the manual or the number of the
online help facility panel involved. The manual numbers display on the back cover
in the form xxxx-xxxx-y, where xxxx-xxxx is the order number and y is a 1- or
2-digit revision number. The panel number is displayed in the upper left corner of
the screen.
Locate the pages in the document or the panels that contain incorrect or
incomplete information, and prepare a description of the problem.
All applicable information in the list under “Documenting Any Problem” on page
19.
Record how the output differs from what was expected. Answer the following
questions:
v Is all or part of the output missing?
v Is the output duplicated?
v Is there more output than expected?
v Is the information inaccurate?
Examine the internal trace and isolate the problem to a specific operation or
module.
If the problem is a database recording failure, a recording filter might not be set
appropriately, or an installation parameter can be causing the problem.
The message processing facility can filter out some messages that are routed to the
NetView program. Check the message entries in the MPF table.
32 Troubleshooting Guide
The message revision table can affect message text, color, route codes, descriptor
codes, display, and logging attributes for messages to be written on MVS consoles.
This information is necessary for SNA topology manager VTAM CMIP services
problems.
Locate and save a copy of the VTAM and NCP definitions being used when the
problem occurred. Only provide the definitions used to establish the
communications path between NetView, VTAM, and the agent nodes. These
definitions are documented in the VTAM library.
Locate and save a copy of the configuration of the agent nodes that are involved in
the problem. This information can be obtained by saving a copy of the
configuration files being used by the communications manager. In addition, the
communications manager provides the DISPLAY command, which can be used to
capture the current configuration information:
DISPLAY > file.out
Locate and save a copy of the configuration and topology databases of the agent
nodes that are involved in the problem. The communications manager provides the
DISPLAY command, which can be used to capture the current configuration and
topology information:
DISPLAY > file.out
Capture NetView management console views that are related to the problem.
Explain the information in the view that relates to the problem.
Also include views that are incorrect, describing in detail objects that are missing
or are incorrect in the view.
Print and save a copy of the network log containing several TASKUTIL or
TASKMON command outputs for the time period preceding and during the loop.
If the NetView trace is not already running, use the NetView TRACE command to
turn it on while the system is still running. Code the TRACE command as follows:
'TRACE ON,MODE=INT,SIZE=4000'
After the trace has run at least one minute, request a console dump of the NetView
address space and the NetView internal trace address space (see “Locating the
Trace When MODE=INT Is Specified” on page 100 for additional information
about the trace records).
Print and save NetView SMF record 38 subtype 2 data. The TASKURPT sample can
be used to write this data to the NetView network log.
Write down any messages that are displayed on the terminal at the time of the
loop.
34 Troubleshooting Guide
h A console dump
Use the MVS DUMP command to dump the entire NetView address space and, if
you requested it, the NetView internal trace dataspace CNMTRACE. Look for
repetitive entries for NetView tasks in the trace tables to determine what NetView
modules are in the loop. NetView modules begin with the AAU, BNJ, CNM, DSI,
DUI, or DWO identifiers. If the module you locate does not begin with DSI, BNJ,
AAU, DUI, or CNM, the problem is probably not with the NetView program.
If a command list or user-written command processor was running at the time the
loop occurred, include a copy of the command list or command processor. Retain a
copy of all applicable command lists and command processors that were
processing at the time of the loop.
h Module name, compilation date, PTF level, and offset into module of the loop
(if applicable)
After collecting all of the required documentation, report the problem to IBM
Software Support.
All applicable information in the list under “Documenting Any Problem” on page
19.
h Description of the operation attempted, the results expected, and the results
received
Record the actual performance, the expected performance, and the source of
information for the expected performance. Obtain a network log showing messages
without a command or command response. If a document is the source of expected
performance information, note the order number and page number of the
document.
h The size and type of operating environment, and the number of devices being
monitored
Print these records to see a performance history for tasks in NetView. The
TASKURPT sample can display statistics for a single task or all tasks in NetView.
Archived SMF data might provide information about resource usage trends leading
up to a failure.
This information is necessary for SNA topology manager VTAM CMIP services
problems.
Locate and save a copy of the VTAM and NCP definitions being used when the
problem occurred. You only have to provide the definitions used to establish the
communications path between NetView, VTAM, and the agent nodes. These
definitions are documented in the VTAM library.
36 Troubleshooting Guide
This information is necessary for SNA topology manager problems.
Locate and save a copy of the configuration of the agent nodes that are involved in
the problem. This information can be obtained by saving a copy of the
configuration files being used by the communications manager. In addition, the
communications manager provides the DISPLAY command, which can be used to
capture the current configuration information:
DISPLAY > file.out
Locate and save a copy of the configuration and topology databases of the agent
nodes that are involved in the problem. The communications manager provides the
DISPLAY command, which can be used to capture the current configuration and
topology information:
DISPLAY > file.out
After collecting all of the required documentation, report the problem to IBM
Software Support.
All applicable information in the list under “Documenting Any Problem” on page
19.
To identify which task is in the wait state, examine the trace record, and research
the activity that took place before the wait.
For RODM and GMFHS, also obtain the RODM log and GMFHS trace information.
For RODM and GMFHS, use the MVS DUMP command to dump both the RODM
and GMFHS address spaces. The DUMP command is described in MVS library.
For NetView, you can dump the data using the MVS DUMP command with the
CSA, NUC , RGN, SQA, and TRT options. Use IPCS to search the dump as
follows:
1. Find the TVB that was having the problem.
After you locate the TVB, get the TCB address from the TVB and examine the
TCB/RB structure. Normally, the first request block is in a wait state from
DSIWAIT. This is normal because NetView tasks wait on a list of event control
blocks (ECBs) until one or more ECBs are posted. The posting of one or more
ECBs signals the NetView task that there is work to process.
To determine whether a task is in a normal wait state, use the save areas to
determine what called DSIWAIT. DSIWAIT is a service routine that is invoked
by the DSIWAT macro. For an operator task, DSIOST is the dispatching
module. This means if DSIOST called DSIWAIT, the task is in its normal wait
state waiting for work.
2. DSIWAIT is called with a four-word parameter list. The first word is the ECB
address or a pointer to the ECB list. The parameter list can be pointed to by
register 1 in the current save area or register 1 in the previous save area. The
previous save area is pointed to by the current save area plus 4. Determine
whether the task is waiting on only one ECB. The parameter list X'C' indicates
whether the wait is on a single ECB or a list of ECBs. If X'C' is X'80', the first
word of the parameter list is pointing to an ECB list. This can be causing the
problem, because the task waits on the entire ECB list rather than on one ECB.
3. The TIB control block contains a standard parameter list and save area. This
parameter list and save area are often used by DSIGMN and DSIFMN, and by
DSIWAIT and DSIPOST.
The TIB control block is pointed to by the TVB and is built when the task is
initialized. The TIB contains a parameter list and save area for mainline
processing and exit processing.
Note: This exit processing applies to an immediate request block (IRB) exit and
not NetView installation exit processing.
Examine the following parameter lists and save areas to determine the last
GET, FREE, WAIT, or POST:
TIB X'3BC'
Contains the mainline parameter list.
X'3CC'
Contains the mainline save area.
X'414' Contains the first exit parameter list.
X'424' Contains the first exit save area.
38 Troubleshooting Guide
The save areas follow standard save area conventions. Table 3 describes fields
of interest in the TIB.
Table 3. Fields of Interest in the TIB
Location Explanation
X'2C' Pointer to the normal CWB. The CWB contains a save area at X'4' that is for
the current command processor or the last one run (TIBNCCWB).
X'328' Pointer to a command list block (CLB) that contains the current command
list name (TIBCLBWK).
Determine the module in which the wait occurred by locating the address of the
last instruction run under the problem task. The last instruction run is WAIT SVC
(0A01). If this is not true, do further analysis to determine whether the program is
in a loop or the code is running as expected.
If the module issuing the wait is DSIWAIT, you can find the issuer of the wait
routine (command facility DSIWAT macro) by finding register 13 in the current
save area (the save area at the time of the SVC 1) and backing up one save area.
This save area is that of the issuer of the NetView DSIWAT macro. Record the
following:
1. The name and the compilation date of the module.
2. The hexadecimal offset into the module.
A wait condition has many external symptoms, including a locked keyboard and
no response to commands. If this happens, request a console dump while the
system is in the wait condition.
It is important that you request the console dump before issuing any commands or
trying to clear up the wait condition. The dump indicates what the task was doing
and why it is in a wait state.
h Module name, compilation date, PTF level, and offset into module (if
applicable)
Locate and save a copy of the VTAM and NCP definitions being used when the
problem occurred. You only have to provide the definitions used to establish the
communications path between NetView, VTAM, and the agent nodes. These
definitions are documented in the VTAM library.
After collecting all of the required documentation, report the problem to IBM
Software Support.
40 Troubleshooting Guide
Part 2. Diagnosing the NetView Program
Chapter 4. NetView Program Problem Worksheet. . . . . . . . . . . . . . . . . . . . . . 45
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
System Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Installation Exits and Command Lists . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
Problem Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Abend Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Message Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Loop Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Wait Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
Incorrect Output Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
Documentation Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
42 Troubleshooting Guide
Correlation Identifiers Between VTAM Messages and the Status Monitor . . . . . . . . . . . . 136
Security Authorization Facility Trace Record . . . . . . . . . . . . . . . . . . . . . . . 137
SAF Trace Record Descriptions . . . . . . . . . . . . . . . . . . . . . . . . . . 137
SNA Topology Manager NetView Trace Entries . . . . . . . . . . . . . . . . . . . . . . 144
Allocate Storage Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
Allocate Storage for an Array Request . . . . . . . . . . . . . . . . . . . . . . . . 145
Reallocate Storage Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Free Storage Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
First Failure Data Capture Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
Program-to-Program Interface (PPI) Trace Facility . . . . . . . . . . . . . . . . . . . . . . 148
Understanding the PPI Trace Anchor Block and the PPI Trace Table . . . . . . . . . . . . . . . 148
Understanding the Program-to-Program Interface Trace Record. . . . . . . . . . . . . . . . . 149
Locating the Program-to-Program Interface (PPI) Trace Table . . . . . . . . . . . . . . . . . 151
Locating the Oldest Program-to-Program Interface Trace Record . . . . . . . . . . . . . . . . 152
Generalized Trace Facility (GTF) Output Files . . . . . . . . . . . . . . . . . . . . . . 152
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
Record the following general information:
1. Date:
2. Problem Number:
3. Component ID:
4. Recommended service update (RSU) level:
5. Installation Option:
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
Problem Classification
Check one of the appropriate problem categories below that matches the symptoms
associated with your problem:
Abend Problems
For abends or processor exception problems, complete the following:
1. What is the abend code?
2. What processes were taking place at the time of the abend?
3. Online help facility (type HELP ABEND and use the scroll function to locate
the abend code).
4. Gather the following documentation before contacting IBM Software Support:
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A copy of the trace log. See “NetView Trace” on page 99.
v The first unformatted dump of the abend.
v A completed NetView problem worksheet.
5. Gather the following information from the dump:
a. What is the program status word (PSW) at the time of the abend?
b. In what module did the abend occur?
c. What was the module compiled?
d. What is the PTF level of the module pointed to by the abend?
e. What is the offset into the module pointed to by the PSW at the time of the
abend?
f. List the registers at the time of the abend.
Message Problems
For message problems, complete the following:
1. Record the message ID and any error codes displayed.
v Message ID:
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands:
v Other:
4. If the message was unexpected and cannot be corrected by following the
actions in the NetView online help, gather the following documentation before
calling IBM Software Support:
v A hard copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v The message ID:
46 Troubleshooting Guide
v The exact text of the message on the log.
v A completed NetView problem worksheet.
5. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
6. Did the message text differ from what was published?
v Has local modification been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Loop Problems
For loop problems, complete the following:
1. What events led up to the loop?
2. What data was being displayed?
3. What was the last command entered?
4. If this is an enabled loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v After obtaining a console dump, cancel the NetView program with a dump.
Note: If the loop is still occurring after the NetView program has been
canceled, look for a problem other than NetView.
5. If this is a disabled loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v A document describing the scenario leading to the problem.
v A hard copy of the system log.
v A hard copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A hard copy of the trace log. See “NetView Trace” on page 99.
v The addresses of instructions within the loop.
v A dump obtained by using the CPU RESTART function.
Note: If ABEND071 does not occur in the NetView program and normal
processing resumes, this is not a NetView problem.
6. What are the modules involved in the loop?
7. What are the dates that the modules were compiled?
8. What are the PTF levels of the modules involved in the loop?
Wait Problems
For wait problems, complete the following:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the system console log.
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
Performance Problems
For performance problems, complete the following:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A copy of the NetView trace. See “NetView Trace” on page 99.
v Information describing your NetView operating environment:
v Descriptions of any modifications to your system:
Documentation Problems
For documentation problems, complete the following:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program, call IBM
Software Support.
48 Troubleshooting Guide
5. If the problem is with an online help panel, call IBM Software Support.
If you cannot solve your problem by using the examples, follow the instructions in
Chapter 2, “Classifying Problems” and Chapter 3, “Documenting and Reporting
Problems” before contacting IBM Software Support.
The following table indicates where to find information about a particular problem
(category):
Table 4. Problem Categories and Scenarios
Problem Category Problem Scenario Page
Abend A NetView subtask ends abnormally. 52
Abend occurs while you are using a high-level language (HLL), and 56
you receive message CNM983E, CNM998E, or CNM999E.
Abend A78 is received at task or NetView termination. 53
Abend 301 is received. 55
Abend U0258, U0268, or U0269 is received. 54
Automation NetView automation is unexpectedly driven. 62
NetView automation is not driven when expected. 63
Commands Logon/bind problems with command facility. 52
RMTCMD RUNCMD response from a distributed mainframe server is 67
displayed on the MVS console of the destination mainframe server.
Logon Cannot log on to a command facility terminal. 52
Messages CNM983E, CNM998E, or CNM999E 56
DSI124I 56
DWO049W is received for a DSIFRE request. 57
DWO049W is received for a DSIGET request. 59
DWO158W 60
DWO627E 62
MS Transport MS Transport Cancels (Message DWO627E is received). 62
Recording EP/Local errors are not being recorded at the hardware monitor 64
database.
Distributed mainframe server errors are not being recorded. 65
Solicited data is not being recorded. 66
Unsolicited remote errors are not being recorded. 65
®
Security Problems with RACF or an SAF product. 67
Reference:
Where:
luname Is the name of the logical unit.
operatorid
Is the operator identifier.
52 Troubleshooting Guide
code Is the code used for problem classification. The abend code has 6
alphanumeric characters, yyyzzz, where:
yyy Is the system completion code.
zzz Is the application program (the NetView program and your
application) completion code.
The subtask identified by luname/operatorid has ended with the indicated abend
code. If the subtask is an operator station task (OST), this message is displayed on
the screen when the task is reinstated. For any other type of task, this message is
queued to the authorized receiver.
Where:
task Is the name of the task for which the NetView ESTAE/ESTAI exit
(DSIMSX) is driven.
v For the NetView main task, it is SYSOP.
v For an operator station task (OST), it is the ID of the operator that is
logged on.
v For a data services task (DST), it is the task name defined in
CNMSTASK.
v For any task name that cannot be determined, UNKNOWN is used.
hhhhhh Is the hexadecimal completion code. The first three digits comprise the
system completion code and the last three digits are the user completion
code.
If the first three digits are non-zero, this is a system completion code. If the
last three digits are non-zero, this is a user completion code.
If both sets of 3 digits are non-zero, it usually means that a subtask module
ended and passed trash in register 15.
An abend occurred. The NetView ESTAE/ESTAI exit gets control and performs a
supervisor call (SVC) dump. The routing code for this message is 2 (master console
information), 10 (system error/maintenance), or 11 (programmer information).
1. Save the dump data set.
2. See “Documenting ABEND Problems” on page 24.
To determine if this coding error occurred, use the diagnostic aids described for
return code 20 in the documentation for the DSIFRE macro.
Each time a command facility subtask issues a VTAM request parameter list-based
macro and an error occurs, the NetView program enters the command facility or
terminal access facility (TAF) SYNAD/LERAD exit routine. In a TAF environment,
if the request parameter list (RPL) is valid, register 10 contains the address of the
RPL. In a non-TAF environment, if the RPL is valid, register 3 contains the address.
VTAM passes major and minor return codes to SYNAD/LERAD in register 0. If
VTAM receives an RPL that is not valid (return code = X'18'), the subtask abends.
The termination code in a non-TAF environment is abend code U0258 (X'102'). The
termination code in a TAF environment is U0269 (X'10D').
VTAM passes a return code (register 0=X'102') indicating that the RPL is not valid
for one of the following reasons:
v The RPL is already in use.
v A check is issued before the RPL exit routine is scheduled.
v The RPL has been overlaid.
Message DSI625I
Message DSI625I is not issued when the RPL is not valid because the contents of
the RPL are not reliable or do not pertain to the request causing the abend.
At the time of the abend, register 0 does not contain the return code passed to
DSISYN or DSISYNX because register 0 is used as an abend work register.
Where:
taskid Specifies the ID of the task issuing the macro
sourcelu
Specifies the LU issuing the macro
macroname
Specifies the name of the macro
snocode
Specifies the should-not-occur code
54 Troubleshooting Guide
Table 5. Return Codes for User Abend U0258 (X'102') and U0269 ('10D')
Return Code Explanation
2 The request type (RPLREQ) field is outside the range expected by
DSISYN, or the macro is not issued by the command facility. Some macros
have request type codes within the numerical range of DSISYN tables, but
the command facility does not use them.
6 The communication identifier (CID) in the receive-any RPL does not match
any CID in RPLs pointed to by DSINAT. The command facility does not
recognize the session.
7 A CID that is not valid was passed to or received from VTAM (no abend
occurs in the TAF environment).
Any error messages and applicable return codes that are issued are listed in the
network log. In the NetView dump, register 13 points to the save area of the
module that issued the RPL CHECK macro before the abend occurred. In the
dump, use register 14 from the save area to find the module that called module
DSISYN or DSISYNX.
56 Troubleshooting Guide
For information about: Refer to:
The RESOURCE, TASKUTIL, TASKMON, NetView online help
and SESSMDIS commands
If the dump has successfully completed, use the following resolution steps to
debug the storage overlay problem.
1. Use the IPCS dump-formatting utility to format the dump.
2. From the IPCS Browse Option Pointer panel or IPCS Storage panel, issue the
IPCS STATUS subcommand to verify that the Dump Title indicates the error
occurred during a DSIFRE service request. For example, enter:
IPCS STATUS
If the Dump Title indicates the error occurred during a DSIGET service request,
see “Message DWO049W Is Received for a DSIGET Request” on page 59.
3. From the IPCS Browse Option Pointer panel or IPCS Storage panel, issue the
IPCS SUMMARY subcommand to display the general purpose registers at the
time the data was dumped. For example, enter the following command:
IPCS SUMMARY REGISTERS
The SUMMARY subcommand displays the summary output panel.
4. Issue the FIND command from the summary output command line to find the
registers at the time the data was dumped in the summary output. For
example, enter the following command:
FIND 0033
The FIND subcommand displays the problem request block (PRB) with WLIC
field 0033. See reference 1 in Figure 6 on page 58.
The general purpose registers at the time the data was dumped are pointed to
by the supervisor request block (SVRB), the request block before the problem
request block in the chain. See reference 2 in Figure 6 on page 58.
5. Use the general purpose registers pointed to by the SVRB to find the program
that caused the storage overlay. Record the contents of these registers by
writing them down or printing the IPCS panel.
The following list shows the general purpose registers that contain diagnostic
information; they are circled in Figure 6 on page 58 for easy reference:
Register 2 The return address of the program that issued the NetView
DSIFRE service macro to free the storage.
Register 3 The length of the storage specified on the DSIFRE macro.
Register 4 The address of the storage being freed by the NetView DSIFRE
service macro.
SVRB: 00ABCAF8
PRB: 00ABEA38
6. From the IPCS Browse Option Pointer panel, select the pointer to see the IPCS
Storage Panel.
Locate the program that issued the NetView service macro DSIGET and
DSIFRE.
v If the program that issued the NetView DSIGET/DSIFRE service macro is a
NetView module, contact IBM Software Support.
v If the program that issued the NetView DSIGET/DSIFRE service macro is not
a NetView module, try to determine why the storage freed by the DSIFRE
macro has overlaid the storage. You can do this by looking at the program
that issued the DSIGET and DSIFRE macro, the length of the storage, and the
storage address.
58 Troubleshooting Guide
Message DWO049W Is Received for a DSIGET Request
You receive message DWO049W (without message DWO115W) when a program
attempts to get storage using the NetView DSIGET service macro, and the NetView
program detects one of the following situations:
v The storage pooling structures in memory are damaged.
v NetView internal storage maps are inconsistent.
v A possible storage overlay was detected while attempting to get storage.
The caller of the DSIGET macro gets a zero return code in register 15 if the
NetView program can get the storage after detecting the error. The caller gets a
non-zero return code in register 15 if the NetView program did not obtain the
storage.
If the dump has been successfully completed, use the following resolution steps to
debug the problem.
1. Use the IPCS dump-formatting utility to format the dump.
2. From the IPCS Browse Option Pointer panel or IPCS Storage panel, issue the
IPCS STATUS subcommand to verify that the Dump Title indicates the error
occurred during a DSIGET service request. For example, enter the following
command:
IPCS STATUS
3. If the Dump Title indicates the error occurred during a DSIFRE service request,
see “Message DWO049W Is Received for a DSIFRE Request” on page 57.
4. If the Dump Title indicates that the error occurred during DSIGET, perform the
following steps:
a. Review the sequence of events prior to the failure.
b. Review the NetView log to determine the active commands and tasks.
c. Review the NetView trace data for DSIGET/DSIFRE activity that might
point out the failing program.
d. Review any recently changed user-written programs for storage overlay
problems. The problem detected during DSIGET generally indicates that
NetView storage management control blocks and maps have been overlaid.
For example, a program stores data far enough beyond the storage obtained
that NetView data in adjacent storage is overlaid.
5. From the IPCS Browse Option Pointer panel or IPCS Storage panel, issue the
IPCS SUMMARY REGISTERS subcommand to display the general purpose
registers at the time the data was dumped. The SUMMARY subcommand
displays the summary output panel. Capture the contents of these registers by
writing them down or by printing the IPCS panel.
The following list shows the general purpose registers containing diagnostic
information for a NetView DSIGET service macro failure:
Register 2 The return address (GPR 14) of the program that issued the
NetView DSIGET service macro to get the storage.
If the program did not issue a DSIGET macro, it might have
called a common service routine that issued a DSIGET on its
behalf.
If the dump has been successfully completed, use the following resolution steps to
diagnose the control block overwrite condition:
1. Use the IPCS dump-formatting utility to format the dump.
From the IPCS Browse Option Pointer panel or the IPCS Storage panel, issue
the IPCS SUMMARY subcommand to display the general purpose registers at
the time the data was dumped. For example, enter the following command:
IPCS SUMMARY REGISTERS
The SUMMARY subcommand displays the summary output panel.
2. Issue the FIND command from the summary output command line to find the
registers at the time the data was dumped in the summary output. For
example, enter the following command:
FIND 0033
The FIND subcommand displays the Problem Request Block (PRB) with WLIC
field 0033. See Figure 7 on page 61 for an example of the display.
The general purpose registers at the time the data was dumped are pointed to
by the supervisor request block (SVRB). The SVRB is the request block before
the problem request block in the chain. To display the SVRB, use the UP PF
key.
3. Use the following general purpose registers pointed to by the SVRB to find the
program that caused the control block overwrite condition.
v Register 2 contains the caller’s base register (GPR 12) of the program that
issued the NetView service macro DSILCS to free the command work block
(CWB) or service work block (SWB).
60 Troubleshooting Guide
v Register 6 points to the work block being freed by the NetView service
macro DSILCS.
v Record the contents of these registers.
4. From the IPCS Browse Option Pointer panel, select the pointer to see the IPCS
Storage panel.
5. Locate the program that issued the NetView service macro DSILCS to free the
work block.
v If the program that issued the NetView service macro DSILCS is a NetView
module, contact IBM Software Support.
v If the program that issued the NetView service macro DSILCS is not a
NetView module, determine why the work block freed by the DSILCS macro
has been overwritten. You can do this by looking at the program that freed
the work block.
a. Locate the work block that DSILCS was trying to free.
1) Look at the first word at the work block address. It is structured in
the following way:
Byte 0 = CBH ID of work block
(X'D1' for SWB, X'C9' for CWB)
Byte 1 = In-use block (X'FF')
Byte 2-3 = Work block length
(X'0258' = SWBEND-DSISWB for SWB and
X'0170' = CWBEND-DSICWB for CWB)
PRB: 00AB2140
WLIC... 00020033 FLCDE... 00AEE398 0PSW... 070C1000 8291A45C
LINK... 00AB4200
GPR0-3... 029FF088 00047968 000479C8 02987800
GPR4-7... FFFFFFFC 0291ACD0 00047968 00000000
GPR8-11.. 00007AA0 0290D204 00000000 0290D698
GPR12-15. 02905450 00029038 02905840 0290D62C
62 Troubleshooting Guide
For information about: Refer to:
Corrective automation statements IBM Tivoli NetView for z/OS Automation
Guide
Information on using message CNM493I IBM Tivoli NetView for z/OS Automation
with the DEFAULTS and OVERRIDE Guide
commands, and the AUTOTBL command
The DEFAULTS and OVERRIDE commands NetView online help
If EP/local errors are not being recorded at the hardware monitor database, take
the following actions:
1. Use the NetView command LIST STATUS=TASKS to ensure that BNJMNPDA
is an active task. Start the task if it is not active. Also, determine whether a task
abend was recorded at the time the error should have been recorded.
2. Ensure that recording filters have not been set to block these records from
being recorded.
3. Determine whether any errors are being recorded.
4. Check SYS1.LOGREC to determine if the error is recorded there. If the error is
not recorded, this is not a hardware monitor defect.
If you follow the preceding steps and do not identify your problem, document it in
the following way:
64 Troubleshooting Guide
1. Obtain the following documentation:
| v Listings of the CNMSTYLE member for NetView installation.
v Data from SYS1.LOGREC for the error in question.
v The network log and the NetView trace from the time of the failure.
2. Follow the instructions in Chapter 2, “Classifying Problems” and Chapter 3,
“Documenting and Reporting Problems” before contacting IBM Software
Support.
If remote device errors are not being recorded and you are using focal point
alerting, check each step at the focal point and the distributed mainframe server in
the following way:
1. Ensure that DSICRTR, BNJDSERV, and xxxxxLUC (where xxxxx is a 1–5
character domain name such as CNM01) are active tasks for the correct
NetView domain, using the NetView command LIST STATUS=TASKS. Start the
tasks at both the focal point and distributed servers if the tasks are not active.
Also determine whether a task abend was recorded at the time of the error.
2. Ensure that recording filters have not been set to block these records at the
focal point or the distributed mainframe server.
3. Determine whether any errors are being recorded.
4. Check SYS1.LOGREC to determine whether the error is recorded there. If the
error is not recorded, this is not a NetView defect.
5. Ensure that VTAM CSECT ISTMGC00 is link-edited as reusable in
NETVIEW.V5R4USER.VTAMLIB. Also, be sure there are no concatenated
libraries containing versions of ISTMGC00.
6. If you change the APPLID of the NetView hardware monitor, ensure that you
specify it in ISTMGC00 and code ACBNAME=BNJHWMON.
If you followed the preceding steps and do not identify your problem, use the
following steps:
1. Obtain the following information:
| v Listings of the CNMSTYLE member for NetView installation (focal point and
the distributed mainframe server).
v From SYS1.LOGREC, data for the error in question
v A dump of CSECT ISTMGC00
v A VTAM buffer trace of task DSICRTR, or user-defined APPLID
v The network log and the NetView trace at the time of the failure
2. Follow the instructions in Chapter 2, “Classifying Problems” and Chapter 3,
“Documenting and Reporting Problems” before contacting IBM Software
Support.
If you followed the preceding steps and do not identify your problem, document
the problem in the following way:
1. Obtain the following documentation:
| v Listings of the CNMSTYLE member for NetView installation from the focal
point and the distributed mainframe server
v A VTAM buffer trace of BNJHWMON and DSICRTR, or of the user-defined
APPLID, from the focal point and the distributed mainframe server
v A VTAM path information unit (PIU) trace of the unit for which the
solicitation was performed
v The complete text for any message issued because of the solicitation
v The network log and the NetView trace from the time of the failure
2. Follow the instructions in Chapter 2, “Classifying Problems” and Chapter 3,
“Documenting and Reporting Problems” before contacting IBM Software
Support.
66 Troubleshooting Guide
RMTCMD RUNCMD Command Response Is Displayed on MVS Console
RMTCMD RUNCMD commands are sent to a service point from a distributed
NetView program. The responses from the service point for the RUNCMD are sent
to the MVS console of the mainframe server where the service point resides,
instead of being returned to the distributed NetView program where the remote
RUNCMD was issued. For example, you issue the following command from
network A01NV:
RMTCMD LU=B01NV,RUNCMD SP=B0488LAA,APPL=APPLNAME,LOG OPER1
If the service point B0488LAA resides under LU B02NV instead of under B01NV,
the RUNCMD can get to B0488LAA but the response might not be returned to
A01NV. The RMTCMD logs on the operator issuing the command. If OPER1 is
logged on to A01NV and sends the RMTCMD to B01NV, OPER1 is logged on to
B01NV. If the RMTCMD finds B0488LAA on B02NV, OPER1 might not be logged
on to B02NV. Therefore, if the NetView program does not have an authorized
receiver, the response is returned to its MVS console.
This problem might occur when you issue log commands that reflect the responses
of the command on service points.
If responses are not being returned to the NetView program that issued the
command, verify that the service point specified in the RUNCMD is under the LU
specified in the RMTCMD.
If you use a system authorization facility (SAF) product such as Resource Access
Control Facility (RACF), and you experience performance problems, a possible
cause might be excessive security authorization calls. To enhance performance of
security within the NetView environment, refer to the IBM Tivoli NetView for
z/OS Tuning Guide.
If you cannot solve the problem, gather detailed information about your security
setup and processing by using an SAF TRACE record, as described in “NetView
Trace” on page 99 and in “Security Authorization Facility Trace Record” on page
137, and contact IBM Software Support.
DSIDIAGG
Where:
STORAGE
Required keyword that starts or stops a diagnostic storage accounting
mechanism for all tasks or the tasks whose names match the task name,
LU name, or opid pattern.
ON Turns the specified accounting on.
OFF Turns the specified accounting off.
* Enables or disables the accounting for all tasks.
task/LU or name/opid
Identifier of up to 8 characters which can include ″?″ or ″*″ wild card
characters.
Notes:
1. Using an asterisk (*) or other general task name patterns causes higher than
normal CPU utilization. Use these only when the severity of the problem
requires them.
2. Commands are cumulative, and more than one pattern can be used by using
the command over to add more patterns.
3. DSIDIAGG STORAGE with no other operands causes the active settings to be
displayed.
4. If you want to diagnose a BNH160I message condition, enter a DSIDIAGG
command using the LU name, task ID, or operator ID for the task named in
BNH160I. Then, start the task and rerun the scenario that causes BNH160I.
BNH160I contains additional data about the storage in question. The DSIFRE
service also issues diagnostic messages if discrepancies are found in the use of
DSIGET and DSIFRE for the specified task.
5. When BNH160I messages are being diagnosed, it is best to issue the
DSIDIAGG command to set ON each task indicated by a BNH160I message.
This command provides information about which program issued DSIGET
and which program issued DSIFRE.
6. Diagnostics are issued by DSIFRE using the message IDs TRACEFMN,
TRACEGMN, and TRACEDIA.
7. While a task is running, you can use the RID stop command to stop a task
during a storage discrepancy. The RID stop command is shown here:
RID TASK=opid,ID=DSIGMN
RID stop provides additional information and stops a task during various
DSIGET and DSIFRE discrepancies. If it does not stop, information is written
to the log and the operator. These diagnostics are useful for testing new
applications running on the NetView program.
8. DSIDIAGG STORAGE OFF * removes the diagnostic command, and eliminates the
diagnostic CPU utilization.
9. Take note of the additional data produced by the BNH160I messages
produced when a task ends, and any TRACEFMN data. IBM Software
Support might ask you to send a NetView log containing the data if the
modules seem to be Tivoli programs.
68 Troubleshooting Guide
10. The DSIDIAGG command uses message DSI633I to indicate that the command
was processed. Numeric return codes are used to indicate problems with the
operands. To display the return codes, use PIPE NETV MOE DSIDIAGG STORAGE
ON/OFF luname | CONS. The return code has the following values:
Return Code Meaning
100 Too few operands
104 Command name (token 1) too long
108 Fourth operand missing
112 Second operand length error
116 Second operand name error
120 Third operand length error
124 Third operand name error
128 Fourth operand length error
200 No working storage for tables left
BNH161I
This message is issued when a task exceeds criteria based on the DEFAULTS or
OVERRIDE command settings. You can review the cause of BNH161I messages
and take the following actions:
v Add automation to suppress the BNH161I messages that are a result of limits
you want to enforce.
v Add automation to take corrective action in the event of excessive CPU, storage,
or other excessive activity.
BNH162I
This message indicates that the NetView region below the 16 MB line is depleted.
This can be caused by the following situations:
v Starting the NetView program with a region that is too small. MVS starts using
storage below the 16 MB line after the above 16 MB area is depleted.
v Problems in programs using below-the-line storage. Use the TASKMON
command to review task storage use.
BNH163I
This message indicates that the NetView region above the 16 MB line is depleted.
This can mean that the region is too small for the workload, or a task is looping or
has other storage management problems. Use the TASKMON and TASKURPT
commands to review the storage usage. Consider increasing the NetView region
size the next time the NetView program is started. Use the OVERRIDE command
to set limits for the storage a task uses if a loop is suspected.
Both the Common Event Infrastructure support and the correlation engine require
that task DSICORSV is active and connected to the correlation engine code running
under UNIX System Services. By default, the NetView program does not activate
the task. It must be started either explicitly or by coding INIT=YES in CNMSTASK.
If DSICORSV is active, the CORRSERV STATUS command can be used to check
the status of the connection.
The client must be started using the startClient batch file or shell script. The status
of the client can be checked by examining the client’s log file in the directory
where it was started. A common problem with the client is an inability to subscribe
to the event server’s event topic. This is indicated by exception messages in the
client log, located in the directory where the client was started. The subscription
problem can be caused by the event server application being inactive under
WebSphere, or by WebSphere’s name server using a bootstrap port other than 2809
(this can occur when multiple WebSphere profiles are in use). If a non-default
bootstrap port is being used, the lauchClient command in the startClient batch or
script file might need to be modified to include the -CCBootstapPort parameter, as
well as modifying the PROVIDERPORT property in the client’s properties file.
70 Troubleshooting Guide
Problems with the client-correlation engine connection can also be caused by
incorrect configuration in the client or correlation engine properties files. The
LCLPORT property for the client must have the same value as the CLIENTPORT
property in the correlation engine’s properties file. The NVPORT property for the
client must have the same value as the CLIENTLISTPORT property in the
correlation engine’s properties file. The NVHOST client property must be the
network name or IP address of the TCP stack that the correlation engine is using
for TCP support. The PROVIDERHOST client property must be the name or
address of the WebSphere Application Server where the event server application is
running.
IPCS also provides a verb exit interface so that you can write a verb exit routine to
generate a unique diagnostic report that is not available in IPCS.
The NetView program provides an IPCS verb exit routine for analyzing NetView
dumps from an MVS system. Use the routine with NetView Version 2 Release 4 or
later. The routine assists you in analyzing a NetView dump before you contact IBM
Software Support and during the analysis of a problem while you are in contact
with IBM Software Support.
| The IPCS verb exit routine that is provided with the NetView program has both a
command-line interface and a panel interface. The panel interface is available if the
environment is set up under TSO to allow ISPF panels to be displayed. The panel
interface provides more powerful functions than the command-line interface, such
as the ability to select multiple tasks or the ability to specify an IPCS symbol
wherever a storage address is required. It also provides help text through the ISPF
help interface.
The ISPF panels used with the NetView IPCS code are installed in the data set
defined with the SCNMPLIB qualifier. The default for this is
NETVIEW.V5R4M0.SCNMPLIB; however, your data set can be different.
For information about how to enable the NetView IPCS code to run in a TSO IPCS
environment, see IBM Tivoli NetView for z/OS Installation: Configuring Additional
Components.
Operation
NetView provides a verb exit routine, CNMIPCS, that functions similarly to any
standard IPCS verb exit routine. The output from CNMIPCS is written to both the
terminal and the IPCS print file. All numeric values displayed in error messages
are in hexadecimal.
When you run the CNMIPCS verb exit routine, the routine reads the IPCS symbols
CNMASID and MVT.
If CNMASID is found, CNMIPCS uses this variable as the address space identifier
(ASID) for running the command entered.
If the MVT symbol is found and contains the same ASID as CNMASID, the symbol
MVT is used as the pointer to the NetView main vector table (MVT) control block.
If CNMASID is not found, the ASID portion of CNMASID will be set to the MVT
ASID.
If both symbols are not found, CNMIPCS searches for the NetView MVT control
block in the default ASID passed from IPCS. If this search is unsuccessful,
CNMIPCS continues searching for the control block in the remaining ASIDs.
If the NetView MVT control block is found, the IPCS symbol CNMASID and MVT
are set and CNMIPCS runs the requested command. Otherwise, CNMASID is set
to the default ASID, a message is issued indicating the MVT was not found, and (if
it is a non-NetView specific command) the command is run.
If CNMIPCS cannot find the NetView MVT control block and you are able to
locate it, you can set the MVT symbol manually using the IPCS LITERAL
command.
Note: You can modify the address space against which CNMIPCS runs (including
a non-NetView address space) by specifying the ASID verb.
By default or if you specify MENU, CNMIPCS runs the panel interface. The
actions on the main panel correspond to the verbs listed in the syntax. Some verbs
do not have a corresponding selection in the panel interface, and some functions
that are available in the panel interface are not available in the verb interface.
Notes:
1. If you start CNMIPCS either with the MENU option or without any options
and if an ISPF environment is not active or the main panel is not available to
TSO, CNMIPCS runs using the SUMMARY verb.
74 Troubleshooting Guide
2. If you specify a verb other than MENU, the output for the specified verb is
displayed and the menu is not displayed.
3. When you specify the MENU option, a single formatting action is performed,
after which CNMIPCS ends. To perform another formatting action, run
CNMIPCS again.
The syntax for the CNMIPCS routine is shown in Figure 8 on page 76.
TraceOp:
ALL
DISP
FRE
GET
LOST
MENT
MENTMXIT
MQS
MXIT
POS
PSS
SAF
STOR
SUM
TRTVB ( SelOp )
TCP
WAT
SelOp:
ACTIVE
ABEND
ALL
LU(lu_name)
OP(operator_id)
TCB(address)
TIB(address)
TVB(address)
76 Troubleshooting Guide
Summary of VERBX CNMIPCS Verbs
The following list shows the verbs that can be specified on the CNMIPCS
command. Unless indicated otherwise, a selection on the main menu in the panel
interface provides the same information as the verb.
MENU
Displays the main menu for the panel interface if CNMIPCS is run in an
ISPF environment and the CNMIPCS panels are available to TSO. From the
main menu, you can select an action to perform. The selections on the
main menu correspond to other verbs that are available. Any information
that can be retrieved using a verb other than MENU can be retrieved using
the panel interface. MENU is the default verb.
78 Troubleshooting Guide
Notes:
1. If a TVB is found for options ABEND, TVB, TIB, TCB, OP, or LU, the IPCS
symbols TVB, TIB, and TCB are set for the task found. For the ABEND option
TVB, TIB, and TCB are set to the last abending task found.
2. If you are entering a variable that contains single quotation marks, preserve the
single quotation marks by enclosing them in another set of quotation marks.
For example the address variable for a TVB can be entered as a blank, null, zero,
decimal value, or a hexadecimal value. When specified as a hexadecimal value,
the following quotation mark rule is observed:
...'DISPLAY(TVB)' Blank
...'DISPLAY(TVB())' Null
...'DISPLAY(TVB(0))' Zero
...'DISPLAY(TVB(100))' Decimal value
...'DISPLAY(TVB(X''64''))' Hexadecimal value
Summary Output
Figure 9 on page 81 shows an example of the IPCS output when you issue the
SUMMARY command (VERBX CNMIPCS 'SUMMARY') or select Summary on the
main menu. In the example, the IPCS symbol MVT is defined if the main vector
table (MVT) is found. IPCS symbols TVB, TIB, and TCB are defined for the last
abending NetView task found.
80 Troubleshooting Guide
| CNMIPCS SUMMARY
|
| ASID in hex = 00000021 Job name = NV54PROC
|
| MVT address in hex = 00008190 NetView version = NV54
|
|
| TVB 40250 IND1-4 00008100 TIB 1179A0 TCB 8D5BB0 OPT 5 DSIQTSK DSIQTSK
|
| ABENDING TASK DSIQTSK TCB 8D5BB0 RTM 7F70A090 TCB CC 940C4000
|
| Registers at time of abend from RTM2
|
| REG 0 06BA2610
| REG 1 00000000 a b
| REG 2 0000AB90 DSIMVT +0000
| REG 3 FFFFFFFF
| REG 4 06D10030
| REG 5 06BDE030
| REG 6 00114F74
| REG 7 00114F74
| REG 8 06BA2100
| REG 9 00114F74 c d e f
| REG A 06560D7E DSIQTSKI 2009.096 +1FFE TIVNV54
| REG B 0655FD7F DSIQTSKI 2009.096 +0FFF TIVNV54
| REG C 0655ED80 DSIQTSKI 2009.096 +0000 TIVNV54
| REG D 06BA2580
| REG E 8655FBC8 DSIQTSKI 2009.096 +0E48 TIVNV54
| REG F 00000001
|
| PSW at time of abend from RTM2 7F70A090
| PSW 078D2000 8655FCE4 00040011 00115070 DSIQTSKI 2009.096 +0F64 TIVNV54
|
| NetView IPCS version NV54 PTF level TIVNV54 Time-Date 08.32 05/18/09
If the MVT is found in the specified ASID, the SUMMARY command is run. If the
MVT is not found in the specified ASID, a warning message is issued and the
SUMMARY command is run.
CPOOL Output
Issuing the CPOOL command or selecting Task CPOOL information on the main
menu displays CPOOL storage allocation by task, subpool, and CPOOL size.
Figure 11 shows an example of the output when you run the CNMIPCS routine
with the CPOOL command using the default option of ACTIVE:
VERBX CNMIPCS 'CPOOL'
See “Options for Some CNMIPCS Verbs” on page 78 for other options that can be
specified. The following list shows the field descriptions:
Field Description
a TVB address - if zero, then this is for non-queued storage
b Subpool
c Size of individual cells
d Total number of cells in use
e Total number of cells allocated
f Maximum number of cells ever allocated
g Amount of above the line storage in use
h Amount of above the line storage allocated but not in use
i Amount of below the line storage in use
j Amount of below the line storage allocated but not in use
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j)
CELLS TOTAL HIGH ABOVE ABOVE STG BELOW BELOW STG
TVB SP SIZE USED CELLS WATER STG USED NOT USED STG USED NOT USED
000000 00 8 5 3B 5 50 1B0 0 0
000000 00 18 5 9 6 A0 60 0 0
000000 00 30 4 4 5 E8 0 0 0
000000 00 60 3C8B 3CDE 3C8B 16EDF0 F60 28 FC0
82 Troubleshooting Guide
Note: The D command displays storage only in 4-word multiples; therefore, it
truncates X''33'' to X''30''.
Instead, you can select Task summary on the main menu; when you do that, the
Task Selection panel is displayed so that you can select the active TVBs or any
subset of tasks.
The example shows the output produced when you run the CNMIPCS routine
with the DISPLAY command using the default option of ACTIVE. See “Options for
Some CNMIPCS Verbs” on page 78 for other options that can be specified. The
following list shows the field descriptions:
Field Description
a The TVB address.
b Flags from TVB (TVBIND1 — TVBIND4).
c The TIB address.
d The TCB address.
e The task type.
f The TVB task priority. For MVS dispatching priority, subtract this number
from 255.
g The LU name of the task.
h The operator ID or task name of the task.
Figure 13. Example for Output from the IPCS DISPLAY Command
DISPMOD Output
Figure 14 shows an example of the LMOD and CSECT information displayed when
you run CNMIPCS with the DISPMOD command:
VERBX CNMIPCS 'DISPMOD'
Instead, you can select Load module/CSECT (DISPMOD) on the main menu.
84 Troubleshooting Guide
Notes:
1. If no CSECT name is found in the load module, the CSECT column entry
contains the load module name; the ADDRESS column entry will be zero and
the DATE column entry will contain LMOD.
2. The DISPMOD command uses a best guess algorithm and therefore might
display erroneous information for some CSECTs or load modules.
Instead, you can select TCB and RB structure on the main menu. If you do that,
the Task Selection panel is displayed so that you can select the active TVBs or any
subset of tasks, as long as the NetView program is the target address space. If the
NetView program is not the target address space, you must specify the address of
a TCB on the Task Selection panel.
The following example shows the output when you run the CNMIPCS routine
using the LEVEL command:
NetView IPCS version NV54 PTF level TIVNV54 Time-Date 08.32 5/18/09
LRCE Output
Figure 16 on page 86 shows an example of the IPCS output when you run the
CNMIPCS routine with the following LRCE command using the operator_id option:
VERBX CNMIPCS 'LRCE(OP(KATIEF))'
See “Options for Some CNMIPCS Verbs” on page 78 for other options that can be
specified on the LRCE command. The following list shows the field descriptions:
Field Description
a The TVB address
b The flags from TVB (TVBIND1 - TVBIND4)
c The TIB address
d The TCB address
e The task type
f The task priority
g The LU name of the task
h The operator ID or task name of the task
i The LRCE address
j The name associated with LRCE
k The command list block address
l The address of the first of a chain of blocks containing the command
procedure in storage
m The name of the procedure represented by this CLIST block (CLB)
n The type (CLIST or REXX)
o The load mode (LOCAL if loaded for this execution, or GLOBAL if loaded
with LOADCL)
MAP Output
The MAP command and the Storage map and Storage map summary selections on
the main menu display the storage usage. To use the MAP command, enter the
following command:
VERBX CNMIPCS 'MAP'
Figure 17 on page 87 shows an example of the output when you run the CNMIPCS
routine using the MAP command:
86 Troubleshooting Guide
| MVS LEVEL SP7.0.9 HBB7740
|
| ASID 50
|
| REGION SIZE REQUESTED
|
| < 16M 7FB000
| > 16M 4800000
|
| REGION SIZE ALLOCATED
|
| < 16M 7D000
| > 16M 1507000
|
| ************************************
| SUBPOOLS 245 & 255 ABOVE 16 MEG
|
| DFE 7F643460 AREA 794000 SIZE 8
| DFE 7F6C4610 AREA 794400 SIZE 8
| DFE 7F6643D0 AREA 7945D0 SIZE 8
|
Figure 17. Example of Output from the MAP Command
Figure 18 is an example of the output when you run the CNMIPCS routine using
the MAP(sum) command:
VERBX CNMIPCS 'MAP'(sum)
NLDM Output
Figure 19 on page 89 shows an example of the IPCS output when you run the
CNMIPCS routine with the NLDM command:
VERBX CNMIPCS 'NLDM'
Figure 19 on page 89 is an example of the output when you run the CNMIPCS
routine using the NLDM command.
88 Troubleshooting Guide
**** EVENT COUNTERS ****
No of PIU buffers proc-ed 1F5D9 NO of PIUs processed 2082C1
No of SAW buffers proc-ed 482C No of SESS STARTS proc-ed 2E95
No of SESS ENDS processed 23CB No of SESS rec-ed to VSAM 149E
** SESSION COUNTERS **
No of ASB control blocks AD0 ASB cnt blk highwater mark AE5
No of SESS being filtered 0 SESS filter highwater mark 0
No of SESS with host endpt AC9 No SESS keeping RTM data 1F
No of SESS keep-g XNET dat 0 No of SESS keep-g DOM dat 95C
No of SSCP-SSCP sessions 4 SSCP-SSCP highwater mark 4
No of SSCP-PU sessions 5 SSCP-PU highwater mark 5
No of SSCP-LU sessions F3 SSCP-LU highwater mark F5
No of LU-LU sessions 9CD LU-LU highwater mark 9E0
No of SESS wait-g for VSAM 6 Record queue h-water mark 17
No of SESS KEEP-G acnt dat AD0
** RESOURCE COUNTERS **
No of ARB control blocks 88C ARB cnt blk highwater mark 8A4
No of SSCP ARBS 5 SSCP ARB highwater mark 5
No of PU ARBS 5 PU ARB highwater mark 5
No of LU ARBS 869 LU ARB highwater mark 881
No of LINK ARBS 19 LINK ARB highwater mark 19
** STORAGE COUNTERS **
No of bytes for RTM data 9B0 No of bytes for SESS parms 3C8CA
No of bytes for TRACE data 6758C2 No of bytes for ACCT data 22000
No of bytes ASB cnt blk B6000 No of bytes ARB cnt blk 3D000
NPDA Output
Figure 20 on page 90 shows an example of the hardware monitor output from IPCS
when you run the CNMIPCS routine with the NPDA command:
VERBX CNMIPCS 'NPDA'
Hardware Monitor
Instead, you can select Task message queue information on the main menu. If you
do that, the Task Selection panel is displayed so that you can select the active TVBs
or any subset of tasks.
See “Options for Some CNMIPCS Verbs” on page 78 for other options that can be
specified. The following list shows the field descriptions:
Field Description
a The TVB address.
b The operator ID (task name) of the task.
c TVBMPUBQ - The number of messages on the TVB public message queue.
d TVBMPUBH - The number of messages on the TVB high priority public
queue. The data services tasks (DSTs) high priority message queue is the
TIDOSTQ. TVBMPUBH is not used by DSTs.
e TVBMPUBL - The number of messages on the TVB low priority public
queue. The data services tasks (DSTs) low priority message queue is the
TIDPPTQ. TVBMPUBL is not used by DSTs.
f TVBMPRIQ - The number of messages on the TVB private message queue.
90 Troubleshooting Guide
g TVBMPRQH - The number of messages on the TVB high priority private
queue.
h TVBMPRQL - The number of messages on the TVB low priority private
queue.
Note: Some queues apply only to certain tasks. These special queues are displayed
under the task to which they apply if they have any items on the queues.
S/A 06BA2580
R14= DSIQTSKI 2009.096 +0E48 TIVNV54 R15=
00000000 BAC 06BA2060 FOR 06A54938 R14 8655FBC8
R15 00000000 R0 06BA2610 R1 06A548F8 R2 00000000
R3 00000000 R4 06A54A4C R5 00114F08 R6 00040250
R7 00006DC8 R8 06BA2100 R9 000060C8 R10 06560D7E
R11 0655FD7F R12 0655ED80
S/A 06BA2060
R14= DSIQTSK 2009.096 +0258 TIVNV54 R15=DSIQTSKI 2009.096 +0000 TIVNV54
00000000 BAC 000372B0 FOR 06BA2580 R14 8655AB40
R15 0655ED80 R0 06BA2466 R1 06BA20F0 R2 00000001
R3 00040250 R4 00000000 R5 000060C8 R6 00040250
R7 00006DC8 R8 00000000 R9 06BA2100 R10 06BA2108
R11 0655B8E7 R12 8655A8E8
Instead, you can select Task storage counters on the main menu. If you do that,
the Task Selection panel is displayed so that you can select the active TVBs or any
subset of tasks.
See “Options for Some CNMIPCS Verbs” on page 78 for other options that can be
specified. The following list shows the field descriptions:
92 Troubleshooting Guide
a b c d e f
OPID TVBCUPOL TVBCUSTR TVBGUSTR TVBQCNT
TVB 5968 SYSOP 6FFA 0 10A20D 0
TVB 4B918 C01NVPPT 10FF3 E828 B9C0 1
TVB 21E00 DSIDCBMT 0 0 C00 0
TVB 21C00 DSIHLLMT 0 0 C00 0
TVB 21A00 DSISTMMT 0 0 C00 0
TVB 21800 SYSOP 0 0 C00 0
TVB 21600 C01NV 0 0 276C 0
TVB 21400 DSILOGMT 0 9C CA8 0
TVB 4BB10 DSILOG 4FFB 1F28 1800 0
TVB 4BD08 DSICRTR 3FFC 284A 18C8 0
TVB 4BF00 DSITRACE 2FFD 1F28 1800 0
TVB 4C0F8 CNMCSSIR 2FFE 0 23E0 0
TVB 4C2F0 CNMCALRT 0 0 EA6 0
TVB 4C4E8 DSISVRT 7FF9 12DE6 1800 0
TVB 4C6E0 DSIGDS 8FF8 284A 1800 0
TVB 4C8D8 DSIAMLUT 3FFC 1D3C 1BB0 0
TVB 4CAD0 BNJDSERV DFF5 FA92 3210 0
TVB 4CCC8 BNJMNPDA 0 6000 C00 0
TVB 4CEC0 C01NVLUC 4FFB 2ED4 9281 0
TVB 4D0B8 C01NVVMT 6FF9 3355 17419 0
TVB 4D2B0 C01NVBRW 1FFE 0 C00 0
TVB 4D4A8 DSIUDST 2FFD 2E12 1800 0
TVB 4D6A0 CNMTAMEL 31FE3 54A04 1E566 B
TVB 4D898 DSI6DST 6FF9 4E12 5FE0 0
TVB 4DA90 DSIHPDST 3FFC 4E12 1800 0
TVB 4E270 DSIROVS 1FFE 247A 1800 0
TVB 4E468 DSIELTSK FFF 2C2A 1800 0
TVB 4E660 AAUTSKLP 14FF3 13523A 8214 0
TVB 4E858 AAUTCNMI 7FF9 1F3A6 1800 0
TVB 4EA50 BNJDSE36 4FFB 22A2 1800 0
TVB 4EC48 DSIKREM 1FFE 3956 1800 0
TVB 4F038 DSIQTSK 4FFC 1000 2A00 0
TVB 4F230 DUIFSSCO 2FFD 0 3146 0
TVB 74010 KATHI2 19FF1 10038 CA51 0
TVB 21200 AUTO1 13FF3 E848 4934 0
TVB 21000 AUTO2 7FF9 0 D24 0
TVB 22080 DUIFEAUT 6FFA 0 9282 0
TVB 25080 DUIFCSGW BFF5 0 F53 0
TVB 5B080 DBAUTO1 FFF4 E828 4AD8 0
TVB 5C010 DBAUTO2 FFF4 E828 47F4 0
TVB 5D010 SSMMON 13FF3 28B5C 299A 0
TVB 60010 AUTONET 17FF2 FFB8 2504 0
TVB 646D0 KATIE EFF5 E828 D24 0
TVB 644D0 ALEX 7FF9 0 D24 0
TVB 642D0 ROB 7FF9 0 D24 0
TVB 640D0 MARYANNE 7FF9 0 D24 0
TVB 67010 SADIE 7FF9 0 D24 0
TVB 736D0 THOMAS AFF6 E828 D24 0
TVB 72080 TARA AFF6 0 D5A 0
TOTALS 1A0EC1 29AA6F 1BA243 C
Instead, you can select Auto table usage on the main menu.
MESSAGE TABLE
SEQ# HIT COUNT COMPARE COUNT
PN10969 00000000 00000076
PN10969 00000000 00000076
04410IMS 00000000 00000076
PN10969 00000000 00000076
Trace Output
The following symbols are set if the NetView internal trace is found:
TRACHEAD Contains the address of the trace table header.
TRACETOP Contains the address of the first trace table entry.
TRACENXT Contains the address of the next available entry in the trace table.
TRACEBOT Contains the address of the last trace table entry.
Figure 25 shows an example of the output you receive from the CNMIPCS routine
when you run IPCS with a select option as in the following TRACE command,
which can narrow your selection to a specific TVB:
VERBX CNMIPCS 'TRACE trtvb(x"7F080")'
Instead, you can select NetView Internal Trace on the main menu. If you do that,
the Trace Option Selection panel is displayed where you must choose one or more
trace options, and then the Task Selection panel is displayed so that you can select
the active TVBs or any subset of tasks.
Figure 25. Example of Output from IPCS TRACE Command Using a Select Option
Figure 26 on page 95 shows the output you receive from the CNMIPCS routine
when you run IPCS with this TRACE command using a trace option:
VERBX CNMIPCS 'TRACE (SUM)'
94 Troubleshooting Guide
Trace table is in data space : CNMTRACE
Address of trace table header : 1000
Address of TOP of trace table : 1040
Address of NEXT available entry : 6EC160
Address of BOTTOM of trace table: FA0FE0
Figure 26. Example of Output from IPCS TRACE Command Using a Trace Option
Instead, you can select the Find module/control block name on the main menu.
Note: If no CSECT name is found in the load module, the CSECT column entry
contains the load module name, the ADDRESS column entry contains
zeroes, and the DATE column entry contains the abbreviation LMOD.
Network Log
With the exception of some full-screen activities, the network log is a record of all
operator station activity, including commands entered and messages received. The
network log can also record the output of the TASKUTIL or TASKMON command.
Use the network log to correlate operator console activities with other events in the
network. Figure 30 on page 97 is an example of the printed network log.
96 Troubleshooting Guide
N E T V I E W PRINT LOG/TRACE UTILITY 08/18/09 1
******** 08/18/09 NTV90 N 13:11:54 * N E T V I E W DISK LOG
NETOP1 13:11:55 - DSI556I DSILOG : VSAM DATASET 'OPEN' COMPLETED, DDNAME = 'DSILOGS'
RETURN CODE = X'00', ACB ERROR FIELD = X'00'
13:11:55 - DSI547I DSILOG : SECONDARY VSAM DATA SET IS NOW ACTIVE
13:11:55 - DSI556I DSILOG : VSAM DATASET 'CLOSE' COMPLETED, DDNAME = 'DSILOGP'
RETURN CODE = X'00', ACB ERROR FIELD = X'00'
13:11:55 - DWO520I DSILOG : VSAM DATASET 'CLOSE' COMPLETED, DDNAME = 'DSILOGP'
RETURN CODE = X'00', ACB ERROR FIELD = X'00'
LU32706 13:12:20 DSI022A INVALID PASSWORD, REENTER
NETOP1 NTV90 % 13:12:20 - DSI029I INVALID LOGON ATTEMPT FROM TERMINAL LU32706, ERROR IN
THE 'PASSWORD' FIELD
LU32706 NTV90 13:12:25 DSI022A INVALID PASSWORD, REENTER
NETOP1 NTV90 % 13:12:25 - DSI029I INVALID LOGON ATTEMPT FROM TERMINAL LU32706, ERROR IN
THE 'PASSWORD' FIELD
OPER2 NTV90 13:12:34 - DSI020I OPERATOR OPER2 LOGGED ON FROM TERMINAL LU32706
USING PROFILE (DSIPROFA), HCL ( )
13:12:45 * AUTOWRAP
13:12:45 DSI082I AUTOWRAP STARTED
13:13:12 * TASKUTIL
13:13:14 ' DWO022I
13:13:14 ' TASKNAME TYPE DPR CPU-TIME N-CPU% S-CPU% MESSAGEQ STORAGE-K CMD
13:13:14 ' -------- ---- --- ----------- ------ ------ -------- --------- --------
13:13:14 ' OPER2 OST 251 0.04 66.83 0.13 0 73 **NONE**
13:13:14 ' DSITIMMT OPT 255 0.01 9.77 0.02 N/A 4 N/A
13:13:14 ' DSILOG DST 254 0.10 6.86 0.01 0 26 N/A
13:13:14 ' CNMCSSIR OPT 250 0.01 3.99 0.01 0 11 N/A
13:13:14 ' DSIDCBMT OPT 255 0.18 0.00 0.00 N/A 3 N/A
13:13:14 ' DSIHLLMT OPT 255 0.01 0.00 0.00 N/A 7 N/A
13:13:14 ' DSISTMMT OPT 255 0.00 0.00 0.00 N/A 7 N/A
13:13:14 ' SYSOP OPT 255 0.00 0.00 0.00 N/A 7 N/A
13:13:14 ' NTV90 OPT 255 0.01 0.00 0.00 N/A 9 N/A
13:13:14 ' DSILOGMT OPT 255 0.02 0.00 0.00 N/A 8 N/A
13:13:14 ' NTV90PPT PPT 255 0.05 0.00 0.00 0 122 **NONE**
13:13:14 ' DSICRTR DST 249 0.01 0.00 0.00 0 33 N/A
13:13:14 ' DSIMONIT OPT 255 0.02 0.00 0.00 N/A 4 N/A
13:13:14 ' CNMCALRT OPT 249 0.00 0.00 0.00 N/A 3 N/A
13:13:14 ' BNJDSERV DST 249 0.04 0.00 0.00 0 84 N/A
13:13:14 ' NTV90BRW OPT 250 0.00 0.00 0.00 0 11 N/A
13:13:14 ' NETOP1 OST 251 0.60 0.00 0.00 0 166 **NONE**
13:13:14 ' MNT MNT 255 0.00 0.00 0.00 0 4752 N/A
13:13:14 ' AUTO1 AUTO 250 0.02 0.00 0.00 0 39 **NONE**
13:13:14 ' AUTO2 AUTO 250 0.04 0.00 0.00 0 102 **NONE**
13:13:14 ' NETVIEW OTHR N/A N/A 0.00 0.00 N/A N/A N/A
13:13:14 ' NETVIEW SRB N/A 0.45 12.56 0.02 N/A N/A N/A
13:13:14 ' NETVIEW TOTL 33 2.75 100.00 0.20 0 5471 N/A
13:13:14 ' SYSTEM TOTL N/A N/A N/A 8.28 N/A N/A N/A
13:13:14 ' END DISPLAY
13:13:17 * NPDA
13:13:19 " N E T V I E W SESSION DOMAIN: NTV90 OPER2 08/18/09 13:13:17
13:13:19 " NPDA-01A * MENU * HOST DOMAIN: NTV90
13:13:19 " SEL# PRODUCES:
13:13:19 " ( 1) ALERTS-DYNAMIC DISPLAY
13:13:19 " ( 2) TOTAL EVENTS DISPLAY
13:13:19 " ( 3) TOTAL STATISTICAL DATA DISPLAY
0 4 8 C E 10 12 13 14 1C 20 28 30
Bytes In Hex
b No error ! Immediate message The last byte(s) may The first 2 bytes
(cross-domain) indicate contain message
N Header or indicator. Total
trailer - Message generated % Message was sent to message length
record (N) by command facility authorized receiver can be up to
255 bytes.
• Command input from P Message originated
a terminal at the PPT
Q Unsolicited message
98 Troubleshooting Guide
Turn on the MSGMODID option using the DEFAULTS command with
MSGMODID=YES. The message DSI799I is written to the network log using
DSIWLS. This message provides the following:
v The original error message number
v The name of the module that issued the original message
v The occurrence within the module (necessary when the module issues a message
more than once)
These records can be viewed using TASKURPT, or you can use a standard SMF
reporting tool to format the records. These records give you many NetView task
resource usage statistics, such as CPU, I/O, storage, message queueing rates, and
amount of penalty time assessed. Use these statistics when a loop condition,
storage outage, or other performance problems are evident.
NetView Trace
NetView trace captures the sequence of internal processing. The trace provides
information you can use in resolving NetView problems and user errors. The trace
also provides records of key problem determination data such as parameter values,
addresses, return codes, and flag settings. Trace output can be recorded internally
in virtual storage, externally in the DSITRACE data sets, or to the generalized trace
facility (GTF).
Keep the NetView internal trace active at all times. This can slightly degrade
system performance, but having the trace on at all times is important in diagnosis.
Use the default options as shown in the following example:
OPTIONS=(DISP,PSS,QUE,STOR,UEXIT)
You can dynamically specify the events to be traced using the TRACE command.
Use the trace with available service aids, such as the network log and a dump, to
assist in resolving a problem.
Use NetView trace to identify the source of command facility problems or user
errors, and to provide information useful for resolving these problems.
The MOD option usually results in a large number of trace entries. If you run the
MOD trace option, use it only for a short time to trap specific data.
v When you run the trace internally (MODE=INT), entries wrap quickly if you
specify a small storage size. INT is the default.
v When you run the trace externally (MODE=EXT), it can use additional storage.
Also, the DSITRACE data sets (primary and secondary) must be large enough to
provide adequate storage.
v When you run the trace to the generalized trace facility (MODE=GTF), trace
record formats might be different.
You can restrict use of the TRACE command by limiting which operators can use
it.
You can use IPCS to view the internal trace table online.
The internal trace table is a wraparound table. The SIZE operand of the TRACE
command specifies the number of pages in storage to be allocated for the table.
The default setting is SIZE=4000, although you can increase this value to the
maximum value of 524286, the limit for a dataspace.
Dump system data to locate the in-storage trace table. To locate the trace table in
the dump, find the command facility main vector table (MVT) control block. To
locate the MVT in a dump, use the following DISPMOD command to locate the
entry point of load module DSIMNTEX:
DISPMOD DSIMNTEX
The field MVTITDSI (at offset X'AA8' in the MVT) contains the address of a control
block that contains Internal Trace Dataspace Information (ITDSI). The ITDSI
contains the name, token, and ALET of the dataspace, as well as the size and
starting address of the trace table in the dataspace. If this address is zero (0), the
You can use the command facility utility program DSIPRT to print the trace data
from the trace log. You can also use CNMPRT, which contains the job control
language (JCL), to print the trace log.
MXITTA2.*...Q............DSIFMN
DSILOG 08/18/09 NCAB 12:48:55 L MENT D4C5D5E3 81F216C8 00069BD8 0009ED14 0009ED24 81F2B3FE C4E2C9C7 D4D54040
MENTA2.H...Q........A2..DSIGMN
L 12:48:55 L GET. C7C5E300 81F2B3FE 00069BD8 00000000 02297410 00000070 00000000 4100D440
GET.A2.........Q..............M
12:48:55 L MXIT D4E7C9E3 81F21CA4 00069BD8 00000000 0009ED24 81F2B3FE C4E2C9C7 D4D54040
MXITA2.U...Q........A2..DSIGMN
M 12:48:55 L MQS D4D8E240 81F04EEC 00069BD8 02297410 C4E2C9D3 D6C74040 E5C1D340 40404040
MQS A0+....Q....DSILOG VAL
004C0070 00C90024 1248550C D5C3C1C2 40404040 00000000 00000000 E5C1D340
.<...I......NCAP ........VAL
40404040 00100086 086F1248 550C0000 00000000 00000032 404ED5C3 C1C2F0F0
...F.?.............. +NCAB00
N 12:48:55 L MENT D4C5D5E3 81F32518 00069BD8 0009ED14 0009ED24 81F28FD8 C4E2C9D7 D6E2E340
MENTA3.....Q........A2.QDSIPOST
DSILOG 12:48:55 L DISP C4C9E2D7 82D80618 00067270 80067290 40000000 00000000 C4D2C9D3 D6C74040
DISPB........... .......DSILOG
12:48:55 L MXIT D4E7C9E3 81F49906 00067270 806D96D8 0001075C 82080618 C4E2C9E6 C1C9E340
MXITA4R0....._OQ..P*B...DSIWAIT
12:48:55 L MENT D4C5D5E3 81F21058 00067270 0001D74C 0001075C 80072EDE C4E2C9C6 D4054040
MENTA2........P<..P*....DSIFMN
O 12:48:55 L FRE. C6D9C500 80072EDE 00067270 00000000 0005F8B0 000000F4 8007299C 8000D4D8
FRE...............8....4......MQ
12:48:55 L MXIT D4E7C9E3 81F2165C 00067270 00000000 0001D75C 80072EDE C4E2C9C6 D4054040
MXITA2.*..........P*....DSIFMN
12:48:55 L MXIT D4E7C9E3 82084854 00067270 00000000 0001D40C 82080AB8 C4E2C9E6 D3D4D7E3
MXITB.............M.B...DSIWLMPT
12:48:55 L MENT D4C5D5E3 81F497B0 00067270 0001D74C 0001D75C 82080618 C4E2C9E6 C1C9E340
MENTA4P.......P<..P*B...DSIWAIT
P 12:48:55 L WAT E6C1E340 82080618 00067270 00016A70 00000000 00000000 C4E2C9D3 D6C74040
WAT B...................DSILOG
12:48:55 L DISP C4C9E2D7 82080618 00067270 000169D0 40000000 00000000 C4E2C9D3 D6C74040
DISP............ .......DSILOG
This section describes some of the entries that can be displayed at the operator task
when the MONOPER keyword has been specified.
For comparison, here are module entry and exit trace entries as they are displayed
when formatted in a dump by the NetView CNMIPCS IPCS verb exit routine.
0E5E4040 MENT DSIFMN 09.093 +0000 TIVNV54 RET=8D63B8C8 TVB=001410D0
R1/R15 0014355C R13 0014356C R14 8D4B860C DSIFMSGM 09.093
0E5E4080 MXIT DSIFMN 09.093 +03D6 TIVNV54 RET=8D63BC9E TVB=001410D0
R1/R15 00000000 R13 0014356C R14 8D4B860C DSIFMSGM 09.093
The following table is a cross reference to the various trace record descriptions:
Description: See:
DISPATCH trace record “DSIWAT/DSIPOS/DISPATCH Trace
Record” on page 113
DSIFRE trace record “DSIGET/DSIFRE Trace Record” on page
113
DSIGET trace record “DSIGET/DSIFRE Trace Record” on page
113
DSIMQS trace record “Message Queuing Service (MQS) Trace
Record” on page 110
DSIPOS trace record “DSIWAT/DSIPOS/DISPATCH Trace
Record” on page 113
DSIPSS trace record “DSIPSS Trace Record” on page 114
DSIWAT trace record “DSIWAT/DSIPOS/DISPATCH Trace
Record” on page 113
Installation exit trace record “Installation Exit (UX) Trace Record” on
page 111
Internal trace dataspace information “Internal Trace Dataspace Information” on
page 108
The trace table header record contains status information about the trace records.
The information includes the time that the current record was entered, the last two
times that the table wrapped, and the addresses of the current and last entries in
the table.
For hexadecimal offset X'20', if the buffer is an AIFR, the trace record field
represents a chain of buffers and the trace entry is repeated for each buffer in the
chain.
The exit number is designated in hexadecimal X'01' to X'15' for DSIEX01 through
DSIEX21 (DSIEX02A is traced with X'02'). For data services task (DST) exits, this
field is designated in hexadecimal as follows:
v XITDI (DST initialization exit)=X'E9'
v XITVN (VSAM initialization exit)=X'EA'
v XITVI (VSAM input exit)=X'EB'
v XITVO (VSAM output exit)=X'EC'
v XITCI (CNM interface input exit)=X'ED'
v XITCO (CNM interface output exit)=X'EE'
v XITXL (External log exit)=X'F0'
v XITBN (Sequential log initialization installation exit)=X'F1'
v XITBO (Sequential log output installation exit)=X'F2'
In case of a nonzero return code, a trace entry is generated if the trace is on.
When this trace entry is for DSIFRE Q=YES, the length shown at offset X'14' in the
trace entry will be zero.
Note: The last byte in this group indicates the DSIGET or DSIFRE reason code. If
the value is an odd number greater than 100, it indicates an internal failure
code for a DSIGET. If the value is an odd number less than 100, it indicates
an internal failure code for a DSIFRE.
For hexadecimal offset X'20', if the buffer is an AIFR, the trace record field
represents a chain of buffers and the trace entry is repeated for each buffer in the
chain.
For trace entries with a NetView buffer, the buffer has a NetView buffer header
followed by the first section of the buffer text starting at the offset given by
HDRTDISP. Refer to IBM Tivoli NetView for z/OS Programming: Assembler for more
information about BUFHDR.
Constants for Option Bytes: Table 19 and Table 20 on page 114 list the constants
for option byte 1 and option byte 2 of the DSIPSS trace record shown in Table 21
on page 115.
Table 19. Constants for Option Byte 1
Constants Hexadecimal Description
PSMSEGMT X'40' Data message segment has no message
header
PSMNOOP X'00' Do not change ready message
PSMFRSTF X'06' Begin full-line mode
PSMMIDF X'04' Continue full-line mode
PSMLASTF X'05' End full-line mode
PSMONLYF X'07' One full-line message
Additional Data (64 bytes) that varies according to the type code:
Type Code
Additional Data Description
INPUT (X'01')
64 bytes of data formatted as follows:
X'20' 4-byte length of input area
X'24' 4-byte length of input received
X'28' 1-56 bytes of input data
OUTPUT (X'02')
First 64 bytes of NetView buffer.
CMDLINE (X'05')
First 64 bytes of NetView buffer.
IMMEDIATE (X'05')
First 64 bytes of NetView buffer.
ASYPANEL (X'0F')
64 bytes of data formatted as follows:
X'20' First 20 bytes of ASYPANEL parameter list
X'34' 4 bytes of non meaningful data
X'38' 0-40 bytes of the data to be sent to the terminal
CANCEL (X'10')
Non meaningful data
PSSWAIT (X'11')
Non-meaningful data
| These are some, but not all, of the tasks for which the trace record is generated on
| completion of an IP Service request:
| CNMTAMEL
| DSIIPLOG
| DSIREXEC
| DSIRSH
| DSIRTTR
| DSITCPIP with the NMC-3270
| DSIUDST (when RMTCMD over IP is enabled)
| DSIWBTSK
| DUIDGHB
| These trace records will have eye-catchers of TCxx. Tasks issuing the NCCF
| SOCKET command might generate two trace records. If the IP Service request is
| asynchronous, a trace record is generated following the invocation of the IP Service
| request (eye-catcher TCxx) and a trace record is generated on completion of the IP
| Service request (eye-catcher TAxx). If the NCCF SOCKET command request is
| synchronous, only one trace record is generated (TCxxx). In addition, these are
| some, but not all, commands that use NetView IP services:
| IPLOG
| REXEC
| RSH
| TN3270
| Note: For asynchronous IP Service requests, the TAxx completion records contain
| information returned by TCP/IP. Consult the z/OS Communications Server: IP
| Sockets Application Programming Interface Guide and Reference in the section
| ″Using the Macro Application Programming Interface (API)″ to identify the
| corresponding EZASMI macro invocation. For example, TCGH and TAGH
The Field Type column indicates whether a particular field is input, output, or
both. For the NCCF SOCKET command trace entries, only input fields will be
displayed in the trace entries. The Field Type is not specified for the header
section of each trace record (X'00'–X'17').
Note: This does not trace interfaces using the REXX SOCKET function.
With the exception of the Select Exit (SE) trace record, the record types correspond
to the types of IP Services calls. The SE trace record is generated to indicate
SELECT request completion.
The following IOCTL REQARG tables map to the REQARG Details entry (offset
X'34') in the IOCTL Trace record.
The following SETSOCKOPT option value tables map to the option value entry
(X'34') in the SETSOCKOPT trace record.
The following SETSOCKOPT option value mapping is for the following options:
v IP_ADD_MEMBERSHIP
v IP_DROP_MEMBEFRSHIP
Table 58. SETSOCKOPT Option Value Mapping 1
Hexadecimal Offset Field Type Trace Record Field
X'34' Input IPv4 multicast address
X'38' Input IPv4 interface address
The following SETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_IF
v IPV6_MULTICAST_IF
Table 59. SETSOCKOPT Option Value Mapping 2
Hexadecimal Offset Field Type Trace Record Field
X'34' Input IPv4 interface address or IPv6
interface index number
The following SETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_LOOP
v IPV6_MULTICAST_LOOP
v IPV6_V6ONLY
v SO_BROADCAST
v SO_KEEPALIVE
v SO_OOBINLINE
v SO_RCVBUF
v SO_REUSEADDR
v SO_SNDBUF
v TCP_KEEPALIVE
v TCP_NODELAY
Table 60. SETSOCKOPT Option Value Mapping 3
Hexadecimal Offset Field Type Trace Record Field
X'34' Input Enable/Disable Field (see request for
details), Buffer Data size, or timeout
value
The following SETSOCKOPT Option Value Mapping is for the following options:
v IPV6_JOIN_GROUP
v IPV6_LEAVE_GROUP
The following SETSOCKOPT Option Value Mapping is for the following options:
v IPV6_MULTICAST_HOPS
v IPV6_UNICAST_HOPS
Table 62. SETSOCKOPT Option Value Mapping 5
Hexadecimal Offset Field Type Trace Record Field
X'34' Input Number of hops
The following SETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_TTL
Table 63. SETSOCKOPT Option Value Mapping 6
Hexadecimal Offset Field Type Trace Record Field
X'34' Input Time-to-live value
The following SETSOCKOPT Option Value Mapping is for the following options:
v SO_LINGER
Table 64. SETSOCKOPT Option Value Mapping 7
Hexadecimal Offset Field Type Trace Record Field
X'34' Input Enabling Field
X'38' Input Seconds to Linger
The following GETSOCKOPT option value tables map to the Option Value entry
(offset X'18') in the GETSOCKOPT trace record.
The following GETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_IF
v IPV6_MULTICAST_IF
Table 75. GETSOCKOPT Option Value Mapping 1
Hexadecimal Offset Field Type Trace Record Field
The following GETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_LOOP
v IPV6_MULTICAST_LOOP
v IPV6_V6ONLY
v SO_BROADCAST
v SO_KEEPALIVE
v SO_OOBINLINE
v SO_RCVBUF
v SO_REUSEADDR
v SO_SNDBUF
v TCP_NODELAY
Table 76. GETSOCKOPT Option Value Mapping 2
Hexadecimal Offset Field Type Trace Record Field
X'18' Output Enable/Disable Field (see request for
details), Buffer Data size, or timeout
value
The following GETSOCKOPT Option Value Mapping is for the following options:
v IPV6_MULTICAST_HOPS
v IPV6_UNICAST_HOPS
Table 77. GETSOCKOPT Option Value Mapping 3
Hexadecimal Offset Field Type Trace Record Field
X'18' Output Number of hops
The following GETSOCKOPT Option Value Mapping is for the following options:
v IP_MULTICAST_TTL
Table 78. GETSOCKOPT Option Value Mapping 4
Hexadecimal Offset Field Type Trace Record Field
X'18' Output Time-to-live value
The following GETSOCKOPT Option Value Mapping is for the following options:
v SO_LINGER
Table 79. GETSOCKOPT Option Value Mapping 5
Hexadecimal Offset Field Type Trace Record Field
X'18' Output Enabling Field
X'1C' Input Seconds to Linger
The following GETSOCKOPT Option Value Mapping is for the following options:
The following GETSOCKOPT Option Value Mapping is for the following options:
v SO_TYPE
Table 81. GETSOCKOPT Option Value Mapping 7
Hexadecimal Offset Field Type Trace Record Field
X'18' Output Socket Type
The following IOCTL RETARG tables map to the RETARG Details entry (offset
X'1C') in the IOCTL trace record.
These records document certain internal flows in status monitor processing. They
can be useful in solving status monitor problems.
The SAF trace record is generated if NetView is using an SAF product for operator
verification, command authorization, or span authority checking.
For more information about the SAF return and reason codes for RACF V2R1, refer
to the External Security Interface (RACROUTE) Macro Reference for MVS.
Note: Multiple type-3 records might be required if the value is greater than 80
bytes. For the RMTOPS class, the POST record will not contain profile
Note: Multiple type-2 records might be required if the value is greater than 80
bytes.
Table 98. Security Authorization Facility EXTRACT Trace Record - Type 2
Hexadecimal Offset Trace Record Field
X'00' SAF EXTRACT Trace Record ID = "EXT2"
X'04' Return address of caller
Note: Multiple type-3 records might be required if the value is greater than 80
bytes.
Table 99. Security Authorization Facility EXTRACT Trace Record - Type 3
Hexadecimal Offset Trace Record Field
X'00' SAF EXTRACT Trace Record ID = "EXT3"
X'04' Return address of caller
X'08' TVB address
X'0C' PRE RACROUTE trace record ID = "PRE" POST RACROUTE trace
record ID = "POST"
X'10' For PRE, the list of field names whose values are to be extracted
For POST, this value contains pairs of 4-byte length fields followed
by their related data. If the length field is zero, the next field is a
subfield length.
Subfield
Data
IC initial command name
CTL X'00'=specific, X'40'=general, X'80'=global
MSGRECVR
X'00'=no, X'80'=yes
DOMAINS
domain name
CONSNAME
console name
NGMFADMN
X'00'=no, X'80'=yes
Note: Multiple type-3 records might be required if the value is greater than 80
bytes.
Table 102. Security Authorization Facility FASTAUTH Trace Record - Type 3
Hexadecimal Offset Trace Record Field
X'00' SAF FASTAUTH Trace Record ID = "FST3"
X'04' Return address of caller
X'08' TVB address
X'0C' PRE RACROUTE trace record ID = "PRE" POST RACROUTE trace
record ID = "POST"
X'10' For PRE only, the resource to be checked
The HLL trace area has eight entries, each of which is 6 bytes, for a total 48 bytes.
Each trace point is identified by a unique 16-bit ID. The first 12 bits of each trace
entry represent the module ID. The next 4 bits are the location ID within the
module. By convention, X'0' is the location ID value for the module entry and X'F'
is the location ID value for the module exit. The module ID value corresponds to
the HLL service routine. See Table 114 on page 147 for module ID values.
The next 4 bytes of each trace entry contain useful diagnostic information captured
at a diagnostic point. Use the trace entries that are recorded at entry to and exit
from HLL service routines to determine the location of the error. The diagnostic
information recorded at entry is the return address of the caller of the service, from
register 14 of the caller. The information recorded at exit is the return code from
the HLL service routine. The information recorded at other location IDs is only for
use by IBM Software Support.
For example, consider the HLL command procedure TEST, which consists of four
subroutines: SR01, SR02, SR03, and SR04. The command procedure records a
4-character subroutine name entry in HLBFFDCA at entry to, and exit from, these
routines. SRE1 is the value at the entry of the first subroutine (SR01) and SRX1 is
the value at the exit of SR01. If the TEST command procedure abends, a panel
similar to the panel in Figure 34 on page 147 is displayed at the NetView operator
console.
???
In Figure 34, message CNM998E indicates that the abend occurred in the command
procedure TEST that has an entry point address 02522BA8. Message CNM983E
indicates that the command procedure successfully entered and exited subroutine
SR01 and then entered subroutine SR03, but did not exit it. This indicates that the
abend occurred in subroutine SR03.
Message CNM999E indicates that the ID of the last HLL service routine entered is
X'001'. The final entry in the last CNM999E message is only for use by IBM
Software Support. See Table 114 to correlate the ID (X'001') with the service routine
CNMCMD. This is the service routine that the TEST command procedure was
running at the time of the abend. The return address from the service routine is
82522F4E. From the return address, you can compute the offset in the user code
from which the service was run.
Table 114. Module ID Used by FFDC Trace
Module ID HLL Service Routines Module ID HLL Service Routines
001 CNMCMD 015 CNMALTD
002 CNMSMSG 016 CNMCELL
003 CNMNAMS 019 CNMC2T
004 CNMGETD 020 CNMSMU
005 TIMEP (see note) 021 CNMRGS
006 WAIT command 022 CNMAUTO
007 CNMINFC 023 CNMQAPI
008 CNMINFI 024 CNMPMDB
009 CNMGETA 026 CNMIPXL
00A CNMVARS 051 CNMETIN
00B CNMMEMO 052 CNMETRM
00C CNMMEMR 053 CNMEWAT
00D CNMMEMC 054 CNMEGTP
00E CNMSCAN 055 CNMENTR
Note: HLL service routine TIMEP is for IBM Software Support use only.
The PPI trace table and the GTF trace record chain are anchored by the PPI trace
anchor block.
Understanding the PPI Trace Anchor Block and the PPI Trace
Table
The PPI trace table resides in the subsystem interface (SSI) address space and is
anchored by the PPI trace anchor block. The PPI trace anchor block:
v Resides in the common storage area in MVS
v Contains the following pointers:
– The first pointer points to the address of the PPI trace table.
– The second pointer points to the last PPI trace record written to the trace
table.
v Contains information about the status of the PPI trace
The PPI trace records follow the 12-byte header. The trace records are variable
length and are linked with backward and forward pointers. If a trace record is
longer than the space available after the most current record in the trace table, the
new entry is written at the beginning of the table, overwriting any records that are
already there. Figure 35 on page 149 shows the format of each trace record.
| When using the Interactive Problem Control System (IPCS) to read the GTF trace
| records, you can use the CNMS4501 sample supplied with the NetView product to
| format the PPI trace records. Sample CNMS4501 can be run either as an IPCS GTF
| filter exit or an IPCS GTF formatter exit. To run CNMS4501 as an IPCS GTF filter
| exit, go to the IPCS command line and enter the following command:
| GTF EXIT(CNMS4501) DSNAME(gtf_data_set_name)
To run CNMS4501 as an IPCS GTF formatter exit, link edit the CNMS4501 sample
as IMDUSRDB into the system link library, then go to the IPCS command line and
enter the following command:
GTF DSNAME(gtf_data_set_name)
Figure 36. Example of the Output from the CNMS4501 Installation Exit
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting
and Reporting Problems” before contacting IBM Software Support.
Table 115. Sysplex Problem Scenarios
Problem Category Problem Scenario Page
| DVIPA management No DVIPA SNMP traps received 156
| DVIPA management No DVIPA configuration changes received 156
| DVIPA management No DVIPA statistics recorded 157
| DVIPA management No data is returned from a DVIPA 3270 command or its associated 157
| sample
| DVIPA management A command issued with DOMAIN=ALL from a master NetView 157
| returns incomplete data
| DVIPA management Distributed DVIPA Connection Routing data is incomplete in the EMA 158
| workspace
| OSA or HiperSockets™ OSA or HiperSockets data is not returned at an NMC client. 158
| management
| Stack and Interface Data is not returned 158
| Management
| Sysplex topology Sysplex Topology is not being presented correctly 160
| Telnet management The Telnet server port active connections count is 0 158
| XCF services No data returned using LIST STATUS=XCFGRPS command 159
| XCF services BNH638I message issued per stack for Discovery Manager Resource 159
| XCF services BNH587I message is received 159
| XCF services PLEXCTL command fails 159
| XCF services START XCFGROUP problems 160
| XCF services Discovery commands fail 160
| XCF services BNH067I message is received; unexpected switch of master NetView 160
| XCF services BNH558E message is received; master NetView unable to contact 160
| enterprise system
| Note: If traps are being generated for ITNM, use the same task name for both.
| For information on configuring the NetView program to receive and process SNMP
| traps from the z/OS Communications Server, see IBM Tivoli NetView for
| z/OS Installation: Configuring Additional Components.
| The following z/OS Communications Server DVIPA traps can then be received by
| the NetView program:
| – ibmMvsDVIPAStatusChange
| – ibmMvsDVIPARemoved
| Note: If traps are being generated for ITNM, it is recommended to use the
| same task name for both.
| For information on configuring the NetView program to receive and process
| configuration updates from the z/OS Communications Server, see IBM Tivoli
| NetView for z/OS Installation: Configuring Additional Components.
| Note: This is not the case with DVIPA discovery. You will only receive message
| BNH638I and the corresponding DWO050E message once per interval.
| There is a problem writing to the NetView for z/OS Enterprise Management Agent
| data space.
CNMTRACE
CNMTRACE provides tracing information for host commands related to EMA
functions, 3270 DVIPA commands, and some sysplex data processing execs.
To see the DVIPA event (z/OS Communications Server DVIPA SNMP trap, DVIPA
TCP/IP profile update, or sysplex monitoring message) you received for which
NetView will rediscover DVIPA information, enable the CNMTRACE.DVIPEVNT
or CNMTRACE.DVIPEVNT.opid DEBUG option.
The following REXX exec can be used to control the CNMTRACE function:
/* rexx */
/* This exec starts debug tracing for EMA-related and DVIPA host commands. */
/* Input is as follows: */
/* command , option, opid */
/* The following commands are currently supported: */
/* NACMD, DVIPSTAT, DVIPPLEX, DVIPTARG, DVIPCONN, DVIPHLTH, */
/* DVIPDDCR, VIPAROUT, DVIPEVNT, COLLCTL, and DVIPA. */
/* The following options are valid: */
/* YES or ON : provides information upon entry and exit of the exec */
/* DEBUG : provides YES-level information plus additional debugging */
/* information such as data returned from data collector execs. */
/* OFF or NO : turns off tracing */
/* The opid parameter is optional; if omitted, all operator IDs will be traced. If */
/* provided, only the opid provided will be traced. */
arg input
parse var input cmd ',’ debugopt ',’ operid
if operid <> '’ then
operid = '.’||strip(operid)
'pipe lit /’debugopt’/ | var (common) cnmtrace.’cmd||operid
exit 0
RXTRACE
RXTRACE provides entry/exit and program trace capability for REXX execs and
command lists. This support is shipped in most IP management commands, as
well as in a large number of other base NetView commands. Tracing can be set for
a single operator, all operators, or for a series of operators.
Use the RXTRACE command for control; note that by default this function is
shipped with a setting of NONE. To use the trace, a user will first have to choose
option 3 on the RXTRACE panel to change the setting. RXTRACE sends its output
to the netlog using EZL260I messages.
Workspace issues
There are log and trace functions available on the workstation for issues related to
the workspace.
Chapter 10. Graphic Monitor Facility Host Subsystem Problem Worksheet . . . . . . . . . . . . 169
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
System Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
GMFHS Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
RODM Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
RODM Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Problem Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170
Abend problems (processor exception/trap/fault) . . . . . . . . . . . . . . . . . . . . . 170
Message Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Loop Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
Wait Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Incorrect Output Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172
Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Documentation Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS . . . . 175
Alert and Alert History Problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
Alerts Are Not Listed in the Event Viewer at the NetView Management Console Workstation . . . . . . 177
Alerts Are Not Listed in the Hardware Monitor Alerts History Panel . . . . . . . . . . . . . . . 178
Alerts Do Not Change Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
Command Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
Cannot Initiate an IP Session Using NETCONV . . . . . . . . . . . . . . . . . . . . . . 180
Cannot Initiate an LU 6.2 Session Using NETCONV . . . . . . . . . . . . . . . . . . . . 181
Command Results Are Unexpected from Network Management Gateways . . . . . . . . . . . . . 182
Commands Failed to Run Because of COS Gateway Errors . . . . . . . . . . . . . . . . . . 182
Commands Failed to Run Because of OST Errors . . . . . . . . . . . . . . . . . . . . . 182
Commands Failed to Run Because of PPI Gateway Errors . . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of RODM Attribute Errors . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of Service Point Errors . . . . . . . . . . . . . . . . . . 183
Commands Failed to Run Because of Time-out Errors . . . . . . . . . . . . . . . . . . . . 183
Commands Failed with Message IHS2069W, Command Exit Not Installed . . . . . . . . . . . . . 184
GMFHS Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Errors Are Received During GMFHS Configuration Initialization . . . . . . . . . . . . . . . . 185
GMFHS Status Solicitation Fails . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
Instrumentation (z/OS-based only) Problems . . . . . . . . . . . . . . . . . . . . . . . . 186
Events Are Not Received from z/OS Instrumentation . . . . . . . . . . . . . . . . . . . . 186
Component or Connection Status is not Properly Reflected on the Topology Console . . . . . . . . . 186
Status Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
Resource Status Is Incorrect. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
The Resource Exists but the Status Is Not Updated . . . . . . . . . . . . . . . . . . . . . 188
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS . . . . . . . . . . . 209
Diagnostic Tools for the NetView Management Console . . . . . . . . . . . . . . . . . . . . 209
Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
Access to Online Help without the Console . . . . . . . . . . . . . . . . . . . . . . . 209
Topology Console Environment Information Window . . . . . . . . . . . . . . . . . . . . 210
Message Help for the Topology Server . . . . . . . . . . . . . . . . . . . . . . . . . 210
Diagnostic Tools for GMFHS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
GMFHS Message Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
GMFHS Output Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
Determining Which GMFHS Output Log to Use . . . . . . . . . . . . . . . . . . . . . 211
Console Log Window . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
GMFHS Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
Starting and Stopping the GMFHS Trace . . . . . . . . . . . . . . . . . . . . . . . . 214
Viewing the GMFHS Trace Online . . . . . . . . . . . . . . . . . . . . . . . . . . 215
Using the GMFHS Internal Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
IPC Task Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Event Manager Task Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Host
v Component ID:
v Recommended service update (RSU) level:
4. Workstation Service Level:
Notes:
a. To determine the service level for NetView management console, see
Environment Information in the NetView management console online help.
b. To determine the service level for the topology console, refer to file
tds\client\bin\duimnt01.gen
c. To determine the service level for the topology server, refer to file
tds\server\bin\duimnt02.gen
System-related Information
Record the following system-related information:
1. The platform and level you are using:
2. Are you using the Tivoli desktop?
3. The personal computer you are using:
4. How much memory is installed on your personal computer?
5. How many bytes of free disk space do you have for each drive being used?
6. Have you recently changed the system? If so, have you:
v Changed or added hardware?
v Applied software maintenance?
v Added user written code (plug-ins or Java applications)?
v Other changes?
7. The speed of the computer you are using:
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
6. If you have more than one workstation, does the problem occur consistently on
all workstations?
Problem Classification
This section addresses the following problem classifications:
v Processor traps
v Message problems
v Loop problems
v Wait problems
v Incorrect output problems
v Performance problems
v Documentation problems
Look at the problem classification that matches the symptoms associated with your
problem.
Processor Traps
For abends or processor exception problems, respond to the following questions or
tasks and, if appropriate, record the answers:
1. What is the trap code?
2. What processes were taking place at the time of the abend or trap?
3. For the topology console, what is the Java stack trace for exceptions? Recreate
the problem by setting the TCONSOLE_JAVAOPTS environment variable to
-Djava.compiler=NONE.
Message Problems
For message problems, respond to the following questions and, if appropriate,
record the answers:
1. Record the message ID and any error codes displayed.
v Message ID:
v Error codes:
2. Review the message in the NetView management console online help to
determine user action.
v What processes were taking place when the message occurred?
– Commands:
– Other:
Performance Problems
For performance problems, respond to the following questions and, if appropriate,
record the answers:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Obtain a copy of the appropriate workstation error logs.
Documentation Problems
For documentation problems, respond to the following questions and, if
appropriate, record the answers:
1. Identify the order number, revision level, and title of the manual.
2. Identify the location (chapter and section name) of the error in the manual.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program, call IBM
Software Support.
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Component ID:
4. Recommended service update (RSU) level:
GMFHS Information
1. Did the GMFHS data model load successfully?
2. Have you modified the GMFHS data model? If so, what was added or
changed?
3. Did you receive a GMFHS message at the system console?
GMFHS messages are in the range between DUI3900-DUI4099 and
DUI4200-DUI4499.
RODM Applications
1. Are you running any other RODM applications?
2. Can you remove one or more RODM applications and re-create the problem?
RODM Methods
1. Are you running any user-written methods with RODM? If so, which ones?
2. Can you bypass these and successfully run the function you are attempting?
Problem Classification
This section addresses the following problem classifications:
v Abend problems (processor exception/trap/fault)
v Message Problems
v Loop Problems
v Wait Problems
v Incorrect Output Problems
v Performance Problems
v Documentation Problems
Look at the problem classification that matches the symptoms associated with your
problem:
Message Problems
For message problems, complete the following:
1. Record the message ID and any error codes displayed.
v Message ID:
v Does the message contain any return codes, reason codes, feedback codes,
error codes, or sense information? List the codes or information.
2. Check the message in the online help to determine user action.
3. What processes were taking place when the message occurred?
v Methods:
v RODM Load Utility:
v Other:
4. If the message was unexpected and cannot be corrected by following the
actions in the online help, collect the following documentation before calling
IBM Software Support:
v A hard copy of the network log
v The message ID:
v The exact text of the message on the log
v A completed GMFHS problem worksheet
v A copy of the GMFHS job output
v The GMFHS initialization member (DSIPARM member DUIGINIT)
v A copy of the RODM log
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v The customization member (EKGCUST)
5. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
6. Did the message text differ from what was published?
v Has local modification been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Loop Problems
For loop problems, complete the following:
1. What events led up to the loop?
2. What data was being displayed?
3. What was the last command entered?
4. If this is a method loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v A document describing the scenario leading to the problem
v A hard copy of the system log
Chapter 10. Graphic Monitor Facility Host Subsystem Problem Worksheet 171
v The addresses of instructions within the loop:
v A dump obtained by using the CPU RESTART function
v The GMFHS initialization member (DSIPARM member DUIGINIT)
v A copy of the RODM log
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v The customization member (EKGCUST)
5. What are the modules involved in the loop?
6. What are the dates that the modules were compiled?
7. What are the PTF levels of the modules involved in the loop?
Wait Problems
For wait problems, complete the following:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. Collect the following documentation before calling IBM Software Support:
v A copy of the system console log
v A copy of the system console dump
v A completed GMFHS problem worksheet
v A copy of the GMFHS job output
v The GMFHS initialization member (DSIPARM member DUIGINIT)
v A copy of any GMFHS trace output
v A copy of the RODM log
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v The customization member (EKGCUST)
5. What is the name of the module in which the wait occurred?
6. What is the date that the module was compiled?
7. What is the PTF level of the module involved?
8. What is the offset into the module where the wait occurred?
Performance Problems
For performance problems, complete the following:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Collect the following documentation before calling IBM Software Support:
v A copy of the GMFHS job output
v The GMFHS initialization member (DSIPARM member DUIGINIT)
v A copy of the RODM trace
v The customization member (EKGCUST)
v A copy of the RODM log containing log record type 8 lock and storage
statistics
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v Information describing your RODM operating environment
v Descriptions of any modifications to your system
Documentation Problems
For documentation problems, complete the following:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program, call IBM
Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
Chapter 10. Graphic Monitor Facility Host Subsystem Problem Worksheet 173
174 Troubleshooting Guide
Chapter 11. Troubleshooting and Initial Diagnosis for NetView
Management Console and GMFHS
This section provides problem scenarios and resolutions that you can use to locate
examples of problems you might encounter when using the NetView management
console.
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems,” on page 9 and Chapter 3,
“Documenting and Reporting Problems,” on page 19 before contacting IBM
Software Support.
Table 116. NetView management console and GMFHS Problem Scenarios
Problem Category Problem Scenario Page
Alerts Alerts are not listed in the Event Viewer at the 177
NetView management console workstation.
Alerts are not listed in the hardware monitor 178
alerts history panel.
Alerts do not change status. 178
Commands Cannot initiate an IP session using the 180
NETCONV command.
Cannot initiate an LU 6.2 session using the 181
NETCONV command.
Command results are unexpected. 182
Commands failed to run because of common 182
operation services (COS) gateway errors.
Commands failed to run because of operator 182
station task (OST) errors.
Commands failed to run because of 183
program-to-program interface (PPI) errors.
Commands failed to run because of RODM 183
attribute errors.
Commands failed to run because of service 183
point errors.
Commands failed to run because of time-out 183
errors.
GMFHS Errors are received during GMFHS 185
configuration initialization.
GMFHS Status solicitation fails. 185
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 177
3. Ensure that the scope checker (DUIFSSCO) and hardware monitor (BNJDSERV)
tasks are active.
Check the NetView log for messages related to these tasks.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 179
For information about: Refer to:
How GMFHS identifies RODM objects using IBM Tivoli NetView for z/OS Resource Object
alerts Data Manager and GMFHS Programmer’s
Guide
How GMFHS identifies RODM objects using IBM Tivoli NetView for z/OS Resource Object
the alert processor module Data Manager and GMFHS Programmer’s
Guide
The GMFHS initialization parameters IBM Tivoli NetView for z/OS Administration
Reference
Setting dispatching priorities MVS/ESA Installation and Tuning Guide
The RUNCMD and GMFHS TASK NetView online help
commands
Command Problems
The following sections describe problem scenarios and their resolutions for
command problems. Potential problems can include the following:
v “Cannot Initiate an IP Session Using NETCONV”
v “Cannot Initiate an LU 6.2 Session Using NETCONV” on page 181
v “Command Results Are Unexpected from Network Management Gateways” on
page 182
v “Commands Failed to Run Because of Service Point Errors” on page 183
v “Commands Failed to Run Because of RODM Attribute Errors” on page 183
v “Commands Failed to Run Because of COS Gateway Errors” on page 182
v “Commands Failed to Run Because of OST Errors” on page 182
v “Commands Failed to Run Because of PPI Gateway Errors” on page 183
v “Commands Failed to Run Because of Time-out Errors” on page 183
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 181
To create an LU 6.2 session, you might need to change the VTAM and NCP
definitions. Verify that these changes were made correctly.
11. See “INCORROUT” on page 11.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 183
For information about: Refer to:
Timeout values specified in IBM Tivoli NetView for z/OS User’s Guide:
SERVER.PROPERTIES NetView Management Console
Misconceptions about which RODM attribute values are being loaded for GMFHS
can occur if attributes are defined at both the class level and object level.
If the attributes are defined at the class level, the values are used in GMFHS only
if no object is defined for that specific attribute. For example, the definitions shown
in Figure 37 on page 185 are coded correctly.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 185
Even though a session might have been established with the domain (in the
case of a DOMS010 session protocol), the domain might not have become
inactive before the status solicitation was completed.
2. Verify that the RODM definition for the domain entity correctly matches the
actual domain.
3. If the preceding steps have been verified and the problem persists, increase the
value of the CommandTimeoutInterval field.
To resolve this:
1. Verify that the NETCONV connection is active. Enter the following command
at the NetView command prompt:
NETCONV ACTION=LIST,OPID=ALL
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 187
For information about: Refer to:
The fields used by GMFHS IBM Tivoli NetView for z/OS Resource Object
Data Manager and GMFHS Programmer’s
Guide
Alerts cannot flow to the NetView program (and through to GMFHS) until the
domain receives this authorization, but GMFHS continues (based on a timer) to
attempt to establish a session with any domain under the NMG. The alerts
generated as a result of these requests stack up and flow to the mainframe server
when the domain receives the focal point alert authorization.
This is not an error, but you can avoid the situation by following these steps:
1. Initialize the NMG.
2. Before you start GMFHS, issue a NetView FOCALPT authorization command
for all domains that have a DOMS010 session protocol.
This error can occur if the time in the command response received by GMFHS is
earlier than the time kept by the mainframe server on which GMFHS is running.
To solve this problem, ensure that the workstation clock corresponds to the focal
point mainframe server clock, not including the Greenwich Mean Time (GMT)
offset.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 189
v “Unable to Connect to the Topology Server from the Topology Console”
v “Topology Console Hangs During Sign-on”
v “Topology Console Hangs When Accessing a View”
v “There Is a Duplicate GMFHS Resource on the Topology Console” on page 191
v “Problems Occur with Minimized Windows” on page 191
v “Property Changes Are Lost” on page 191
If you did not perform the following steps, you will notice an additional GMFHS
resource:
1. Code a NetView domain name in the DUIGINIT initialization file.
2. Select the GMFHS resource.
3. Select Start GMFHS.
This occurs because GMFHS does not know the NetView domain name, so a
default value of CNM01 is used. Therefore, this resource has a different name from
the original GMFHS resource.
These problems occur because of problems with the Java environment. Avoid
minimizing the topology console window for extended periods of time. Secondary
windows that are minimized can be restored by using the operating system
mechanism for doing so. For example, in Windows, use the task bar.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 191
Topology Server Problems
The following sections describe problem scenarios and their resolutions for
command problems. Potential problems can include the following:
v “Server Does not Start and setup_env.cmd Is not Found”
v “Setup_env.cmd Is Found but BINDIR Is not Set”
v “Topology Server Starts but Then Closes (Windows)”
v “Topology Server Does Not Complete Initialization on AIX” on page 193
v “Server Windows Disappear on AIX Platform” on page 193
v “Incorrect Timestamps If the Topology Server is on the Windows Platform” on
page 193
To resolve this make sure that the path is correct in the BINDIR variable
This can happen if interprocess communications (IPC) resources have not been
cleaned up.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 193
SET TZ
SET TZ=xxxyyzzz
Where:
xxx Is any 3-letter time zone acronym (for example, EST for Eastern Standard
Time, CST for Central Standard Time, or PST for Pacific Standard Time).
yy Is a one- or two-digit number that is the difference in hours between
Greenwich Mean Time (GMT) and the local time. If the local time is west
of GMT, this number is unsigned. For example, the following statement
sets the time zone variable, TZ, for central standard time (CST), which is 6
hours west of GMT.
Example for setting TZ to central standard time:
SET TZ=CST6
If the local time is east of GMT, this number has a preceding minus (-)
sign. For example, the following statement sets the time zone variable, TZ,
for Germany, 1 hour east of GMT.
Example for setting the TZ variable for Germany:
SET TZ=CET-1
Use a negative (−) sign for the negative numbers, but do not use a positive
(+) sign for the positive numbers.
When you set this field, remember to take into account the setting of the
time and time zone offset on your NetView host. If you run your NetView
host using the local time (instead of GMT) and a time zone offset of 0,
code a 0 for the offset on the workstation. If you do not code 0, the status
timestamps will not be correct. Set this value to the offset from GMT on
your workstation only if you use GMT and a time zone offset on your
host.
To check the GMT setting on your host (and whether the local time is
different) enter the MVS display time command from a NetView operator
screen, as shown below.
Example for checking the GMT setting on the host:
MVS D T
View Problems
The following sections describe problem scenarios and their resolutions for
problems with views. Potential problems can include the following :
v “Expected Configuration or More Detail View Does Not Exist” on page 196
v “Resource Icon Is Missing from View” on page 197
v “Tree View List Is Incorrect” on page 198
v “View Layout Is Incorrect” on page 199
v “Unable to Open View” on page 200
v “Unable to Monitor Views of Your Network” on page 200
v “View Does Not Show Correct Connectivity” on page 201
v “View Does Not Contain Resource” on page 202
v “Multiple Correlated Aggregate Objects Contain the Same Object” on page 204
v “Real Resource Is Not Shown as a Member of a Correlated Aggregate Object” on
page 205
v “Information Displayed for Correlated Aggregate Object Changes” on page 206
v “Cannot Navigate Between Correlated Aggregate Object and Contained
Resources” on page 207
v “Pop-up Menu in the Business Tree Is Not Displayed on AIX Platform” on page
207
v “View Problems”
v “The Topology Display Subsystem View Is Not Complete” on page 208
The status area can indicate a change in the view without an actual change
occurring. This can occur for one of the following reasons:
v A customized view in the NetView management console server is customized
and saved without any actual changes occurring.
All consoles monitoring this view receive a message in their status area
indicating the view definition has changed.
v A change is made in RODM to alter a view followed by a change which restores
the view to its previous state.
If this occurs, the status area indicates that multiple changes have been made to
the view. The view remains the same when refreshed.
v A session between the mainframe and NetView management console server is
temporarily disconnected while non-customized views are opened.
When the session is restored, the status area of each non-customized view
indicates a change whether or not the view has actually changed.
v A change is made in RODM to a GMFHS presentation data model attribute
which changes the definition of a view without modifying the view display.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 195
Expected Configuration or More Detail View Does Not Exist
The requested configuration or more detail view is missing. To determine if this
might occur, ask the following questions:
v Was this view generated by GMFHS?
If so, one of the following conditions might exist:
– You have not correctly defined the view in the RODM view definition file.
– The RODM view definition file did not load with a return code of 0.
– The operator is not authorized to display the view.
In this case, the operator receives an error message.
v Was the view created in RODM by a topology manager?
If so, one of the following conditions might exist for one of the following:
– SNA topology manager
- The SNA topology manager autotask named FLBTOPO is not started.
- The SNA topology manager is not monitoring sufficient topology in your
network.
- The following monitoring problems occurred:
v The SNA topology manager was previously monitoring the relevant
topology in your network.
v The monitoring was stopped.
v A TOPOSNA PURGE command was run that deleted the relevant
resources from RODM.
- The operator is not authorized to display the view. In this case, the
operator receives an error message.
– MultiSystem Manager
The following monitoring problems occurred:
- The MultiSystem Manager was previously monitoring the relevant topology
in your network.
- The monitoring was stopped.
- Because monitoring stopped, updates to resources in your network did not
occur.
The operator is not authorized to display the view. In this case, the operator
receives an error message.
Ensure that the span-of-control definitions for NetView management console views
are correctly defined by reviewing the following:
1. Does the SPANAUTH keyword specify to use the NetView span table?
2. Does the CTL attribute for the operator give the operator authority to control
resources and views?
3. Does the NGMFVSPN attribute for the operator specify to apply
span-of-control for view names, resource names, or both?
4. Do the spans that are active for the operator include the view names, the
resource names, or both? View and resource names are specified with
SPANDEF statements.
Finding the exact field to check depends on the exact view type requested.
However, you must define at least one of these fields for the resource. These are
the resolution steps for views created in RODM by the SNA topology manager:
1. Verify that the SNA topology manager autotask named FLBTOPO has started.
2. Verify that the SNA topology manager is monitoring the relevant topology in
your network.
Ensure that the RODM definition file loaded with a return code of 0. If RODM
view definitions did not load with a return code of 0, refer to the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 197
Tree View List Is Incorrect
The tree view list can be incorrect because:
v RODM is not loaded properly.
v Span-of-control limits the operator’s view list.
v GMFHS is not available.
v There is a problem with a network view collection definition object that you
created for use by the RODM Collection Manager facility of GMFHS.
1. Ensure that the RODM view definitions are correct and loaded with a return
code of 0.
If RODM view definitions did not load with a return code of 0, refer to the
IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS
Programmer’s Guide.
2. Ensure that all of the following are true:
v The NetView management console session is active
v GMFHS is available
v Any manager you are communicating with is still available
3. Ensure that the span-of-control definitions for NetView management console
views are correctly defined by checking the following:
a. Does the SPANAUTH keyword specify to use the NetView span table?
b. Does the CTL attribute for the operator give the operator authority to
control resources and views?
c. Does the NGMFVSPN attribute for the operator specify to apply
span-of-control for view names, resource names, or both?
d. Do the spans that are active for the operator include the view names, the
resource names, or both?
View and resource names are specified with SPANDEF statements.
4. Verify that the view name (MyName field of the view object) of the missing
view is not greater than 32 characters.
If the view name is greater than 32 characters, it is omitted from the tree view.
An entry is written to the RODM log that specifies which view or views were
omitted from the tree view and why they were omitted.
5. For Exception Views, look for duplicate ExceptionViewNames.
If more than one view object has the same ExceptionViewName, only one of
the views is displayed in the graphical list. An entry, which specifies the view
or views that were omitted from the tree view, is written to the RODM log. If
the view was omitted because of a duplicate ExceptionViewName, an entry is
written to the RODM log indicating the value of the ExceptionViewName field
of the view omitted from the tree view.
6. For network view collection-definition objects, look for the following situations
and take action if appropriate:
v If the collection definition object was loaded after GMFHS was started, the
Trigger field of the object must be set to any value in order for GMFHS to
process the object.
This is not a problem if the object was created by the RODM Collection
Manager wizard of the NetView management console.
v Check for errors that the RODM Collection Manager facility might have
encountered while processing the collection definition object.
If there is an error that prevents the network view from being created,
system console messages are logged . If the collection definition object was
For example, you might have specified a layout parameter list that contains the
correct layout parameters for the resource in question, but you might have linked
the layout parameter list to the wrong view object. If this happens, RODM loads
with a return code of 0, but the layout parameters apply to the wrong view.
GMFHS uses only the one layout object that is common between the layout
object list associated with the displayable object and the layout object list
associated with the view. In general, a displayable object might link to any
layout object as long as only one of the layout objects are linked to any
given view object. Default layout parameters are used if more than one
layout object is found.
If you follow the resolution steps and both of the following statements are true,
you might have incorrectly specified optional layout parameters:
v The view still does not layout correctly.
v There are no additional error messages.
To resolve the situation,
1. Take any action specified by the error panels.
2. Ensure that the layout parameters are coded correctly.
3. Ensure that layout parameters are correctly linked to the view object.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 199
Unable to Open View
If NetView management console issues an error message (for example:
GC_BAD_HEADER_VIEWSIZE) when you attempt to open a view, the view
cannot be displayed. This can happen if any of the following conditions exist:
v Your view is too complex to be displayed by NetView management console.
v You are not authorized to display the view.
v You are not authorized to display any of the resources in the view.
v You selected the Locate Resource option for a resource you are not authorized to
display.
v You selected the More Detail option for a resource you are not authorized to
display.
Ensure that the span-of-control definitions for NetView management console views
are correctly defined by checking the following:
1. Does the SPANAUTH keyword specify to use the NetView span table?
2. Does the CTL attribute for the operator give the operator authority to control
resources and views?
3. Does the NGMFVSPN attribute for the operator specify to apply
span-of-control for view names, resource names, or both?
4. Do the spans that are active for the operator include the view names, the
resource names, or both?
View and resource names are specified with SPANDEF statements.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 201
Loading RODM view definitions IBM Tivoli NetView for z/OS Resource Object
Data Manager and GMFHS Programmer’s
Guide
Ensure that the span-of-control definitions for NetView management console views
are correctly defined by checking the following items:
1. Does the SPANAUTH keyword specify to use the NetView span table?
2. Does the CTL attribute for the operator give the operator authority to control
resources and views?
3. Does the NGMFVSPN attribute for the operator specify to apply
span-of-control for view names, resource names, or both?
4. Do the spans that are active for the operator include the view names, the
resource names, or both?
View and resource names are specified with SPANDEF statements.
To solve problems with GMFHS views, including views that were created with the
RODM Collection Manager facility of GMFHS, follow these steps:
1. Verify that all required tasks are active.
2. Did the RODM definition file load with a return code of 0?
If RODM view definitions did not load with a return code of 0, refer to the
IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS
Programmer’s Guide.
3. Determine whether you have correctly defined the view in the RODM
definition file.
Look at the following fields in the RODM definition file:
v For More Detail views, review the following:
– ComposedOfLogical
– ComposedOfPhysical
v For configuration views, review the following:
– ParentAccess
– ChildAccess
– PhysicalConnPP
– PhysicalConnUpstream
– PhysicalConnDownstream
– LogicalConnPP
– LogicalConnUpstream
– LogicalConnDownstream
v For network views, perform one of more of the following steps:
– Determine whether the view object is defined.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 203
To solve problems with Exception views, perform the following steps:
1. Verify that the view is current.
If necessary, refresh the view.
2. Ensure that the NetView management console session is active.
If communicating with a topology manager, ensure that the manager is
available.
3. Look for the following conditions:
v The ExceptionViewList field on the resource object matches the
ExceptionViewName field on the view object.
v The ResourceTraits field on the resource object contains values that map to
the ExceptionViewFilter field on the view object.
To determine if either the ExceptionViewList or the ExceptionViewFilter field is
incorrect, change the ExceptionViewFilter field on the view object to X’0000’.
v If the resource object is now displayed in the view, your previous
ExceptionViewFilter field was filtering the object out of the view.
Ensure that the DisplayStatus, UserStatus, and ResourceTraits fields of the
resource object are as you expected.
v If the resource object is still not displayed in the view, compare the values in
the ExceptionViewList field on the resource object to the
ExceptionViewName field on the view object.
Most likely, there is not a match.
4. Changing the ExceptionViewList at the class level does not trigger updates to
exception views, even though the change is made.
A message is written to the RODM log to inform you of this situation. Close
and reopen the view to see if a class level change caused any updates.
5. Ensure that the DisplayResourceType field for this resource object is defined
correctly.
If the DisplayResourceType field is incorrect, the resource object cannot be
displayed in the view. Different results are received, depending on the view
being open or closed.
For example, assume that you have an exception view with an
ExceptionViewName value of FAILURE. You create a resource object and
change the ExceptionViewList field of the resource object to FAILURE, but the
DisplayResourceType of this resource object is defined incorrectly and cannot
be displayed in the view:
v If the exception view is open, a message is written to the RODM log
indicating the failure of the update.
v If the exception view is closed when the view is opened, a message is
written to the RODM log and the workstation issues message DUI1700I
stating the view is not complete.
This problem is usually encountered when the agent provided a limited set of
network addresses for the real resource.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 205
v If a MAC address or IP address is shown and the Configuration-Parents
navigation did not display a correlated aggregate, report the problem to IBM
Software Support.
| This situation might occur when you are using a custom application, such as
| an application provided by a Tivoli partner. If this is the case, ask the
| application developer to enable support for the topology correlation function
| as described in the IBM Tivoli NetView for z/OS Resource Object Data Manager
| and GMFHS Programmer’s Guide.
This new or learned information can add to the information displayed in the Data2
field. Based upon default settings, it can also change the name or type of the
aggregate objects. You can change some of these defaults.
Refer to the IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS
Programmer’s Guide for more information.
Note: The only way to alter the displayed additional information in Data2 is to
have your systems administrator stop loading the topology correlation
function.
2. The Resource Type of the correlated aggregate can change, based upon the field
used to correlate objects to the aggregate.
v If the first field used for correlation is LAN MAC address, the initial Type is
LAN workstation aggregate.
v If the field used is IP Address, the initial Type will be IP System Aggregate.
v If correlation is by the (free-form) Correlater field, the Type will be Open
System Aggregate.
v If resources monitored by different agents are correlated (a cross-correlation),
the Type will be Open System Aggregate.
v If a topology feature creates aggregate objects of the above Resource Types,
subsequent topology correlation might alter the Type even if topology is not
collected from any other components.
The topology component affected by this is the Tivoli management region.
Refer to the IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS
Programmer’s Guide or the FLCSDM8 customization file for more information.
Note: For more information about topology correlation operations and how
you can customize them, see the IBM Tivoli NetView for z/OS User’s
Guide: NetView Management Console and the IBM Tivoli NetView for
z/OS Resource Object Data Manager and GMFHS Programmer’s Guide.
Chapter 11. Troubleshooting and Initial Diagnosis for NetView Management Console and GMFHS 207
The Topology Display Subsystem View Is Not Complete
The topology display subsystem view does not show the NETCONV connection,
GMFHS, RODM, and the RODM managers. This can happen when the
instrumentation on IBM Tivoli NetView for z/OS has not been enabled. To resolve
this problem, enable the instrumentation on IBM Tivoli NetView for z/OS that
populates this view. For more information, refer to the IBM Tivoli NetView for
z/OS Customization Guide.
Log Files
The following table lists where to look for messages related to the:
v Topology server
v Topology console
Table 117. Network Management Console Log Files
Problem Area Where to Look
topology server The ihserror.log, ihsmessage.log, and the ihsecped.log files are located in one of the
following directories:
v For Windows: %BINDIR%\TDS\server\log
v For UNIX: $BINDIR/TDS/server/log
The ihserror.log file contains messages that provide diagnostic information. If you
want to contact IBM Software Support for help, you might need to provide the
ihserror.log file. The ihsecped.log file contains messages from the cpebatch utility.
Start the topology server with the -b option to write additional information to the
ihsmessage.log file. You can obtain help for messages logged in the ihsmessage.log
file.
Note: The online help for the Command Profile Editor (CPE) is not available as
HTML.
1. Go to the appropriate directory on the NetView management console Console
workstation to locate the message:
v For Windows: \usr\local\Tivoli\bin\generic_unix\TDS\client\help
v For UNIX: /usr/local/Tivoli/bin/generic_unix/TDS/client/help
2. Use a tool such as grep to locate the message number.
v For Windows: Type grep IHS1006 *.html
v For UNIX: Type cat *.html | grep IHS006
3. Use a browser to display the located file.
Select Log to place all information in the log window. The log can be saved in a
file on the topology server.
Select Print to print all the information to STDOUT. This is usually a separate
command prompt window.
GMFHS logs information to these output logs in the form of protocol data units
(PDUs). The only PDU logged by GMFHS is a PDU38, which is also referred to as
a system error synopsis PDU. This PDU carries error message and trace
information if tracing has been enabled. Note that trace information can be
optionally logged, but error information is always logged. By default, trace
information is not logged. Remember this distinction when you determine which
type of output log to use.
Internal Trace Log: By default, PDU38 information is sent to the internal trace
log. Logging of data to this output log can be disabled by specifying the FILE,
YES, or GTF option of the PRINTPDU38 parameter and not additionally specifying
the NO or INTERNAL option of this statement. Similarly, it can be disabled by
specifying the FILE, YES, or GTF option of the PRINT keyword on the TRACE
command and not additionally specifying the NO or INTERNAL option.
The internal trace log is a wrapping log. This means that if the log is full,
additional entries overwrite the oldest entries in the log and the log has wrapped.
The internal trace log has a default size of 100 pages of storage, where a page is 4
KB of data.
You can control both the overall size of this log and the number of bytes for each
entry written to the log. The TRACEPAGES initialization parameter controls the
number of 4 KB pages in the log; the default value is 100. The TRACEBYTES
initialization parameter controls the number of bytes for each entry logged.
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS 211
When GMFHS is stopped, the data in the internal trace log is automatically
flushed to an output data set (unless the TRACEPAGES value is set to a value
other than the default of 100). The output data set is defined by the CNMT DD
statement in the GMFHS startup procedure (sample CNMSJH10). By default, this
data set is the job SYSOUT data set CNMT.
To flush the internal trace log before GMFHS stops, use the FLUSH option of the
GMFHS TRACE command. This command flushes the data to the output data set
defined by the CNMT DD statement and clear the PDU38 information from the
internal trace log.
Output Data Sets: Logging of PDU38 data to an output data set can be enabled
by specifying the FILE or YES option on the PRINTPDU38 parameter, or by
specifying the FILE or YES option on the PRINT keyword on the TRACE
command. The log data sets used with this type of logging are defined by DD
statements in the GMFHS startup procedure (CNMSJH10). Each GMFHS
component uses its own DD statement to specify the data set. The GMFHS
components and corresponding DD statements are as follows:
CNMC Network command manager (NETCMD)
CNMD Database server (DBSERVER)
CNME Event manager (EVENTMGR)
CNMF Network configuration manager (NETCON)
CNMI Inter-processor communication (IPC)
CNMM GMFHS main (control) task (MAINTASK)
CNMN RODM Collection Manager (RCMGR)
CNMO Operator interface manager (OPERIF)
CNMP IPC-RODM event manager (IRMGR)
CNMR Resource traits manager (RTMGR)
CNMS View status manager (VSTATMGR)
CNMT CNMT trace data set
CNMV View manager (VIEWMGR)
By default, each of these DD statements uses a job SYSOUT data set. Unlike the
internal trace log, output data sets are not wrapped. They continue to grow until
GMFHS is stopped. When GMFHS is started, the output data sets are cleared of
previous data and logging begins at the beginning of the data set.
If you are using this type of output logging to the default SYSOUT data set, be
aware that you are using spool space for each of the PDU38s logged. PDU38 error
information cannot be disabled, and over an extended period of execution GMFHS
can log a large number of error messages (including informational messages). If
tracing is enabled, the amount of PDU38 information that is logged in a short
amount of time can be very large. In general, do not enable output data set logging
to SYSOUT data sets with tracing enabled.
The internal trace log uses the SYSOUT data set CNMT. and the GMFHS
automatically flushes data to this data set when stopping. The TRACEPAGES value
is set to a default of 100. If you do not have tracing enabled, the internal trace log
does not fill up unless a large number of console messages are issued by GMFHS.
Only console messages are sent to the internal trace log if tracing is not enabled. To
If a sequential data set fills up with PDU38 information, output logging for that
component switches to the internal trace log if the internal trace log was not
previously enabled.
| If you are using the SYSOUT data sets and are using JES2, you can view output
| data while GMFHS is executing with ISPF as follows:
| 1. From ISPF, select System Display and Search Facility (SDSF).
| 2. Select the Display Active (DA) jobs option to display active jobs on your
| system. Find your GMFHS job.
| 3. Type a question mark (?) next to the GMFHS job. The panel displays the active
| SYSOUT files for that job.
| Note: This methodology works only with JES2; it does not work with JES3.
Generalized Trace Facility: The Generalized Trace Facility (GTF) can be used as
an output log for PDU38 data. To enable logging of PDU38 data to GTF, specify
the GTF option on the PRINTPDU38 parameter in member DUIGINIT, or specify
the GTF option on the TRACE command (PRINT parameter).
GMFHS uses GTF event ID X'5E2' for logging PDU38 data. If GTF output logging
is enabled, the GTF must be started. If it is not started, GMFHS issues error
messages DUI3985I and DUI3986E and routes any succeeding PDU38 information
to the internal trace log (if no other logging facility is active). When the GTF starts,
GMFHS issues error message DUI3987I and begins logging PDU38 information to
the GTF.
The Console Log window contains responses to commands. All responses are
displayed at the NetView operator console.
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS 213
The Console Log window can hold only 500 lines; so if many commands are sent,
some responses eventually wrap off the top of the Console Log window. This
number can be customized on the Console Properties window.
The Console Log window does not have to be active or visible for responses to be
inserted. All command responses are kept in a repository as they are received, and
are visible when you invoke the Console Log window from the NetView
management console Console.
GMFHS Trace
The GMFHS TRACE command controls the type and level of tracing done by
GMFHS subtasks. Trace entries are written to the task trace-print data sets or to the
generalized trace facility (GTF), depending on the setting of the PRINTPDU38
parameter in DUIGINIT.
If trace entries are being issued to the task trace-print data sets, they are written to
each subtask output DD member. The entries are written in time sequence within
each DD member.
If trace entries are being issued to the GTF, the trace entries are written in time
sequence.
Use the GMFHS TRACE command PRINT parameter to control whether trace
entries are made to the internal trace log, the task trace-print data sets, or to the
generalized trace facility (GTF). Although the PRINT parameter is listed as a trace
parameter, it also controls where error information is written for a specific subtask,
regardless of whether tracing for the subtask is enabled. Both error and trace
information flows to the location indicated by the PRINT parameter. The difference
is that you cannot stop the flow of error messages, but you can stop the tracing.
For example, if you set tracing off for the interprocess communication (IPC)
subtask and specify PRINT=FILE, and if console message DUI4024A is issued for
the IPC subtask, the error information associated with that message is written to
the task trace-print data set. Specify PRINT=GTF to send the error information to
the GTF, if it is active. To see the output, browse the GTF data set. You can also
format the GTF data set with the event identifier (EID) X'5E2'. Specify
PRINT=INTERNAL to send the information to the internal trace log.
See “Viewing the GMFHS Trace Online” on page 215 for more information about
the GMFHS TRACE command PRINT parameter and subtask output DD member
names.
Note: Tracing the view manager task (VIEWMGR) with a LEVEL value greater
than 50 generates large amounts of data and can severely degrade system
performance.
Figure 38. Example of Current Trace Settings Displayed by the GMFHS TRACE Command
Note: If you want to save the trace information internally to the in-storage trace
table, see “Using the GMFHS Internal Trace” on page 216.
Note: This methodology works only with JES2; it does not work with JES3.
1. From ISPF, select System Display and Search Facility (SDSF).
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS 215
2. Select the Display Active (DA) jobs option to display the active jobs on your
system. You are looking for your GMFHS job.
3. Enter a question mark (?) next to the GMFHS job. The panel displays the active
SYSOUT files for that job.
If you are tracing GMFHS with PRINT=YES or PRINT=FILE, depending on
which components you are tracing, GMFHS puts the component traces in the
following DD statements that are defined in the GMFHS JCL or PROC:
CNMC Network command manager (NETCMD)
CNMD Database server (DBSERVER)
CNME Event manager (EVENTMGR)
CNMF Network configuration manager (NETCON)
CNMI Interprocess communication (IPC)
CNMM Main task (MAINTASK)
CNMN RODM Collection Manager (RCMGR)
CNMO Operator interface (OPERIF)
CNMP IPC-RODM manager subtask (IRMGR)
CNMR Resource traits manager
CNMS CNMS status manager (VSTATMGR)
CNMT CNMT trace data set
CNMV View manager (VIEWMGR)
For example, if you are tracing NETCON and NETCMD, and CNMC and
CNMF are displayed after you enter a question mark next to the GMFHS job,
you can browse the CNMC and CNMF files.
To dump the contents of the in-storage trace table, issue the GMFHS TRACE
FLUSH command. The GMFHS TRACE FLUSH command writes the contents of
the trace table in the data set specified in the CNMT DDNAME in the GMFHS JCL
and reinitializes the in-storage trace table. To prevent data loss when you issue a
GMFHS TRACE FLUSH command, a new in-storage trace table of n pages is
allocated prior to printing and releasing the current table.
Chapter 12. Diagnostic Tools for NetView Management Console and GMFHS 217
218 Troubleshooting Guide
Part 4. Diagnosing RODM Problems
Chapter 13. RODM Problem Worksheet . . . . . . . . . . . . . . . . . . . . . . . . . 221
General Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
System Related Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
RODM Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
RODM Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
Problem Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Abend Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Message Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
Loop Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Wait Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
Incorrect Output Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Performance Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
Documentation Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Component ID:
4. Recommended service update (RSU) level:
RODM Applications
Record the following information:
1. Are you running GMFHS with RODM?
2. Are you running any other RODM applications?
3. Can you remove one or more RODM applications and re-create the problem?
RODM Methods
1. Are you running any user-written methods with RODM? If so, which ones?
2. Can you bypass these and successfully run the function you are attempting?
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
Problem Classification
Complete the following problem category that matches the symptoms associated
with your problem:
Abend Problems
For abends or processor exception problems, complete the following:
1. What is the abend code?
2. What processes were taking place at the time of the abend?
3. Gather the following documentation before contacting IBM Software Support:
v A copy of the network log
v A copy of the trace log
v The first unformatted dump of the abend
v A completed RODM problem worksheet
v A copy of the RODM log
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v The customization member (EKGCUST)
4. Gather the following information from the dump:
a. What was the program status word (PSW) at the time of the abend?
b. In what module did the abend occur?
c. What date was the module compiled?
d. What is the PTF level of the module pointed to by the abend?
e. What is the offset into the module pointed to by the PSW at the time of the
abend?
f. List the registers at the time of the abend.
Message Problems
For message problems, complete the following:
1. Record the message ID and any error codes displayed.
v Message ID:
v Does the message contain any return codes, reason codes, feedback codes,
error codes, or sense information? List the codes or information.
2. Use NetView online help for the message to determine user action.
3. What processes were taking place when the message occurred?
v Methods:
v RODM Load Utility:
v Other:
Loop Problems
For loop problems, complete the following:
1. What events led up to the loop?
2. What data was being displayed?
3. What was the last command entered?
4. If this is a method loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v A document describing the scenario leading to the problem
v A hard copy of the system log
v A hard copy of the network log
v A hard copy of the trace log
v The addresses of instructions within the loop
v A dump obtained by using the CPU RESTART function
v A copy of the RODM log
v The RODM checkpoint data sets (if applicable)
v The RODM loader input data sets and output listing (if applicable)
v The customization member (EKGCUST)
5. What are the modules involved in the loop?
6. What are the dates that the modules were compiled?
7. What are the PTF levels of the modules involved in the loop?
Wait Problems
For wait problems, complete the following:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
Performance Problems
For performance problems, complete the following:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the network log
v A copy of the RODM trace
v The customization member (EKGCUST)
v A copy of the RODM log containing log record type 8 lock and storage
statistics
Documentation Problems
For documentation problems, complete the following:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program, call IBM
Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
The return code with the reason codes described in this chapter are provided on
the assumption that you know the application failed and at least one of the
following has occurred:
v The application issues an error message.
v You receive incorrect output.
v The application abends.
v You discover a return code with reason code in the RODM log.
v The application reacts unexpectedly.
To use Table 118 on page 228 to locate examples of problems you might encounter
when using RODM, take the following steps:
1. Locate your problem scenario using the first two columns.
v Problem Category
Arranged alphabetically
v Problem Scenario
– Arranged (first) according to where the symptom shows
– (Then) arranged alphabetically
2. Go to the indicated page for a description of the problem and resolution steps
for correcting the problem. These steps might include the use of specific RODM
diagnostic tools or might refer you to other documentation.
3. Follow the resolution steps to correct your problem.
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems,” on page 9 and Chapter 3,
“Documenting and Reporting Problems,” on page 19 before contacting IBM
Software Support .
Debugging Methods
If you know that you have a problem with a user-written method, follow these
steps:
1. Unit-test the method. Create a dummy PL/I or C main procedure to call the
method and a dummy EKGMAPI module to dump all of the data that is
passed to RODM. Verify that the data you received or passed is correct.
2. If the method is intended to run synchronously, ensure that it does not violate
MVS cross-memory restrictions by issuing supervisor calls (SVCs).
3. If the method is intended to run asynchronously, use WTO instructions by way
of an assembler routine to examine program flow in the method.
4. Use the Output to Log (2008) MAPI function to write user-provided data to the
RODM log file. You can use log record types 1, 9, and 10 in the RODM log file
to trace your method execution.
5. Enable method tracing by setting log-level values as described under
“Log-Level Values” on page 244.
6. Code the method to set return and reason codes that indicate the execution
result.
Application Failure
Use this section for troubleshooting when you are running an application and the
application fails. For example, if you receive an error message, incorrect output, or
the application abended, RODM will write a return code and reason code to the
RODM log.
This problem can occur if the translation-window, checkpoint data set is too small.
To solve the problem, do the following:
This problem can occur if the total size of the data-window checkpoint data set is
too small.
1. If you have defined checkpoint data sets:
a. Take a checkpoint of RODM and end RODM.
b. Add an additional checkpoint data set to the RODM start JCL and warm
start RODM using the new data-window data set.
2. If the checkpoint data sets you have defined are not large enough, use the IBM
Tivoli NetView for z/OS Tuning Guide to compute the size of the data-window
data set.
3. If you have not defined checkpoint data sets, use the IBM Tivoli NetView for
z/OS Installation: Configuring Graphical Components to size the data-window data
set.
4. Add data-window data sets (to a maximum of 512) and warm start RODM.
5. If you cannot solve your problem, follow the instructions in Chapter 2,
“Classifying Problems,” on page 9 and Chapter 3, “Documenting and Reporting
Problems,” on page 19 before contacting IBM Software Support .
Note: You can also find this information in the RODM dump.
11. If the type 7 log record indicates the following abends, perform the steps
listed:
Abend 0C1
a. Subtract the value in the base register from the value in register 14 to find
the offset of the instruction following the branch and link register (BALR)
instruction.
b. Add this to the offset of the control section (CSECT) entry point as
determined from the compiled listing to determine the effective offset of
the instruction following the BALR instruction.
Note:
If you receive a return/reason code of 12/213, look for the first pointer
that was logged as X'FFFFFFFF', or look for an address that is not valid.
5. Correct your application using the corrective action information listed for that
return code and reason code.
6. Review the RODM log listing for error records associated with the transaction
in error.
Depending on the severity of the error, you might need to modify the value of
the EKG_LogLevel parameter to ensure that all transactions get logged.
7. If you cannot solve your problem, follow the instructions in Chapter 2,
“Classifying Problems,” on page 9 and Chapter 3, “Documenting and Reporting
Problems,” on page 19 before contacting IBM Software Support.
To solve this problem, replace the method that is exhausting the Language
Environment for z/OS storage.
To solve this problem, gather the dump (and all other associated problem
information) and contact IBM Software Support .
Even though you choose '3' to cancel the request, RODM and possibly the SNA
topology manager and GMFHS abend.
CPU utilization does not change, except for the task in the wait state, which does
not use the CPU.
The following resolution steps apply to all of the symptom sets described above.
1. End RODM.
Note: All data is lost because the checkpoint function is disabled because of the
checkpoint error.
2. Correctly specify all DD names and data set names of the checkpoint data sets
in the start JCL.
3. Change the suspect checkpoint data sets in the start JCL, or ensure that all
checkpoint data sets specified in the start JCL are error-free.
4. Increase the region size of the RODM program.
5. Replace the damaged checkpoint data sets.
6. If a set of checkpoint data sets from a previous successful checkpoint exists,
warm start RODM using those checkpoint data sets.
Otherwise, cold-start RODM.
Attention: When you cold-start RODM, the checkpoint data sets are
reinitialized and RODM starts with an empty data cache.
7. If you cannot solve your problem, follow the instructions in Chapter 2,
“Classifying Problems,” on page 9 and Chapter 3, “Documenting and Reporting
Problems,” on page 19 before contacting IBM Software Support .
These reactions can happen if a method destroys RODM control blocks by using
incorrect pointers or by passing function blocks that are not valid.
Attention: Methods that run in the RODM address space are APF-authorized.
Allow enough time for paging to complete before doing the following:
1. Reduce the number of notification queues, as follows:
a. Remove all notification subscriptions that reference the notification queue
you want to delete.
b. Delete these queues using the Delete Object API function.
2. Increase the log level.
3. Adjust the lock parameters and reload the customization member using the
RELOAD command.
4. Use API calls at run time to reduce the depth of the vertical classes using the
Delete Class API function to delete classes.
User-supplied information can be written to the RODM log through the Output to
Log method application program interface (MAPI) function.
You can customize member EKGCUST to specify which log records to write to the
RODM log, or you can invoke an MAPI call from a RODM method to write
records to the RODM log. After customizing EKGCUST, you can use the MVS
MODIFY command to reload member EKGCUST or to query the current RODM
log file.
| Also, loss of read integrity results in down-level records and erroneous “no
| record found” conditions.
| The MVS MODIFY command with the LOGF option clears the local buffers
| and forces a CLOSE TYPE=T. This provides read integrity that is current up
| to the time the MODIFY command is issued.
| The MVS MODIFY command enables you to specify RODM logging options.
Log-Level Values
When an error occurs within RODM, you can review the RODM log listing for
error records associated with setting the transaction in error. Depending on the
Note: If you pass a FAIB, EAIB, FIELD NAME, CLASS NAME, or OBJECT NAME
pointer that is not valid, the pointer that is not valid is often logged as
X'FFFFFFFF' and the API will receive a return/reason code of 12/213.
If you receive a return/reason code of 12/213, look for the first pointer that was
logged as X'FFFFFFFF', or look for an address that is not valid.
Your application can also update log-level values. The default values are the values
of the log-level parameters specified in RODM customization member EKGCUST.
If the transaction return code is greater than or equal to the value of a log-level
parameter, RODM writes a log record. You can specify the following values:
Log Level Log Record Written for Transaction Codes
0–3 All
4–7 Warning, error, or severe
8–11 Error or severe
12–999 Severe only
Note: Do not use a log-level of zero (0). Log-level zero (0) logs all RODM API
requests. There is a potential for an auxiliary storage shortage to occur if
log-level zero (0) is used in a high-stress environment.
The rules for method tracing also determine which log records are written to the
RODM log and when they are output. The following fields are used as input to
RODM method tracing:
v EKG_MLogLevel in the associated user object initially set from MLOG_LEVEL in
customization member EKGCUST
v EKG_MTraceFlag in each method object
v EKG_MTraceType in the associated user object initially set from MTRACE_TYPE
in customization member EKGCUST
v EKG_LogLevel in the associated user object initially set from LOG_LEVEL in
customization member EKGCUST
Use these parameters in conjunction with the type of method that is triggered, as
well as the type of API request, to determine the log record that is to be written to
the RODM log.
The following log records are sent to the RODM log regardless of log-level settings:
v Log record type 0 (Log Version Record) is the log version record.
This is the first record written to the log file when you start RODM.
v Log record type 1 (Output to Log MAPI request) is sent to the RODM log when
a method invokes the Output to Log (2008) MAPI request.
v Log record type 5 (RODM system services failure) is sent to the RODM log
when RODM encounters a system services error.
The following log records are sent to the RODM log based on the value of
EKG_LogLevel:
v Log record type 2 (UAPI transaction request) is sent to the RODM log when the
return code from a UAPI function is greater than or equal to EKG_LogLevel
v Log record type 3 (object-specific method) and log record type 4
(object-independent method) are sent to the RODM log when the following
conditions exist:
– The Set Return/Reason Code (2006) API function is called from an
object-specific or object-independent method.
– The return code is successfully set in the MAPI function call.
– The return code is greater than or equal to EKG_LogLevel.
Log record type 9 (MAPI transaction request) is sent to the RODM log based on
EKG_MLogLevel. Log record type 9 is sent to the RODM log when the return code
from a method MAPI request is greater than or equal to EKG_MLogLevel and one
of the following conditions exists:
v EKG_MTraceFlag in the associated method object is ON.
v One of the following bits corresponding to the method type is ON in
EKG_MTraceType:
Bit Method Type
24 Object-deletion
25 Object-independent
26 Named
27 Notification
28 Change
29 Query
Note: For the EKG_MTraceType field, bits are numbered 0-31 from left to right,
where bit 0 is the leftmost bit and bit 31 is the rightmost bit.
EKG_MTraceType is a field on each user object. Its default value is the value of the
MTRACE_TYPE parameter specified in the RODM customization member
EKGCUST. EKG_MTraceFlag is a field on each method object. Its default is 0
(method tracing disabled).
Log record type 10 (method entry and exit) is sent to the RODM log when a
method is entered, the entry trace bit (bit 31) in EKG_MTraceType is ON, and one
of the following conditions exists:
v EKG_MTraceFlag in the associated method object is ON.
v One of the following bits corresponding to the method type is ON in
EKG_MTraceType:
Bit Method Type
25 Object-independent
26 Named
27 Notification
Log record type 10 (method entry and exit) is also sent to the RODM log when a
method is exited, the exit trace bit (bit 30) in EKG_MTraceType is ON, and one of
the following conditions exists:
v EKG_MTraceFlag in the associated method object is ON.
v One of the following bits corresponding to the method type is ON in
EKG_MTraceType:
Bit Method Type
25 Object-independent
26 Named
27 Notification
28 Change
29 Query
All examples of formatted log record entries are shown in uppercase letters, but
MVS output for the RODM program is originally in mixed case.
Samples EKGLG000 and EKGRLOG are installed with RODM through SMP/E.
You can use the following operands with the PARM keyword:
TYPE Specifies the entry types that are to be printed.
You can specify any digits in the range of 1–10. You cannot specify TYPE 0
because type 0 is always printed. If you omit this operand, all entries are
printed.
METHODN
Specifies the name of the method for which type 1 log entries are printed.
Only the entries for the specified methods are printed. The asterisk (*) wild
card character is valid only at the start and end of the string. The
exclamation mark (!) wild card character is not valid. You can specify a
maximum of 10 names.
If you specify METHODN and TYPE without specifying type 1 for TYPE,
type 1 printing is assumed.
NOHEADER
Specifies that the log entry headers are not included in the formatted log
output.
STIME
Specifies the start date and time of log records.
Records logged at and after this time will be included in the formatted log
output. The time the records are logged is local time.
The operands of the STIME keyword are specified %STIME MM/DD/YYYY
HH:MM:SS where:
MM Starting month
DD Starting day
YYYY Starting year. The short form of YY is also supported.
HH Starting hour
MM Starting minute
SS Starting second
If the STIME keyword is not specified, all log records will be formatted
based on the keywords that are specified.
Some operands of the STIME keyword can be omitted. See Table 120 on
page 249 for a list of operands that can be omitted and the default value
used.
ETIME
Specifies the end date and time of log records.
Records logged at and before this time will be included in the formatted
log output. The time the records are logged is local time.
The operands of the ETIME keyword are specified %ETIME MM/DD/YYYY
HH:MM:SS where:
MM Ending month
DD Ending day
If the ETIME keyword is omitted, all log records will be formatted based on the
keywords that are specified. Some operands of the ETIME keyword can be
omitted. Table 120 on page 249 lists the operands that can be omitted and the
default values:
Table 120. Default Values for STIME and ETIME Keyword Operands
Operand Default Value Used
MM/DD/YYYY The current date
YYYY The current year
YY The current year
HH:MM:SS (for STIME) 00:00:00
HH:MM:SS (for ETIME) 23:59:59
SS 00
Example of Coding the PARM Keyword: For this example of coding the PARM
keyword, assume that you want log records that meet the following criteria:
v Type 1, 3, and 9
v Method ABCDE and all methods that begin with FGH
v Entries from 1 P.M. on 05/02/2009 to 5 P.M. on 05/05/2009
EKGLG000 is the load module for the log formatter program. The STEPLIB data
sets contain EKGLG000 and the necessary run-time libraries that are related to
Language Environment for z/OS. The EKGLOG data set contains the unformatted
RODM log used as input to this program.
When you have the required data sets, do the following steps:
1. Specify your input parameters under the STEP1 EXEC statement.
2. Specify the input log file under the EKGLOG DD name in the JCL.
3. Specify the output file under the SYSPRINT DD name in the JCL.
When failing return codes are received from VSAM I/O functions, the _amrc
structure is accessed to help diagnose these errors. The _amrc structure, defined in
the C standard I/O header file, contains diagnostic information returned by
VSAM. Some important fields are _amrc._code._feedback and
_amrc._code._feedback. The _rc field contains the VSAM R15, and the _fdbk field
contains the VSAM error code or reason code.
The following messages are issued when an unrecognized log type is encountered:
250 Troubleshooting Guide
UNKNOWN TYPE OF LOG RECORD
CANNOT FORMAT LOG SPECIFIC DATA
You can prevent this header from printing by specifying the NOHEADER
parameter on the PARM keyword in the EXEC statement:
PARM='%NOHEADER'
Following are descriptions of the fields in the formatted log record header:
RECORD NUMBER
Specifies the record number in the log file.
RECORD NUMBER is generated by the RODM log formatter and does not
map to an unformatted log record.
RBA Specifies the relative byte address (RBA) of the unformatted log type in
VSAM.
RBA is generated by the RODM log formatter and does not map to an
unformatted log record.
LOG_TYPE
Specifies the log record type, as follows:
Log Type Log Record Name
0 Log version record
1 Write-to-log MAPI log record
2 UAPI trace log record
3 Object-specific method log record
4 Object-independent method log record
5 RODM system services (SS) log record
6 Operator request log record
7 Abend log record
The Data Type field in the tables contains RODM abstract data types (for example,
Integer, Smallint, and TimeStamp).
The formatted log record sections contain examples of each log record after they
are formatted by the RODM log formatter. Each formatted log record contains a
primary header with data that is common to all of the log records. The log-type
specific fields are described following each example of the formatted log record.
Unformatted Log Record Type 0: Log record type 0 is the log version record. This
is the first record written to the log file when you start RODM. Log-level values do
not determine when RODM generates this log record.
Table 121 on page 253 provides descriptions of the fields, data types, and offsets in
log record type 0.
Table 121. Information in Unformatted Log Record Type 0
Field Description Data Type Decimal Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
RODM name Char(8) 040 X'28'
RODM version level Integer 048 X'30'
RODM release level Integer 052 X'34'
RODM point release level Integer 056 X'38'
Log file DD name Char(8) 060 X'3C'
Name of data set containing log file Char(44) 068 X'44'
Time conversion in hours Integer 112 X'70'
Time conversion in seconds Integer 116 X'74'
Notes:
1. The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
2. The time conversion fields indicate the time difference between local time and
Greenwich Mean Time (GMT). For example, for a time zone 4 hours west of
GMT, the time conversion in hours value is X'FFFFFFFC' and the time
conversion in seconds value is X'FFFFC7C0'
Each time field is a 4-byte signed integer. A positive value indicates a local time
zone east of Greenwich mean time (GMT), while a negative value indicates a
local time zone west of GMT.
Formatted Log Record Type 0: Log record type 0 is the log version record. This is
the first record written to the log file when you start RODM.
| DATE: 08/07/2009 N e t V i e w
| TIME: 17:31 Resource Object Data Manager
| Log Print Utility
| Log_type : 0 (Log Version Record) RBA : 0
| Record number : 1 Record Length : 120
| Transaction ID: 0000000000000000x Timestamp : Fri Aug 07 17:30:42 2009
| User Appl ID :
| API Version : 1
| RODM Name : RODMNAME
| RODM Version : 5
| RODM Release : 4
| RODM Point Rel: 0
| RODM Log DD : EKGLOGS
| RODM Log DSN : VSAMC540.EKGLOGS
| RODM GMT Value: -4
Unformatted Log Record Type 1: Log record type 1 is the write-to-log MAPI log
record. It records information about RODM that you can use to help debug
methods. To debug methods, you can issue an MAPI call to send the output from
the method to the RODM log. Ensure that enough information is provided in this
type of log record so that problems in methods can be isolated and diagnosed.
Log record types 9 and 10 also have information for diagnosing methods.
Log-level values do not determine when RODM generates this log record.
Table 122 on page 255 provides descriptions of the fields, data types, and offsets in
log record type 1:
Table 122. Information in Unformatted Log Record Type 1
Field Description Data Type Decimal Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
Method name MethodName 040 X'28'
Message CCSID Smallint 048 X'30'
User supplied data AnonymousVar 050 X'32'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 1: Log record type 1 is the write-to-log MAPI log
record. It records information about RODM that you can use to help debug
methods.
Log record type 1 also records GMFHS non-console error messages. Each message
describes the following items:
v Message number
v The message text
v An explanation of the message
v Whether a dump was taken:
None No dump was taken.
RODM A dump of the RODM address space was taken.
GMFHS A dump of GMFHS was taken.
Both A dump of the RODM address space and GMFHS was taken.
Figure 45 on page 256 is an example of log record type 1 that has been formatted
by the RODM log formatter:
Unformatted Log Record Type 2: Log record type 2 is the UAPI trace log record.
You can use log record type 2 to help debug applications. If the return code of a
UAPI transaction is greater than or equal to EKG_LogLevel, the related
information is written to the RODM log file.
After you set the proper log-level (the default for EKG_LogLevel is 8), the selected
type 2 log records (output from RODM) are written to the RODM log after each
transaction. You can check the return code, the reason code, and the function block
contents in the log record.
The Function_Block portion of the RODM log record is dependent on the type of
function being run. Any data being pointed to is resolved if RODM already knows
the value of the data. If the length value for data being pointed to is zero or if a
field pointer is zero, no field data is contained in the log record. The following
restrictions also apply:
Table 123 on page 257 provides descriptions of the fields, data types, and offsets in
log record type 2.
Table 123. Information in Unformatted Log Record Type 2
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
Secondary Header:
Return code Integer 040 X'28'
Reason code Integer 044 X'2C'
Function block Anonymous 048 X'30'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
EAIB
Object Name
Class Name
Object Name Pointer
Object Name Length
Object ID
Class Name Pointer
Class Name Length
Class ID
Naming Count
Reserved
FAIB
Unformatted Log Record Type 3: Log Record Type 3 is the object-specific method
log record. You can use log record type 3 to trace an object-specific method.
An object-specific method can issue a MAPI call to set return and reason codes.
This causes the program to pass the return and reason codes back to the caller. If
the specified return code is greater than or equal to the value of EKG_LogLevel, a
record is placed in the RODM Log.
Table 124 on page 260 provides descriptions of the fields, data types, and offsets in
log record type 3:
Table 124. Information in Unformatted Log Record Type 3
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
Return code Integer 040 X'28'
Reason code Integer 044 X'2C'
Function Integer 048 X'30'
Class ClassID 052 X'34'
Object ObjectID 056 X'38'
Field FieldID 064 X'40'
Subfield Smallint 068 X'44'
Method name MethodName 070 X'46'
Short lived parm SelfDefining 078 X'4E'
Long lived parm SelfDefining 078+n X'4E'+n
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 3: Log Record Type 3 is the object-specific method
log record. You can use log record type 3 to trace an object-specific method.
Figure 50 on page 261 is an example of log record type 3 that has been formatted
by the RODM log formatter:
An object-independent method can issue an MAPI call to set return and reason
codes. This causes the program to pass the return and reason codes back to the
caller. If the specified return code is greater than or equal to EKG_LogLevel, a
record is placed in the RODM Log.
Table 125 on page 262 provides descriptions of the fields, data types, and offsets in
log record type 4:
Table 125. Information in Unformatted Log Record Type 4
Field Description Data Type Decimal Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
Return code Integer 040 X'28'
Reason code Integer 044 X'2C'
Function Integer 048 X'30'
Method name MethodName 052 X'34'
Short lived parm SelfDefining 060 X'3C'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Unformatted Log Record Type 5: Log record type 5 is the RODM system services
(SS) log record. You can use log record RODM type 5 to track operating system
macros.
Log record type 5 contains information for the operating system macros that fail
during the transactions shown in the Transaction ID field.
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 5: Log record type 5 is the RODM system services
(SS) log record. You can use log record type 5 to track operating system macros.
Figure 54 is an example of log record type 5 that has been formatted by the RODM
log formatter:
Unformatted Log Record Type 6: Log Record Type 6 is the operator request log
record. It is written to the log file after the operator has completed a successful
action, such as starting RODM or using the MVS MODIFY command.
A bit setting for the type of operator action is on if the condition is true.
Log-level values do not determine when RODM generates this log record.
Table 127 provides descriptions of the fields, data types, and offsets in log record
type 6.
Table 127. Information in Unformatted Log Record Type 6
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Total record length Integer 000 X'0'
Log type Smallint 004 X'4'
Reserved Smallint 006 X'6'
Time stamp TimeStamp 008 X'8'
Transaction ID TransID 016 X'10'
User application ID ApplicationID 024 X'18'
API version Integer 032 X'20'
Reserved Integer 036 X'24'
Last checkpoint transaction ID TransID 040 X'28'
Bit setting for: MVS START commandChar(1) 048 X'30'
warm start is X'80'
MVS START command cold start is X'40'
MVS MODIFY command checkpoint request is X'10'
MVS MODIFY command termination request is X'10'
NOTE: Lower 4 bits reserved
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 6: Log Record Type 6 is the operator request log
record. It is written to the log file after the operator has completed a successful
action, such as starting RODM or using the MODIFY command.
Figure 56 on page 266 shows an example of log record type 6 that has been
formatted by the RODM log formatter:
Unformatted Log Record Type 7: Log record type 7 is the abend log record.
During operation, RODM might encounter error conditions that are recorded. If an
abend condition occurs, a type 7 log record (abend log record) is written to the
RODM log. The type 7 log record indicates the name of the abend module and
system diagnostic work area (SDWA) information.
Note: Only the first 56 bytes of data are described in the log record header.
Table 128 on page 267 provides descriptions of the fields, data types, and offsets in
log record type 7.
Table 128. Information in Unformatted Log Record Type 7
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Integer 000 X'0'
Total record length Smallint 004 X'4'
Log type Smallint 006 X'6'
Reserved TimeStamp 008 X'8'
Time stamp TransID 016 X'10'
Transaction ID ApplicationID 024 X'18'
User application ID Integer 032 X'20'
API version Integer 036 X'24'
Reserved
Return code set by MVS at abend Integer 040 X'28'
Reason code set by MVS at abend Integer 044 X'2C'
Abend module name Char(8) 048 X'30'
SDWA INFO DATAAREA 056 X'38'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 7: Log record type 7 is the abend log record.
Figure 58 on page 268 is an example of log record type 7 that has been formatted
by the RODM log formatter:
Unformatted Log Record Type 8: Log record type 8 is the statistics log record.
The type 8 log record is a table with statistical information about each RODM cell
pool stored as an entry in that table. The table can have multiple RODM cell pool
entries.
This log record is written to the RODM log when you issue the MVS MODIFY
command with the STATCELL parameter. Log-level values do not determine when
RODM generates this log record.
The statistics log record lists the status of cell pool usage for segments or windows,
and the lock word usage. The format of the log record is different for each
log_type_flag as follows:
Flag Meaning
0 For cell pool usage information for segments. Table 130 on page 271
describes the cell pool usage information for segments.
1 For cell pool usage information for windows. Table 130 on page 271
describes the cell pool usage information for windows.
5 For API statistics. Table 131 on page 272 describes API statistics.
Table 129 on page 271 provides descriptions of the fields, data types, and offsets in
log record type 8.
0 = Cell pool usage information for segments. Table 130 on page 271 provides
descriptions of cell pool usage information for segments.
1 = Cell pool usage information for windows. Table 130 on page 271 provides
descriptions of cell pool usage information for windows found in log record type
8.
5 = API statistics. Table 131 on page 272 describes information for API statistics.
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
The number of entries in log record type 8 specifies how many cell pools are
printed in the type 8 log record. Statistical information for each cell pool contains:
v Cell size
v Pool size
v Number of cells in use
v High water mark
v Percentage of cells in use
v Total percentage of cells in use
v Percentage of high water
v Segment histogram counter (eight integer fields)
Table 130. Log_type_flag=0 or 1: Cell Pool Usage Information for Segments and Windows
Field Description Data Type Decimal Offset Hex Offset
Current pocket count Integer 044 X'2C'
Available pocket active count Integer 048 X'30'
Table 131 provides descriptions of API statistics found in the unformatted log
record type 8.
Table 131. Log_type_flag=5: API Statistics
Field Description Data Type Decimal Offset Hex Offset
Last Clear Time stamp TimeStamp 044 X'2C'
Output Time stamp TimeStamp 052 X'34'
Number of query methods triggered Integer 056 X'38'
Number of change methods triggered Integer 060 X'3C'
Number of notification methods triggered Integer 064 X'40'
Number of object deletion methods triggered Integer 068 X'44'
Number of permanent entries (N1) Integer 072 X'48'
Number of regular entries (N2) Integer 076 X'4C'
*
Permanent function call identifier Integer 20 (N1-1)+4+76 (See note 1) See note 2
*
Total successful calls for permanent function ID Floating 20 (N1-1)+12+76 (See note 1) See note 2
through user API
Total successful calls for permanent function ID Floating 20*(N1-1)+20+76=x (See note See note 2
through method API 1)
Regular function call identifier Integer 20*(N2-1)+4+x (See note 1) See note 2
*
Total successful calls for regular function ID through Integer 20 (N2-1)+8+x (See note 1) See note 2
user API
Total unsuccessful calls for regular function ID through Integer 20*(N2-1)+12+x (See note 1) See note 2
user API
Total successful calls for regular function ID through Integer 20*(N2-1)+16+x (See note 1) See note 2
method API
Total unsuccessful calls for regular function ID through Integer 20*(N2-1)+20+x (See note 1) See note 2
method API
Notes:
1. Calculating Decimal Offsets for API Statistics:
N1 is equal to the value of the Number of Permanent Entries field. N2 is equal
to the value of the Number of Regular Entries field. x is equal to the value of
the Successful calls for permanent function ID through user API field.
Formatted Log Record Type 8: Log record type 8 is the statistics log record. The
type 8 log record is a table with statistical information about each RODM cell pool
stored as an entry in that table. It supplies segment and window statistics as well
as lock level statistics.
Segment and Window Statistics: Figure 60 on page 273 is an example of the output
from the RODM log formatter for log record type 8 segment and window statistics:
|
Figure 60. Formatted RODM Log Record Type 8 for Segment and Window Statistics
API Statistics: Figure 61 on page 275 is an example of the output from the RODM
log formatter for log record type 8 API statistics:
Figure 61. Formatted RODM Log Record Type 8 for API Statistics
Chapter 15. Diagnostic Tools for RODM 275
The following are descriptions of the fields in log record type 8:
STAT TYPE
Specifies that API statistics be gathered.
LAST CLEAR TIMESTAMP
Specifies the time when the regular data was cleared. The time displayed
in this field is one of the following times:
v The last time the MODIFY STATAPI CLEAR command was issued
v The last time RODM was cold-started
v The last time a checkpoint was taken, if that checkpoint was followed by
a warm start
OUTPUT TIMESTAMP
Specifies the time when the API statistics were output.
NO. OF QUERY TRIGGERED
Specifies the number of calls for query the methods triggered.
NO. OF CHANGE TRIGGERED
Specifies the number of calls for change the methods triggered.
NO. OF NOTIFY TRIGGERED
Specifies the number of calls for notification the methods triggered.
NO. OF PERMANENT ENTRIES
Specifies the number of different function identifiers that RODM keeps
track of and reports on in the “Permanent Count Data” section of the
formatted log record.
All the function identifiers and their counts are listed in the unformatted
log record.
However, in the formatted log record, any function identifiers that have a
total count of zero are not displayed.
PERMANENT COUNT DATA
Array of permanent data kept by RODM.
FUNCTION ID
Specifies the function ID of permanent data.
PERM UAPI COUNT
Specifies the number of calls through the user API with a return
code of zero (0) for the function ID.
PERM MAPI COUNT
Specifies the number of calls through the method API with a return
code of 0 for the function ID.
NO. OF REGULAR ENTRIES
Specifies the number of function identifiers that RODM keeps track of and
reports on in the “Regular Count Data” section of the formatted log record.
All the function identifiers and their counts are listed in the unformatted
log record.
However, in the formatted log record, any function identifiers that have a
total count of zero are not displayed.
REGULAR COUNT DATA
Array of regular data kept by RODM.
The data counters for the regular entries are cleared when:
Note: After a warm start, the counters for the API statistics are restored
from the last checkpoint before the warm start.
Unformatted Log Record Type 9: Log record type 9 is the MAPI trace log record.
You can use this log record to help debug a method.
Log record type 9 is written to the RODM log only if the return code of the
method API function is greater than or equal to EKG_MLogLevel.
Table 132 on page 278 provides descriptions of the fields, data types, and offsets in
log record type 9.
Table 132. Information in Unformatted Log Record Type 9
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Total record length Integer 000 X'0'
Log type Smallint 004 X'4'
Reserved Smallint 006 X'6'
Time stamp TimeStamp 008 X'8'
Transaction ID TransID 016 X'10'
User application ID ApplicationID 024 X'18'
API version Integer 032 X'20'
Reserved Integer 036 X'24'
Method name MethodName 040 X'28'
Return code Integer 048 X'30'
Reason code Integer 052 X'34'
Method type Char: 056 X'38'
C, I, N, O, Q, or X
C Specifies
change method
I Specifies
object-
independent
method
N Specifies named
method
O Specifies
object-deletion
method
Q Specifies query
method
X Specifies
notification
method
* Three reserved bytes Three bytes 057 X'39'
Function block 060 X'3C'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 9: Log record type 9 is the MAPI trace log record.
You can use this log record to help debug a method. Figure 63 on page 279 is an
example of log record type 9 that has been formatted by the RODM log formatter:
Unformatted Log Record Type 10: Log record type 10 is the method entry and
exit log record. You can use this log record to diagnose a method.
Figure 64 on page 280 shows an example of an unformatted log record type 10.
Table 133 on page 280 provides descriptions of the fields, data types, and offsets in
log record type 10:
Table 133. Information in Unformatted Log Record Type 10
Field Description Data Type Decimal Hex Offset
Offset
Primary Header:
Total record length Integer 000 X'0'
Log type Smallint 004 X'4'
Reserved Smallint 006 X'6'
Time stamp TimeStamp 008 X'8'
Transaction ID TransID 016 X'10'
User application ID ApplicationID 024 X'18'
API version Integer 032 X'20'
Reserved Integer 036 X'24'
Method name MethodName 040 X'28'
Return code Integer 048 X'30'
Reason code Integer 052 X'34'
Note: The time stamp is in modified Lilian time format. It is a 64-bit floating point
number that is the number of milliseconds since midnight October 14, 1582.
Formatted Log Record Type 10: Log record type 10 is the method entry and exit
log record. You can use this log record to help debug a method.
Figure 65 shows an example of log record type 10 that has been formatted by the
RODM log formatter.
This internal trace uses the MVS Component Trace facility and is controlled by the
MVS TRACE command. The trace is written to a wrap around table in a trace
dataspace (EKGTRDSP).
By default, the amount of virtual storage which is used to contain the trace data is
256K. More storage can be allocated (up to 1 gigabyte) by specifying the size on
the TRACE command in place of the ON operand. For example, to allocate one
megabyte for the trace data, enter:
TRACE CT,1M,COMP=rodmname
The size of the trace area cannot be altered when the trace has been started. You
must stop and restart the trace in order to change the size of the trace area.
After issuing the TRACE command to start the RODM internal trace, MVS will
issue a write-to-operator with reply (WTOR) message ITT006A to solicit trace
parameters:
*nn ITT006A SPECIFY OPERAND(S) FOR TRACE CT COMMAND.
Use the OPTIONS operand to designate which of several RODM trace events are
to be recorded. The events and their codes are:
Code Event
CC Console communications (commands received from the console and
messages issued to the console)
CF RODM module flow
ALL All implemented trace events
Note: The CF trace option has a severe performance impact on RODM and RODM
applications. Avoid activating a CF trace on a production RODM unless
absolutely necessary for problem diagnosis.
To deactivate the RODM internal trace, use the TRACE command with the OFF
operand:
TRACE CT,OFF,COMP=rodmname
When the trace is stopped, all recorded trace data is discarded. The trace is
automatically stopped when RODM ends.
If the MVS DUMP command is used to dump RODM and the trace data is
required, you must include the name of the trace dataspace in the list of
dataspaces to be dumped. You can spool the trace data to a data set.
You can use the RODM dump utility to generate five types of reports to print the
contents and structure of classes and objects. The reports include the following
items:
v Class listing
v Class index
v Object listing
v Object index
v Statistical report
The contents of the output depend on the input parameters. The input parameters
follow the SYSIN DD * control statement. You can provide the SYSIN DD
statement as instream values or in a data set.
If you provide a class name, the reports begin at the requested class. Otherwise,
the reports begin at the highest class level, which is the universal class. If you
enter an object name, only the object and class are printed. It is implied that the
requested object belongs to the requested class. If the requested object does not
belong to the requested class, an error is indicated.
If you do not enter an object name, the utility prints all objects and classes
subordinate to the requested or default class. The statistical report is generated to
show the types and numbers of user API queries issued against RODM when
running the dump utility.
EKGDUMP is the compiled and link-edited dump utility program. The STEPLIB
data set contains the load modules of the RODM dump utility and the EKGUAPI
module. The SYSIN data set contains the control statement input.
The CLASSES, CLASSNDX, OBJECTS, and OBJECNDX data sets are the reports
produced by the utility. These data sets contain the DCB operands of LRECL=133
and RECFM=FBA. BLKSIZE can be provided on the DD statements or in the
SYSOUT data sets and is handled by JES.
The SYSPRINT data set contains a statistical report indicating the number and type
of user API calls made while the dump utility is running. This data set contains
the DCB operands of LRECL=80 and RECFM=FBA.
When you have the required data sets, do the following actions:
v Specify your input parameters under the SYSIN DD * in the JCL.
v Specify the output report files.
v Specify your message output file under the SYSPRINT DD name in the JCL.
– If you do not specify this DD name, the messages are written to the JES log in
your MVS system.
– If you use a SYSOUT file under this DD name, the SYSOUT file is kept in the
held output queue in the MVS system where RODM is active.
Usage Notes: You can enter control parameters in any of the following formats:
v PARAMETER=value PARAMETER(value) PARAMETER value
v Blanks, equal signs, and parentheses are delimiters. The first
non-delimiter is the parameter and the next non-delimiter after
the parameter is the value. Anything after the delimiter ending the
value is ignored.
v Each of the control parameters is intended to be entered one time.
If you enter any parameter more than once, the last occurrence is
used.
RODMNAME=RODM1
APPLID=USER1
PASSWORD=USERPW
CLASS=CLASS1
OBJECT=OBJ
REPORT=NO
Following each of the subfield types is a 2-byte value. This value specifies the data
type in a hexadecimal format. To determine the data type, convert the hexadecimal
value to decimal.
Note:
Bit Meaning
0 Specifies the value subfield
1 Specifies the query subfield
2 Specifies the change subfield
3 Specifies the notification subfield
4 Specifies the previous value subfield
5 Specifies the TimeStamp subfield
6-31 Not used
Only the first six bits of the subfield map are used.
LOCAL COPY MAP
Specifies a bit map that indicates which of the subfields in the SUBFIELD
bit map have been locally defined and which have not.
RODM sets a local copy map bit in an output block to 1. This indicates
that the corresponding subfield contains locally defined data.
Bits that do not have a value of 0 indicate subfields that have values or
contents inherited from a parent class.
Valid values for the bit map are:
Bit Meaning
Where:
HH Specifies the hour.
MM Specifies the minutes.
SS.SSS
Specifies the number of seconds followed by decimal fractions of a
second.
This field is converted from modified Lilian time by the RODM log
formatter.
Statistical Report
The statistical report describes the types and number of user API queries issued
against RODM during execution of the dump utility. Figure 72 on page 292 is the
expected output from the RODM dump utility each time you request a report:
If you are using RACF, you need to use the PASSWORD parameter with the
APPLID.
The rodmname parameter is required to specify the particular RODM from which
the data cache is to be dumped.
Check the input specified after the SYSIN DD * statement for parameters or values
that are not valid.
An error was detected by RODM on the connect request from the dump utility.
This message is followed by message EKGDP009 which reflects the return and
reason codes from RODM.
An error was detected by RODM on the disconnect request from the dump utility.
This message is followed by message EKGDP009 which reflects the return and
reason codes from RODM.
This message supplies return and reason codes from RODM to further identify an
error when running the dump utility.
The dump utility attempted to find, through a RODM API request, a class that was
specified on the input parameter. Message EKGDP009 follows this message with
the return and reason codes from RODM.
The dump utility attempted to find, through a RODM API request, an object you
specified on the input parameter. EKGDP009 follows this message with the return
and reason codes from RODM.
The dump utility attempted a RODM API Query Structure of Entity request and
failed. Message EKGDP009 follows this message with the return and reason codes
from RODM.
The dump utility attempted a RODM API Query Field for Class Children request
and failed. EKGDP009 follows this message with the return and reason codes from
RODM.
The dump utility attempted a RODM API Query Field for Object Children request
and failed. Message EKGDP009 follows this message with the return and reason
codes from RODM.
The dump utility attempted a RODM API Query Structure of a Field request and
failed. Message EKGDP009 follows this message with the return and reason codes
from RODM.
The dump utility attempted a RODM API Query Subfield request and failed.
Message EKGDP009 follows this message with the return and reason codes from
RODM.
Online help is available for each message through the NetView program.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager . . . . . . . . . 307
Problems During Initialization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Wrong Autotask Error . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
Insufficient Storage for Topology Manager Initialization . . . . . . . . . . . . . . . . . . . 309
Error Reading Initialization File FLBSYSD . . . . . . . . . . . . . . . . . . . . . . . . 309
Error Reading Customization Table FLBOSIDS, FLBSRT, or FLBEXV . . . . . . . . . . . . . . . 309
Severe Error: Topology Manager Ends . . . . . . . . . . . . . . . . . . . . . . . . 310
Warning Error: Topology Manager Initialization Completes . . . . . . . . . . . . . . . . . 310
Cannot Connect to VTAM CMIP Services . . . . . . . . . . . . . . . . . . . . . . . . 311
Cannot Connect to RODM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312
FLBTOPO Task Abends . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
Abend During Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Abend After Initialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
User Abend . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
Abend Error Conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316
Topology Manager Unexpectedly Shuts Down . . . . . . . . . . . . . . . . . . . . . . . 317
Topology Manager Reinitializes Unexpectedly . . . . . . . . . . . . . . . . . . . . . . . 318
Topology Manager Seems to Be Suspended . . . . . . . . . . . . . . . . . . . . . . . . 319
Automatic Monitoring Is Failing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320
SNA Topology Manager Cannot Receive Agent Node Topology Data . . . . . . . . . . . . . . . . 321
RODM Object Missing Some Attribute Values . . . . . . . . . . . . . . . . . . . . . . . 323
Resources Are Not Shown in the Views . . . . . . . . . . . . . . . . . . . . . . . . . 324
Monitor Operation Unexpectedly Ends . . . . . . . . . . . . . . . . . . . . . . . . . . 326
Monitor Operation Ended Normally. . . . . . . . . . . . . . . . . . . . . . . . . . 329
Topology Manager or VTAM CMIP Services Was Ended by an Operator . . . . . . . . . . . . . 329
Topology Manager or VTAM CMIP Services Unexpectedly Ended . . . . . . . . . . . . . . . . 329
An Active VTAM Path or Session Became Inactive . . . . . . . . . . . . . . . . . . . . . 329
Monitor Operation Stopped Because of a Network Problem . . . . . . . . . . . . . . . . . . 330
Blank Status History for a Topology Manager Resource . . . . . . . . . . . . . . . . . . . . 330
A Locate Resource Request Does Not Find the Resource . . . . . . . . . . . . . . . . . . . . 331
Cannot Activate, Deactivate, or Recycle a Resource . . . . . . . . . . . . . . . . . . . . . . 331
Network Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Generic Commands Fail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
Incorrect NetView Management Console Command Profiles . . . . . . . . . . . . . . . . . . . 332
Objects Are Not Purged . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
Objects Unexpectedly Purged . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
The Resource Status Is Unknown . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339
The Resource Is a Node Other than an APPN Network Node . . . . . . . . . . . . . . . . . 340
Resource Is an APPN Network Node . . . . . . . . . . . . . . . . . . . . . . . . . 340
Resource Is a TG That Does Not Connect Two Network Nodes. . . . . . . . . . . . . . . . . 341
Resource Is a TG That Connects Two Network Nodes . . . . . . . . . . . . . . . . . . . . 341
Chapter 18. Diagnostic Tools for the SNA Topology Manager . . . . . . . . . . . . . . . . . 363
SNA Topology Manager Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
SNA Topology Manager Log Entries. . . . . . . . . . . . . . . . . . . . . . . . . . . 364
SNA Topology Manager Log Record Formats. . . . . . . . . . . . . . . . . . . . . . . 365
System Interface Log Entries-Major Code 22 . . . . . . . . . . . . . . . . . . . . . . . 367
SNA Topology Manager Log Entries—Major Code 78 . . . . . . . . . . . . . . . . . . . . 373
Internal RODM Error Code Indicator . . . . . . . . . . . . . . . . . . . . . . . . 396
Internal RODM Class Indicator . . . . . . . . . . . . . . . . . . . . . . . . . . 398
Common Log Entries-Major Code 79 . . . . . . . . . . . . . . . . . . . . . . . . . 399
SNA Topology Manager Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400
External Tracing (GTF) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
GTF Trace Record Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401
GTF Trace Record Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
Tracing Internally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Internal Trace Buffer Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Internal Trace Record Header . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
Trace Events. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405
4000-0002 (CENT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 406
4000-0003 (CEXT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407
4000-0008 (LOGS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
4002-0007 (MSGS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409
4003-000E (CMIP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410
4004-0019 (RTIB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 412
4004-001A (RARY) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 413
4005-0015 (RCLS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414
4005-0016 (RON) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
4005-0017 (ROBJ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416
4005-0018 (RATR) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417
4007-001E (UPDT) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418
4008-0000 (GET) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
4008-0001 (FREE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420
4009-0006 (FSM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 421
400A-0004 (NEW) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
400A-0005 (DEL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
Some SNA topology manager problems can be caused by communications
problems. Use the diagnosis procedures described in the VTAM library to gather
information about problems with VTAM CMIP services.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Component ID:
4. NetView Version and Release:
5. Recommended service update (RSU) level:
6. NetView function modifier ID (FMID):
Problem Information
Gather the following documentation before contacting IBM Software Support. Use
the diagnosis procedures described in the z/OS Communications Server SNA
Diagnosis manuals to gather information about problems with VTAM CMIP
services. For information about VTAM CMIP services, see the z/OS Communications
Server CMIP Services and Topology Agent Guide.
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See Chapter 6, “Diagnostic Tools for the
NetView Program,” on page 73.
v A copy of the system log.
v A copy of the NetView HLL remote interactive debugger (RID) trace logs. See
the IBM Tivoli NetView for z/OS Programming: PL/I and C for more information
about using RID.
v A completed SNA topology manager problem worksheet.
v The RODM START job control language.
v The customization file used to start RODM.
v The GMFHS data model and resource definition files. Refer to the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide
for information about the definitions and their associated files.
v The SNA topology manager data model and resource definition files. Refer to
the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation Guide for
information about the definitions and their associated files.
v The customization file used to start the SNA topology manager.
v The IHSERROR.LOG and IHSERROR.BAK files. See Chapter 12, “Diagnostic
Tools for NetView Management Console and GMFHS,” on page 209.
v The application trace log.
v RODM log records. See Chapter 15, “Diagnostic Tools for RODM,” on page 243
on how to capture this data.
v A dump of the RODM dataspaces. See Chapter 15, “Diagnostic Tools for
RODM,” on page 243 for information about capturing this data.
v The GMFHS output log and trace print data set. See Chapter 11,
“Troubleshooting and Initial Diagnosis for NetView Management Console and
GMFHS,” on page 175.
v Copy of any trace information created using the TOPOSNA TRACE command.
See “SNA Topology Manager Traces” on page 400 for more information about
the trace information provided by the SNA topology manager.
Problem Classification
Check one of the following appropriate problem categories that matches the
symptoms associated with your problem.
Abend Problems
For abends or processor exception problems, complete the following questions:
1. What is the abend code?
2. What processes were taking place at the time of the abend?
3. The user abend codes are described in the online help facility (type HELP
ABEND and use the scroll function to locate the abend code). The system
abend codes are documented in the IBM z/OS library.
4. Gather the following documentation before contacting IBM Software Support:
v The first unformatted dump of the abend.
5. Gather the following information from the dump:
a. What is the program status word (PSW) at the time of the abend?
b. In which module did the abend occur? See “SNA Topology Manager” on
page 11.
c. When was the module compiled?
d. What is the PTF level of the module pointed to by the abend?
e. What is the offset into the module pointed to by the PSW at the time of the
abend?
f. List the registers at the time of the abend.
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID:
v The exact text of the message on the log.
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands:
v NetView management console commands:
v Other:
4. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
5. Did the message text differ from what was published?
Loop Problems
For loop problems, complete the following questions:
1. What events led up to the loop?
2. What data was being displayed?
3. What was the last command entered?
4. If this is an enabled loop, collect the information discussed in “Documenting
LOOP Problems” on page 33.
v After obtaining a console dump, close the NetView program with a dump
(use the NetView CLOSE DUMP command).
Note: If the loop is still occurring after the NetView program has been
canceled, this is not an SNA topology manager problem.
5. If this is a disabled loop, collect the information discussed in “Documenting
LOOP Problems” on page 33.
v A scenario describing the events leading to the problem.
v The addresses of instructions within the loop.
v A dump obtained by using the CPU RESTART function.
6. What are the modules involved in the loop?
7. What are the dates that the modules were compiled?
8. What are the PTF levels of the modules involved in the loop?
Wait Problems
For wait problems, complete the following questions:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. Gather the following documentation before calling IBM Software Support:
v A copy of your VTAM resource definitions. Refer to the z/OS Communications
Server SNA Resource Definition Reference for information about where these
definitions are located.
v A copy of the system console dump.
5. What is the name of the module in which the wait occurred?
6. What is the date that the module was compiled?
7. What is the PTF level of the module involved?
8. What is the offset into the module where the wait occurred?
Performance Problems
For performance problems, complete the following questions:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Gather the following documentation before calling IBM Software Support:
v A copy of your VTAM resource definitions. Refer to the z/OS Communications
Server SNA Resource Definition Reference for information about the location of
these definitions.
v Copies of the agent node configurations.
v Copies of the agent node topology data.
v Descriptions of any modifications to your system.
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the SNA topology manager, call
IBM Software Support
5. If the problem is with an online help panel, call IBM Software Support.
If you are unable to solve your problem by using the examples in this chapter,
refer to the examples in the following documents:
v For the NetView program, some examples are described in Chapter 5,
“Troubleshooting and Initial Diagnosis for the NetView Program,” on page 51.
v For the Graphic Monitor Facility host subsystem (GMFHS), some examples are
described in Chapter 11, “Troubleshooting and Initial Diagnosis for NetView
Management Console and GMFHS,” on page 175.
v For the Resource Object Data Manager (RODM), some examples are described in
Chapter 14, “Troubleshooting and Initial Diagnosis for RODM,” on page 227.
For additional reference information about topology manager, see the IBM Tivoli
NetView for z/OS SNA Topology Manager Implementation Guide.
If you are still unable to solve your problem by using the references previously
listed, follow the instructions in Chapter 2, “Classifying Problems,” on page 9 and
Chapter 3, “Documenting and Reporting Problems,” on page 19 before contacting
IBM Software Support.
Table 134. SNA Topology Manager Problem Scenarios
Problem Category Problem Scenario Page
Abend FLBTOPO task abends (message DSI819I). 314
Hang Topology manager seems to be suspended (message FLB486I). 319
Initialization Cannot connect to RODM (messages FLB482E, FLB483W, and 312
FLB485E).
Cannot connect to VTAM CMIP services (message FLB677E or 311
FLB678E).
Error reading or processing customization table FLBOSIDS, FLBSRT or 309
FLBEXV.
Topology manager reinitializes unexpectedly. 318
Error reading or processing initialization file FLBSYSD. 309
Not enough storage (message FLB480E). 309
Wrong autotask (message FLB446E). 308
Followed by messages:
FLB442E SNA TOPOLOGY MANAGER IS SHUTTING DOWN BECAUSE OF AN ERROR
FLB443I SNA TOPOLOGY MANAGER SHUTDOWN IS COMPLETE
If a keyword is missing, a log entry is created with a major code of 78 and a minor
code of 36. A common reason for this is that FLBSYSD is down-level and therefore
can be missing new required keywords.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 309
Type of Error: Page:
Severe error 310
Warning error 310
Followed by messages:
FLB442E SNA TOPOLOGY MANAGER IS SHUTTING DOWN BECAUSE OF AN ERROR
FLB443I SNA TOPOLOGY MANAGER SHUTDOWN IS COMPLETE
The following messages are logged when the VTAM CMIP services are active, but
the attempt to connect fails:
FLB677E SNA TOPOLOGY MANAGER FAILED TO CONNECT TO CMIP SERVICES retcode retflag
Log entries are also created that provide more information about the exact error.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 311
1. Verify that the specifications in the FLBSYSD initialization file match the
TOPOMGR VTAM APPL definition statement.
The FLBSYSD specification for APPLNAME and APPLPASS must match the
TOPOMGR VTAM APPL specifications for ACBNAME and PRTCT,
respectively. For example:
TOPOMGR VTAM APPL statement:
TOPOMGR APPL ACBNAME=TOPOMGR,PRTCT=TOPOPASS
Log entries are also created that provide more detail on the exact error.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 313
This object is created in the RODM data cache by the SNA topology manager
loader file FLBTRDMA. The default name is SuperclusterView. The SNA
topology manager uses the name defined by the
SUPER_CLUSTER_VIEW_NAME keyword in the FLBSYSD initialization file to
reference this object.
The name in the FLBSYSD file must match the name of the object in the RODM
data cache.
If the names do not match, change the names defined in files FLBTRDMA and
FLBSYSD so that they match.
v To use the Network_View_Class object that is already defined in the RODM
data cache, verify that the name used in both files matches this name and
proceed to step 7.
v If you do not want to use the object that is already defined in the RODM
data cache, perform the following steps:
a. Create the object in the RODM data cache again using the new name.
b. Stop RODM and proceed to step 2 to restart RODM and reload the data
models.
c. Cold-start RODM to delete the existing Network_View_Class object, or
warm-start RODM using a checkpoint file that does not have a definition
for the object.
You can use the RODMView tool to verify that an object of the class
Network_View_Class object exists in the RODM data cache with the name
defined in the FLBSYSD file. Defining and customizing these values is
described in the IBM Tivoli NetView for z/OS SNA Topology Manager
Implementation Guide.
7. Restart the SNA topology manager.
An abend occurred in the SNA topology manager. The NetView program performs
an SVC dump. Usually, this indicates a software problem within the SNA topology
manager or an interface problem between the SNA topology manager and another
task.
There can be incorrect (or incomplete) objects in the RODM data cache. The SNA
topology manager is dependent on the objects that it references in the RODM data
cache and related objects being created correctly.
Abends Page
Abend during initialization 315
Abend after initialization 315
User abend 315
Abend error conditions 316
These rules and restrictions are described in the IBM Tivoli NetView for z/OS Data
Model Reference.
To determine whether the abend occurred during warm-start processing, check the
network log and determine whether message DSI819I follows message FLB402I,
and that message FLB440I is not logged. The following messages are issued, in the
order shown, if the abend occurred while working with the objects in the RODM
data cache:
FLB402I SNA TOPOLOGY MANAGER HAS BEGUN WARM-START PROCESSING
DSI819I NETVIEW IS DUMPING FOR TASK FLBTOPO. COMPLETION CODE= X'hhhhhh',
DOMAIN=domainid
These rules and restrictions are described in the IBM Tivoli NetView for z/OS Data
Model Reference.
To determine whether the abend occurred after the SNA topology manager
initialized, check the network log and determine whether message DSI819I follows
message FLB440I. The following messages are issued, in the order shown, if the
abend occurred after the SNA topology manager was initialized:
FLB440I SNA TOPOLOGY MANAGER INITIALIZATION IS COMPLETE
DSI819I NETVIEW IS DUMPING FOR TASK FLBTOPO. COMPLETION CODE= X'hhhhhh',
DOMAIN=domainid
User Abend
For enhanced serviceability of the SNA topology manager, a user abend can be
initiated to dump the FLBTOPO autotask for diagnostic purposes. Whether this
user abend is taken depends on the setting of the ABEND_AND_DUMP parameter
in the FLBSYSD initialization file. The default setting is YES.
When the SNA topology manager detects a severe processing error condition, it
checks the setting of the ABEND_AND_DUMP parameter in FLBSYSD:
v If it is set to YES, the following message is issued:
FLB694E SNA TOPOLOGY MANAGER DETECTED A SEVERE ERROR CONDITION,
ABEND X'abendcode' TAKEN FOR FLBTOPO TASK, PROBE probeid
The NetView address space is dumped and the SNA topology manager abends
with a user abend. Use the online help facility (type HELP ABEND and use the
scroll function to locate the abend code).
v If set to NO, the following message is issued:
FLB693E SNA TOPOLOGY MANAGER DETECTED A SEVERE ERROR CONDITION,
BUT A STORAGE DUMP WAS NOT REQUESTED, PROBE probeid
ABEND CODE X'abendcode'
The user abend is not initiated; the SNA topology manager initiates shutdown
and logoff.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 315
Abend Error Conditions
Most incorrect object data, such as incorrect attribute values, do not cause the SNA
topology manager to abend. However, if SNA topology manager objects are linked
to incorrect objects, or if some of the objects and links required by an object are not
created, the SNA topology manager can abend, depending on the severity of the
problem. These error conditions can be caused by the following conditions:
v The SNA topology manager abended while it was creating an object in the
RODM data cache, and the SNA topology manager was restarted without
restarting RODM.
v The SNA topology manager was stopped by an operator issuing a command
other than TOPOSNA STOPMGR, while the SNA topology manager was
creating an object in the RODM data cache, and the SNA topology manager was
restarted without restarting RODM.
v An operator took a checkpoint of RODM while the SNA topology manager was
creating an object in the RODM data cache, and the SNA topology manager was
warm-started after warm-starting RODM with the checkpoint data.
v A user created a SNA topology manager object in the RODM data cache without
setting the correct links to other objects or without creating the other required
objects.
When you have verified that the topology data is valid, it is a good idea to
archive previous versions of your checkpoint data sets.
If the SNA topology manager is not processing updates that require the
creation or deletion of objects within the RODM data cache, try to
checkpoint the RODM data cache . Updates that only change the status of
an object do not usually cause problems.
Wait until the network is stable (no other resources are being added to, or
removed from, the network) or temporarily stop all monitor operations.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 317
FLB481E SNA TOPOLOGY MANAGER DISCOVERED THAT RODM
'rodmname' IS TERMINATING/QUIESCING
FLB442E SNA TOPOLOGY MANAGER IS SHUTTING DOWN BECAUSE OF AN ERROR
FLB443I SNA TOPOLOGY MANAGER SHUTDOWN IS COMPLETE
Review the NetView and system log. If an operator has ended the RODM
task, warm-start the RODM again and then start the SNA topology manager
FLBTOPO task to resume the previous SNA topology manager state.
v Another probable cause is that the SNA topology manager encountered an
unrecoverable error while referring to data in the RODM data cache.
Usually, the SNA topology manager creates one or more log entries
describing the error and logs the following messages:
FLB482E SNA TOPOLOGY MANAGER ENCOUNTERED AN UNRECOVERABLE ERROR ON
A CALL TO RODM 'rodmname'
FLB442E SNA TOPOLOGY MANAGER IS SHUTTING DOWN BECAUSE OF AN ERROR
FLB443I SNA TOPOLOGY MANAGER SHUTDOWN IS COMPLETE
FLB610I TASK FLBTOPO IS STARTING LOGOFF PROCESSING
FLB611I TASK FLBTOPO HAS COMPLETED ITS LOGOFF PROCESSING
Use the information in the associated log entries to diagnose and correct the
problem. Log entries are described in “SNA Topology Manager Log Record
Formats” on page 365.
v A fourth probable cause is the SNA topology manager cannot allocate
enough storage.
In this case, the SNA topology manager creates a log entry (78-0) indicating
the storage allocation problem:
FLB442E SNA TOPOLOGY MANAGER IS SHUTTING DOWN BECAUSE OF AN ERROR
FLB443I SNA TOPOLOGY MANAGER SHUTDOWN IS COMPLETE
FLB610I TASK FLBTOPO IS STARTING LOGOFF PROCESSING
FLB611I TASK FLBTOPO HAS COMPLETED ITS LOGOFF PROCESSING
Use the information in the associated log entries to diagnose and correct the
problem. The log entries are described in “SNA Topology Manager Log
Record Formats” on page 365.
If the cause is not one of the above, the information provided in the log entry,
along with the description of the log entries, enables you to determine the
cause of the error and provide solutions. The log entries are described in “SNA
Topology Manager Log Record Formats” on page 365.
2. After solving the problem, restart the SNA topology manager.
The SNA topology manager goes through the following steps during
reinitialization:
The SNA topology manager logs the following message when it begins the wait for
the RODM checkpoint to complete:
FLB486I SNA TOPOLOGY MANAGER CALLED RODM 'rodmname'
DURING A RODM CHECKPOINT AND WILL RETRY
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 319
To solve the problem, do the following steps:
1. Determine whether a RODM checkpoint is in progress.
v If a RODM checkpoint is in progress, wait for it to complete. The SNA
topology manager accepts new commands and updates as soon as RODM
resumes accepting updates. If the RODM checkpoint hangs, the SNA
topology manager:
– Does not process any new commands or updates
– Seems to be suspended
Correct the RODM hang condition, and the SNA topology manager resumes
processing new commands and updates. The diagnostic procedures for
RODM are described in Chapter 14, “Troubleshooting and Initial Diagnosis
for RODM,” on page 227.
v If a RODM checkpoint is not in progress, see “Documenting WAIT
Problems” on page 37.
The SNA topology manager does not process commands, including the
TOPOSNA STOPMGR command, while it is waiting for RODM to complete
a checkpoint. RODM checkpoints can take a significant amount of time,
depending on the amount of data in RODM.
2. Determine whether the SNA topology manager is collecting initial topology
data for a network, local or LU collection request.
If the SNA topology manager is collecting initial topology data for a network
topology, local topology, or LU collection (LUCOL) request from a VTAM
topology agent, the SNA topology manager might seem to be suspended.
After one or more of these topology requests, the topology agent sends
multiple buffers to the SNA topology manager. These buffers are queued until
the last initial transfer complete signal is sent by the topology agent. The SNA
topology manager starts processing these buffers and creates the objects in the
RODM data cache.
Upon completion, an initial transfer complete message is issued. The time
required for completion of this process depends on the number of objects
reported by the agent.
Verify that the SNA topology manager is actively processing topology data by
using the TASKUTIL FLBTOPO command to check the number of messages
queued to the FLBTOPO task.
If the number of messages queued is high but changing, it is an indication that
the SNA topology manager is processing data.
If the number of messages queued has not changed for a long period of time, it
is a good indication that the SNA topology manager is suspended.
The TASKMON command can be used to check all performance statistics for
FLBTOPO. The CPU usage and message queueing statistics might also be
indicators of task activity or task suspension conditions.
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide for a description of a monitor operation.
The following messages are issued when the SNA topology manager cannot
monitor the network or local topology of a node:
v For network topology:
FLB403I REQUESTED MONITORING OF SNA NETWORK TOPOLOGY FROM
NODE nodename
FLB409W MONITORING OF SNA NETWORK TOPOLOGY FROM NODE nodename WILL
BE RETRIED
FLB685W NO ACTIVE PATH TO NODE nodename OR CMIP SERVICES NOT
ACTIVE ON THIS NODE OR INCORRECT NODE NAME
v For local topology:
FLB420I REQUESTED MONITORING OF SNA LOCAL TOPOLOGY FROM
NODE nodename
FLB426W MONITORING OF SNA LOCAL TOPOLOGY FROM NODE nodename WILL
BE RETRIED
FLB685W NO ACTIVE PATH TO NODE nodename OR CMIP SERVICES NOT
ACTIVE ON THIS NODE OR INCORRECT NODE NAME
v For LU collection:
FLB540I REQUESTED MONITORING OF LU COLLECTION FROM nodename
FLB544W MONITORING OF THE LU COLLECTION FROM NODE nodename
WILL BE RETRIED
FLB685W NO ACTIVE PATH TO NODE nodename OR CMIP SERVICES NOT
ACTIVE ON THIS NODE OR INCORRECT NODE NAME
Most of these problems result in associated VTAM CMIP services log entries being
created or SNA topology manager messages being issued. Refer to the z/OS
Communications Server SNA Diagnosis manuals for more information about
diagnosing VTAM CMIP services problems. For information about VTAM CMIP
services, see the z/OS Communications Server CMIP Services and Topology Agent
Guide.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 321
To solve the problem, take the following actions:
1. Verify that you specified the correct node name when you issued the
TOPOSNA MONITOR command.
The SNA topology manager retries monitor operations to unknown nodes. The
retry procedures are the same as those for communication problems.
If the wrong node name was specified, issue a TOPOSNA STOP command for
the unknown node, and issue the TOPOSNA MONITOR command again using
a valid node name.
2. Verify that the VTAM topology agent is installed on this node and that the
VTAM CMIP services are active on this node.
Refer to the z/OS Communications Server CMIP Services and Topology Agent Guide
for information about installing the VTAM topology agent.
From VTAM you can enter the following commands:
D NET,VTAMOPS,OPT=OSIMGMT
or
D NET,VTAMOPS,OPTION=OSIMGMT
The expected message is IST1189I, indicating that the OSIMGMT option is YES
or NO. If NO, you can turn it on by entering:
F NET,VTAMOPTS,OSIMGMT=YES
For more information about VTAMOPTS, refer to z/OS Communications Server
SNA Operation.
3. Verify that the topology agent is active.
If not active, start the topology agent and CMIP services, and then proceed to
step 7. Refer to the z/OS Communications Server CMIP Services and Topology
Agent Guide for information about starting the VTAM topology agent.
4. Verify that the mainframe server can establish a session with the agent node.
If you are using the APPN function of VTAM, verify that the agent node can be
located by VTAM. This means:
v The agent node must be in the same APPN subnetwork as your VTAM node.
v The agent node must be in a subnetwork adjacent to the VTAM subnetwork
and the two subnetworks are connected by peripheral border nodes or
extended border nodes.
v The agent node must be in a subnetwork that is not adjacent to the VTAM
subnetwork and the subnetworks between the agent node subnetworks and
the VTAM subnetwork are connected by extended border nodes.
If the APPN function of VTAM is enabled and the agent node is explicitly
defined to VTAM, the definition requirements are the same as those in effect
when not using the APPN function of VTAM.
The SNA topology manager logs the sense code returned from VTAM when a
session cannot be established with the agent node. Refer to the z/OS
Communications Server SNA Diagnosis manuals for more information about
diagnosing VTAM problems. Also, see the z/OS Communications Server CMIP
Services and Topology Agent Guide.
Solve the network problem and proceed to step 7.
5. Verify that you are not trying to obtain network topology data from an APPN
end node.
The topology agent at an end node rejects requests for network topology. If
network topology information was requested from an end node, issue a
TOPOSNA STOP,NETWORK command for the end node.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 323
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide for more information.
4. Verify that the SNA topology manager supports the missing attribute.
The resources and attributes supported by the SNA topology manager are
described in the IBM Tivoli NetView for z/OS Data Model Reference.
The SNA topology manager discards all attributes it does not support. The first
time the SNA topology manager receives an unsupported attribute from an
agent, it creates an informational log entry in the network log, with a major
code of 78 and a minor code of 25.
See “SNA Topology Manager Log Record Formats” on page 365 for more
information about this log entry.
5. Verify that the topology agent reports the missing attribute.
All topology agents must report all mandatory attributes; otherwise, VTAM
CMIP services rejects the data received from the agent node and creates a log
entry identifying the data being discarded.
Some of the attributes supported by the SNA topology manager are optional,
and might not be reported by the installed topology agent. Refer to the
documentation for the topology agent installed at the agent node to determine
what attributes it reports.
The VTAM topology agent is described in the z/OS Communications Server CMIP
Services and Topology Agent Guide.
Another way to determine whether the topology agent is reporting the attribute
is to trace the information being received by the SNA topology manager from
the agent node. To trace the information, perform the following steps:
a. Enable the CMIP trace category, using the TOPOSNA TRACE command, to
trace all CMIP data received by the SNA topology manager.
b. Locate the replies received from the node reporting the resource with the
missing attribute to determine whether the attribute is being reported by
the agent node.
The NetView online help facility describes how to use the TOPOSNA TRACE
command. The format of the trace records is described in GTF Trace Record
Format. The format of the CMIP-linked replies received by the SNA topology
manager is described in the IBM SystemView library.
If the missing resource is a node resource, another possibility is that the node
might not be in any views to which the NetView management console operator
can navigate.
1. Verify that the SNA topology manager is monitoring the topology of the agent
node that is reporting the missing resource.
Use the TOPOSNA LISTREQS command to determine which nodes are being
monitored and the type of topology being monitored for each node.
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide and to the NetView online help facility for more information about using
the TOPOSNA LISTREQS command.
2. Verify that the SNA topology manager is monitoring the correct type of
topology. Many resources are reported only when the local topology of the
owning node of a resource is monitored:
v The resources and attributes reported by network and local topology are
described in the IBM SystemView library.
v The resources and attributes supported by the SNA topology manager are
described in the IBM Tivoli NetView for z/OS Data Model Reference.
3. Verify that an active path exists between the node owning the resource and a
node being monitored by the SNA topology manager.
The SNA topology manager ignores updates for a resource if an active path
does not exist between the node and any of the nodes being monitored by the
SNA topology manager. The resource still exists, but the attribute values are
out-of-date (see “RODM Object Missing Some Attribute Values” on page 323)
and the status of the resource is set to unknown (see “The Resource Status Is
Unknown” on page 339).
The resource is not created by the SNA topology manager if the following
conditions are true:
v The resource is not reported by the local topology of any node.
v An active path of CP-CP sessions does not exist between any nodes adjacent
to the node owning the resource and any of the nodes whose network
topology is being monitored by the SNA topology manager.
“The Resource Status Is Unknown” on page 339 describes how the SNA
topology manager determines if an active path exists for each class of resource.
4. Verify that the object was not purged by the SNA topology manager. See
“Objects Unexpectedly Purged” on page 335.
5. Verify that the topology agent reported the missing resource.
All topology agents must report all mandatory attributes; otherwise, VTAM
CMIP services rejects the data received from the agent node and creates a log
entry identifying the data being discarded.
Some of the attributes supported by the SNA topology manager are optional,
and might not be reported by the installed topology agent. Refer to the
documentation for the topology agent installed at the agent node to determine
what resources it reports.
The VTAM topology agent is described in the z/OS Communications Server CMIP
Services and Topology Agent Guide.
Another way to determine whether the topology agent is reporting the attribute
is to trace the information being received (by the SNA topology manager from
the agent node). To trace the information, perform the following steps:
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 325
a. Using the TOPOSNA TRACE command, enable the CMIP trace category to
trace all CMIP data received by the SNA topology manager.
b. Locate the replies received from the node to determine whether the resource
is being reported by the agent node.
The IBM Tivoli NetView for z/OS SNA Topology Manager Implementation Guide
describes how to use the TOPOSNA TRACE command.
The format of the trace records is described in “GTF Trace Record Format” on
page 401. The format of the CMIP-linked replies received by the SNA topology
manager are described in the IBM SystemView library.
Note: VTAM CMIP traces can be used to collect the same information. Refer to
the z/OS Communications Server CMIP Services and Topology Agent Guide
for more information.
6. Resources obtained from monitoring the local topology of a node might not be
displayed in any views to which the NetView management console operator
can navigate.
The navigation views display all network nodes obtained from monitoring
network topology, along with all nodes adjacent to the network nodes. When a
node can be displayed, all resources owned by that node can also be displayed.
A problem results when the node owning the resource (including the node
itself) is not displayed. The resource might not be displayed if you request the
local topology of a node and you are not monitoring the network topology of
the subnetwork the node belongs to, or if the node is not adjacent to a network
node reported by the network topology. In either case, the node is created in
the RODM data cache, and can be found and displayed by using the locate
resource function.
To locate a resource, specify the DisplayResourceName of the resource. Use the
configuration parents option. The Any View with the Resource option does not
find the resource because it does not search the SNA topology manager views.
Using the locate resource function and formatting of the display resource
names for all resources (objects) created by the SNA topology manager are
described in the IBM Tivoli NetView for z/OS SNA Topology Manager
Implementation Guide. Also, refer to the IBM Tivoli NetView for z/OS Data Model
Reference for more information about display resource names.
To solve the problem, check with the other operators to determine whether the
monitor operation must be restarted.
Check with the operator who stopped the task as to the reason for ending the task.
Restart the SNA topology manager, and VTAM CMIP services if necessary, and
restart the monitor operation.
When the VTAM CMIP services ends, the following message is issued:
FLB684E SNA TOPOLOGY MANAGER DISCOVERED THAT CMIP SERVICES IS TERMINATING
Use the descriptions of the messages and log entries to diagnose and correct the
problem. Restart the SNA topology manager, and VTAM CMIP services if
necessary, and restart the monitor operation.
For example, an active CDRM to the remote VTAM became inactive or was
deactivated by an operator, or if this remote VTAM had a session through an
NCP-NCP connection, the link between two NCPs became inactive or was
deactivated by an operator.
To solve the problem, determine whether the CDRM and the link to this node are
active. If not, activate the CDRM, the link to this node, or both. When the session
is active, the SNA topology manager resumes the monitor operation providing the
monitor request retry is still in effect.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 329
Monitor Operation Stopped Because of a Network Problem
The SNA topology manager begins retrying the monitor operation, using the retry
values specified by the TOPOSNA SETDEFS command. If it is a temporary
network problem, the SNA topology manager will probably be able to restart the
monitor operation. Some network problems can require operator intervention to
solve.
Refer to the z/OS Communications Server SNA Diagnosis manuals for more
information about diagnosing VTAM CMIP services problems. For information
about VTAM CMIP services, see the z/OS Communications Server CMIP Services and
Topology Agent Guide.
If you solve the problem before the SNA topology manager exhausts its retries, the
monitor operation is restarted the next time the SNA topology manager tries to
restart the monitor operation.
The amount of time until the next retry can be very long, depending on the values
specified by the TOPOSNA SETDEFS command. To determine the amount of time
until the SNA topology manager retries the operation, use the TOPOSNA
LISTREQS command. If the time period until the next retry is too long, issue the
TOPOSNA MONITOR command again; the SNA topology manager retries the
operation immediately.
If the SNA topology manager has exhausted its retries before the problem is
solved, the monitor operation is ended. Issue the TOPOSNA MONITOR command
again to restart the monitor operation. The SNA topology manager sends the
following message to the operator that started the monitor operation, and logs it in
the network log, when the retries for a monitor operation are exhausted:
v For network topology:
FLB462E MONITORING OF SNA NETWORK TOPOLOGY FROM NODE nodename
FAILED ALL RETRIES
v For local topology:
FLB463E MONITORING OF SNA LOCAL TOPOLOGY FROM NODE nodename
FAILED ALL RETRIES
v For LU collection:
FLB545E MONITORING OF THE LU COLLECTION FROM NODE nodename
FAILED ALL RETRIES
If the error is an unrecoverable error, the SNA topology manager does not retry the
operation and the monitor operation is ended. Issue the TOPOSNA MONITOR
command again to restart the monitor operation.
The SNA topology manager commands are described in the NetView online help
facility. Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager
Implementation Guide for more information about monitoring topology information.
Network Problems
Most of the problems are related to communications problems or setup problems at
the agent node. Refer to the z/OS Communications Server SNA Diagnosis manuals for
more information about diagnosing VTAM CMIP services problems. For
information about VTAM CMIP services, see the z/OS Communications Server CMIP
Services and Topology Agent Guide.
To solve the problem, check the CommandIndicator field as defined in the SNA
topology data model to ensure it has the correct value for the resource selected.
Refer to the IBM Tivoli NetView for z/OS Data Model Reference for more information
about the CommandIndicator field and the valid values for each object.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 331
If the wrong value is filled in by the SNA topology manager, contact IBM Software
Support and provide the value, the object name, and the object class.
The steps needed to modify the NetView management console command profiles,
and add customized commands are described in the IBM Tivoli NetView for
z/OS User’s Guide: NetView Management Console.
The SNA topology manager can purge an object from the RODM data cache when
any of the following is true:
v An update is received from a topology agent indicating that the object no longer
exists.
v An update is received that indicates that a link is now associated with another
TG.
v An update is received for a node that specifies a new node type (class) for a
node. The old node in the RODM data cache is deleted and replaced with a new
node object in the correct class.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 333
between the nodes and the network nodes being monitored. “The Resource
Status Is Unknown” on page 339 describes what monitor operations provide
information about a resource.
4. Verify that the resource has not been updated within the specified time period.
Verify again that the resource is not being reported by any active monitor
operations. See steps 2 and 3.
If the resource is not being monitored and is not being purged, the status of the
object has probably been updated within the time period specified by the purge
operation. The PURGDAYS parameter of the TOPOSNA PURGE command and
the PURGDAYS keyword in the FLBSYSD file (used during SNA topology
manager initialization) specify the maximum age for resources.
Any resource that has not been updated within the specified time period is
purged (if it is not being monitored). Objects are not purged if the SNA
topology manager has received an update for the resource within the specified
time period.
To determine when a resource was last updated, query the TIMESTAMP
subfield of the states field of the object. Query the RODM data cache directly or
query the information by requesting more information about the object from an
NetView management console workstation. If the time stamp is within the
period of time specified by the purge operation, the object is not purged.
Either decrease the time period and issue the purge operation again, or issue
the purge operation again with the same values when the object is old enough
to be purged.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 335
Determine whether one of the previous scenarios explains why the resource was
purged. If none of them are applicable, check for GMFHS errors or problems at
the NetView management console workstation.
To resolve the problem, do the following steps:
1. The resource was purged by a TOPOSNA PURGE command.
The TOPOSNA PURGE command purges resources (the objects used to
represent the resources) from the RODM data cache. When the object is
removed from the RODM data cache, it is also deleted from any NetView
management console views it is in.
To determine whether an object was purged, look to see if it still exists in the
RODM data cache. You can use the RODMView function to check the existence
of the object.
The object is purged by the TOPOSNA PURGE command because it is not
being monitored and the number of days since its last update exceeds the
number of days specified by the PURGDAYS parameter. Refer to the IBM Tivoli
NetView for z/OS SNA Topology Manager Implementation Guide for more
information about purging objects.
If the object still exists in the network and if you want to restore it to your
NetView management console views, proceed to step 7. Otherwise, no action is
required.
2. The resource was purged when the SNA topology manager was warm-started.
The SNA topology manager purges all resources that have not received updates
within the amount of time specified by the PURGDAYS keyword in the
FLBSYSD file.
The processing is similar to that performed when a TOPOSNA PURGE
command is issued:
v The resources (the objects used to represent the resources) are removed from
the RODM data cache.
v When the object is removed from the RODM data cache, it is also deleted
from any NetView management console views where it appears.
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide for more information about purging objects.
To determine whether an object was purged, verify whether it still exists in the
RODM data cache. You can use the RODMView function to check the existence
of the object.
If the object still exists in the network and you want to restore it to your
NetView management console views, proceed to step 7. Otherwise, no action is
required.
3. The resource was purged when the SNA topology manager was cold-started.
The SNA topology manager purges all resources that have not received updates
within the amount of time specified by the PURGDAYS keyword in the
FLBSYSD file.
The SNA topology manager is cold-started when this value is set to zero; in
other words, all objects created by the SNA topology manager are purged.
The processing is similar to that performed when a TOPOSNA PURGE
command is issued:
v The resources (the objects used to represent the resources) are removed from
the RODM data cache.
v When the object is removed from the RODM data cache, it is also deleted
from any NetView management console views where it appears.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 337
This happens when a back level NCP (t4Node) is known to a VTAM host
contact only, meaning the NCP has never been activated but VTAM knows
about the NCP.
VTAM reports this NCP by its subarea number (NETA.00000097), and the SNA
topology manager creates this t4Node in RODM as NETA.00000097.
When the NCP is activated, VTAM reports this NCP by name (NETA.NCP1)
and also reports its subarea information.
This indicates to the SNA topology manager that NETA.00000097 is really
NETA.NCP1; therefore, the SNA topology manager deletes node
NETA.00000097 from RODM and creates node NETA.NCP1 and message
FLB690I is logged.
The same scenario is possible for a t5Node.
6. A topology agent sent an update deleting a resource.
Transmission groups (TGs) associated with dynamically created links are
deleted when the underlying link is deactivated. TGs are also deleted when the
link associated with the TG is assigned to another TG.
Objects that are members of a VTAM topology agent’s definition group are
deleted when the definition group is deactivated.
If the resource is still defined at the agent node, it was not deleted. If the object
still exists in the network and you want to restore it to your NetView
management console views, proceed to step 7. Otherwise, no action is required.
7. If the object still exists, restore it to the views by monitoring the correct
topology.
Do one of the following to create an object again that has been deleted in the
RODM data cache and in the NetView management console views:
v For any resource, monitor the local topology of the node owning the
resource.
If the local topology of the node is already being monitored or the resource
is not restored when the monitor operation is started, the resource is no
longer defined at that node.
v If the resource is a node, monitor the local topology of a node adjacent to the
missing node.
If the local topology of the node is already being monitored or the resource
is not restored when the monitor operation is started, the missing node is not
connected to the node being monitored or the connection is inactive.
v If the resource is a network node or a TG between two network nodes,
monitor the network topology of any network node in the same subnetwork.
If the network topology of the subnetwork is already being monitored or the
resource is not restored when the monitor operation is started, one of the
following is true:
– The resource no longer exists.
– The resource is no longer a part of the network topology of the
subnetwork.
A network node might have been changed to an end node, removing the
node and any TGs to the node from the network topology of the
subnetwork.
– An active path of CP-CP sessions does not exist between the network
nodes being monitored and any of the network nodes adjacent to the
missing node.
Try monitoring the network topology of the missing node (or for TGs the
node owning the TG) or one of the nodes adjacent to the missing node.
Attention: Starting RODM with checkpoint data removes all data created
by the SNA topology manager since the checkpoint data set was created
from the RODM data cache. To rebuild this data, issue monitor operations
for all missing resources. User-created objects and objects that no longer exist
in the network that are not in the checkpoint data cannot be restored without
the objects being explicitly created by the user.
For more information about reporting the status of resources, refer to the IBM
Tivoli NetView for z/OS SNA Topology Manager Implementation Guide.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 339
Table 136. Resource Status Is Unknown (continued)
A resource has a status of unknown because it is: Page:
A link 342
A port 342
An interchange node or a migration data host 342
A t5Node 343
To solve the problem, monitor the local topology of the node or the local topology
of one or more of its adjacent nodes. The status of the node remains unknown if the
local topology of the node is not monitored, and there are no links active between
the node and any of the adjacent nodes being monitored.
The local topology conditions are the same as any other node in the network.
APPN network nodes can also be reported by monitoring the network topology of
any other network node in the subnetwork because APPN propagates the status of
all network nodes in a subnetwork to all other network nodes in that subnetwork.
An active path of CP-CP sessions must exist between the node and the network
node reporting the status of the node using network topology because the network
nodes use CP-CP sessions to propagate the status of the network nodes throughout
the network.
If an active path does not exist, the information being reported for the node is not
considered reliable because more recent updates are not being received by the
network node reporting the status.
To solve this problem, monitor the local topology of the node owning the TG.
The local topology conditions are the same as any other TG in the network.
TGs between network nodes can also be reported by monitoring the network
topology of any other network node in the subnetwork because APPN propagates
the status of all TGs between network nodes in a subnetwork to all other network
nodes in that subnetwork.
An active path of CP-CP sessions must exist between the node owning the TG and
the network node being monitored because the network nodes use CP-CP sessions
to propagate the status of the TGs throughout the network.
If an active path does not exist, the information being reported for the TG is not
considered reliable because more recent updates are not being received by the
network node reporting the status.
To solve the problem, monitor the local topology of the node owning the TG or the
network topology of one or more network nodes in the same subnetwork as the
node owning the TG.
The status of the TG remains unknown if all of the following are true:
v The local topology of the node owning the TG is not monitored.
v A path of active CP-CP sessions does not exist between the node owning the TG
and any of the network nodes in the subnetwork whose network topology is
being monitored.
Note: All TGs between network nodes are placed in the network topology
database. This includes TGs that do not support CP-CP sessions. Some TGs
that are defined to connect two network nodes might not be defined in the
network topology database until the link associated with the TG is activated.
Resource Is a TG Circuit
The status of TG circuits is derived from the status of the underlying TGs. If the
status of both TGs is unknown, the status of the TG circuit is also unknown.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 341
If the SNA topology manager knows only about one of the TGs associated with the
TG circuit, the status of the TG circuit matches the status of the TG.
Refer to “Resource Is a TG That Does Not Connect Two Network Nodes” on page
341 and “Resource Is a TG That Connects Two Network Nodes” on page 341 for
information about obtaining the status of the TGs associated with the TG circuit.
Obtaining the status of a TG associated with a TG circuit updates the status of the
circuit.
Resource Is a Link
Links are reported only by monitoring the local topology of the node owning the
link.
To solve the problem, monitor the local topology of the node owning the link.
Resource Is a Port
Links are reported only by monitoring the local topology of the node owning the
port.
To solve this problem, monitor the local topology of the node owning the port.
The subarea side is considered active when there is an active CDRM to this node
from the reporting VTAM agent. The APPN side is considered active when the
node is reachable or there is an active APPN connection from the reporting VTAM
agent.
If WORST is specified and one side (subarea or APPN) is not active, the status of
the node is unknown. Use the WORST status specification to determine whether
either side (APPN or subarea) of a node is not active.
When only LUCOL topology is requested, the node is created as an snaNode, and
the status is set to unknown.
To solve the problem, change the specification in the FLBSYSD file and restart the
SNA topology manager or actively monitor the node from both sides (subarea and
APPN).
When only LUCOL topology is requested, the node is created as an snaNode, and
the status is set to unknown.
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide for more information.
The most common reasons why the status of the resource is not set to the expected
value are:
v The SNA topology manager or agent node encounters a problem while
processing the resource control command.
See “Cannot Activate, Deactivate, or Recycle a Resource” on page 331 for a
description of the failures that can occur, along with suggested solutions.
v The status of the resource is not changed by resource control commands.
v The SNA topology manager is not receiving the status update for the resource.
v The SNA topology manager is not aware of a configuration change in the
network.
v Mapping of the OSI status and states to the DisplayStatus of a resource is
incorrect in the FLBOSIDS table.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 343
v Mapping of OSI status for the resolved status of a multiply-owned resource is
incorrect in FLBSRT table.
For more information about processing updates and the interpretation of the status
of resources, refer to the IBM Tivoli NetView for z/OS SNA Topology Manager
Implementation Guide.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 345
different adjacent node). The old transmission group (the one previously
associated with the TG) no longer exists.
APPN does not delete resources from the network topology, explicitly. It deletes
resources if an update is not received for the resource within a set period of
time (usually 15 days). Even after the time limit, the resource can still be in the
network topology database (see “Unexpected Resources Are Displayed” on
page 356).
Therefore, the status of the TG reflects its status in the network topology
database (probably unknown or unsatisfactory). The status of the link reflects the
status received from the topology agent. As soon as the link is activated, the
SNA topology manager determines that the link is associated with a new TG,
and no longer shows the link as being associated with the old TG.
Eventually the old TG is purged from the topology databases in the network.
Issue the TOPOSNA PURGE command to delete the old TG from the NetView
management console views. The transmission group is deleted when the
TOPOSNA PURGE command is issued, but only if APPN has purged the TG
from the network topology databases in the network.
Refer to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation
Guide for more information about purging resources.
6. The customized status of a resource is incorrect.
Check the NetView log for the following messages:
FLB660W SNA TOPOLOGY MANAGER ENCOUNTERED AN INCLUDE ERROR
'code' IN CUSTOMIZATION TABLE table
WITH ENTRY 'record'
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 347
The nnDomainNetwork object is used to represent the status of the session
routing capability of the APPN subnetwork. The aggregate objects that
contribute to the aggregation status of the nnDomainNetwork object are as
follows:
– All nnDomain objects in the APPN subnetwork
– All interDomain Circuit objects used to connect the nnDomain objects in the
subnetwork
v interDomainNetwork Circuits (all inter subnetwork links between two
subnetworks)
Intersubnetwork links are TG circuits used to connect border nodes in one
subnetwork to a network node or border node in another subnetwork. They are
used to route session requests between subnetworks. They are not TG circuits
that are used to provide APPN casual connections between a network node in
one subnetwork and an end node in another subnetwork. Also, they are not TG
circuits used for LEN connections between nodes in two subnetworks.
The resources that contribute to the aggregation status of the
interDomainNetwork Circuit object are all the interDomain Circuit objects that
connect nnDomain objects in different subnetworks.
v nnDomainNetworkCluster object (APPN network)
The nnDomainNetworkCluster object is used to represent the status of the
session routing capability of the entire APPN network. The aggregate objects
that contribute to the aggregation status of the nnDomainNetworkCluster object
are:
– All nnDomainNetwork objects
– All interDomainNetwork Circuit objects
These objects are only aggregated into the status of the
nnDomainNetworkCluster object if the AGGREGATE_TO_CLUSTER=YES is
specified in the FLBSYSD file.
Table 137. Aggregate Resource Status
Aggregate resource status is not being updated because: Page:
Status of the aggregate resource is unknown. 348
Status of the aggregate resource is known, but incorrect. 349
Some of the resources displayed when you request more detail of an aggregate
object with unknown status might have a status value other than unknown. Some
of the resources shown in the generated view might not aggregate their status into
the status of the aggregate object.
For example, an interDomain Circuit object displays all TG circuits between two
network nodes. If none of the TG circuits support CP-CP sessions, the status of the
interDomain Circuit object is unknown, even though the status of the TG circuits
might be different.
Some of the resources displayed when you request more detail of an aggregate
object might not aggregate their status into the status of the aggregate object. The
aggregate status of the aggregate object is set using only the status of the objects
that contribute to the aggregation of that object.
The setting of the status of the aggregate object also depends on the settings of the
aggregation thresholds for that object.
If you suspect the status to be incorrect, investigate the settings of the aggregate
thresholds for the object in question. You might have set the aggregation
thresholds to values that conflict with one another.
For example, the unknown aggregation threshold level is set to 100%. This means
that the status of the aggregate threshold is set only to unknown when the status of
all aggregated objects is unknown. The degraded aggregation threshold is set to 2,
which means the status of the aggregate threshold is set to degraded when the
status of two or more of the aggregated resources is unsatisfactory. If two resources
aggregate to the resource, one with unknown status and the other with
unsatisfactory status, a conflict arises, because the current values for the underlying
resources do not fit any of the aggregation thresholds defined for the object.
Views of topology objects are created when you request more detail for a SNA
topology manager object.
In most cases, the object is an aggregate resource. The objects shown include all
objects represented by the aggregate object, including those that do not aggregate
their status. The object can represent a real resource also, such as a node or TG
circuit object.
The views generated by the SNA topology manager, and the objects they contain,
are described in the IBM Tivoli NetView for z/OS SNA Topology Manager
Implementation Guide.
NetView management console views stay open until one of the following occurs:
v The NetView management console operator closes the view.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 349
v The object used to generate the view is purged from the RODM data cache.
v All of the objects in the view are purged.
See “Objects Unexpectedly Purged” on page 335 for more information about
purging and recovering objects.
Table 138. Displaying Topology Objects
Type of View: Page:
InterDomainNetworkCircuit 350
NnDomainNetwork 350
InterDomainCircuit View 351
NnDomain 351
SnaLocalTopology 352
Link-Port 352
Real Resource View 353
InterDomainNetworkCircuit View
An interDomainNetworkCircuit view is generated by requesting more detail of an
interDomainNetworkCircuit object. The interDomainNetworkCircuit object is used
to represent a intersubnetwork link between two APPN subnetworks.
They are not TG circuits used to provide APPN casual connections between a
network node in one subnetwork and an end node in another subnetwork. Also,
they are not TG circuits used for LEN connections between nodes in two
subnetworks.
The resources that are shown in this view are all the interDomainCircuit objects
that connect nnDomain objects in different subnetworks.
This view is not displayed if all the interDomainCircuit objects shown in the view
are purged. See “Objects Unexpectedly Purged” on page 335 for more information
about purging and recovering objects.
NnDomainNetwork View
An nnDomainNetwork view is generated by requesting more detail of an
nnDomainNetwork object. The nnDomainNetwork object is used to represent the
status of the session routing capability of the APPN subnetwork. The resources
that are shown in its view are:
v All nnDomain objects in the APPN subnetwork
v All interDomainCircuit objects used to connect the nnDomain objects in the
subnetwork
The view might no longer be displayed for one of the following reasons.
v All the resources shown in the view are purged.
To create the view again, monitor the network topology of the subnetwork and
request more detail of the nnDomainNetwork object.
InterDomainCircuit View
The interDomainCircuit view is generated by requesting more detail of an
interDomainCircuit object. The interDomainCircuit object is used to represent the
status of the connection between network nodes. The resources that are shown in
an interDomainCircuit view are all the TG circuits used to connect the two
network nodes. The network nodes can be in the same subnetwork or different
subnetworks.
Unless a connection between the two network nodes no longer exists or one of the
nodes is no longer a network node, monitoring the network topology of the
subnetwork containing the network node and requesting more detail of the
interDomain Circuit object will create the view again. See “Objects Unexpectedly
Purged” on page 335 for more information about purging and recovering objects.
NnDomain View
An nnDomain view is generated by requesting more detail of an nnDomain object.
The nnDomain object is used to represent the status of the network node and the
resources in its domain. The resources that are shown in an nnDomain view are:
v The network node whose domain is represented by the nnDomain object
v All TG circuits that support CP-CP sessions used to connect the network node to
its served end nodes
v All end nodes for which the network node provides network node services
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 351
v The class of the network node was changed from a network node to another
node type.
See “Objects Unexpectedly Purged” on page 335 for more information purging and
recovering objects. Unless the node no longer exists or is no longer a network
node, monitoring the network topology of the subnetwork containing the network
node and requesting more detail of the nnDomain object creates the view again.
SnaLocalTopology View
An snaLocalTopology view is generated by requesting more detail of a node object.
The snaLocalTopology object is used to represent all resources owned by the node
and all resources the node is aware of (the local topology of the node). Requesting
more detail of a node displays a view containing an snaLocalTopology object.
Requesting more detail of this object shows the resources associated with the node.
The intermediate view is generated to circumvent the NetView management
console restriction that a more detail view cannot contain the parent resource used
to build the view. To show a node as part of its local topology, the intermediate
view is used.
The links and ports owned by the node are shown in the Link-Port view.
Requesting more detail of a node creates both these views.
This view is not displayed if all resources shown in the view are purged or if the
node is purged. Also, this view might to be displayed if the class of the node was
changed.
Usually, monitoring the local topology of the node and requesting more detail of
the node object creates the view again. See “Objects Unexpectedly Purged” on
page 335 for more information about purging and recovering objects.
Link-Port View
A Link-Port view is generated by requesting more detail of a node object. The
Link-Port view shows all links and ports owned by the node (obtained from the
local topology of the node).
The node, its associated TG circuits, and adjacent nodes are shown in the
snaLocalTopology view. Requesting more detail of a node creates both these views.
This view is not displayed if all links and ports associated with the node are
purged or if the node is purged. Also, this view might not be displayed if the class
of the node was changed.
See “Objects Unexpectedly Purged” on page 335 for more information about
purging and recovering objects.
As the SNA topology manager monitors the network topology of network nodes, it
assumes that each network node being monitored is in a unique subnetwork and
creates an nnDomainNetwork object to represent the subnetwork.
When the SNA topology manager discovers that two network nodes being
monitored are a part of the same subnetwork, it merges the nnDomainNetwork
objects into a single nnDomainNetwork object containing all the resources
contained in both views. The SNA topology manager merges the
nnDomainNetwork objects when it detects an active link connecting network
nodes in each subnetwork that supports CP-CP sessions.
This link enables the two subnetworks to exchange topology information between
the network nodes in each subnetwork, effectively creating a single subnetwork.
When the SNA topology manager has merged two subnetworks, it does not split
them into separate subnetworks. Even though the links that connected the
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 353
subnetworks are deactivated or purged, the SNA topology manager assumes the
resources form disjointed parts of the same subnetwork.
Therefore, as soon as two subnetworks are connected by a CP-CP session, they are
merged and remain merged until all resources in the subnetworks are purged.
In some cases, this information does not specify the type of the node being
reported for example:
v Local topology reports for adjacent nodes where the link between the nodes is
not active
The reported adjacent node information is the representation of the node from
the perspective of the local node and might not be correct.
v Local topology reports for adjacent network nodes or end nodes where the link
indicates a LEN-level connection
Even after the link to an adjacent node is activated, the reported node type
might still be inaccurate. Nodes can define that a link to an adjacent node is to
be treated as a LEN-level connection (no CP-CP sessions or APPN network
services). The node reports the adjacent node as a LEN node because that is how
the node appears to the node being monitored.
v Border nodes are reported as end nodes by the network nodes in the adjacent
subnetwork (to which the border nodes are connected). This disparity is a result
of APPN protocols, where border nodes appear to the adjacent network node as
casually-connected end nodes.
v The SNA topology manager has out-of-date information about a node
The SNA topology manager displays a node using the last reliable information it
received about the node. If the configuration of the node was changed, the node
must be monitored (either directly or indirectly) for the SNA topology manager
to learn of the change.
v The network topology of a subnetwork is erroneously reporting a node as a
network node.
Even after a node has been changed from a network node to another node type,
the network topology databases of the network nodes in the subnetwork might
still represent the node as a network node and report the node as a network
node in the network topology of the node.
See “Status of a Nonexistent Resource Is Not Unknown” on page 359 for more
information.
v The network topology from a migrationDataHost node created as a t5Node
object
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 355
The VTAM agent on the migrationDataHost node reports only the CDRMs
during network topology, and does not report the type of this node. The SNA
topology manager creates this node as a t5Node object. The SNA topology
manager also creates a t5Node for each active CDRM reported with a
realSSCPname, the class of these nodes might not be accurate.
Collect the local topology to reflect the correct class of these nodes.
v The LUCOL monitor from a VTAM agent is created as an snaNode
The SNA topology manager creates a snaNode for the VTAM agent during the
monitoring of the LU collection if it is not monitoring local or network topology
from this VTAM agent.
Collect the local topology to reflect the correct class of this node.
After the SNA topology manager creates a resource, the resource remains in the
RODM data cache (and is displayed) until it is purged using the TOPOSNA
PURGE command or the topology agent informs the SNA topology manager that
A topology agent sends updates to delete dynamically created links and their
associated TGs when the link is deactivated. These updates are only sent as a part
of the local topology of the node.
For more information about updating and interpreting resource status, refer to the
IBM Tivoli NetView for z/OS SNA Topology Manager Implementation Guide.
For network topology, the topology agent reports all resources in the network
topology database. Some of the resources that are reported by the node probably
are not owned by that node, but are contained in the network topology database of
the node.
APPN propagates the status of all network nodes, and the TGs between network
nodes to all other network nodes in the same subnetwork.
To summarize:
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 357
v Links and ports are reported only as part of the local topology of the owning
node.
v All nodes are reported as part of their local topology and as part of the local
topology of any of their adjacent nodes. Network nodes are also reported in the
network topology of any network node in the same subnetwork.
v All TGs are reported as part of the local topology of the owning node. TGs
between network nodes are also reported in the network topology of any
network node in the same subnetwork.
v LU topology can only be collected from VTAM topology agents.
To solve this problem, issue the TOPOSNA PURGE command to delete the
resource from the RODM data cache and the NetView management console views.
If the resource is not purged, see “Objects Are Not Purged” on page 332.
For the adjacent network nodes and TGs between network nodes, the resource is
defined if the network topology is not being monitored. If the network topology is
being monitored, use the following methods to determine whether the resource is
defined:
v Stop monitoring the network topology of the subnetwork.
The resource is defined if the status of the resource is not changed to unknown.
v Stop monitoring the local topology of the subnetwork.
The resource is defined if the status of the resource changes to unknown. If the
status of the resource does not change, the resource is being reported either:
– By the network topology of the subnetwork, which means it might or might
not be defined
– by the local topology of another node, which means it is defined
v Query the local topology of the node using a local command.
APPN does not send commands to remove resources from the APPN network
node topology databases when a resource is deleted. It relies on each APPN
network node aging the resource out of its network topology database. Each
network node removes a resource from its topology database if an update is not
received for the resource within a set period of time (usually 15 days). Until this
time period expires, the resource remains in the APPN network topology database
of the node and is reported to the SNA topology manager, even though the
resource no longer exists.
The entire content of the database is sent, including resources that no longer exist
in the network but have not yet been aged out of the network node database.
Depending on the content of the information, some of these updates might be
propagated throughout the network, updating the time stamps of the resources
within the database of each node. Updates for nonexistent resources can be
propagated in this way, extending the amount of time required to age the resource
out of the network topology database.
In some networks, the resources are not removed from the databases, because new
network nodes are being added to the network or network nodes relearn their
topology databases.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 359
2. Stop all network topology monitor operations, and restart the monitor
operations after all monitors have been stopped.
This enables the SNA topology manager to learn the resource no longer exists.
3. If the status of the resource is not unknown, the resource either exists in the
network, or it was not successfully removed from all network topology
databases.
“Status of an Existing Resource Is Not Unknown” on page 358 suggests
procedures that can be used to remove existing resources.
4. If the status of the resource is unknown, the resource was successfully deleted
and the SNA topology manager is no longer receiving updates for it.
Issue the TOPOSNA PURGE command to delete the resource from the RODM
data cache and the NetView management console views. If the resource is not
purged, see “Objects Are Not Purged” on page 332.
Network topology can also report old information for resources. For example, if a
network node was changed to an end node, the network topology databases of the
network nodes in the subnetwork will probably continue to represent the node as a
network node. The SNA topology manager shows the node as a network node
unless it discovers that the node is really an end node (by monitoring the local
topology of the node). The SNA topology manager might create extraneous
nnDomainNetwork objects to represent a separate subnetwork containing the
incorrectly represented node. The node is really not a network node, so the
network topology will not contain any active connections to the node with active
CP-CP sessions; the SNA topology manager assumes the node is in a separate
subnetwork.
1. See “Status of a Nonexistent Resource Is Not Unknown” on page 359 for
suggested ways to eliminate extraneous information from the network topology
databases of the network nodes in the subnetwork.
2. If a node is incorrectly being shown as a network node, monitor the local
topology of the node or the serving network node of the node.
The SNA topology manager detects that the node is no longer a network node
and updates the node in the RODM data cache. It deletes the extraneous
nnDomain and nnDomainNetwork objects if they do not contain any other
resources.
See “Class of Node Object Does Not Match Node Type” on page 355 for more
information.
Chapter 17. Troubleshooting and Initial Diagnosis for the SNA Topology Manager 361
If any of these messages are found in the log, use the NetView online help facility
for the message to find more information about how to correct the problem. Refer
to the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation Guide for
information.
Other diagnostic tools that can be used to help diagnose problems with the
topology manager include:
v The interactive problem control system (IPCS)
v The network log
v The TASKMON command
v The TASKUTIL command
v The NetView internal trace
v VTAM CMIP traces
v The program-to-program interface (PPI) trace facility for NetView
These tools are described in Chapter 6, “Diagnostic Tools for the NetView
Program,” on page 73.
All messages issued by the topology manager use the prefix FLB. All existing
message utilities, such as automation, the ASSIGN command, and the network log
browse utility (BROWSE command) can be used with topology manager messages.
The identity of the component that generated the message is usually within the
message text, but the message numbers can also be used to determine the
component.
The message numbers have been assigned so that each component uses a specific
range. The topology manager is assigned 400 through 599 and 620 through 700.
The topology manager can also issue message numbers 600 through 619.
Messages are issued whenever events occur that might require operator attention,
including useful informational messages. Many of the messages are not related to a
Online help exists for all topology manager messages. To view this help
information, issue the HELP command followed by the message number (including
the FLB prefix) at a NetView operator console. The help information for the
message is displayed as one or more panels. The help information includes:
v A copy of the message text
v An explanation of why the message was generated
v An explanation of any message variables (variable data provided in the message
to clarify the event and its source)
v An explanation of any significant actions the topology manager performs as a
result of the event
v Any recommended responses (by the operator or others)
Log entries identify the specific event, along with all related information. In
addition, the topology manager creates log entries whenever an error is detected,
even if a message is also generated. These log entries contain detailed information
that describes the error in greater detail, and provide any related information that
can be used to diagnose the problem. For example, in the case where the topology
manager retries a monitor operation 10 times, one retry message is issued, but 10
log entries are created, each one containing information about why a particular
attempt failed.
The topology manager log entries are actually messages containing the log
information. There are three messages used to indicate the creation of a log entry.
They correspond to the type of event being logged (an Error, Warning, or
Informational event). The message numbers used are the same for each type of log
entry as follows:
v FLB600E for logging errors, including retry errors, along with any related error
information
v FLB601W for logging warning events that might require operator attention,
along with any related information
v FLB602I for logging informational events
In addition to these messages, the data associated with a log entry is provided
using messages FLB603I and FLB604I. These messages are placed in the network
log. They are also sent to the operator associated with the topology manager task
that created the log entry. These messages are not intended to be viewed by
operators, but are issued so that automation table entries can be created to
interpret them, or an ASSIGN command can be used to route them.
You can route all of these messages to an operator station that is set up specifically
to track the events logged by the topology manager components. The sample
automation table entries in the FLBAUT file provided with topology manager
NOTICE
For any topology manager log entry containing a major-minor code that is
not described in this section, contact IBM Software Support.
Each log entry contains a probe ID, major code, minor code, and log data size as
follows:
Probe ID
This identifies which specific section of code created the log entry. This ID
is used mostly by IBM Software Support when diagnosing problems with
the topology manager program. It can also be used by customers to
identify and correlate multiple occurrences of the same event. The same
event can be logged in several different sections of a program, with each
event having a different probe ID, and the same major and minor codes.
Major code
This code identifies the component that detected the event (which is
probably an error). This can be one of the topology manager components, a
NetView program system call issued by one of the topology manager
components, or one of the utility functions used by the topology manager
components. The following major codes are used by the topology manager
components:
Code Description
22 The event was reported by one of the NetView program system
calls invoked by a component of the topology manager.
78 The event was detected by the topology manager task or command
processor.
79 The event was detected by one of the topology manager utility
functions (such as the interface to the NetView program message
facilities).
This code, when combined with the minor code, uniquely identifies the
event being logged.
Minor code
This code identifies the type of event being logged. Each component has its
own set of events, so this code, when combined with the major code,
identifies the event being logged.
A notation convention is used to identify log entries in this book. The
major and minor codes are combined, separated by a hyphen (-) or a slash
(/). For example, the log entry with major code 78 and minor code 25 is
identified as log entry 78-25 or as log entry 78/25.
Log data size
The amount of additional information provided that is related to the event.
The topology manager components can include up to 4096 bytes of log data within
a log entry. Log entries with data are placed in the network log using a
Chapter 18. Diagnostic Tools for the SNA Topology Manager 365
multiple-line message, with each message containing up to 32 bytes of log data (64
hexadecimal characters). All of the messages associated with a log entry (all parts
of the multiline message) use the same probe ID. Specifically:
v Log entries with no additional data are created using one message (either
FLB600E, FLB601W, or FLB602I):
FLB600E PROBEID 0B510511 MAJOR CODE 78 MINOR CODE 59 LOG DATA SIZE : 0 BYTES
v Log entries with 1–32 bytes of additional data are created using two messages.
The first message (FLB600E, FLB601W, or FLB602I), identifies the event being
logged, and the amount of additional data. The last message (FLB604I) provides
the additional data. Both messages use the same probe ID:
FLB600E PROBEID 0B520247 MAJOR CODE 78 MINOR CODE 92 LOG DATA SIZE : 12 BYTES
FLB604I PROBEID 0B520247 DATA 0000: 0001005340B90EA0000700B3
v Log entries with greater than 32 bytes of additional data are created using
multiple messages, with the number of messages dependent on the amount of
additional data. The first message (FLB600E, FLB601W, or FLB602I), identifies
the event being logged, and the amount of additional data. This message is
followed by one or more FLB603I messages, which provide 32 bytes of
additional data each. As many FLB603I messages are logged as needed to
provide all but the last 1–32 bytes of additional data associated with the log
entry. The last message (FLB604I) provides the last 1–32 bytes of additional data,
and signifies the end of messages associated with the log entry. All messages use
the same probe ID. Following is an example of a log entry:
FLB600E PROBEID 0B510514 MAJOR CODE 78 MINOR CODE 187 LOG DATA SIZE : 60 BYTES
FLB603I PROBEID 0B510514 DATA 0000: 00000005D5C5E3C14BC1F5F7D4000000
FLB603I PROBEID 0B510514 DATA 0010: 00000000000000000000000000000000
FLB603I PROBEID 0B510514 DATA 0020: 00000000000000000000000000000000
FLB604I PROBEID 0B510514 DATA 0030: 00000005046BD50020000000
The example below demonstrates padding character data with a period to align to
a byte boundary. Although no other data follows the character data in the log data,
the period is added to make the total number of characters in the field even (18).
The size of the data is shown in bytes, where the size of the 18 character string
data is 9 bytes.
FLB600E PROBEID 0B300701 MAJOR CODE 77 MINOR CODE 8 LOG DATA SIZE : 9 BYTES
FLB604I PROBEID 0B300701 DATA 0000: (result-code 817).
These log entries are created whenever a component of the topology manager
receives an unexpected result from a system function. A system function is a
function provided by the NetView program or the MVS system (for example the
generalized trace facility (GTF)). These log entries can be created by any
component of the topology manager. Usually, there are associated log entries that
describe the consequences of the failure. In most cases, the task that detects the
problem will end.
22-22
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMINFC. The additional data contains the return code from the CNMINFC function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMINFC function reads the contents of a NetView global variable. The additional data identifies the name of
the variable being read.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMINFC function
0004 8 Name of the NetView variable read
Chapter 18. Diagnostic Tools for the SNA Topology Manager 367
Hexadecimal
Offset Decimal Length Description
000C n For IBM Software Support use
22-23
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMNAMS. The additional data contains the return code from the CNMNAMS function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMNAMS function allocates, frees, locates, and reallocates named areas of virtual storage. The additional data
identifies the actual function, as well as the name of the virtual storage area.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information and
the information from related log entries to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMNAMS function
0004 n For IBM Software Support use
22-24
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMSMSG. The additional data contains the return code from the CNMSMSG function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMSMSG function is used to send messages, and send data between the tasks that make up the topology
manager. The additional data identifies the destination of the data or message, the type of data or message, and the
contents of the data or message that cannot be sent.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMSMSG function
0004 n For IBM Software Support use
22-25
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMVARS. The additional data contains the return code from the CNMVARS function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMVARS function is used to set or retrieve the value of a global variable. Global variables are used by the
topology manager to preserve information when a task ends, and to exchange information between tasks. The
additional data identifies the global variable being read, or set, and the type of operation.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMVARS function
0004 16 For IBM Software Support use
0014 n The name of the variable. The name is formatted as a NetView high-level
language (HLL) varying length field. Record the information associated with
this log entry and contact IBM Software Support.
0014+n m For IBM Software Support use
22-26
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMGETD. The additional data contains the return code from the CNMGETD function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMGETD function is used to read and manipulate the data on the inbound data queues of the task. The data
on these queues is sent to the receiving task by other topology manager tasks and command processors. The
additional data identifies the type of operation. It might also identify the origin of the data and the contents of the
data, depending on the error.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMGETD function
0004 n For IBM Software Support use
22-27
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMSMU. The additional data contains the return code from the CNMSMU function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMSMU function is used to send multiple domain support message units (MDS-MUs) to agent nodes. These
MDS-MUs are used to send CMIP requests to the agent nodes. The additional data identifies the data to be sent, the
destination of the MDS-MU, and any other parameters required by the NetView program to send the MDS-MU.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMSMU function
0004 n For IBM Software Support use
Chapter 18. Diagnostic Tools for the SNA Topology Manager 369
22-29
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMSCOP. The additional data contains the return code from the CNMSCOP function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMSCOP function is used to determine whether an operator is authorized to issue a command, specify a
command keyword, or use a particular value for a command keyword.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMSCOP function
0004 n For IBM Software Support use
22-30
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMCMD. The additional data contains the return code from the CNMCMD function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMCMD function is used to issue a NetView command.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMCMD function
0004 n For IBM Software Support use
22-31
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMLK. The additional data contains the return code from the CNMLK function. These return codes are described
in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMLK function is used to request, release, or query the status of a named lock.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMLK function
0004 8 For IBM Software Support use
22-32
Event Description: An unexpected return code was received from the NetView high-level language (HLL) function
CNMSUBS. The additional data contains the return code from the CNMSUBS function. These return codes are
described in IBM Tivoli NetView for z/OS Programming: PL/I and C.
The CNMSUBS function is used to request substitution of symbolics in a data string.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the CNMSUBS function.
0004 n Data passed to CNMSUBS for substitution of system symbolics.
22-37
Event Description: A failure occurred while attempting to issue a NetView command.
Response: Record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 An error code used to identify the problem.
0004 8 A buffer containing the command to be issued. The buffer is formatted as a
NetView high-level language (HLL) varying length field. A varying length
field consists of a two-byte integer containing the size of the data in the
field, followed by the data within the field.
22-38
Event Description: A problem occurred while a topology manager task was trying to access a global data variable.
Response: Record the information associated with this log entry and contact IBM Software Support.
22-39
Event Description: An unexpected return code was received from the assembler macro DSIWAT. The additional
data contains the return code from the DSIWAT macro. These return codes are described in IBM Tivoli NetView for
z/OS Programming: Assembler.
The DSIWAT macro is used to wait for the completion of an event.
Response: Check for related log entries or messages that provide more information on the consequences of this
Chapter 18. Diagnostic Tools for the SNA Topology Manager 371
failure. In most cases, the task that issued this macro ends. Use the return code description and the data contained in
the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
22-40
Event Description: An unexpected return code was received from the assembler macro DSIPUSH. The additional
data contains the return code from the DSIPUSH macro. These return codes are described in IBM Tivoli NetView for
z/OS Programming: Assembler.
The DSIPUSH macro is used to establish recovery procedures for the topology manager tasks.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that issued this macro ends. Use the return code description and the data contained in
the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 Return code from the DSIPUSH function
0004 n For IBM Software Support use
22-47
Event Description: An unexpected return code was received invoking a RODM function using the RODM user
application program interface. The additional data contains the return code from the EKGUAPI function. These
return codes are described in the IBM Tivoli NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s
Guide.
The EKGUAPI function is used to access the RODM data cache.
Response: Check for related log entries or messages that provide more information on the consequences of this
failure. In most cases, the task that called this function ends. Use the return code description and the data contained
in the additional data associated with this log entry to determine the cause of the problem. Use this information, and
the information from related log entries, to correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The RODM function ID. This identifies the RODM function being invoked.
0004 4 Return code from the EKGUAPI function
0008 4 Reason code from the EKGUAPI function
22-56
Event Description: An unexpected error occurred while attempting to store a topology manager trace record in GTF.
The additional data contains the GTF category of the trace record, and an internal error code.
Message FLB637E is also logged.
Response: The information in the trace record is lost. Use the error code description and the data contained in the
additional data associated with this log entry to determine the cause of the problem. Use this information to correct
the problem.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Hexadecimal
Offset Decimal Length Description
0000 2 The GTF event ID of the trace record. The GTF event ID used by the
topology manager is X'05E8'.
0002 4 An internal error code.
Code Description
X'00000077'
Software problem with a component of the topology manager.
Record the information associated with this log entry and contact
IBM Software Support.
X'00000088'
The required GTF trace category is not active. Either the GTF or
the indicated GTF trace category was stopped after the topology
manager traces were started. The task that created this log entry
continues to try to trace information, but will not create another of
these log entries until it successfully stores some trace information.
To resolve the problem, stop the topology manager traces or restart
the indicated GTF trace category.
X'00000099'
Software problem with the topology manager; record the
information associated with this log entry and contact IBM
Software Support.
All others
An internal GTF error occurred. The error code provided is the one
received from the MVS GTRACE DATA macro. Refer to the MVS
library for more information about the macro and its return codes.
Following is a list of the most frequently received return codes:
0 The data was recorded in GTF trace buffers.
4 GTF is not active. No data was recorded. Activate GTF
and enable the appropriate GTF event IDs.
8 Incorrect parameter. Record the information associated
with this log entry and contact IBM Software Support.
12 Incorrect parameter. Record the information associated
with this log entry and contact IBM Software Support.
16 Incorrect parameter. Record the information associated
with this log entry and contact IBM Software Support.
24 All GTF buffers are full. No data was recorded. The
topology manager traces are overflowing the GTF trace
buffers. Increase the size of the trace buffers or decrease
the amount of data being captured by turning off some of
the topology manager trace categories.
28 Incorrect parameter. Record the information associated
with this log entry and contact IBM Software Support.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 373
NOTICE
Contact IBM Software Support for any SNA topology manager log entry
containing a major-minor code that is not described in this chapter.
78-0
Event Description: The topology manager cannot allocate enough memory to successfully complete a function.
If this probe is issued by the TOPOSNA command processor, the command is not processed. If this probe is issued
by the FLBTOPO task during initialization, the topology manager ends; otherwise, the command that caused the
probe ends and is not retried.
Response: Release any allocated memory that is not in use within the NetView program address space. Some
suggestions are to stop any unneeded tasks or to release any data storage not in use. If this problem persists, restart
the NetView program in a larger address space. If you suspect that the memory shortage is caused by a software
problem, such as a NetView task not freeing unused memory, dump the NetView address space, and follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting and Reporting Problems” before
contacting IBM Software Support.
Use the TOPOSNA LISTSTOR and TOPOSNA LISTRODM commands to examine storage usage for the topology
manager. Include the output of these commands with any other information associated with this log entry when
reporting the problem to IBM Software Support.
The storage estimates for the topology manager are described in the IBM Tivoli NetView for z/OS SNA Topology
Manager Implementation Guide.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The size of the storage area that cannot be allocated.
78-20
Event Description: The topology manager cannot allocate enough memory to successfully complete a function.
If this probe is issued by the FLBTOPO task during initialization, the topology manager ends; otherwise, the
command that caused the probe ends and is not retried.
Response: Release any allocated memory that is not in use within the NetView program address space. Some
suggestions are to stop any unneeded tasks or to release any data storage not in use. If this problem persists, restart
the NetView program in a larger address space. If you suspect that the memory shortage is caused by a software
problem, such as a NetView task not freeing unused memory, dump the NetView address space, and follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting and Reporting Problems” before
contacting IBM Software Support.
Use the TOPOSNA LISTSTOR and TOPOSNA LISTRODM commands to examine storage usage for the topology
manager. Include the output of these commands with any other information associated with this log entry when
reporting the problem to IBM Software Support.
The storage estimates for the topology manager are described in the IBM Tivoli NetView for z/OS SNA Topology
Manager Implementation Guide.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
78-25
Event Description: The topology manager received an incorrectly formatted message from VTAM CMIP services.
The header portion of the message contains incorrect data. The topology manager discards the message and
continues processing.
Response: Refer to the z/OS Communications Server CMIP Services and Topology Agent Guide for more information. In
most cases, there is a software problem in the interface between the topology manager and VTAM CMIP services.
Record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server SNA Diagnosis manuals for more information on diagnosing VTAM
CMIP services problems. For information about VTAM CMIP services, see
the z/OS Communications Server CMIP Services and Topology Agent Guide.
0004 10 For IBM Software Support use
000E 1 Size of the header information (k).
000F 1 For IBM Software Support use
0010 k The message header information received from VTAM CMIP services.
0010+k 2 Size of the message. The message is shown in character format (not
hexadecimal format). This is the number of characters in the message.
0012+k l The message received from VTAM CMIP services. The message is shown in
character (not hexadecimal) format.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 375
78-26
Event Description: The topology manager received an incorrectly formatted message from VTAM CMIP services.
The topology manager cannot parse the contents of the message. The topology manager discards the message and
continues processing.
Response: The message received from VTAM CMIP services contained data that the topology manager did not
recognize. In most cases, there is a software problem in the interface between the topology manager and VTAM
CMIP services. Record the information associated with this log entry and contact IBM Software Support.
It is also possible that the agent node sent incorrect topology information. If the message was received from the agent
node, verify that the contents of the message are correct.
For more information on the format of the information, refer to:
v IBM SystemView library
v CCITT Rec.X.710 | ISO/IEC 9595:1991 (ISO/IEC 9595:1991, Information technology - Open Systems Interconnection
- Common management information service definition)
v CCITT Rec.X.711 | ISO/IEC 9596-1:1991 (ISO/IEC 9596-1:1991, Information technology - Open Systems
Interconnection - Common management information protocol - Part 1: Specification)
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 14 For IBM Software Support use
000E 1 Size of the header information (k).
000F 1 For IBM Software Support use
0010 k The message header information received from VTAM CMIP services.
0010+k 2 Size of the message. The message is shown in character format (not
hexadecimal format). This is the number of characters in the message.
0012+k l The message received from VTAM CMIP services. The message is shown in
character (not hexadecimal) format.
78-27
Event Description: An error occurred when the topology manager tried to end its association with VTAM CMIP
services. This error occurred while the topology manager was ending. The topology manager continues shutdown
processing by releasing all allocated resources and then ending.
Response: Use the VTAM CMIP services error code to determine the cause of the error. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more information. The most probable cause of this
error is that VTAM CMIP services are not active. In most cases, this error can be ignored because the topology
manager is already ending.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server SNA Diagnosis manuals for more information on diagnosing VTAM
CMIP services problems. For information about VTAM CMIP services, see
the z/OS Communications Server CMIP Services and Topology Agent Guide.
0004 4 The VTAM CMIP services error field value. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more
information.
78-30
Event Description: An unexpected error occurred when the topology manager attempted to send a CMIP message
to an agent node. The topology manager sends CMIP messages to begin a monitor operation, end a monitor
operation, or activate, inactivate, or recycle a resource. The requested function fails. The topology manager continues
to process other requests. If the function was initiated by an operator command, the operator receives an error
message.
Response: Use the VTAM CMIP services error code to determine the cause of the error. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more information. Correct the problem and retry
the operation.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
0004 4 Size of the message data. The message data is shown in character format
(not hexadecimal format). This is the number of characters in the message.
0008 k The message being sent. The message is shown in character (not
hexadecimal) format. The entire message is shown, including the routing
information at the beginning of the message (the message starts with
src-type, dest-type, or msg).
Chapter 18. Diagnostic Tools for the SNA Topology Manager 377
78-31
Event Description: An unexpected error occurred when the topology manager attempted to receive a CMIP
message from the agent node or VTAM CMIP services. The topology manager receives CMIP messages containing
the results of monitor operations or resource control requests. It also receives special messages from VTAM CMIP
services to inform it of error conditions and other operation results. The topology manager reinitializes. Message
FLB684E, FLB677E, or FLB678E might also be logged.
Response: Use the VTAM CMIP services error code to determine the cause of the error. Refer to the VTAM library
for more information. The most probable cause of this error is that VTAM CMIP services ended unexpectedly. If
VTAM CMIP services are not active, start them. If the topology manager ends, restart the topology manager (using
the AUTOTASK OPID=FLBTOPO command). If necessary, you can use the TOPOSNA SETDEFS,CMPRETRY
command to change the VTAM CMIP services connection retry values.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
0004 4 The ID of the association between the topology manager and VTAM CMIP
services. A value of zero indicates that the error occurred before the
association was completed.
0008 4 Extended error information from VTAM CMIP services. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more
information.
000C 2 Offset within the message information where the actual message received
from VTAM CMIP services begins.
000E k The message received from VTAM CMIP services. The first part of the
message is the routing information. The beginning of the actual message
within this data is indicated by the offset information in the previous field.
78-32
Event Description: The topology manager received an incorrectly formatted message from VTAM CMIP services.
The topology manager cannot parse the contents of the message. The difference between this log entry and the log
entry with minor code 26 is the topology manager recognizes the message, but cannot parse the topology data or
error information within the message. The topology manager discards the message and continues processing.
This log entry is also created when the topology manager receives an unexpected message from VTAM CMIP
services. Examples of unexpected messages are CMIP responses before any CMIP requests were sent out, or internal
completion messages when the corresponding operation is not outstanding. The topology manager discards the
message and continues processing.
The message data helps indicate which error occurred. For message syntax errors, the message data contains the
portion of the data where the syntax error was found. For unexpected messages, the entire message is shown,
including the routing information at the beginning of the message (the message starts with src-type, dest-type, or msg).
Response: The message received from VTAM CMIP services contained data that the topology manager did not
recognize. In most cases, the agent node sent incorrect topology information. If the message was received from the
agent node, verify that the contents of the message are correct.
For more information on the format of the information, refer to:
v IBM SystemView library
v CCITT Rec.X.710 | ISO/IEC 9595:1991 (ISO/IEC 9595:1991, Information technology - Open Systems Interconnection
- Common management information service definition)
v CCITT Rec.X.711 | ISO/IEC 9596-1:1991 (ISO/IEC 9596-1:1991, Information technology - Open Systems
Interconnection - Common management information protocol - Part 1: Specification)
Hexadecimal
Offset Decimal Length Description
0000 4 Size of the message data. The message data is shown in character format
(not hexadecimal format). This is the number of characters in the message.
0004 k The message received from VTAM CMIP services. The message is shown in
character (not hexadecimal) format.
78-35
Event Description: The topology manager cannot open the initialization file FLBSYSD. Initialization of the topology
manager fails.
Response: Determine why the topology manager cannot open the initialization file. The file is installed in the data
set NETVIEW.V5R4M0.DSIPARM. A modified copy might be in a user DSIPARM data set. Place the file in the correct
data set, and restart the topology manager.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 k Name of the initialization file (FLBSYSD). This name is shown in character,
not hexadecimal, format.
78-36
Event Description: The topology manager encountered an error while reading the initialization file FLBSYSD.
Initialization of the topology manager fails.
Response: Use the internal error indicator supplied in the log entry to determine the cause of the problem.
Disregarding I/O errors, the problem is caused by an incorrectly formatted FLBSYSD file. Correct the syntax error,
and restart the topology manager. Modifying the initialization file is described in the IBM Tivoli NetView for
z/OS SNA Topology Manager Implementation Guide.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 379
Hexadecimal
Offset Decimal Length Description
0000 4 An integer indicating the cause of the problem.
Value Problem
1 The indicated section identifier is missing. The FLBSYSD file is
divided into sections, identified by unique identifiers followed by a
colon (:).
2 The indicated keyword was not found in the indicated section. The
FLBSYSD file is divided into sections, with one or more keywords
in each section.
3 The value for a keyword exceeds the maximum allowed value for a
keyword.
4 The value for a keyword is incorrectly formatted. The value
contains a double quotation mark (") with no ending double
quotation mark.
0004 2 The number of characters in the section identifier.
0006 k Name of the section identifier being referenced when the error was
detected. The name is shown in character, not hexadecimal, format.
0006+k 2 The number of characters in the keyword identifier.
0008+k l Name of the keyword identifier being referenced when the error was
detected. The name is shown in character, not hexadecimal, format.
78-37
Event Description: The topology manager received a topology update from an agent node containing an object
attribute that it does not support. The attributes supported by the topology manager are described in the IBM Tivoli
NetView for z/OS Data Model Reference.
The attribute received is an optional attribute, For more information on the mandatory and optional attributes for
APPN topology, refer to the IBM SystemView library.
Response: The topology manager ignores the unknown attribute and continues processing the other data in the
received update. This log entry is created the first time the topology manager receives each unknown attribute
during a monitor operation. The topology manager continues to ignore the attribute in all other updates, but the
problem is not logged.
The attribute and its value are not stored in the RODM data cache by the topology manager. You can choose to
ignore this log entry, because the topology manager continues to process the received updates. This does warn you
that some of the data being reported by a topology agent is not stored in the RODM data cache.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 Number of characters in the object identifier.
0002 k The object identifier of the unknown attribute. The identifier is shown in
character (not hexadecimal) format.
78-38
Event Description: The topology manager received a topology update from an agent node containing unsupported
management extensions. Management extensions can be added to the update by the agent nodes to indicate optional
information. The topology manager does not support any management extensions in the update information. It
usually ignores this data without logging any information. The log entry is created when the update indicates that
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 Number of characters in the object identifier.
0002 k The object identifier of the attribute in the management extension marked as
significant. The identifier is shown in character (not hexadecimal) format.
78-40
Event Description: A software problem has been detected by the topology manager. A RODM function completed
with an unknown RODM return code. The topology manager expects the RODM return code to be 0, 4, 8, or 12. The
topology manager ends.
Response: Record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The RODM return code
78-41
Event Description: RODM indicated that the response block used by RODM was not large enough to hold all of the
data related to a query function issued by the topology manager. The topology manager allocates a RODM response
block large enough to hold the data and issues the RODM function again. After the RODM function is completed, the
topology manager releases the allocated response block.
Response: This log entry is for information only. No action is required.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The size of the existing RODM response block.
0004 4 The size of the RODM response block needed to hold the data.
78-42
Event Description: The topology manager encountered an unexpected RODM error while trying to access the
topology manager defaults object. The topology manager defaults object (class FLB_DEFAULTS, object name
FLBDEF) is used to store the default settings defined using the TOPOSNA SETDEFS command.
The provided RODM error code indicator is an internal indicator used to map the error codes received from RODM
Chapter 18. Diagnostic Tools for the SNA Topology Manager 381
into a contiguous set of values. See the “Internal RODM Error Code Indicator” on page 396 for the table showing the
mapping of the RODM return codes and reason codes to this internal error indication.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
If the internal RODM error code indicates an unrecoverable error, the topology manager stops and must be restarted.
Retry the command that failed (TOPOSNA QUERYDEF or TOPOSNA SETDEFS).
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 For IBM Software Support use
0002 2 The RODM function ID.
0004 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
78-43
Event Description: The topology manager encountered an unexpected RODM error while trying to create an
aggregate object. The RODM error code indicator provided is an internal indicator used to map the error codes
received from RODM into a contiguous set of values. See “Internal RODM Error Code Indicator” on page 396 for the
table showing the mapping of the RODM return codes and reason codes to this internal error indication.
The object that cannot be created is identified by its name and its graph type. These objects are created when
topology information is received from the agent nodes. Depending upon the severity of the error, the topology
manager ends or retries the related monitor operation based on the ERRLIMIT setting. The ERRLIMIT value can be
changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
78-44
Event Description: The topology manager encountered an unexpected RODM error while trying to initialize the
attributes of an aggregate object. The RODM error code indicator provided is an internal indicator used to map the
error codes received from RODM into a contiguous set of values. See “Internal RODM Error Code Indicator” on page
396 for the table showing the mapping of the RODM return codes and reason codes to this internal error indication.
The object that cannot be initialized is identified by its RODM object ID and its graph type. These objects are created
and initialized when topology information is received from the agent nodes. The topology manager ends when this
error occurs.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Restart the topology manager, and restart the monitor operations for the agent nodes. If you warm-start the topology
manager, the existing monitor operations are restarted by the topology manager.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 383
Hexadecimal
Offset Decimal Length Description
0000 2 The type of aggregate object.
Value Description
1 NN domain. RODM class aggregateGraph2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.6708), graph type nnDomain.
2 NN domain network. RODM class aggregateGraph2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
3 NN domain network cluster. RODM class aggregateGraph2, ASN.1
object identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
4 SNA local topology. RODM class snaLocalTopo, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2152), graph ID
SnaLocalTopo.
5 Super cluster view. RODM class Network_View_Class RODM class
name (Network_View_Class).
6 Interdomain circuit. RODM class circuit2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.2186).
7 Interdomain network circuit. RODM class circuit2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2186).
12 NN domain (for virtual nodes). RODM class aggregateGraph2,
ASN.1 object identifier and RODM class name (1.3.18.0.0.6708),
graph type nnDomain.
0002 8 The RODM object ID of the aggregate object.
000A 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
78-46
Event Description: The topology manager encountered an unexpected RODM error while trying to set or reset the
view links for an object. These links are used to specify how the object is displayed at the workstation. The RODM
error code indicator provided is an internal indicator used to map the error codes received from RODM into a
contiguous set of values. See “Internal RODM Error Code Indicator” on page 396 for the table showing the mapping
of the RODM return codes and reason codes to this internal error indication.
The objects being linked or unlinked are identified by their RODM object IDs. The view links for these objects are set
or reset when topology information is received from the agent nodes and when the objects are purged. Depending
upon the severity of the error, the topology manager ends or retries the related monitor operation based on the
ERRLIMIT value. The ERRLIMIT value can be changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 The RODM function ID. This identifies the RODM function being invoked.
0002 8 The RODM object ID of the first object.
000A 8 The RODM object ID of the second object.
78-47
Event Description: The topology manager encountered an unexpected RODM error while trying to set or reset the
aggregation links for an aggregate object. These links are used to control the aggregation of the status of the object.
The RODM error code indicator provided is an internal indicator used to map the error codes received from RODM
into a contiguous set of values. See “Internal RODM Error Code Indicator” on page 396 for the table showing the
mapping of the RODM return codes and reason codes to this internal error indication.
The aggregate object and the object it was being linked to or unlinked from are identified by their RODM object IDs.
The aggregation links for these objects are set or reset when topology information is received from the agent nodes
and when the objects are purged. Depending upon the severity of the error, the topology manager ends or retries the
related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be changed by the TOPOSNA
SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 An indicator identifying the RODM function ID.
Value RODM function
1 Link objects (DUIFCUAP)
2 Unlink objects (DUIFCUAP)
0002 8 The RODM object ID of the aggregate object.
000A 8 The RODM object ID of the object the aggregate object was being linked to
or unlinked from.
0012 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
78-48
Event Description: The topology manager encountered an unexpected RODM error while trying to read or update
the DisplayResourceOtherData attribute of an aggregate object. The RODM error code indicator provided is an
internal indicator used to map the error codes received from RODM into a contiguous set of values. See “Internal
RODM Error Code Indicator” on page 396 for the table showing the mapping of the RODM return codes and reason
codes to this internal error indication.
The object that cannot be read or updated is identified by its RODM object ID and its graph type. These objects are
updated when topology information is received from the agent nodes. Depending upon the severity of the error, the
topology manager ends or retries the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value
can be changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 385
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 The type of aggregate object.
Value Description
1 NN domain. RODM class aggregateGraph2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.6708), graph type nnDomain.
2 NN domain network. RODM class aggregateGraph2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
3 NN domain network cluster. RODM class aggregateGraph2, ASN.1
object identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
4 SNA local topology. RODM class snaLocalTopo, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2152), graph ID
SnaLocalTopo.
5 Super cluster view. RODM class Network_View_Class RODM class
name (Network_View_Class).
6 Interdomain circuit. RODM class circuit2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.2186).
7 Interdomain network circuit. RODM class circuit2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2186).
12 NN domain (for virtual nodes). RODM class aggregateGraph2,
ASN.1 object identifier and RODM class name (1.3.18.0.0.6708),
graph type nnDomain.
0002 8 The RODM object ID of the aggregate object.
000A 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
000C k The data to be added or removed from the DisplayResourceOtherData
attribute. The format of this attribute for the topology manager RODM
objects is described in the IBM Tivoli NetView for z/OS Data Model Reference.
The data is shown in character (not hexadecimal) format.
78-56
Event Description: A problem occurred while creating and initializing the topology manager aggregate objects. The
topology manager, during initialization, attempts to locate the Network_View_Class object identified in the FLBSYSD
initialization file by the label SUPER_CLUSTER_VIEW_NAME. The value of that label is used as the MyName
attribute of the Network_View_Class object used by the topology manager.
The topology manager also tries to locate the aggregateGraph2 object identified in the FLBSYSD initialization file by
the label NN_DOMAIN_NETWORK_CLUSTER_DRN. The value of that label is used as the DisplayResourceName
attribute of the nnDomainNetworkCluster object used by the topology manager. If the object already exists in the
RODM data cache, the topology manager uses the existing object. When the object is located or created, the object is
linked to the Network_View_Class object.
If any of these operations fail, initialization of the topology manager fails.
Response: The log entry contains an error indicator that identifies the error encountered while initializing these
objects. In addition, there are additional log entries created to further define the problem. Use this information to
determine the cause of the problem. Correct the problem and restart the topology manager.
The most probable error is that the information in the FLBSYSD initialization file is incorrect. For example, the name
of the Network_View_Class object must match the name of the object created by the topology data model. The
Hexadecimal
Offset Decimal Length Description
0000 2 The type of error.
Value Description
1 Storage cannot be allocated within the topology manager to
represent the objects.
2 Storage cannot be allocated within the topology manager for the
tables needed to manage the other aggregate objects.
3 The Network_View_Class object cannot be located.
4 The nnDomainNetworkCluster object cannot be located or created.
5 The nnDomainNetworkCluster object cannot be linked to the
Network_View_Class object.
78-58
Event Description: The topology manager cannot allocate enough memory to successfully complete a function. The
topology manager ends.
Response: A corresponding log entry (79-0) is also created, describing the memory allocation failure in more detail.
Use the information and description of that log entry to resolve the problem. Restart the topology manager.
78-59
Event Description: The topology manager encountered an error processing an update for a resource. The most
common problems are:
v An unexpected RODM error occurred while the topology manager was trying to update or query a RODM object.
Depending on the error, the topology manager might be able to recover.
v A software problem has been detected by the topology manager while processing the update. A problem occurred
in the internal interfaces within the topology manager.
The topology manager either ends, ends the related monitor operation, retries the related monitor operation, or
continues processing. Associated log entries are created to identify the specific cause of the error. Refer to the
description of these log entries to determine what actions the topology manager takes when this error occurs.
Response: Use the error information in the associated log entry to resolve the problem.
If a RODM error occurred, a corresponding log entry with minor code 76 or 77 is created, identifying the objects that
failed. Another log entry is created, major code 22, minor code 47 containing the RODM error codes. Use the
information and description of these log entries to resolve the problem.
If a software error occurred, record the information associated with this log entry and contact IBM Software Support.
If the topology manager ended, restart the topology manager and the monitor operations for the agent nodes. If you
warm-start the topology manager, the existing monitor operations are restarted by the topology manager. If the
topology manager ended a monitor operation, restart the monitor operation that ended. Otherwise, the topology
manager recovered and no further action is required.
78-65
Event Description: The topology manager encountered a problem trying to link the nnDomainNetwork and
nnDomainNetworkCluster objects to create the views seen at the workstation. Depending upon the severity of the
error, the topology manager ends or retries the related monitor operation based on the ERRLIMIT value. The
ERRLIMIT value can be changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Other log entries are created describing the error in more detail. Use the information and description of
these log entries to resolve the problem.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 387
78-66
Event Description: The number used to uniquely identify each nnDomainNetwork object has wrapped. This
probably should not happen because the number is a very large number and is recalculated when the topology
manager is started. The topology manager ends.
Response: This number is combined with the SNA network ID to form the DisplayResourceName of the
nnDomainNetwork objects in the RODM data cache. This number is set during topology manager initialization to the
highest existing value currently in the RODM data cache, and is incremented each time a new nnDomainNetwork
object is created. The maximum value is 231-1. Purge or renumber the nnDomainNetwork objects in the RODM data
cache, and then restart the topology manager.
78-69
Event Description: The topology manager encountered an unexpected RODM error while trying to read the list of
objects for a class during warm-start processing. The RODM error code indicator provided is an internal indicator
used to map the error codes received from RODM into a contiguous set of values. See “Internal RODM Error Code
Indicator” on page 396 for the table showing the mapping of the RODM return codes and reason codes to this
internal error indication.
The object class that cannot be read is identified by an internal indicator representing the RODM class.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
Restart the topology manager.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
0002 2 The RODM function ID.
0004 2 The internal class indicator used by the topology manager to represent the
RODM object classes. See “Internal RODM Class Indicator” on page 398 for
the table showing the mapping of the RODM object classes to this internal
indication.
78-71
Event Description: The topology manager encountered an unexpected RODM error while trying to read the
attributes of an object during warm-start processing, while creating a graph object, or while deleting an object. If the
error occurred while attempting to create a graph object, an associated log entry is created with minor code 81. If the
error occurred while attempting to delete an object, an associated log entry is created with minor code 84. The
RODM error code indicator provided is an internal indicator used to map the error codes received from RODM into
a contiguous set of values. See “Internal RODM Error Code Indicator” on page 396 for the table showing the
mapping of the RODM return codes and reason codes to this internal error indication.
The object is identified by its RODM object ID.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
Hexadecimal
Offset Decimal Length Description
0000 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
0002 2 The RODM function ID.
0004 8 The RODM object ID of the object.
78-72
Event Description: This log entry is created for two different error conditions. The first is a storage allocation error.
If this occurs, a corresponding log entry is created (minor code = 0). If the error is not a storage allocation error, a
software problem has been detected by the topology manager. A problem occurred in the internal interfaces within
the topology manager. Initialization of the topology manager fails.
Response: If there is a corresponding storage allocation failure log entry, use the information in that log entry to
correct the problem. Otherwise, record the information associated with this log entry and contact IBM Software
Support.
Trace Data: For IBM Software Support use only.
78-73
Event Description: A RODM object was read during warm-start processing that has an incorrectly formatted
MyName attribute. The format of this attribute for the topology manager objects is described in the IBM Tivoli
NetView for z/OS Data Model Reference. Initialization of the topology manager fails.
Response: The RODM object ID of the incorrectly formatted object is provided, along with the value of the
MyName attribute. Correct the value for the attribute, or delete the object. Restart the topology manager.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the object.
0008 k The name of the object (the contents of the MyName attribute).
78-74
Event Description: A RODM object was read during warm-start processing that has an incorrectly formatted
MyName attribute. The format of this attribute for the topology manager objects is described in the IBM Tivoli
NetView for z/OS Data Model Reference. Initialization of the topology manager fails.
Response: The incorrectly formatted object is identified by the value of the MyName attribute. Correct the value for
the attribute, or delete the object. Restart the topology manager.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 389
Hexadecimal
Offset Decimal Length Description
0000 k The name of the object (the contents of the MyName attribute). This name
has been converted from the RODM format to its SystemView format. In
most cases, the formats are the same. See the IBM Tivoli NetView for
z/OS Data Model Reference for details on the required conversions between
RODM names and SystemView names.
78-75
Event Description: The topology manager had to truncate the value of the DisplayResourceOtherData attribute of
an object. The size of this attribute is limited to 255 characters, and the received values exceed that size. The topology
manager truncates the value. This attribute is updated when topology information is received from the agent nodes
for the object.
Response: The topology manager truncates the value and continues processing. This log entry serves as a warning
to indicate that the value of the DisplayResourceOtherData attribute for an object cannot be updated with all the
data related to that attribute.
Note: This minor code might not provide sufficient information to resolve the problem. It is anticipated that some
additional initial problem determination and diagnosis will be done by the user. If the problem cannot be
resolved, record the information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the object.
0008 k The data that had to be truncated from the DisplayResourceOtherData
attribute. The format of this attribute for the topology manager RODM
objects is described in the IBM Tivoli NetView for z/OS Data Model Reference.
The data is shown in character (not hexadecimal) format.
78-76
Event Description: The topology manager encountered an unrecoverable RODM error. Another log entry is created
(major code 22, minor code 47) that provides the actual RODM return code and reason code. Other log entries might
be created that provide more information on the error, such as the operation that failed when the error occurred. This
log entry is used to identify what RODM object was being referenced when the failure occurred.
The log entry provides the RODM class ID of the object, the RODM object ID of the object if the error is related to a
specific object in that class, and the RODM field ID of the attribute if the error is related to a specific object attribute.
The format of these identifiers is described in the IBM Tivoli NetView for z/OS Resource Object Data Manager and
GMFHS Programmer’s Guide.
Response: Use the information in this log entry along with the information in the related log entries to diagnose
and correct the problem.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The RODM class ID of the object.
0004 8 The RODM object ID of the object.
000C 4 The RODM field ID of the object attribute.
Hexadecimal
Offset Decimal Length Description
0000 4 The number of characters in the class name.
0004 k The name of object class. The name is shown in character, not hexadecimal,
format.
0004+k 4 The number of characters in the object name.
0008+k l The name of the object. The name is shown in character, not hexadecimal,
format.
0008+k+l 4 The number of characters in the attribute name.
000C+k+l m The name of the object attribute. The name is shown in character, not
hexadecimal, format.
78-78
Event Description: The topology manager encountered an unexpected RODM error while trying to initialize the
attributes of a logical link or port object. The RODM error code indicator provided is an internal indicator used to
map the error codes received from RODM into a contiguous set of values. See “Internal RODM Error Code Indicator”
on page 396 for the table showing the mapping of the RODM return codes and reason codes to this internal error
indication.
The object that cannot be initialized is identified by its RODM object ID. These objects are created and initialized
when topology information is received from the agent nodes. Depending upon the severity of the error, the topology
manager ends or retries the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be
changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
If the topology manager ended, restart the topology manager and the monitor operations for the agent nodes. If you
warm-start the topology manager, the existing monitor operations are restarted by the topology manager. Otherwise,
restart the monitor operation that ended.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 391
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the link or port object.
0008 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
78-79
Event Description: The topology manager encountered an unexpected RODM error while trying to create an object.
The failure occurred while the topology manager was creating the object in the RODM data cache, updating the
DisplayResourceOtherData field, or setting the initial values of the object attributes. The RODM error code indicator
provided is an internal indicator used to map the error codes received from RODM into a contiguous set of values.
See “Internal RODM Error Code Indicator” on page 396 for the table showing the mapping of the RODM return
codes and reason codes to this internal error indication.
The object that cannot be created is identified by its name. These objects are created when topology information is
received from the agent nodes. Depending upon the severity of the error, the topology manager ends or retries the
related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be changed by the TOPOSNA
SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
The RODM return codes, reason codes, function IDs, and other API information are described in the IBM Tivoli
NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s Guide. Refer to the topology data model,
which describes the topology manager RODM objects in the IBM Tivoli NetView for z/OS Data Model Reference. Use
this information to diagnose and correct the problem.
If the topology manager ended, restart the topology manager and the monitor operations for the agent nodes. If you
warm-start the topology manager, the existing monitor operations are restarted by the topology manager. Otherwise,
restart the monitor operation that ended.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
0002 k The name of the object that cannot be created. This is the value of the
MyName field in the RODM data cache. The format of this name for the
topology manager RODM objects is described in the IBM Tivoli NetView for
z/OS Data Model Reference. The name is shown in character (not
hexadecimal) format.
78-80
Event Description: The topology manager encountered an unexpected RODM error while trying to update the
DisplayResourceOtherData attribute of a logical link or port object.
The object that cannot be read or updated is identified by its RODM object ID. These objects are updated when
topology information is received from the agent nodes. Depending upon the severity of the error, the topology
manager ends or retries the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be
changed by the TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the mapping of the internal error indicator and the RODM function ID to determine the probable
RODM return and reason codes. If the internal error indicator is an unrecoverable error, two other log entries are
created. The first (22-47) provides the actual RODM return code and reason code. The second (78-76 or 78-77)
identifies the targeted object (and possibly the field) within the RODM data cache.
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the aggregate object.
0008 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
000A k The value of the DisplayResourceOtherData attribute that cannot be stored.
The format of this attribute for the topology manager RODM objects is
described in the IBM Tivoli NetView for z/OS Data Model Reference. The data
is shown in character (not hexadecimal) format.
78-81
Event Description: The topology manager encountered an unexpected RODM error while trying to read the
FLB_Creator attribute of a graph (aggregate) object. An associated log entry is created with minor code 71. This
associated log entry contains the RODM error indicator.
The object that cannot be read is identified by its name and its graph type. These objects are created when topology
information is received from the agent nodes. Depending upon the severity of the error, the topology manager ends
or retries the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be changed by the
TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the error information in the associated log entry to resolve the problem.
If the topology manager ended, restart the topology manager and the monitor operations for the agent nodes. If you
warm-start the topology manager, the existing monitor operations are restarted by the topology manager. Otherwise,
restart the monitor operation that ended.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 393
Hexadecimal
Offset Decimal Length Description
0000 2 The type of aggregate object.
Value Description
1 NN domain. RODM class aggregateGraph2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.6708), graph type nnDomain.
2 NN domain network. RODM class aggregateGraph2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
3 NN domain network cluster. RODM class aggregateGraph2, ASN.1
object identifier and RODM class name (1.3.18.0.0.6708), graph type
nnDomain.
4 SNA local topology. RODM class snaLocalTopo, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2152), graph ID
SnaLocalTopo.
5 Super cluster view. RODM class Network_View_Class RODM class
name (Network_View_Class).
6 Interdomain circuit. RODM class circuit2, ASN.1 object identifier
and RODM class name (1.3.18.0.0.2186).
7 Interdomain network circuit. RODM class circuit2, ASN.1 object
identifier and RODM class name (1.3.18.0.0.2186).
12 NN domain (for virtual nodes). RODM class aggregateGraph2,
ASN.1 object identifier and RODM class name (1.3.18.0.0.6708),
graph type nnDomain.
0002 k The name of the aggregate object that cannot be created. This is the value of
the MyName field in the RODM data cache. The format of this name for the
topology manager RODM objects is described in the IBM Tivoli NetView for
z/OS Data Model Reference. The name is shown in character (not
hexadecimal) format.
78-82
Event Description: The topology manager encountered an error processing an update for an intersubnetwork TG.
Possible causes are as follows:
v The network IDs of the nodes connected by the intersubnetwork TG are the same. APPN enables a network to be
divided into subnetworks based on network IDs. APPN also enables the network to be divided into subnetworks
where the nodes in the subnetworks have the same network ID. This feature is called clustering. Clusters are
connected by extended border nodes, and do not share topology information. The topology manager supports this
feature of APPN, but requires topology agents on the extended border node nodes to actually divide the network.
The topology manager does not provide agents for any nodes that are can be extended border nodes.
v The topology manager encountered an internal error while processing the update for the intersubnetwork TG.
v The topology manager encountered a RODM error while processing the update for the intersubnetwork TG.
Associated log entries are created to identify the specific cause of the error. Refer to the description of these log
entries to determine what actions the topology manager takes when this error occurs.
Response: Use the error information in the associated log entry to resolve the problem.
78-83
Event Description: The topology manager encountered an unexpected RODM error while trying to update or query
an object in the RODM data cache. Depending upon the severity of the error, the topology manager ends or retries
the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be changed by the TOPOSNA
SETDEFS,ERRLIMIT command.
The RODM error code indicator that is provided is an internal indicator used to map the error codes received from
RODM into a contiguous set of values. See the “Internal RODM Error Code Indicator” on page 396 for the table
showing the mapping of the RODM return codes and reason codes to this internal error indication.
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the object.
0008 2 The internal RODM error code indicator used by the topology manager to
map the RODM return and reason codes.
78-84
Event Description: The topology manager encountered an unexpected RODM error while trying to delete a node
object in the RODM data cache. An associated log entry is created with minor code 71. This associated log entry
contains the RODM error indicator.
The object that cannot be deleted is identified by its name and RODM object ID. The topology manager attempted to
delete the object because an update was received that changed the node type (class) of the node. The topology
manager cannot process the update because the new node object (in the new class) cannot be created while the old
object still exists in the RODM data cache. Depending upon the severity of the error, the topology manager ends or
retries the related monitor operation based on the ERRLIMIT value. The ERRLIMIT value can be changed by the
TOPOSNA SETDEFS,ERRLIMIT command.
Response: Use the error information in the associated log entry to resolve the problem. This error, along with
possible resolution actions, is also described in “Objects Are Not Purged” on page 332.
If the topology manager ended, restart the topology manager and the monitor operations for the agent nodes. If you
warm-start the topology manager, the existing monitor operations are restarted by the topology manager. Otherwise,
restart the monitor operation that ended.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 8 The RODM object ID of the object.
0008 k The name of the node object that cannot be deleted. This is the value of the
MyName field in the RODM data cache. The format of this name for the
topology manager RODM objects is described in the IBM Tivoli NetView for
z/OS Data Model Reference. The name is shown in character (not
hexadecimal) format.
78-173
Event Description: When creating an object in RODM, the topology manager discovered more than one object in
RODM with the same DisplayResourceName. This condition occurs if a user-created object in RODM has the same
DisplayResourceName as the topology manager created object. The RODM object identifier is logged with this log
entry.
Response: If the object was created by user, delete the object and create it with a different DisplayResourceName. If
Chapter 18. Diagnostic Tools for the SNA Topology Manager 395
the object was not user-created, contact IBM Software Support. The monitor action trying to create this object might
fail.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 16 The object identifier of the RODM object which has the duplicate
DisplayResourceName.
78-191
Event Description: During initialization, the SNA topology manager was unable to read a required class or field
from RODM (this implies that the data model is not entirely loaded). The name of the class or field is logged.
Response: Load the GMFHS data model, then load the SNA topology data model or wait until the data model is
entirely loaded before starting the SNA topology manager. When this error is detected, message FLB686E is also
issued, and the SNA topology manager will retry to read the RODM data model based on the RODM retry and the
retry limit values specified in FLBSYSD or with the SETDEFS command.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 A value of 0 indicates a class, a value of 1 indicates a field.
0004 Variable The name of the missing class or field.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 397
23 All functions
7 Unrecoverable error. Set when the RODM return code is 8, 12, or any
reason code other than the ones listed for the other internal codes. When
this error is set, the topology manager ends. The actual RODM return code
and reason code are logged using a log entry with major code 22 and
minor code 47. Might cause FLBTOPO user abend (X'185') to help IBM
Software Support in debugging the problem.
8 Data truncated. Set when the following reason codes are returned:
Code RODM function
208 EKG_QueryField, EKG_QuerySubfield
The RODM error code indicator provided is an internal indicator used to map the
error codes received from RODM into a contiguous set of values. See “Internal
RODM Error Code Indicator” on page 396 for the table showing the mapping of
the RODM return codes and reason codes to this internal error indication.
These log entries are created whenever a component of the topology manager
detects an error. These log entries can be created by any of the components of the
topology manager. Usually, there are associated log entries or messages that
describe the consequences of the failure. In most cases the task that detected the
problem ends.
79-0
Event Description: The task cannot allocate enough memory to successfully complete a function. The task stops
processing the current request, and possibly ends.
Response: Release any allocated memory that is not in use within the NetView program address space. Some
suggestions are to stop any unneeded tasks or to release any data storage not in use. If this problem persists, restart
the NetView program in a larger address space. If you suspect that the memory shortage is caused by a software
problem, such as a NetView task not freeing unused memory, dump the NetView address space, and follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting and Reporting Problems” before
contacting IBM Software Support. The storage estimates for the components of the topology manager are described in
the IBM Tivoli NetView for z/OS SNA Topology Manager Implementation Guide.
79-1
Event Description: An unexpected error occurred using the C Program library functions.
Response: Record the information associated with this log entry and contact IBM Software Support.
79-2
Event Description: A software problem has been detected in one of the components of the topology manager.
Response: Record the information associated with this log entry and contact IBM Software Support.
79-3
Event Description: A software problem has been detected in one of the components of the topology manager.
Response: Record the information associated with this log entry and contact IBM Software Support.
79-64
Event Description: A software problem has been detected in one of the components of the topology manager.
Response: Record the information associated with this log entry and contact IBM Software Support.
79-65
Event Description: An error occurred when a topology manager task tried to end its association with VTAM CMIP
services. This error occurred while the task was ending. The task continues shutdown processing by releasing all
allocated resources and then ending.
Response: Use the VTAM CMIP services error code to determine the cause of the error. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more information. The most probable cause of this
error is VTAM CMIP services are not active. In most cases, this error can be ignored because the task is already
ending.
The task might have a problem establishing an association with VTAM CMIP services when the task is restarted and
Chapter 18. Diagnostic Tools for the SNA Topology Manager 399
VTAM CMIP services remains active. When this happens, stop and restart VTAM CMIP services. Record the
information associated with this log entry and contact IBM Software Support.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
0004 8 For IBM Software Support use
79-66
Event Description: An unexpected error occurred when a topology manager task attempted to send the CMIP
message to an agent node. The CMIP message that failed was being sent to cancel an existing CMIP operation. This
error occurred while the task was ending. The task continues shutdown processing by releasing all allocated
resources and then ending.
Response: Use the VTAM CMIP services error code to determine the cause of the error. Refer to the z/OS
Communications Server CMIP Services and Topology Agent Guide for more information.
The most probable cause of this error is that VTAM CMIP services are not active. In most cases, this error can be
ignored because the task is already ending.
Trace Data: The format of the additional data provided in the log entry. The offsets are specified in hexadecimal and
are based from the beginning of the log data. The lengths are specified in decimal.
Hexadecimal
Offset Decimal Length Description
0000 4 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
0004 k For IBM Software Support use
The TOPOSNA TRACE command can be issued at any time, even when the
topology manager is not active. This enables you to turn on traces before starting
the topology manager so that initialization events can be traced. The trace settings
are not changed by the starting or stopping of the topology manager.
The types of events that are traced, along with the format of the trace records, is
unique for the topology manager. The trace information is used by IBM Software
Support to diagnose problems reported by customers. The availability of trace
information significantly helps when IBM Software Support is diagnosing a
problem.
The trace events are grouped into trace categories. These trace categories are
turned on or off by the commands. A topology manager trace record is created
when an event occurs, its associated trace category is turned on. For external
The description of the command describes the trace categories available for the
topology manager. Most trace categories cause significant amounts of trace data to
be captured, possibly affecting performance of the topology manager and
overflowing the GTF trace buffer. The amount of data captured by the trace
categories can be limited by the use of the CLASS parameter.
These trace categories, along with any data that can be gathered with a VTAM
CMIP services trace, capture the trace information most useful when reporting
problems to IBM Software Support.
All external trace records created by the topology manager use GTF format ID
(X'D8'). For the topology manager to actually store external trace data, GTF must
be enabled and the topology manager GTF event ID (X'05E8'). For information
about using GTF, refer to the MVS library.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 401
record format of the data is identified by the combination of the trace
category with the event ID. All GTF trace data is shown in hexadecimal as
well as character format, providing the hexadecimal value represents a
valid EBCDIC character.
The topology manager and VTAM CMIP services cooperate when setting the trace
event number. Each traced event will have a unique trace event number, unless the
number wraps.
1 2
GMT-01/14/2009 15:58:11.341367 LOC-01/14/2009 10:58:11.341367
3 4
HEXFORMAT AID FF FID D8 EID E5E8
5 6
+0000 00FB6B80 E2F5F4F0 C5C5D5E5 0000003D | ..,.S540EENV.... |
7 89 10
+0010 00010001 4003000E E2D5C160 E3D44040 | .... ...SNA-TM |
11 12
+0020 E3C4D4D5 40404040 00000399 00100F90 | TDMN ...r.... |
+0030 0000 | .. |
record 1 of 10
1 2
GMT-01/14/2009 15:58:12.530275 LOC-01/14/2009 10:58:12.530275
3 4
HEXFORMAT AID FF FID D8 EID E5E8
5 6
+0000 00FB6B80 E2F5F4F0 C5C5D5E5 0000003E | ..,.S540EENV.... |
7 89 10
+0010 0001000A 4003000E E2D5C160 E3D44040 | .... ...SNA-TM |
11 12
+0020 E3C4D4D5 40404040 00000000 00100F90 | TDMN ........ |
+0030 00200000 00000006 00070000 00060000 | ................ |
+0040 00013C43 00130000 00000000 00000ED8 | ...............Q |
+0050 9384A299 8360A3A8 978540F1 6B40A299 | ldsrc-type 1, sr |
+0060 834081F1 6B4094A2 8740C3D4 C9D760F1 | c a1, msg CMIP-1 |
+0070 4BD9D6C9 E5819784 A4404D89 95A59692 | .ROIVapdu (invok |
+0080 85C9C440 F1F3F1F1 F7F26B40 93899592 | eID 131172, link |
+0090 858460C9 C440F3F9 F3F2F2F3 6B409697 | ed-ID 393223, op |
+00A0 859981A3 89969560 A58193A4 8540F26B | eration-value 2, |
+00B0 40819987 A4948595 A3404D81 83A38996 | argument (actio |
+00C0 95D985A2 A493A340 4D948195 81878584 | nResult (managed |
+00D0 D6829185 83A3C393 81A2A240 F14BF34B | ObjectClass 1.3. |
+00E0 F1F84BF0 4BF04BF2 F2F9F16B 40948195 | 18.0.0.2291, man |
record 2 of 10
1 2
GMT-01/14/2009 15:58:12.530743 LOC-01/14/2009 10:58:12.530743
3 4
HEXFORMAT AID FF FID D8 EID E5E8
5 6
+0000 00FB6B80 E2F5F4F0 C5C5D5E5 0000003E | ..,.S540EENV.... |
7 12
+0010 0002000A 89A28885 84D58194 85404DD9 | ....ishedName (R |
+0020 859381A3 89A585C4 89A2A389 9587A489 | elativeDistingui |
+0030 A2888584 D5819485 404DC1A3 A3998982 | shedName (Attrib |
+0040 A4A385E5 8193A485 C1A2A285 99A38996 | uteValueAssertio |
+0050 95404D81 A3A39989 82A4A385 E3A89785 | n (attributeType |
+0060 40F14BF3 4BF1F84B F04BF24B F44BF66B | 1.3.18.0.2.4.6, |
+0070 4081A3A3 998982A4 A385E581 93A48540 | attributeValue |
+0080 7FE4E2C9 C2D4D5E3 7F5D5D6B 40D98593 | "USIBMNT")), Rel |
+0090 81A389A5 85C489A2 A3899587 A489A288 | ativeDistinguish |
+00A0 8584D581 9485404D C1A3A399 8982A4A3 | edName (Attribut |
+00B0 85E58193 A485C1A2 A28599A3 89969540 | eValueAssertion |
+00C0 4D81A3A3 998982A4 A385E3A8 978540F1 | (attributeType 1 |
+00D0 4BF34BF1 F84BF04B F04BF2F0 F3F26B40 | .3.18.0.0.2032, |
+00E0 81A3A399 8982A4A3 85E58193 A485407F | attributeValue " |
+00F0 D5E3C4F5 D4E5E27F 5D5D6B40 D9859381 | NTD5MVS")), Rela |
+0100 A389A585 C489A2A3 899587A4 | tiveDistingu |
1 When the event occurred, in Greenwich Mean Time (GMT).
2 When the event occurred, in local time.
3 The GTF format ID of the record.
4 The GTF event ID of the record. This specifies the format of the data in the
trace record. Ignore the high-order nibble (half-byte).
5 Twelve bytes of GTF information.
6 The trace event number. All the trace records used to capture the trace data
for an event have the same record number. Each event traced is assigned a
unique trace event number.
7 The multiple record trace data information. The first two bytes are the
record number (x) within a traced event and the next two bytes are the
total number of records (y) used to capture the data associated with the
event. The value can be read as record x of y. The beginning of the data for
a traced event is indicated by a value of one for the record number (x) and
the end of the data is reached when the record number equals the total
number of records (x=y).
8 The trace category of the traced event. Only present in the first trace record
of a multiple record event.
9 The event ID of the traced event. Only present in the first trace record of a
multiple record event.
10 The 8-character name of the component within topology manager that
generated the record:
SNA-TM
Chapter 18. Diagnostic Tools for the SNA Topology Manager 403
11 The 8-character name of the internal topology manager subcomponent that
generated the record. For use by IBM Software Support.
Only present in the first trace record of a multiple record event.
12 The data associated with the traced event. The format depends on the
event, identified by the combination of the trace category and event ID.
The beginning of the trace data within a record varies. The trace data starts
at X'28' within the first trace record created for an event. The data starts at
X'14' in all other records.
Tracing Internally
The topology manager can trace events to an internal wrap-buffer by using the
TOPOSNA TRACE,MODE=INT command. A SIZE parameter is also available to
specify the size of this buffer in 4096-byte page increments. Although the traced
event contains the same data, the internal trace format differs from the external
(GTF) trace.
Trace Events
These are the events traced by the topology manager that can be used to diagnose
problems. The events are identified by the associated trace category and the event
ID followed by the internal trace eye-catcher enclosed in parenthesis.
The trace data offsets are specified in hexadecimal from the start of the trace event
data. An offset of zero actually starts at X'0028' in the GTF trace record and at
X'000C' in the internal buffer trace record.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 405
4000-0002 (CENT)
Event Description: Traces processing signals between internal topology manager
objects. Traces the entry point for a particular signal. Trace using the TOPOSNA
ON=SIGNALS command. Note that this traces an enormous amount of data and
that the CLASS keyword of the TOPOSNA TRACE command can subset which
target object classes get traced.
Table 142. Trace Data for Event 4000-0002 (CENT)
Hexadecimal Description
Offset
0000 Class of target object
0004 Address of target object
0008 Name of target object
0038 Type of signal
003C Size of variable length parameter list
0040 Start of variable length parameter list
Chapter 18. Diagnostic Tools for the SNA Topology Manager 407
4000-0008 (LOGS)
Event Description: Traces topology manager log entries. This event is traced
using the TOPOSNA TRACE,ON=LOG command.
Table 144. Trace Data for Event 4001-0008 (LOGS)
Hexadecimal Description
Offset
0000 Log entry major code
0004 Log entry minor code
0008 ProbeID
000C Length of first data area
000E Length of second data area
0010 Length of third data area
0012 Length of fourth data area
0014 Length of fifth data area
0016 Length of sixth data area
0018 Length of seventh data area
001A Length of eighth data area
001C Length of ninth data area
001E Data areas according to above lengths
Chapter 18. Diagnostic Tools for the SNA Topology Manager 409
4003-000E (CMIP)
Event Description: The topology manager received data from VTAM CMIP
services or received a command from the topology manager command processor.
This event is also triggered when the time period expires for a pending operation.
The topology manager processes the received data and any pending operations
that have timed out. This event is traced using TOPOSNA TRACE,ON=CMIP.
This trace record shows the received data, which can be:
v A TOPOSNA command to be processed.
The first byte of the received data is X'7F'. The issued command is converted
into the internal command buffer shown in this trace record by the topology
manager command processor. The command buffer is reserved for the use of
IBM Software Support. The occurrence of these records indicates the occurrence
of a command. The network log contains the actual command issued.
v An inbound CMIP message.
The entire message is shown, including the internal routing information at the
beginning of the string. The actual CMIP message begins:
RORSapdu
The message is a response to a previous request sent by the topology
manager. The invokeID field in the message identifies the invoke ID of
the transaction. This is the final response for the transaction.
ROIVapdu
The message is a linked reply to a previous request sent by the topology
manager. The linked-ID field in the message identifies the invoke ID of
the transaction started by the request. Other responses will be
forthcoming.
v A VTAM CMIP services message.
The entire message is shown, including the internal routing information at the
beginning of the string. The actual message begins with:
Service-accept
A requested operation successfully completed.
Service-reject
VTAM CMIP services encountered an error processing an topology
manager request.
Table 146. Trace Data for Event 4003-000E (CMIP)
Hexadecimal Description
Offset
0000 The VTAM CMIP services error code. Refer to the z/OS Communications
Server CMIP Services and Topology Agent Guide for more information.
0004 Internal routing information. For IBM Software Support use
0008 If the received data is a CMIP message from VTAM CMIP services, this
value is the Hexadecimal Offset of the start of the CMIP message in the
data area. If the received data is not a CMIP message, this value is not
defined.
0010 The received data.
Response: If the received data is a CMIP message, the offset of the start of the
message in the received data is specified by the preceding field in the trace data.
The format of the CMIP response is described in the IBM SystemView library.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 411
4004-0019 (RTIB)
Event Description: The results from a RODM function invoked by the topology
manager. This event is traced using the TOPOSNA TRACE,ON=RODM command.
The RODM function IDs, return codes, and reason codes are described in the IBM
Tivoli NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s
Guide. The RODM return codes and reason codes are also described in NetView
online help.
Table 147. Trace Data for Event 4004-0019 (RTIB)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the invoked RODM function.
0004 The RODM return code.
0008 The RODM reason code.
000C The size of the response buffer.
0010 Up to the first 100 bytes of the response buffer. No data is present if the
size of the response buffer is zero (0).
The RODM function IDs, return codes, and reason codes are described in the IBM
Tivoli NetView for z/OS Resource Object Data Manager and GMFHS Programmer’s
Guide . The RODM return codes and reason codes are also described in the
NetView online help.
Table 148. Trace Data for Event 4004-001A (RARY)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the invoked RODM function.
0002 The RODM return code.
0006 The RODM reason code.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 413
4005-0015 (RCLS)
Event Description: The topology manager is preparing to invoke a RODM
function. This trace event, along with the corresponding events 4005-0016,
4005-0017, and 4005-0018, identify the RODM object class, object instances, and
object attributes specified in the function. This trace record, which is traced using
the TOPOSNA TRACE,ON=RODMDUMP command, identifies the class of the
objects.
The RODM function IDs are described in the IBM Tivoli NetView for z/OS Resource
Object Data Manager and GMFHS Programmer’s Guide.
Table 149. Trace Data for Event 4005-0015 (RCLS)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the RODM function to be invoked.
0002 Some RODM functions operate on two objects (such as LinkTrigger and
UnlinkTrigger). This field identifies which object class is being traced. A
value of one indicates this trace record is related to the first object and a
value of two indicates information about the second object is being traced.
0004 The internal class indicator used by the topology manager to represent the
RODM object classes. See “Internal RODM Class Indicator” on page 398 for
the table showing the mapping of the RODM object classes to this internal
indication.
0006 A 4-byte value that indicates the RODM class ID. Every class in RODM is
assigned an ID after RODM is started and the class is defined. This value
can change if a class is added or deleted.
The RODM function IDs are described in the IBM Tivoli NetView for z/OS Resource
Object Data Manager and GMFHS Programmer’s Guide.
Table 150. Trace Data for Event 4005-0016 (RON)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the RODM function to be invoked.
0002 Some RODM functions operate on two objects (such as LinkTrigger and
UnlinkTrigger). This field identifies which object class is being traced. A
value of one indicates this trace record is related to the first object and a
value of two indicates information about the second object is being traced.
0004 The RODM object ID of the object. This value might not be defined.
000C The name of the RODM object.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 415
4005-0017 (ROBJ)
Event Description: The topology manager is preparing to invoke a RODM
function. This trace event, along with the corresponding events 4005-0015,
4005-0016, and 4005-0018, identify the RODM object class, object instances, and
object attributes specified in the function. This trace record which is traced using
the TOPOSNA TRACE,ON=RODMDUMP command, along with 4005-0016,
identifies the object instances. This record is created when the topology manager
specifies the object using the RODM object ID of the object.
The RODM function IDs are described in the IBM Tivoli NetView for z/OS Resource
Object Data Manager and GMFHS Programmer’s Guide.
Table 151. Trace Data for Event 4005-0017 (ROBJ)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the RODM function to be invoked.
0002 Some RODM functions operate on two objects (such as LinkTrigger and
UnlinkTrigger). This field identifies which object class is being traced. A
value of one indicates this trace record is related to the first object and a
value of two indicates information about the second object is being traced.
0004 The RODM object ID of the object.
The RODM function IDs are described in the IBM Tivoli NetView for z/OS Resource
Object Data Manager and GMFHS Programmer’s Guide.
Table 152. Trace Data for Event 4005-0018 (RATR)
Hexadecimal Description
Offset
0000 The RODM function ID. This identifies the RODM function to be invoked.
0002 Some RODM functions operate on two objects (such as LinkTrigger and
UnlinkTrigger). This field identifies which object class is being traced. A
value of one indicates this trace record is related to the first object and a
value of two indicates information about the second object is being traced.
0004 An internal indicator used by the topology manager to identify the object
attribute. For IBM Software Support use
0006 A 4-byte value that indicates the RODM field ID (also called the RODM
attribute ID). Every attribute in RODM is assigned an ID after RODM is
started and the attribute is defined. This value can change if an attribute is
added or deleted.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 417
4007-001E (UPDT)
Event Description: Traces a status flow. This event is traced using the TOPOSNA
TRACE,ON=UPDATE command. Note that the update tracing can be limited to
particular classes using the CLASS keyword of the TOPOSNA TRACE command.
Table 153. Trace Data for Event 4007-001E (UPDT)
Hexadecimal Description
Offset
0000 ProbeID of invoker
0004 Timestamp
0008 Name of target object
0036 Class of target object
0038 Update attributes in the following format:
X'00' Attribute Identifier
X'02' Size of the attribute value
X'04' Varying length attribute value
Note that more than one set of update attributes can be present.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 419
4008-0001 (FREE)
Event Description: An internal topology manager storage free request directed to
the C runtime heap or the topology manager internal storage pool manager. Trace
using TOPOSNA TRACE,ON=STORAGE.
Table 155. Trace Data for Event 4008-0001 (FREE)
Hexadecimal Description
Offset
0000 ProbeID of storage owner (invoker)
0004 Address of storage to be freed
0008 A 4-byte storage identifier. A value of X’FFFF’ indicates the request was for
heap storage, all other values indicate storage pool requests.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 421
400A-0004 (NEW)
Event Description: Traces the allocation of an internal topology manager object.
This event is always traced.
Table 157. Trace Data for Event 400A-0004 (NEW)
Hexadecimal Description
Offset
0000 ProbeID of invoker
0004 Timestamp
0008 Name of allocated object
0036 Class identifier
0038 48 bytes of reserved data
0068 Address of allocated object
006C Variable-length parameter list
Chapter 18. Diagnostic Tools for the SNA Topology Manager 423
400A-001B (CBEG)
Event Description: Traces the start of a topology manager command. This event
is always traced.
Table 159. Trace Data for Event 400A-001B (CBEG)
Hexadecimal Description
Offset
0000 Timestamp
0004 ProbeID of invoker
0008 Variable-length parameter list
Chapter 18. Diagnostic Tools for the SNA Topology Manager 425
400A-001D (XMOG)
Event Description: Traces the transformation of a node from one class to another.
This event is always traced.
Table 161. Trace Data for Event 4008-0000 (GET)
Hexadecimal Description
Offset
0000 ProbeID of invoker
0004 Timestamp
0008 Name of node being transformed
0036 Class of node being transformed
0038 Name of object performing the transformation
0066 New class of the node being transformed
0068 A 4-byte return code
Enable the indicated GTF event IDs, and then issue the trace command again.
The second error is that a problem occurs while you are collecting trace data. A
failure occurs while an topology manager task is storing data in GTF. The
following message is issued:
FLB637E TASK taskname FAILED TO WRITE TRACE DATA USING GTF BECAUSE
OF AN ERROR
A log entry is created when this message is issued. Use the information in this log
entry (major code 22, minor code 56) to resolve the problem. The task continues to
trace information, but does not display this message again until it has successfully
stored trace information.
Chapter 18. Diagnostic Tools for the SNA Topology Manager 427
These TOPOSNA LISTxxxx commands have no optional keywords or parameters.
Refer to NetView online help for the command syntax, complete description, and
output example of each of these TOPOSNA LISTxxxx requests.
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager Program . . . . . . . 435
Routing Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
Improving INITTOPO Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . 436
RODM Errors - FLC070E and FLC076E . . . . . . . . . . . . . . . . . . . . . . . . . . 436
RODM Errors - Return Code 12 and Reason Code 122. . . . . . . . . . . . . . . . . . . . . 437
RODM Error 8/45077 Occurs after Reissuing a GETTOPO TMERES Command . . . . . . . . . . . . 437
Issuing Commands That Contain Case-Sensitive Text . . . . . . . . . . . . . . . . . . . . . 438
Command Support Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
GETTOPO Command Failures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
Tracing GETTOPO Command Processing . . . . . . . . . . . . . . . . . . . . . . . . 438
GMFHS Is Unavailable During GETTOPO Command Processing . . . . . . . . . . . . . . . . 439
| Failures in the IP Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . 439
Command Failures in the SNA Environment . . . . . . . . . . . . . . . . . . . . . . . 440
Failures Caused by RMTCMD Errors . . . . . . . . . . . . . . . . . . . . . . . . 440
Failures Caused by RUNCMD Errors . . . . . . . . . . . . . . . . . . . . . . . . 440
Failures Caused by Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
Failures Caused by Improperly Installed IP Agent on AIX Using LU 6.2 . . . . . . . . . . . . . . 440
Failures After Opening a New Map . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Object Status Update Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441
Missing IP Objects from NetView Management Console Views . . . . . . . . . . . . . . . . . . 443
Extra IP Objects in NetView Management Console Views. . . . . . . . . . . . . . . . . . . . 443
Aggregate Object Contains Identical Real Objects . . . . . . . . . . . . . . . . . . . . . . 443
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Component ID:
4. NetView Version and Release:
5. Recommended service update (RSU) level:
6. NetView function modifier ID (FMID):
7. What MultiSystem Manager features are being run:
8. On which MultiSystem Manager features does the problem occur:
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
Problem Classification
Check one of the following appropriate problem categories that matches the
symptoms associated with your problem.
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID:
v The exact text of the message on the log.
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands:
v NetView management console commands:
v Other:
4. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
5. Did the message text differ from what was published?
v Has local modification been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Wait Problems
For wait problems, complete the following questions:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. What is the name of the module in which the wait occurred?
5. What is the date that the module was compiled?
6. What is the PTF level of the module involved?
7. What is the offset into the module where the wait occurred?
Performance Problems
For performance problems, complete the following questions:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Gather the following documentation before calling IBM Software Support:
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the MultiSystem Manager, call
IBM Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting
and Reporting Problems” before contacting IBM Software Support.
Table 162. MultiSystem Manager Problem Scenarios
Problem Category Problem Scenario Page
Messages Routing messages 436
Processing Improving processing time 436
RODM Errors FLC070E and FLC076E 436
Errors Return Code 12, Reason Code 122 437
RODM Error 8/45077 occurs after reissuing a GETTOPO TMERES 437
command
Commands Case-sensitive text 438
Graphical display command support problems 438
GETTOPO failure 438
Views Object status update failure 441
Missing IP object 443
Extra IP object 443
Aggregate object contains identical real objects 443
To route all messages issued by the AUTOMSM and AUTOIPA tasks to another
OST, add the following statement to your automation table, and ensure that the
operator ID for that OST has been added to the +GRPNAME group:
IF (MSGID ¬= '') &
(OPID = 'AUTOMSM' | OPID = 'AUTOIPA') THEN
EXEC(ROUTE(ONE +GRPNAME));
This example statement routes all messages issued under tasks AUTOMSM and
AUTOIPA to the first logged on operator in the specified group of operators.
Modify the example automation statement by providing values for OPID that are
specific to your environment.
For more information, refer to the IBM Tivoli NetView for z/OS Automation Guide.
FLC076E FLCARODM:2000,x,y
| The message
| FLC076E FLCARODM:2000,8,13
| can mean that the currently active RODM name is not the name specified in the
| COMMON.FLC_RODMNAME statement in the CNMSTYLE member. However, if
| the value specified for RODM name in CNMSTYLE is correct, then the RODM
| name specified in the procedure is not active.
If you receive a FLCARODM return code other than 2000, contact IBM Software
Support.
Note: For more information about topology correlation operations and how you
can customize them, see the IBM Tivoli NetView for z/OS User’s Guide:
NetView Management Console and the IBM Tivoli NetView for z/OS Resource
Object Data Manager and GMFHS Programmer’s Guide.
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager Program 437
Issuing Commands That Contain Case-Sensitive Text
When entering a command from the NetView command line or command list, the
NetView program converts lowercase characters to uppercase prior to processing.
For commands that contain values that are case sensitive, such as resource names,
the uppercase value will cause processing errors and unexpected results. Prefixing
your commands with NETVASIS prevents this conversion and enables you to enter
commands that contain mixed case values.
This applies only to commands that are issued from the NetView command line or
from a NetView command list. GETTOPO statements coded in the MultiSystem
Manager initialization file are processed by MultiSystem Manager and are not
subject to the NetView conversion from lowercase to uppercase. Do not preface
MultiSystem Manager initialization statements with NETVASIS.
Refer to IBM Tivoli NetView for z/OS Installation: Configuring Graphical Components
or to online help for additional details concerning the NetView NETVASIS
command.
In the preceding examples, both service points have the same domain name
(DOM1) even though they reside in different networks (NETA and NETB). If you
want to use the NetView management console to issue commands to your service
points, use unique domain names across all your networks.
To invoke the trace option for GETTOPO processing, specify TRACE=YES on the
GETTOPO command. Specifying TRACE=YES generates an FLC003I message for
each RUNCMD that is issued during GETTOPO command processing. See the
GETTOPO commands in the IBM Tivoli NetView for z/OS command help or
online help for more information on the TRACE parameter.
Then issue:
NCCF START TASK=CNMTAMEL PRI=3
To test TCP/IP connectivity from the service point to the NetView host, use the
PING and TRACERTE commands to ensure that the path is active and available.
| To test TCP/IP connectivity from the NetView host to your service point, use the
| FLCACTIP command to issue commands to the service point. For example, to test
| connectivity to the IBM Tivoli Network Manager agent, you can use the info
| command, as in this example:
| flcactip host=hostname.domain.com port=3333 cmd=info
If TCP/IP connectivity exists between your NetView host and the service point,
but alerts are not being received, check the following items:
v Use the hardware monitor to verify that the alerts are not arriving at the
NetView host.
| v If you are using the MultiSystem Manager for IBM Tivoli Network Manager,
| ensure that an SNMP trap automation task is active and listening on the
| expected port. The LIST taskname command, (where taskname is the SNMP trap
| automation task expected to receive SNMP traps from ITNM), can be used to
| view the status of the task receiver and to verify on which ports and protocols it
| is active.
v Ensure that the event receiver (IHSAEVNT) is active and configured to use the
same port as that specified on the service point.
Refer to the IBM Tivoli NetView for z/OS Installation: Configuring Additional
Components for information on configuring your service point port and receiver
port. This is required only for Tivoli management region (TMR).
v Verify that the service point is configured to send alerts to the NetView host.
Refer to IBM Tivoli NetView for z/OS Installation: Configuring Graphical Components
for information on configuring your service point to send alerts to the NetView
host.
If TCP/IP connectivity exists between your NetView host and the service point,
and alerts are being received, but messages DSI435I and FLC077E are received
after issuing GETTOPO or FLCACTIP commands, increase your NetView
COSTIME to allow the commands enough time to complete before timing out.
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager Program 439
Command Failures in the SNA Environment
This section describes the failures that can occur when GETTOPO uses the
NetView RMTCMD and RUNCMD commands.
This message might indicate that your RUNCMD timeout value is too small.
Refer to the IBM Tivoli NetView for z/OS Command Reference Volume 1 (A-N) or
online help for more information on setting the RUNCMD timeout value.
To determine if the agent code is installed, issue the following command at the
workstation:
ls /usr/lpp/msmip/flcitopo
If the agent code is not installed, you will receive output similar to the following
example:
If the agent is not installed, install the agent code from the product media.
If the agent code is installed, you will receive output similar to the following
example:
% ls /usr/lpp/msmip/flcitopo
/usr/lpp/msmip/flcitopo
If the agent is installed and you still experience errors, ensure you have completed
all the installation and customization steps. Ensure you completed the following
steps:
1. Select Communications Applications and Services.
2. Select NetView for AIX.
3. Select Configure.
4. Select Set options for NetView for AIX daemons (or Set options for AIX
NetView/6000 daemons).
5. Select Set Options for host connection daemons.
6. Select Set Options for spappld daemon.
On this window, you must add the value :/usr/lpp/msmip to the end of the
Execute shell path: field. Note that colons (:) separate these path statements, not
| semicolons (;) as in Windows.
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager Program 441
| b. Verify the SNMP trap gateway files have been updated on the
| workstation. Three files are used to control the nco_g_snmp processes
| that are created during the agent configure:
| MSM_snmp.map
| MSM_snmp_tbl_rep.def
| MSM_GATE.conf
| c. Verify that the nco_g_snmp task is active and started using the
| MSM_GATE.conf configuration file.
| d. Verify that an SNMP gateway, nco_g_snmp, is active and listening on
| the expected port.
| e. Ensure that an SNMP trap automation task is configured and active.
| 2. Verify that the expected alerts show in the hardware monitor:
| a. Determine whether the alerts from the topology agent are being processed
| by the automation table.
| v Examine the automation table to see if it has been correctly modified with
| the statements from sample FLCSTBL.
| Uncomment the existing statement in the automation table that processes
| alerts and resolutions for GMFHS if it has not already been
| uncommented.
| v Determine if the alerts are being filtered from being logged in GMFHS.
| Comment out the filtering statements if you wish these alerts to be
| logged in the NetView hardware monitor and the alert history file.
| Refer to the IBM Tivoli NetView for z/OS Automation Guide for more
| information on the automation table.
| b. Determine whether the alerts are being logged in the alert history file in the
| NetView management console.
| c. Display alert history for the topology agent object.
| If alerts are being forwarded to NetView and are being logged in the
| hardware monitor but they do not show in alert history and do not change
| the status of the object, check to see if NetView task DUIFEAUT is active.
| d. Check the alert forwarding path. The NetView where MultiSystem Manager
| resides must be either the alert focal point for the service point where the
| topology agent resides, or be configured to receive forwarded alerts from
| the NetView which is the focal point.
| 3. Verify topology alerts are causing the appropriate GETTOPO commands to be
| run. Some alerts provide information regarding topology changes.
| These types of alerts cause GETTOPO commands to be issued. If these
| commands are not being driven, it might indicate that the autotask you have
| assigned for such commands is not active or that the command to be driven is
| not a valid NetView command. Verify that the task listed for the ROUTE
| keyword in the automation table for the MultiSystem Manager statements is
| active (you can do this by issuing the LIST autotask command).
| 4. Verify that MultiSystem Manager has not lost contact with the service point.
| Contact can be lost, for example, if the service point loses power and
| disappears from the network without the opportunity to send an alert.
| The HEARTBEAT parameter on the GETTOPO command provides a means for
| checking the connection between the service point and MultiSystem Manager.
| By setting the HEARTBEAT parameter in the GETTOPO command for a
| specified time interval, MultiSystem Manager detects the lost service point and
| notifies the NetView management console operator by changing the status of
| the agent object to either unsatisfactory or unknown.
This can happen if you hid an object on the submap or deleted an object from a
submap. MultiSystem Manager stores only one instance of the object in RODM,
regardless of the number of times it appears on a submap or the number of
submaps in which it appears. Therefore,if an object is displayed in any submap, it
is also displayed in the NetView management console view.
Verify that any object which is hidden or deleted on your IP submap and which
you do not want to show on your NetView management console views is also
hidden or deleted from all the Tivoli NetView for AIX or NetView for AIX
submaps.
For more information about guidelines for cutting, hiding, or deleting objects on
submaps, refer to the Tivoli NetView (for AIX) library.
Select one of the identical objects and select Configuration- Parent. Select another
identical object and select Configuration- Parent. After comparing the results of
the two views, you see that two or more objects were discovered by different
agents. Although their display names can be the same, the objects have different
object names in RODM. You can determine that the objects are different by
comparing the Resource Information displays for those objects.
NetView identifies instances where you define two or more agents that manage the
same resource. This can be corrected by changing the set of resources managed at
the appropriate distributed manager consoles.
Chapter 20. Troubleshooting and Initial Diagnosis for the MultiSystem Manager Program 443
If a network resource has more than one LAN adapter card (MAC address) or IP
address, correlation will occur on the first MAC address and the first IP address
provided by the agent. If another agent later provides a different MAC address or
IP address for that managed resource, it might not correlate to the original
aggregate. This condition can also be caused when a systems administrator
provides an alias at the distributed agent console, such as a ’local’ MAC address.
To prevent these conditions, ensure that every distributed manager specifies the
same primary MAC address and IP address for a managed resource.
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
Record the following general information:
1. Date:
2. Problem Number:
3. Component ID:
4. Recommended service update (RSU) level:
5. Installation Option:
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
Problem Classification
Check one of the appropriate problem categories below that matches the symptoms
associated with your problem:
Abend Problems
For abends or processor exception problems, complete the following items:
1. What is the abend code?
2. What processes were taking place at the time of the abend?
3. Use the online help facility (type HELP ABEND and use the scroll function to
locate the abend code).
4. Gather the following documentation before contacting IBM Software Support:
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A copy of the trace log. See “NetView Trace” on page 99.
v The first unformatted dump of the abend.
v A completed AON problem worksheet.
5. Gather the following information from the dump:
a. What is the program status word (PSW) at the time of the abend?
b. In what module did the abend occur?
c. What was the module compiled?
d. What is the PTF level of the module pointed to by the abend?
e. What is the offset into the module pointed to by the PSW at the time of the
abend?
f. List the registers at the time of the abend.
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID:
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands:
v Other:
4. If the message was unexpected and cannot be corrected by following the
actions in the NetView online help, gather the following documentation before
calling IBM Software Support:
v A hard copy of the network log containing the message.
v The message ID:
Loop Problems
For loop problems, complete the following items:
1. What events led up to the loop?
2. What data was being displayed?
3. What was the last command entered?
4. If this is an enabled loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v After obtaining a console dump, cancel AON with a dump.
Note: If the loop is still occurring after AON has been canceled, look for a
problem other than AON.
5. If this is a disabled loop (see “Documenting LOOP Problems” on page 33),
obtain the following documentation:
v A document describing the scenario leading to the problem.
v A hard copy of the system log.
v A hard copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A hard copy of the trace log. See “NetView Trace” on page 99.
v The addresses of instructions within the loop.
v A dump obtained by using the CPU RESTART function.
Note: If ABEND071 does not occur in AON and normal processing resumes,
this is not an AON problem.
6. What are the modules involved in the loop?
7. What are the dates that the modules were compiled?
8. What are the PTF levels of the modules involved in the loop?
Wait Problems
For wait problems, complete the following items:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the system console log.
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
Performance Problems
For performance problems, complete the following items:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the network log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A copy of the AON Entry/Exit trace.
v Information describing your operating environment:
v Descriptions of any modifications to your system:
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of AON, call IBM Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
The following sections explain AON initialization and recovery to help you in
diagnosing and solving network problems. Extensions to AON automation are
described in the IBM Tivoli NetView for z/OS User’s Guide: Automated Operations
Network.
To change which automation table gets loaded, copy the AUTOCMD statements
| from the CNMSTYLE member to the CNMSTUSR or CxxSTGEN member, and
make the appropriate changes. For more information on this process, refer to IBM
Tivoli NetView for z/OS Administration Reference.
| To change which policy files get loaded, copy the POLICY statements from the
| CNMSTYLE member to the CNMSTUSR or CxxSTGEN member, and make the
appropriate changes. For more information on this process, refer to IBM Tivoli
NetView for z/OS Administration Reference.
| To initialize AON, copy the TOWER statement from the CNMSTYLE member to
| the CNMSTUSR or CxxSTGEN member, and remove the asterisk (*) before AON.
When the policy repository is loaded, an EZL110I message is issued. The policy
repository can be loaded with policy definitions for AON, GRAPHICS, or the
policy for your own application. If AON is uncommented on the TOWER
| statement in the CNMSTYLE member, AON continues initialization by running
command list EZLEAINT, when the EZL110I occurs. If not, AON does not continue
initialization, but the policy repository is loaded with policy definitions for your
other applications, such as GRAPHICS.
The call to the routine dictates whether any of the steps in the routine are omitted,
and which keywords to use to look up the applicable programs in the definition
tables.
EZLEFAIL Routine
AON calls the EZLEFAIL routine when it receives a message or MSU indicating
that a resource has failed (using the NetView automation table), or from any
program run by an operator or timer. The EZLEFAIL routine does the following
actions:
v Confirms that the resource is to be recovered
v Issues a message indicating the resource is unavailable
v Issues a notification describing the failure
v Runs any failure specific programs
v Initiates recovery for the resource
v Marks the resource with Automation in Progress (AIP) status
You can omit any of these steps by specifying the appropriate value in the SKIP
parameter of the EZLEFAIL routine. For more information about the syntax and
parameters for the EZLEFAIL routine, refer to the IBM Tivoli NetView for
z/OS User’s Guide: Automated Operations Network.
Initialization
The EZLEFAIL routine retrieves resource information by running the RESINFO
program defined in the option definition tables. This program returns all required
data in keyword=value list format. The EZLEFAIL routine assigns values to
keyword variables for use by messages and other programs called by the routine.
Next, the EZLEFAIL routine gets any optional processing values from the TBLKEY
parameter. If you do not specify the TBLKEY parameter for the EZLEFAIL routine,
no optional processing or notification occurs. The values on the TBLKEY parameter
specify keywords found in the option definition tables. In the option definition
table, the keywords define the actual processing values used for optional
processing. AON saves the TBLKEY values in the outmsgid and spec_function
variables. Message EZL509I is the default outmsgid. The value of TBLKEY is in the
following format:
tblkey_value=(outmsgid,spec_function_call)
The EZLEFAIL routine gets the values specified on the IST105I keyword in the
option definition table. In the option definition table, you can see the values on the
IST105I keyword as follows:
IST105I=(EZL531,FKVEAIDA(resname restype))
In this example, the EZLEFAIL routine issues the EZL531I message and runs
FKVEAIDA as a function sending the current value of resname (resource name) and
restype (resource type) for optional processing. An optional processing program
performs any automation or processing unique to the resource or failure. No
optional processing is done and no message is issued if SKIP=(0) is specified on
the EZLEFAIL call.
The availability of the immediate higher node of the resource is checked. The
EZLEFAIL routine runs the CHKHIGH program from the option definition table. If
the return code from this program is greater than zero (0), the resource’s
immediate higher node is assumed to be unavailable, and the EZLEFAIL routine
stops processing with a return code of 12. If the connecting higher nodes are
available, recovery processing for the higher nodes reactivate or reconnect all the
subordinate nodes. The EZLEFAIL routine omits this step if SKIP=(H) is specified.
Automation flags defined in the RECOVERY control file statement are analyzed to
see if automation continues for this resource. Automation can be turned off for this
resource or it can be in a NOAUTO window. The EZLEFAIL routine runs the
CHKAUTO program from the option definition table. If the return code from this
program is greater than 0, automated recovery for the resource is assumed to be
undesirable and the EZLEFAIL routine stops with a return code of 13.
The EZLEFAIL routine next checks the current status of the resource. Because this
is a failure processor, the assumption is that if the resource is in an active
(available) status, further processing (recovery) is not necessary. If the status of the
resource is ACTive, CONCTable, NORMal, or ENABLEd, the EZLEFAIL routine
stops with a return code of 14.
The EZLEFAIL routine then determines whether automated recovery for this
resource is already in progress. The timer ID for the recovery timer is found in the
EZLTIMR.resname variable. If a timer already exists to run the RECOVMON
program from the option definition table, the EZLEFAIL routine stops with a
return code of 15. If it does not find a timer ID, it looks for the resource name as a
valid timer ID.
The EZLEFAIL routine returns one of the following return codes from this check:
0 EZLEFAIL processing continues.
1 An infrequent error threshold has been exceeded. The INFRACT program
from the option definition table is run. If the return code from this
program is not zero (0), EZLEFAIL stops with a return code of 21.
2 A frequent error threshold has been exceeded. The FREQACT program
from the option definition table is run. If the return code from this
program is not 0, EZLEFAIL stops with a return code of 22.
3 A critical error threshold has been exceeded. The CRITACT program from
the option definition table is run. If the return code from this program is
not 0, EZLEFAIL stops with a return code of 23.
Optional Processing
This step enables the unique processing requirements of different resource types
and network types to be met. For example, LAN bridge recovery has different
information gathering requirements than TCP/IP node recovery. This routine
gathers the additional data and uses it to decide whether recovery of the resource
continues. The EZLEFAIL routine omits this step if SKIP=(O) is specified.
The EZLEFAIL routine next runs the program specified in the second parameter of
the TBLKEY parameter. If the return code from this program is not 0, EZLEFAIL
stops with a return code of 30. For example, if TBLKEY=REPLYU, the REPLYU
definition from the option definition table is accessed. The format of this variable is
REPLYU=(msgid,prog_name parm1 parm2). In this case, prog_name is run and the
values of parm1 and parm2 are passed as arguments.
For detailed information about the syntax and parameters for EZLEFAIL, refer to
the IBM Tivoli NetView for z/OS User’s Guide: Automated Operations Network.
Any of these steps can be omitted by specifying the appropriate value in the SKIP
parameter of the EZLEFAIL routine.
Initialization
When the EZLERECV routine initializes, it checks the ENABLE flag for the
specified option. If the ENABLE flag is not set to Y, the EZLERECV routine stops
with a return code of 11. If ENABLE is set to Y, the EZLERECV routine gets the
resource information by starting the RESINFO program from the option definition
table. The EZLERECV routine then gets the optional processing values from the
TBLKEY field of the option definition table. If TBLKEY is not specified, no optional
processing or notification is performed. The TBLKEY values are saved in the
outmsgid and spec_function variables. Message EZL504I is the default outmsgid.
Stop Recovery
The EZLERECV routine checks the value of the EZLTIMR.resname variable. If
EZLTIMR.resname contains a timer ID, the EZLERECV routine determines whether
the timer ID still exists. If the timer does exist, the EZLERECV routine purges the
timer and clears the variable to stop any recovery activity on the resource. If it
does not have a timer ID, EZLTIMR uses the resource name as the timer ID. The
EZLERECV routine omits this step if SKIP=(R) is specified.
For more information about the syntax and parameters for EZLERECV, refer to the
IBM Tivoli NetView for z/OS User’s Guide: Automated Operations Network.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service . . . . . . . . . 467
Event/Automation Service Abends . . . . . . . . . . . . . . . . . . . . . . . . . . . 468
Event/Automation Service Task Might be Suspended . . . . . . . . . . . . . . . . . . . . . 468
START, STOP, or RECYCLE Commands Do Not Function Properly . . . . . . . . . . . . . . . . 469
A Service Does Not Complete Initialization . . . . . . . . . . . . . . . . . . . . . . . . 469
Event/Automation Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . 469
Alert Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470
Confirmed Alert Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . 470
Message Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . 471
Confirmed Message Adapter Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . 472
Event Receiver Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472
Trap-to-Alert Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . 473
Alert-to-Trap Service Fails to Initialize . . . . . . . . . . . . . . . . . . . . . . . . . . 473
Unwanted Services are Starting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
Alerts Are Not Forwarded to the Expected Event Server . . . . . . . . . . . . . . . . . . . . 474
Alerts Are Not Converted to the Expected IBM Tivoli Enterprise Console Events . . . . . . . . . . . . 477
An Alert Is Continuously Forwarded . . . . . . . . . . . . . . . . . . . . . . . . . . 477
An Alert Is Incorrectly Cached . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478
| Messages Are Not Forwarded to the Expected Event Server . . . . . . . . . . . . . . . . . . . 479
Messages Are Not Converted to IBM Tivoli Enterprise Console Events . . . . . . . . . . . . . . . 481
A Message Is Incorrectly Cached . . . . . . . . . . . . . . . . . . . . . . . . . . . . 481
IBM Tivoli Enterprise Console Events Are Not Forwarded to the Hardware Monitor . . . . . . . . . . 482
IBM Tivoli Enterprise Console Events Are Not Converted to Alerts . . . . . . . . . . . . . . . . 483
| No Reply from an Event Server to which a Tivoli Enterprise Console Event Was Sent . . . . . . . . . . 484
| Negative Response from an Event Server to which a Tivoli Enterprise Console Event Was Sent . . . . . . . 485
SNMP Traps Are Not Forwarded to the Hardware Monitor . . . . . . . . . . . . . . . . . . . 485
SNMP Traps Are Not Converted to Alerts . . . . . . . . . . . . . . . . . . . . . . . . . 486
Recycling the NetView PPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
Recycling the Event Receiver for IP Connectivity Problems . . . . . . . . . . . . . . . . . . . 487
Recycling the Trap-to-Alert Service for IP Connectivity Problems . . . . . . . . . . . . . . . . . 487
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. Component ID:
4. Recommended service update (RSU) level:
System-Related Information
Record the following system-related information:
1. Operating system and RSU level:
2. Access method and maintenance level:
3. TCP/IP stack and maintenance level:
4. IBM Tivoli Event Console server level (if applicable):
5. Portmapper service level (if applicable; not necessary if you are using the
Portmapper service that was provided with your TCP/IP stack):
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
6. Have you made any recent changes to the system?
v Changed or added hardware:
v Applied software maintenance:
v Other:
7. Can you re-create the problem with the NetView trace running default options
and the E/AS running VERBOSE tracing on the services that are failing?
Problem Classification
Complete the problem category below that matches the symptoms associated with
your problem:
Abend Problems
For abends or processor exception problems, complete the following items:
1. What is the abend code?
2. What processes were taking place at the time of the abend?
3. Gather the following documentation before contacting IBM Software Support:
v A copy of the network log
v A copy of the trace log
v The first unformatted dump of the abend
v A completed E/AS problem worksheet
v A copy of any E/AS trace output
v A copy of the E/AS output log
v A copy of the MVS system log
v The configuration files for the services that are failing. Include your start-up
procedure and global initialization file (IHSAINIT).
4. Gather the following information from the dump:
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID:
v Does the message contain any return codes, reason codes, feedback codes,
error codes, or sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
4. If the message was unexpected and cannot be corrected by following the
actions in the NetView online help, gather the following documentation before
calling IBM Software Support:
v A hard copy of the network log
v The message ID:
v The exact text of the message in the MVS system log
v A completed E/AS problem worksheet
v A copy of the E/AS output log
v The configuration files for the services that are failing. Include your start-up
procedure and global initialization file (IHSAINIT).
5. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
6. Did the message text differ from what was published?
v Have local modifications been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Loop Problems
For loop problems, complete the following items:
1. Are TECROUTE and TRAPROUT filters set to PASS?
2. What events led up to the loop?
3. What data was being displayed?
4. What was the last command entered?
5. What are the modules involved in the loop?
6. What are the dates that the modules were compiled?
7. What are the PTF levels of the modules involved in the loop?
8. Gather the following documentation before calling IBM Software Support:
v A copy of the network log
Wait Problems
For wait problems, complete the following items:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. Gather the following documentation before calling IBM Software Support:
v A copy of the NetView network log
v A copy of the NetView trace log
v A completed E/AS problem worksheet
v A copy of the E/AS output log
v A copy of the MVS system log
v The configuration files for the services that are failing. Include your start-up
procedure and global initialization file (IHSAINIT).
5. What is the name of the module in which the wait occurred?
6. What is the date that the module was compiled?
7. What is the PTF level of the module involved?
8. What is the offset into the module where the wait occurred?
Performance Problems
For performance problems, complete the following items:
1. What were the events that led to the problem?
2. What is the actual performance?
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program, call IBM
Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems” and Chapter 3, “Documenting
and Reporting Problems” before contacting IBM Software Support.
Table 163. Event/Automation Service Problem Scenarios
Problem Category Problem Scenario Page
Abend Event/Automation Service Abends 468
Suspended Task Event/Automation Service Task Might be Suspended 468
START, STOP, or RECYCLE Commands Do Not Function Properly 469
Initialization A Service Does Not Complete Initialization 469
Event/Automation Service Fails to Initialize 469
Alert Adapter Fails to Initialize 470
| Confirmed Alert Adapter Fails to Initialize 470
Message Adapter Fails to Initialize 471
Confirmed Message Adapter Fails to Initialize 472
Event Receiver Fails to Initialize 472
Trap-to-Alert Service Fails to Initialize 473
Alert-to-Trap Service Fails to Initialize 473
Unwanted Services are Starting 474
Alert Problems Alerts Are Not Forwarded to the Expected IBM Tivoli Enterprise 474
Console Server
Alerts Are Not Converted to the Expected IBM Tivoli Enterprise 477
Console Events
An Alert Is Forwarded Continuously 477
An Alert Is Not Cached Correctly 478
Message Problems Messages Are Not Forwarded to the IBM Tivoli Enterprise Console 479
Server
Messages Are Not Converted to IBM Tivoli Enterprise Console Events 481
| If multiple Tivoli Enterprise Console servers or event servers are specified on the
| ServerLocation statement, and there are connectivity problems to each of the
| servers specified by the statement, the recovery time is additive. As a connection is
| attempted to each Tivoli Enterprise Console server or event server (in order), the
| suspended interval for the service seems to be longer for each server that cannot
| be connected.
| The message, confirmed message, alert, and confirmed alert adapter services
| provide TCP/IP state information with the DISPLAY STATUS command. This
| information is helpful in determining if and where a service is suspended. See
| Chapter 25, “Diagnostic Tools for the Event/Automation Service,” on page 489 for
| more information about using the DISPLAY STATUS command.
For the event receiver service, problems accessing the local Portmapper can cause
the Portmapper access function calls to hang up. This is based on a timer
determined within the Portmapper access functions. This problem does not occur if
the UsePortMapper statement is set to NO. These problems usually occur as a
result of the Portmapper service not being active, or the service is terminating
while the Event Receiver service is active.
The Event/Automation Service issues a console message indicating the reason for
the failure. The Event/Automation Service can fail to initialize for the following
reasons:
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 469
v The global configuration file (IHSAINIT is the default) cannot be found. If you
are providing a customized configuration file, make sure you specified it
correctly in the startup procedure. Also ensure that the IHSSMP3 DD statement
in the IHSAEVNT start-up procedure is correct.
v The global configuration file contains incorrect statements. If you are providing a
customized configuration file, make sure that all of the statements in the file are
correct.
v The PPI mailbox identifier used by the Event/Automation Service is in use. The
likely cause is that there is another copy of the Event/Automation Service that
has been started with the same PPI mailbox identifier.
The alert adapter issues a console message indicating the reason for the failure.
The alert adapter can fail to initialize for the following reasons:
v The alert adapter configuration file cannot be found.
IHSAACFG is the default. If you are providing a customized configuration file,
verify that you specified it correctly either on the start-up procedure or in the
global initialization file ALRTCFG statement.
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The alert adapter configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The alert adapter CDS file cannot be found.
IHSAACDS is the default. If you are providing a customized CDS file, verify
that you specified it correctly on the AdapterCdsFile statement in the
configuration file.
v The alert adapter CDS file contains incorrect statements.
If you are providing a customized CDS file, verify that all of the statements in
the file are correct. Additional information in the alert adapter output log
provides the line number and line character position where the error was
detected.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced; the character position is the first place
on the line that was found to be syntactically incorrect.
The confirmed alert adapter issues a console message indicating the reason for the
failure. The confirmed alert adapter can fail to initialize for the following reasons:
v The confirmed alert adapter configuration file cannot be found.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced; the character position is the first place
on the line that was found to be syntactically incorrect.
The message adapter will issue a console message indicating the reason for the
failure. The message adapter can fail to initialize for the following reasons:
v The message adapter configuration file cannot be found.
The default is IHSAMCFG. If you are providing a customized configuration file,
verify that you specified it correctly either on the start-up procedure or in the
global initialization file MSGCFG statement.
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The message adapter configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The message adapter FMT file cannot be found.
The default is IHSAMFMT. If you are providing a customized FMT file, verify
that you specified it correctly on the AdapterFmtFile statement in the
configuration file.
The message adapter FMT file contains incorrect statements. If you are
providing a customized FMT file, make sure that all of the statements in the file
are correct. Additional information in the message adapter output log will
indicate the line number and line character position where the error was
detected.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced. The character position is the first place
on the line that was found to be syntactically incorrect.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 471
Confirmed Message Adapter Fails to Initialize
If the confirmed message adapter fails to initialize correctly, it will end. The
Event/Automation Service DISPLAY STATUS command will display the status of
the message adapter as DOWN.
The confirmed message adapter will issue a console message indicating the reason
for the failure. The confirmed message adapter can fail to initialize for the
following reasons:
v The confirmed message adapter configuration file cannot be found.
The default is IHSANCFG. If you are providing a customized configuration file,
verify that you specified it correctly either on the start-up procedure or in the
global initialization file CMSGCFG statement.
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The confirmed message adapter configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The confirmed message adapter FMT file cannot be found.
The default is IHSANFMT. If you are providing a customized FMT file, verify
that you specified it correctly on the AdapterFmtFile statement in the
configuration file.
The confirmed message adapter FMT file contains incorrect statements. If you
are providing a customized FMT file, make sure that all of the statements in the
file are correct. Additional information in the message adapter output log will
indicate the line number and line character position where the error was
detected.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced. The character position is the first place
on the line that was found to be syntactically incorrect.
The event receiver will issue a console message indicating the reason for the
failure. The event receiver can fail to initialize for the following reasons:
v The event receiver configuration file cannot be found.
The default file is IHSAECFG. If you are providing a customized configuration
file, verify that you specified it correctly either in the start-up procedure or in
the global initialization file ERCVCFG statement.
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The event receiver configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The event receiver CDS file cannot be found. The default file is IHSAECDS.
If you are providing a customized CDS file, verify that you specified it correctly
on the AdapterCdsFile statement in the configuration file.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced; the character position is the first place
on the line that was found to be syntactically incorrect.
The trap-to-alert service will issue a console message indicating the reason for the
failure. The trap-to-alert service can fail to initialize for the following reasons:
v The trap-to-alert service configuration file cannot be found.
The default file is IHSATCFG. If you are providing a customized configuration
file, verify that you specified it correctly either in the start-up procedure or in
the global initialization file TALRTCFG statement.
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The trap-to-alert service configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The trap-to-alert service CDS file cannot be found.
The default file is IHSATCDS. If you are providing a customized CDS file, verify
that you specified it correctly on the AdapterCdsFile statement in the
configuration file.
The trap-to-alert service CDS file contains incorrect statements. If you are
providing a customized CDS file, make sure that all of the statements in the file
are correct. Additional information in the trap-to-alert service output log will
indicate the line number and line character position where the error was
detected.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced; the character position is the first place
on the line that was found to be syntactically incorrect.
The alert-to-trap service will issue a console message indicating the reason for the
failure. The alert-to-trap service can fail to initialize for the following reasons:
v The alert-to-trap service configuration file cannot be found.
The default file is IHSAATCF. If you are providing a customized configuration
file, verify that you specified it correctly either in the start-up procedure or in
the global initialization file ALRTTCFG statement.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 473
Also ensure that the IHSSMP3 DD statement in the IHSAEVNT start-up
procedure is correct.
v The alert-to-trap service configuration file contains incorrect statements.
If you are providing a customized configuration file, make sure that all of the
statements in the file are correct.
v The alert-to-trap service CDS file cannot be found.
The default file is IHSALCDS. If you are providing a customized CDS file, verify
that you specified it correctly on the AdapterCdsFile statement in the
configuration file.
v The alert-to-trap service CDS file contains incorrect statements.
If you are providing a customized CDS file, make sure that all of the statements
in the file are correct. Additional information in the alert-to-trap service output
log will indicate the line number and line character position where the error was
detected.
Note: The first character position is position 0. The actual error can be ahead of
the character position referenced; the character position is the first place
on the line that was found to be syntactically incorrect.
v The alert-to-trap service is not authorized to get the DPI® port number from the
SNMP agent.
The view access defined for the community name provided to the alert-to-trap
service does not allow the alert-to-trap service to retrieve the DPI port number
from the SNMP agent. Ensure that the SNMP agent configuration file allows
access to dpiPort for the community name defined to the alert-to-trap service.
For information about configuring an SNMP agent, see the z/OS Communications
Server IP Configuration Reference.
The sample global initialization file contains NOSTART statements for the
alert-to-trap service and the trap-to-alert service. If you do not plan to use one or
more of the alert adapter, message adapter, or event receiver services, add
NOSTART statements for these services to the global initialization file. You can
receive unexpected error message if you allow a service to start without correctly
configuring the service.
You can start a service after the Event/Automation Service has started without
affecting the operation of currently running services using the Event/Automation
Service START command.
| Where date is the date string trace header and line is a number. Note that if
| the confirmed alert adapter service had been used, the message would contain
| ALERTC.
If the alert has been discarded, verify the changes that were made to the
| adapter CDS file. Also, verify any changes that you might have made to the
data before it was sent from the hardware monitor; check if you added any
variable bindings to the data that are also referenced in the CDS file. If you
have incorrectly specified a variable binding within the NetView address
space, and are matching on that variable binding in the CDS file, the alert will
be discarded if it does not meet the match criteria.
8. Has the converted alert been discarded because of a Filter/FilterMode
statement setting? The default configuration file does not contain any filter
statements, so the alert is not filtered unless you added these statements to the
| adapter configuration file. If you have added one or more of these statements,
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 475
you can determine whether an event has passed the filtering conditions by
| turning on the IP trace option for the adapter service and generating the alert.
Examine the alert adapter output log for the message:
date ALERTA :IHSAACOM:line IP: The event was discarded due to filtering;
filtering mode is mode
Where date is the date string trace header, line is a number, and mode is either
| IN or OUT. Note that if the confirmed alert adapter service had been used, the
| message would contain ALERTC. Check the filter statements and the
corresponding FilterMode setting to verify that you have specified the correct
filter criteria.
| 9. Has the converted alert been discarded or cached because of event server
connection problems? The IHS0192I message IHS0192I Alert Adapter: Server
connections are suspended is sent to the system console whenever an event
| cannot be sent to any of the event servers listed on the ServerLocation
statement. Any event sent after IHS0192I (including the event that caused the
message) is displayed , and before the IHS0193I Alert Adapter: Server
connections have been resumed message is received, is either discarded or
| cached. Note that if the confirmed alert adapter service issued messages
| IHS0192I and IHS0193I, the Confirmed Alert Adapter would be indicated in
| the message text.
| Use the IP trace option of the adapter service to determine why a connection
| cannot be made to an event server. These are some possible causes of
connection problems:
| v TCP/IP is not active on the local host or at the event server.
v The portmapper is not active at the Tivoli Enterprise Console server. This is
only required if the ServerPort that corresponds to the Tivoli Enterprise
| Console server on the ServerLocation statement is zero (0). The confirmed
| alert adapter does not interact with the portmapper.
v The location on the ServerLocation statement or the port on the ServerPort
statement is incorrect.
v The Event Server application is not running.
| 10. The converted alert was sent to one of the event servers specified on the
| ServerLocation statement. If the event is not showing at the expected event
server, check the following items:
| v If you have more than one event server specified on the ServerLocation
| statement, is the order correct? The alert adapter service or the confirmed
| alert adapter service forwards a converted alert to the first event server to
which it can connect.
v Have you installed and activated the .baroc and .rls files at the Tivoli
Enterprise Console server that are required for the server to recognize
converted alert events?
Note: The user1 through user5 slots are preconfigured into the alert adapters
.baroc file for all events that are subclasses of the SNA_Event class. All
classes defined in the default CDS file for the alert adapter are
subclasses of the SNA_Event class. If you changed the CDS file to use
these slots with the predefined classes in the CDS file, or with any
newly defined class that is a subclass of the SNA_Event class, no
modifications are necessary to the .baroc file for these slots.
2. Have you bound additional names and values to the alert data using the
NetView automation table which are not showing in the Tivoli Enterprise
Console event? If so, check the following in addition to the suggestions in the
previous step:
v Make sure the command list that performs the name bindings is being called
from the automation table when the alert is driven.
| v If possible, dump the alert buffer from the PPI PIPE stage with the
| TECROUTE keyword (used to route message data to the message adapter) or
| the TECRTCFM keyword, (used to route message data to the confirmed
| adapter), whichever applies.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 477
The NetView hardware monitor prevents certain instances of conversion loops.
Any alert forwarded to the hardware monitor from the event receiver service will
not be forwarded back to the alert adapter service using the TECROUTE filter.
Likewise, an alert that is forwarded to the hardware monitor from the trap-to-alert
service will not be forwarded back to the alert-to-trap service using the
TRAPROUT filter. As a result, an alert that originates from the NetView hardware
monitor can be looped back to the hardware monitor, but the new alert will not be
sent back to the alert adapter or alert-to-trap service using the same filter on which
it originated.
The NetView hardware monitor does not prevent a conversion loop that involves
both the TECROUTE and TRAPROUT filters together. The NetView hardware
monitor does not prevent an alert that came from the event receiver service from
being forwarded back to the alert-to-trap service using the TRAPROUT filter. It
also does not prevent an alert that came from the trap-to-alert service from being
forwarded back to the alert adapter service using the TECROUTE filter. If you are
using both the TECROUTE and TRAPROUT filters together, you can prevent a
conversion loop from occurring by:
v Not using the event receiver service or the trap-to-alert service.
v Insuring that a Tivoli Enterprise Console event created by the alert adapter
service is not returned to the event receiver.
v Insuring that an SNMP trap created by the alert-to-trap service is not returned to
the trap-to-alert service.
Where date is the date string trace header and line is a number. If the event was
| discarded because of event buffer filtering, the adapter service output log will
contain the message:
date ALERTA :IHSAACOM:line IP: The event was discarded.
| Note that if the confirmed alert adapter service issued the message, then ALERTC
| would be indicated in the message text. You see either of these messages if the
| alert was processed by the adapter service and cannot be sent to any of the event
| servers (for example, Tivoli Enterprise Console server) from the ServerLocation
statement. Use the IP trace output to verify that the alert was processed and not
sent.
To determine why the event is either cached or not cached, check the following
items:
v Is the BufferEvents statement set to the correct value?
v Is the BufEvtPath statement set to the correct value?
v If you have FilterCache statements, are they correctly specified?
v Is the FilterMode statement set to the correct value?
v Did you see any console messages that indicated a problem with accessing the
cache file?
| Where date is the date string trace header and line is a number. Note that if the
| confirmed message adapter service issued the message, then MESSAGEC would
| be indicated in the message text.
If the message has been discarded, verify the changes that were made to the
message adapter FMT file.
7. Has the converted message been discarded because of a Filter or FilterMode
statement setting? The default configuration file does not contain any filter
statements, so the message will not be filtered unless you added these
| statements to the message adapter or the confirmed message adapter
configuration file. If you have added one or more of these statements, you can
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 479
determine whether an event has passed the filtering conditions by turning on
the IP trace option for the adapter service and generating the message. Examine
the adapter output log for the message:
date MESSAGEA:IHSAACOM:line IP: The event was discarded due to filtering;
filtering mode is mode
Where date is the date string trace header, line is a number, and mode is either
| IN or OUT. Note that if the confirmed message adapter service issued the
| message, then MESSAGEC would be indicated in the message text.
Check the Filter statements and the corresponding FilterMode setting to verify
that you specified the correct filter criteria.
| 8. Has the converted message been discarded or cached because of event server
connection problems? Message IHS0192I Message Adapter: Server
connections are suspended is sent to the system console whenever an event
| cannot be sent to any of the event servers listed on the ServerLocation
statement. Any event sent after IHS0192I shows (including the event that
caused the message), and before message IHS0193I Message Adapter: Server
| connections have been resumed is received, is either discarded or cached. Note
| that if the confirmed message adapter service issued the message, then
| Confirmed Message Adapter would be indicated in the message text.
| Use the IP trace option of the adapter service to determine why a connection
| cannot be made to an event server. These are some possible causes of
connection problems:
| v TCP/IP is not active on the local host or at the event server.
v The portmapper is not active at the Tivoli Enterprise Console server. This is
only required if the ServerPort that corresponds to the Tivoli Enterprise
| Console server on the ServerLocation statement is zero (0). The portmapper
| function is not used by the confirmed message adapter.
v The location on the ServerLocation statement or the port on the ServerPort
statement is incorrect.
v The Event Server application is not running.
| 9. The converted message was sent to one of the event servers specified on the
| ServerLocation statement. If the event is not showing at the expected event
| server, check the following items:
| v If you have more than one event server specified on the ServerLocation
| statement, is the order correct? The message adapter service or the confirmed
| message adapter service forwards a converted message to the first event
| server to which it can connect.
v Have you installed and activated the .baroc and .rls files at the Tivoli
Enterprise Console server that are required for the server to recognize
converted message events?
Note: This is not the same as the CDS file matching order, which starts with
the first statement in the file.
v Have you changed the Tivoli Enterprise Console server .baroc file to match
the customization in the FMT file?
Note: The user1 through user5 slots are preconfigured in the message adapter
.baroc file for all events that are subclasses of the NV390MSG_Event
class. All classes defined in the default FMT file for the message
adapter are subclasses of the NV390MSG_Event class. If you have
changed the FMT file to use these slots with the predefined classes in
the FMT file, or with any newly defined class that is a subclass of the
NV390MSG_Event class, then no modifications are necessary to the
.baroc file for these slots.
2. Have you bound additional names and values to the alert data using the
NetView automation table which are not showing in the Tivoli Enterprise
Console event? If so, check the following items:
v Make sure that the command list that performs the name bindings is being
driven from the automation table when the message is driven.
| v If possible, dump the message buffer from the PPI pipe stage (with either the
| TECROUTE or TECRTCFM keyword) to the NetView console. The variable
| binding data is displayed in the EBCDIC translation of the hexadecimal data.
| Verify that the binding is present in this data.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 481
Where date is the date string trace header and line is a number. If the event was
| discarded because of event buffer filtering, the adapter service output log contains
the message:
date MESSAGEA :IHSAACOM:line IP: The event was discarded.
| Note that if the confirmed message adapter service issued the message, then
| MESSAGEC would be indicated in the message text.
| You see either of these messages if the message was processed by the message
| adapter or confirmed message adapter service and cannot be sent to any of the
| event servers from the ServerLocation statement. Use the IP trace output to verify
that the message was processed and not sent.
To determine why the event is either cached or not cached, check the following
items:
v Is the BufferEvents statement set to the correct value?
v Is the BufEvtPath statement set to the correct value?
v If you have FilterCache statements, are they correctly specified?
v Is the FilterMode statement set to the correct value?
v Did you see any console messages that indicated a problem with accessing the
cache file?
| v For a confirmed alert adapter or confirmed message adapter, did the IP trace
| show that a complete response TEC event was received? Was it a positive or
| negative response or was the data not valid?
| Note: The event receiver receives events in a similar manner to the Tivoli
| Enterprise Console server.
2. Is the event receiver service active?
Use the DISPLAY STATUS command to verify that the service is UP.
| 3. If the event sender needs to resolve the event receiver port using the
Portmapper, is the UsePortMapper statement value set to YES?
| If not, the event sender will be unable to determine the port to use to connect
to the event receiver.
4. Is TCP/IP active on the local host? Use the DISPLAY STATUS command to
check the status of TCP/IP (UP or DOWN).
5. If the Portmapper service is required, is it active on the local host?
| 6. If the event sender specifies a fixed port for the event receiver, is that same
port specified on the ServerPort statement?
Note: After the event receiver initializes, it will not create trace entries unless
it has received a Tivoli Enterprise Console event.
8. Has the event been discarded by CDS file processing?
CDS file processing converts the Tivoli Enterprise Console event into an alert.
If none of the event matching criteria is met, the event is discarded. This
should not happen unless the CDS file has been customized. The default CDS
file will not discard the event.
To determine whether a Tivoli Enterprise Console event has been discarded,
turn tracing on for the event receiver service at the NORMAL level. After
sending the event, browse the event receiver output log for the following
message:
date EVENTRCV:IHSAKERN:0332 NORMAL:Default action is <*DISCARD*>
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 483
v Is there another class definition statement in front of the class definition
statement? If so, does it also match the criteria in the SELECT segment of the
class definition statement? Matches are searched in order from the first statement
in the CDS file to the first SELECT segment that is matched.
v Are you using the $CDS_GROUP keyword to transition through the statements
in the CDS file in the correct order?
v Do all slot mappings that carry subvector information start with the characters
SV?
v Are you using the character translation escape characters (#< and #>) correctly?
| If there was no reply from an event server to which a Tivoli Enterprise Console
| event was sent, check the location on the ServerLocation statement. It might be
| that the location exists but is not an event server that can issue a reply. There
| might also be other problems at the event server or in the network.
| Note: All offsets are in hexadecimal. Note also that the Msg. length value includes
| the length of everything after the header, meaning from offset x’24’ to the
| end of the event, including the x’0A01’ sequence.
| 0 <START>>
|
| 8 Msg. ID (0)
| C Msg. from (0)
| 10 Msg. to (0)
| 14 Msg. type (0)
| 18 IPC msg. type (0)
| 1C Msg. length (x'26')
| 20 Header data length (specify any value, because the confirmed adapter
| ignores this specification)
| 24 Class name (IHS+;)
| 29 IHSeventID=C45902AB73920A58;
| 45 END
| 48 x'0A01'
| If the event server wanted the sending adapter to immediately enter retry
| processing for or caching of a TEC event, the event server could send a negative
| response TEC event with this format:
| Note: All offsets are in hexadecimal. Note also that the Msg. length value includes
| the length of everything after the header, meaning from offset x’24’ to the
| end of the event, including the x’0A01’ sequence.
| 0 <START>>
|
| 8 Msg. ID (0)
| C Msg. from (0)
| 10 Msg. to (0)
| 14 Msg. type (0)
| In either case, the character data in the response TEC event (the <START>>, END,
| class name, and the IHSeventID=value; slot should be ASCII.
| A negative response can be sent by an event server when the following conditions
| occur:
| v The event server successfully parses a Tivoli Enterprise Console event such that
| the IHSeventID slot can be isolated or extracted from the Tivoli Enterprise
| Console event.
| v A condition occurs while processing the event for which the event server needs
| to communicate with the Event/Automation Service to try another event server
| or cache the event.
Note: After the trap-to-alert service initializes, it will not create trace entries
unless it has received an SNMP trap.
6. Has the event been discarded by CDS file processing? CDS file processing
converts the SNMP trap into an alert. If none of the trap matching criteria is
met, the event can be discarded This should not happen unless the CDS file
was customized. The default CDS file will not discard the event.
To determine whether an SNMP trap has been discarded, set tracing on
(NORMAL level) for the trap-to-alert service. After sending the event, browse
the trap-to-alert output log for the following message:
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 485
date TRAPALRT :IHSAKERN:0332 NORMAL: Default action is <*DISCARD*>
If the PPI is inactive when the Event/Automation Service is started, none of the
requested services will be started until the PPI becomes active.
The event receiver will issue an error message indicating the cause for entering the
recycle mode, and then issue the following error message:
IHS0181E The Event Receiver will continue recycling until
it can successfully define a socket.
This is the last console message that the event receiver will issue until the socket
can be defined. Further messages will only be sent to the event receiver output log.
The recycle period is 60 seconds.
Note: Although the recycle period is 60 seconds, the recycle period might be
longer if the problem is because of the Portmapper service. The Portmapper
functions are blocking functions that can have time-out periods up to 2 to 3
minutes. This time is in addition to the 60 second recycle period.
To determine whether the event receiver is recycling after the initial console
message is issued, issue the DISPLAY STATUS command. While the event receiver
is recycling, the status of the event receiver will be IPCYCLE.
The trap-to-alert service will issue an error message indicating the cause for
entering the recycle mode, and then issue the following error message:
IHS0181E The Trap-to-Alert Conversion will continue recycling until
it can successfully define a socket.
This is the last console message that the trap-to-alert service will issue until the
socket can be defined. Further messages will only be sent to the trap-to-alert
service output log. The recycle period is 60 seconds.
To determine whether the trap-to-alert service is recycling after the initial console
message is issued, issue the DISPLAY STATUS command. While the trap-to-alert
service is recycling, the status of the trap-to-alert service will be IPCYCLE.
Chapter 24. Troubleshooting and Initial Diagnosis for the Event/Automation Service 487
488 Troubleshooting Guide
Chapter 25. Diagnostic Tools for the Event/Automation
Service
This chapter describes the diagnostic tools that are used to isolate and identify the
source of a problem for the Event/Automation Service. This chapter also describes
how to access error logs and run traces using the following resources:
v Output log files
v Trace files
v Online help support
This chapter also provides information for using diagnostic tools to collect problem
determination information such as the following information:
v Event/Automation Service output logs
v Online help for Event/Automation Service commands and error messages
v Event/Automation Service DISPLAY STATUS and DISPLAY QSTATS command
v Event/Automation Service TRACE command
v GENALERT command
v RPCINFO command (TCP/IP services)
v TestMode statement
v Looping the alert adapter and message adapter directly into the event receiver
v Looping the alert-to-trap service directly into the trap-to-alert service
Output Log
The Event/Automation Service produces messages for errors, warnings, and
information. Error messages and other types of messages are written to the output
log. The output log provides information that is helpful in resolving problems.
For information about the format of the output log, refer to the IBM Tivoli NetView
for z/OS Customization Guide.
Using Commands
The following Event/Automation Service commands are helpful for diagnosing
Event/Automation Service problems.
DISPLAY STATUS
Use the Event/Automation Service DISPLAY STATUS command to help determine:
v Whether a service is active, inactive, or recycling
| v Whether the alert adapter, confirmed alert adapter, message adapter, or
| confirmed message adapter services are experiencing delays using TCP/IP
| connection services
v If the local TCP/IP service is active
© Copyright IBM Corp. 1997, 2009 489
v If the NetView PPI is active
v The list of Tivoli Enterprise Console IP addresses that the alert adapter and
message adapter services are using
| v The list of IP addresses of event servers that the confirmed alert adapter and
| confirmed message adapter are using
v The SNMP agent IP address that the alert-to-trap service is using
v The PPI mailbox identifier that the E/AS uses
v The PPI mailbox that the event receiver and trap-to-alert services use to forward
their alerts
v The ports that the event receiver and trap-to-alert service use
A status of UP, DOWN, or CO-IDLE for the service is an idle status. This indicates
that the service is not processing an event.
v If the service ConnectionMode is connection oriented, a status of UP indicates
| that there is no current connection to an event server (for example, Tivoli
| Enterprise Console server) from the ServerLocation statement. This status is
normal if an alert or message has not yet been sent to the service, and indicates
a connectivity problem if at least one alert or message has been sent. If the
service ConnectionMode has no connection, this status is normal regardless of
how many alerts or messages have been sent to the service.
v A status of CO-IDLE occurs if the service is connection oriented. This status
| indicates that a connection exists to the event server (for example, Tivoli
| Enterprise Console server).
v A status of DOWN indicates that the service is not active.
| Note: The GETPORT status does not apply to the confirmed message or
| confirmed alert adapter.
DISPLAY QSTATS
Use the Event/Automation Service DISPLAY QSTATS command to help determine
whether an event (either an alert, message, SNMP trap, or Tivoli Enterprise
Console event) has been received and forwarded within the E/AS. These events
are counted as follows:
| v The TOTAL SENT count for the CONTROL task represents the total of all alerts
| and messages delivered across the PPI for the alert adapter, confirmed alert
| adapter, alert-to-trap, message adapter, and confirmed message adapter services.
v The TOTAL RCVD count for the CONTROL task represents the total of all
converted Tivoli Enterprise Console events and SNMP traps forwarded from the
trap-to-alert service and the event receiver to the NetView alert receiver task.
| v The TOTAL RCVD count for the alert adapter or confirmed alert adapter
| represents the number of alerts that have been forwarded for translation to
| Tivoli Enterprise Console events.
v The TOTAL RCVD count for the alert-to-trap service represents the number of
alerts that have been forwarded for translation to SNMP traps.
| v The TOTAL RCVD count for the message adapter or confirmed message adapter
| represents the number of messages that have been forwarded for translation to
| Tivoli Enterprise Console events.
Trace
General tracing for the Event/Automation Service is not described in detail in this
document. Tracing the Event/Automation Service using the LEVEL parameter
provides diagnostic information that is to be used by an IBM Software Support
representative to resolve problems that cannot be diagnosed using other methods.
| The IP tracing option for the alert adapter, confirmed alert adapter, message
| adapter, and confirmed message adapter services is described in the following
section. Although this option is not described in detail, you can use it to provide
more information on why an event might not have been sent to its expected
destination.
This option generates a much smaller amount of tracing output per event as
compared to the amount of tracing output generated using the LEVEL parameter
for the same event. IP tracing output for an event begins when the event is ready
to be sent through TCP/IP (in the case of the alert adapter, message adapter, or
alert-to-trap services) or received from TCP/IP (in the case of the event receiver
and trap-to-alert services). The output ends for that event when the event is either
sent, cached, or discarded.
Where taskname is the name of the service task. To disable tracing, issue the
command with IP=OFF.
Note: This tracing option does not provide any output if the event is discarded by
filtering before it is ready to be sent.
The output log sample in Figure 74 on page 493 displays the result of IP tracing for
the alert adapter service. For the format of messages sent to the Event/Automation
Service output log, refer to the IBM Tivoli NetView for z/OS Customization Guide.
The messages presented here refer to the specific data portion of the output
message. The message type (msgtype) is IP for all IP trace messages.
Figure 74. Example output of IP tracing for the alert adapter service
Note: The numbers preceding each line are not a part of the output log. They are
inserted for reference purposes. The module line numbers following the
module name in this example might be different on your system.
In Figure 74, a single alert was sent through the alert adapter service. The
configuration file for the alert adapter service contains the following statements:
ServerLocation=nmpipl06
ServerPort=0
ConnectionMode=CO
FilterMode=IN
Filter:Class=SNA_Performance_Degraded;adapter_host=NMPIPL06
The process of resolving the server port might occur for more than one
server. The resolution occurs for each location on the ServerLocation
statement that has a corresponding ServerPort of 0 (zero), until one of the
The IP trace statements used by the message adapter service are similar to those
used by the alert adapter. The IP tracing output by the other services vary based
on the information that is relevant to that service.
NCCF GENALERT
Use the GENALERT command to drive test alerts through the hardware monitor to
the Event/Automation Service. You can verify that the path from the hardware
monitor to the Tivoli Enterprise Console server and the SNMP agent is active using
these test alerts.
RPCINFO
RPCINFO is a TCP/IP services command that enables you to query information
about active portmappers. Use this command to help you determine the following:
v Whether the portmapper is active on a host anywhere in your IP network.
v The ports that have been defined to the portmapper, and which program
number and program version is associated with each port.
The alert adapter and message adapter expect Tivoli Enterprise Console servers to
be registered as program number 100033057. The Tivoli Enterprise Console servers
are registered with version number 1. Likewise, the event receiver attempts to
register with portmapper with the same program number and version number to
emulate a Tivoli Enterprise Console server.
For information about using the RPCINFO command, refer to the TCP/IP library.
| You can also use the TestMode statement to indicate that converted alerts from the
| confirmed alert adapter or converted messages from the confirmed message
| adapter are to be sent to a debugging file rather than forwarded to an event server.
| Use the value YES in the TestMode statement to place the Tivoli Enterprise
| Console events into the debugging file. At that time, confirmations are not
| expected because the Tivoli Enterprise Console events are not being sent to an
| event server that can confirm them.
For more information about the TestMode and ServerLocation statements, refer to
the IBM Tivoli NetView for z/OS Administration Reference.
To loop either adapter back to the event receiver, specify the local host name or IP
address on the ServerLocation statement. If the event receiver is not configured to
use the portmapper, specify the event receivers ServerPort value on the adapters
ServerPort statement.
If you use the GENALERT command to generate an alert, notice that two alerts
will be displayed on the NPDA Alerts-Dynamic panel: one for the alert originated
with the GENALERT command and another for the same alert after it has been
converted into a Tivoli Enterprise Console event and then converted back into an
alert and forwarded to the NetView hardware monitor. The resource name for all
alerts generated by the event receiver is NV390ALT.
To loop either service, configure the SNMP agent that receives the SNMP trap
generated by the alert-to-trap service to forward the trap to the trap-to-alert
service, which is an SNMP manager. If your SNMP agent provides the capability of
specifying a port, use the same port that is specified in the PortNumber statement
in the trap-to-alert configuration file.
If you use the GENALERT command to generate an alert, notice that two alerts
will be displayed on the NPDA Alerts-Dynamic panel: one for the alert originated
with the GENALERT command and another for the same alert after it has been
converted into an SNMP trap and then converted back into an alert and forwarded
back to the NetView hardware monitor. The resource name for all alerts generated
by the trap-to-alert service is the first 8 characters of the IP address that originated
the trap.
Chapter 27. Troubleshooting and Initial Diagnosis for the NetView Web Application . . . . . . . . . 505
Web Application Cannot Be Started . . . . . . . . . . . . . . . . . . . . . . . . . . . 505
Web Pages Are Not Displaying On a Browser . . . . . . . . . . . . . . . . . . . . . . . 506
Events Are Not Displayed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507
Unexpected Signon Panel or Browser Session Timeout. . . . . . . . . . . . . . . . . . . . . 507
| Problems Viewing NetView Web Services Gateway Data . . . . . . . . . . . . . . . . . . . . 508
Problems Viewing OMEGAMON XE Mainframe Network Data . . . . . . . . . . . . . . . . . 508
CT_Get Request Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509
Performance Data Cannot Be Viewed . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Open Incident Button Not Enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
3270 Console Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 510
Task Assistant and Task Buttons Do Not Work . . . . . . . . . . . . . . . . . . . . . . . 510
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
The following information is required for all problems:
1. Date:
2. Problem Number:
3. ID of the host domain you are trying to access:
4. Web application server name or IP address:
5. Web application build level; locate this information in one of the following
ways:
v If you can open the NetView Web application, obtain the build level from the
About information displayed in the work area.
v Obtain the build level from the netview_installation_dir/doc/
znetview_webapp.gen file.
6. Copies of the current web.xml and nvim.xml files
7. A copy of the current CNMSTYLE %INCLUDE member CNMSTWBM
| 8. A copy of the WebSphere trace log
Problem Classification
Check one of the following appropriate problem categories that matches the
symptoms associated with your problem.
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID:
v The exact text of the message on the log.
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands:
v Other:
4. Did you follow the actions in the NetView online help? If so:
v What occurred?
v Is this what was expected?
v If not, what was expected?
5. Did the message text differ from what was published?
v Has local modification been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Wait Problems
For wait problems, complete the following items:
1. What is the scenario leading to the problem?
2. What data was being displayed?
3. What was the last command entered?
4. If the wait is occurring at the NetView host, see Part 2, “Diagnosing the
NetView Program,” on page 41.
Performance Problems
For performance problems, complete the following items:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, and title of the manual or the number
of the online help panel involved.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the Web application, call IBM
Software Support.
5. If the problem is with an online help panel, call IBM Software Support.
If you are unable to solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems,” on page 9 and Chapter 3,
“Documenting and Reporting Problems,” on page 19 before contacting IBM
Software Support.
Table 164. NetView Web Application Problem Scenarios
Problem Category Problem Scenario Page
Incorrect output Application cannot be started 505
Incorrect output Web pages not displaying on a browser 506
Incorrect output Events are not displayed 507
Incorrect output Unexpected signon panel presentation or browser session timeout 507
| Incorrect output NetView Web Services Gateway data problems 508
®
Incorrect output OMEGAMON XE data display problems 508
Incorrect output Performance data cannot be viewed 510
Incorrect output Open Incident button is not enabled 510
Incorrect output 3270 console problems 510
Incorrect output Task Assistant and task buttons do not work 510
If the correct Web address is being used, check for a port conflict with another
application by looking at the WebSphere Application Server log. For the location of
this log, see the Web application readme file (netview_installation_dir/doc/
znetview_webapp_readme_en.htm).
If the WebSphere Application Server log contains the following message, the port
specified for the Web application is already being used by another application:
If you are using WebSphere Application Server, use the administrative console to
view and change the port settings.
| If you are using the embedded version of the IBM WebSphere Application Server,
display the current port settings by running the following command from the
netview_installation_dir directory:
nvsrvc config -show
The output for this command, which is similar to the following example output,
shows your current port settings:
The following list shows the meaning of some of the values displayed in the
example output:
v WC_defaulthost is the nonsecure application port.
v WC_adminhost is the nonsecure WebSphere administrative console port.
v WC_defaulthost_secure is the secure application port.
v WC_adminhost_secure is the secure WebSphere administrative console port.
v SOAP_CONNECTOR_ADDRESS is the SOAP connector port for IBM Tivoli
OMEGAMON XE for Mainframe Networks.
| To change the port numbers used by the embedded version of the IBM WebSphere
| Application Server, you must uninstall and reinstall the Web application specifying
different port numbers. For more information, see the Web application readme file
(netview_installation_dir/doc/znetview_webapp_readme_en.htm) and the IBM Tivoli
NetView for z/OS Installation: Configuring Additional Components manual.
Chapter 27. Troubleshooting and Initial Diagnosis for the NetView Web Application 507
| Problems Viewing NetView Web Services Gateway Data
| Follow these steps to use the generic SOAP client to verify the output of the
| command you sent:
| 1. Start Internet Explorer version 7 or higher or Firefox 2.0.0.14 or higher.
| 2. Enable the Access data sources across domains option in the security settings
| for the domain in which your server is located. Enter either of the following
| addresses for the SOAP client:
| v http://netviewhost:port/znvsoatx.htm
| v https://netviewhost:port/znvsoatx.htm
| The SOAP client HTML page is displayed.
| 3. In the NetView for z/OS Generic SOAP Client panel, enter the Endpoint, for
| example:
| http://netviewhost:port/znvsoa
| 4. Enter the SOAP method:
| DoCmd
| After you enter the method, the other fields are completed automatically.
| 5. Modify the tags or the text in the Edit Payload (XML) as shown in the
| following example:
| <Name>sysadmin</Name><Password>passwd</Password>
| <NVCMD><cmd><![CDATA[nvcmd]]</cmd></NVCMD>
| where sysadmin and passwd define the NetView operator ID and password
| under which to run the command, and nvcmd is the NetView command to run.
| 6. Click Make SOAP Request. The output of the request is displayed in the
| SOAP Response Payload field.
| If you encounter problems, you can use the SOACTL command to enable tracing.
| The trace entries are written to the network log. For more information on the
| SOACTL command, see IBM Tivoli NetView for z/OS Command Reference Volume 1
| (A-N).
Note: Be sure to enable the Access data sources across domains option in the
security settings.
2. Enter the following address for the SOAP client:
http://localhost:1920///tems/soap/kshhsoap.htm
The localhost address can be used when accessing the SOAP server running on
the same system but must be changed to the proper host name or network
address of a SOAP server running on a different system. The default port
number is 1920. The tems extension refers to the Tivoli Enterprise Management
Server service point name.
The SOAP client HTML page is displayed. In the CT_SOAP Generic SOAP
Client panel, enter Endpoint, which is http://localhost:1920///tems/soap/ for
this example.
508 Troubleshooting Guide
3. Set the SOAP method to CT_Get. After you select the method, the other fields
are filled in automatically.
4. Modify the tags or the text in the Edit Payload (XML) as shown in the
following example:
<CT_Get>
<userid>user_id</userid>
<password>password</password>
<object>TCPIP_details</object>
<target>remote_agent_name</target>
<afilter>local_IP_address</afilter>
<afilter>local_port<afilter>
<afilter>remote_IP_address</afilter>
<afilter>remote_port<afilter>
</CT_Get>
See “CT_Get Request Example” for more information about the tags and values
for the CT_Get request.
5. Click Make SOAP Request. The output of the request is displayed in the
SOAP Request Payload field. Compare this output with one displayed using
the ″Viewing OXEMFN data at NetView web application″ function.
Note: When issuing a CT_Get request for a particular agent type, the Tivoli
Enterprise Management Server where the SOAP server is running must
be configured for that agent type. For example, when issuing a CT_Get
request for a mainframe network agent connected to a z/OS Tivoli
Enterprise Management Server, the Tivoli Enterprise Management Server
running the SOAP server must be configured for that mainframe
network agent.
For TN3270 session availability data, repeat this step and change the <object>
value to TN3270_Server_Sess_Avail.
Where:
<object>
The name of the object to be retrieved. This tag is required. If a value for
<object> is not specified, then all the public elements of an object are retrieved.
<userid>
The user ID to access the Tivoli Enterprise Management Server hub. If a value
for <userid> is not provided, the value of nnn.nnn.nnn.nnn is inserted.
<password>
The password to access the Tivoli Enterprise Management Server hub. This tag
is required for logon validation.
Chapter 27. Troubleshooting and Initial Diagnosis for the NetView Web Application 509
<target>
The name of the agent. This tag is optional, but if it is not specified, then the
default value is *ALL, which retrieves all available targets.
<history>
Specify a value of Y to retrieve historical data, if available.
<attribute>
The attribute name of an object. This tag can be specified multiple times.
<afilter>
Returns rows meeting filter criteria, such as attribute; operator; and value
operators. The following values are valid:
v EQ
v GE
v GT
v LE
v LIKE
v LT
v NE
The following like-pattern characters are also valid:
v A percent sign (%) matches any single character.
v An asterisk (*) matches one or more characters. This is supported only for
character attributes.
Multiple <afilter> tags are supported only as conjuncts, for example, when
they are paired with AND statements.
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS Enterprise Management
Agent. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 519
The NetView agent is not displayed in the Navigator view . . . . . . . . . . . . . . . . . . . 520
The NetView agent node unexpectedly goes offline. . . . . . . . . . . . . . . . . . . . . . 520
NetView for z/OS subnode unexpectedly goes offline . . . . . . . . . . . . . . . . . . . . . 521
The Tivoli Enterprise Monitoring Server becomes inactive while the NetView agent is running . . . . . . . 521
The NetView agent workspace has no data . . . . . . . . . . . . . . . . . . . . . . . . 521
The NetView agent workspace has partial data or incomplete data . . . . . . . . . . . . . . . . 524
Message "KFWITM081E The link target can not be found" when attempting to link to the workspace of another
product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524
No NetView agent workspaces available . . . . . . . . . . . . . . . . . . . . . . . . . 525
Workspace names displayed in navigation tree are unreadable . . . . . . . . . . . . . . . . . . 525
NACMD fails with BNH805I during initialization . . . . . . . . . . . . . . . . . . . . . . 525
No commands available from the Take Action window . . . . . . . . . . . . . . . . . . . . 526
No Tivoli NetView for z/OS Enterprise Management Agent situations available . . . . . . . . . . . . 526
Incorrect results when using the icons in the NetView Command Response Summary view to find or sort data 526
Cross-Product links missing from link list . . . . . . . . . . . . . . . . . . . . . . . . . 527
Problems with cross-product linking when linking to an OMEGAMON XE 3.1.0 workspace . . . . . . . . 527
Security problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 527
NetView agent workspaces have no column headings for the table views . . . . . . . . . . . . . . 528
Cannot start the NetView agent . . . . . . . . . . . . . . . . . . . . . . . . . . . . 528
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent . . . . . . 529
Using NetView online message and command help for the NetView agent . . . . . . . . . . . . . . 529
Using the CNMTRACE function for NetView host components of the NetView agent function . . . . . . . 529
Using the NACTL command to troubleshoot the NetView agent . . . . . . . . . . . . . . . . . 531
Using the DISPPI command to troubleshoot a PPI connection between NetView and the NetView agent . . . . 531
Troubleshooting Data Spaces for a given data collector . . . . . . . . . . . . . . . . . . . . 531
Problem determination for a NetView agent . . . . . . . . . . . . . . . . . . . . . . . . 532
Problem determination flow for the NetView agent. . . . . . . . . . . . . . . . . . . . . 532
Determining if the problem was caused by the NetView agent . . . . . . . . . . . . . . . . . 533
Reproducible problems reported as Tivoli Enterprise Portal client problems . . . . . . . . . . . 533
Unreproducible problems reported as Tivoli Enterprise Portal client problems . . . . . . . . . . . 536
Problems reported as Tivoli Enterprise Portal Server problems . . . . . . . . . . . . . . . . 536
Problems affecting the NetView agent . . . . . . . . . . . . . . . . . . . . . . . . 537
NetView agent communication layer messages and tracing . . . . . . . . . . . . . . . . . . 538
RAS Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
RAS Trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 538
| Using the KDC_DEBUG environment variable . . . . . . . . . . . . . . . . . . . . . . 539
Not all of these questions apply to your situation, but knowing specific
background information makes it easier to report problems and find solutions.
General Information
Record the following general information:
1. Date:
2. Problem Number:
3. Host:
v Component ID
v Tivoli NetView for z/OS operating system and level
v Recommended service update (RSU) level
| v NetView agent build level. This can be found in RKLVLOG. The KLVST045
| BASIC SERVICE DRIVER: message (for example, KLVST045 BASIC
| SERVICES DRIVER: tms_ctbs621:d9098a) should be recorded, and the level of
| the KDS component. The KDS component level can be found by searching
| for Component: kds in the log. The line below this contains the driver level.
| For example:
| Component: kds
| Driver: tms621:d9149a/3940608.4
4. The NetView agent data files version, located in the following locations:
Windows:
<install_dir>\InstallITM\ver\<date>)KNAWICMS.ver
UNIX: Version information on UNIX/Linux is stored in the
$install_dir/registry directory:
v The nat*.ver files are the application support version files
– The natms.ver file is for the Tivoli Enterprise Monitoring Server
– The natps.ver file is for the Tivoli Enterprise Portal
– The natpw.ver file is for the Tivoli Enterprise Portal browser
client
– The natpd.ver file is for the Tivoli Enterprise Portal desktop client
Problem Description
Describe your problem by answering the following questions:
1. What are the symptoms of the problem?
2. What were you trying to do?
3. What should have happened?
4. What actually did happen?
5. Has the function worked before?
6. Other?
7. If the problem seems to be at the NetView host, can you recreate the problem
with the NetView trace running default options?
8. If you have more than one Tivoli Enterprise Portal workstation, does the
problem occur consistently on all workstations?
Problem Classification
Check the appropriate problem category that matches the symptoms associated
with your problem.
Abend problems
For abend problems or processor exception problems, complete the following
items:
Processor Traps
For processor exception problems, respond to the following questions:
1. What is the trap code?
2. Is there any other information related to the exception that can be provided?
3. What processes were occurring at the time of the abend or trap?
4. Gather the logs for the components that failed. See the IBM Tivoli Monitoring
Problem Determination Guide, GC32-8458, for information on where logs are
located for Tivoli Management Services components.
Message Problems
For message problems, complete the following items:
1. Record the message ID and any error codes displayed.
v Message ID.
v The exact text of the message on the log.
v Does the message contain any return codes, feedback codes, error codes, or
sense information? List the codes or information.
2. Check the message in the NetView online help to determine user action.
3. What processes were taking place when the message occurred?
v Commands
v Other
4. Did you follow the actions prescribed in the NetView online help? If so:
v What occurred?
Chapter 28. Tivoli NetView for z/OS Enterprise Management Agent Worksheet 515
v Is this what was expected?
v If not, what was expected?
5. Did the message text differ from what was published?
v Has local modification been made to change the message text?
v Has an update been made to the system that might have changed the
message?
Wait Problems
For wait problems, complete the following items:
1. What events led up to the wait?
2. What data was being displayed?
3. What was the last command entered?
4. If the wait appears to be in the NetView address space, follow the problem
classification instructions for wait problems in Chapter 4, “NetView Program
Problem Worksheet,” on page 45.
5. If the wait appears to be in the NetView agent address space, obtain the
following documentation:
v The scenario leading to the problem
v A system log
v NetView agent RKLVLOG
v A dump of the NetView agent address space and the NetView agent data
space or data spaces. The NetView agent data space is titled CNMEMnnn,
where nnn is a number.
6. What is the name of the module in which the wait occurred?
7. What is the date that the module was compiled?
8. What is the PTF level of the module involved?
9. What is the offset into the module where the wait occurred?
| Note: Note that 3270 commands are real-time commands, while the data on the
| TEP is not necessarily real-time data.
6. If expected messages do not show, have messages been filtered out:
v From the message processing facility (MPF)?
v Using the message revision table (MRT)?
v Through the automation table?
v Through installation exits?
7. Gather the following documentation before contacting IBM Software Support:
v A copy of the NetView log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. See “Network Log” on page 96.
v A description of the events leading to the failure.
v The NetView agent RKLVLOG.
v Screen captures if the data on the Tivoli Enterprise Portal is incorrect.
| v OBEY files
| For more information on the DISPLAY TCPIP,,NETSTAT command, see the z/OS
| Communications Server: IP System Administrator’s Commands manual.
Performance Problems
For performance problems, complete the following items:
1. What were the events that led to the problem?
2. What is the actual performance?
3. What was the expected performance?
Chapter 28. Tivoli NetView for z/OS Enterprise Management Agent Worksheet 517
4. What are the data collection intervals for the enabled TEMA subtowers?
5. What are the row values for the enabled TEMA subtowers?
6. How many Tivoli Enterprise Portal users are sending commands to the same
NetView host?
7. Gather the following documentation before contacting IBM Software Support:
v A copy of the NetView log containing the output of several TASKMON,
TASKUTIL, or TASKURPT commands. The log should also contain the
output of several NACTL LISTINFO commands. See “Network Log” on page
96.
v The NetView trace. See “NetView Trace” on page 99.
v Information describing your NetView operating environment.
v Information describing your Tivoli Management Services environment.
v Descriptions of any modifications to your system.
v The NetView agent RKLVLOG.
v A description of the events leading to the failure.
Documentation Problems
For documentation problems, complete the following items:
1. Identify the order number, revision level, title of the manual, the number of the
online help panel involved, the panel identifier of the Configuration Tool help
panel, or the section in the NetView agent online help.
2. Identify the location of the error in the manual or panel. For manuals, provide
the chapter and section name.
3. Describe the problem the error caused.
4. If the problem affects the operation or use of the NetView program or the
NetView agent, contact IBM Software Support.
5. If the problem is with an online help panel, contact IBM Software Support.
If you are cannot solve your problem by using the examples, follow the
instructions in Chapter 2, “Classifying Problems,” on page 9 and Chapter 3,
“Documenting and Reporting Problems,” on page 19 before contacting IBM
Software Support.
Table 166. NetView for z/OS Enterprise Management Agent Problem Scenarios
Problem Category Problem Scenario Page
Incorrect output The NetView agent is not displayed in the Navigator view 520
Incorrect output The NetView agent node unexpectedly goes offline 520
Incorrect output NetView for z/OS subnode unexpectedly goes offline 521
Incorrect output The Tivoli Enterprise Monitoring Server becomes inactive while 521
the NetView agent is running
Incorrect output The NetView agent workspace has no data 521
Incorrect output The NetView agent workspace has partial data or incomplete 524
data
Incorrect output Message "KFWITM081E The link target can not be found" when 524
attempting to link to the workspace of another product
Incorrect output No NetView agent workspaces available 525
Incorrect output Workspace names displayed in navigation tree are unreadable 525
Incorrect output NACMD fails with BNH805I during initialization 525
Incorrect output No commands available from the Take Action window 526
Incorrect output No Tivoli NetView for z/OS Enterprise Management Agent 526
situations available
Incorrect output Incorrect results when using the icons in the NetView Command 526
Response Summary view to find or sort data
Incorrect output Cross-product links missing from link list 527
Incorrect output Problems with cross-product linking when linking to an 527
OMEGAMON XE 3.1.0 workspace
Incorrect output Security problems 527
Incorrect output NetView agent workspaces have no column headings for the 528
table views
Incorrect output Cannot start the NetView agent 528
The status of any agent node or sub-node can be examined on the Managed
System Status Workspace. To display the Managed System Status workspace,
left-click on the topmost node in the Navigation Tree Enterprise and then
right-click on the topmost node to display a context menu for the Enterprise node.
Select Workspace->Managed System Status. In this scenario, the NetView agent
node is online and the subnode is offline.
Depending on the length of time that the Tivoli Enterprise Monitoring Server is
unavailable, some situational and historical data might be lost. The default
heartbeat timer between the Tivoli Enterprise Monitoring Server and the NetView
agent is 10 minutes. Therefore, it might take as long as 10 minutes before the Tivoli
Enterprise Monitoring Server recovers and you can again see data from the
NetView agent.
See the IBM Tivoli NetView for z/OS Installation: Configuring the Tivoli NetView for
z/OS Enterprise Management Agent for additional information.
All other agent workspaces are populated based on a data collection tower or
subtower, or a TEMA tower or subtower in the CNMSTYLE member. No data is
displayed in these workspaces unless the appropriate TEMA tower or subtower is
enabled. The following table shows this information for primary workspaces. The
table also shows the data collection autotask
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS Enterprise Management Agent 521
| Table 167. Data collection towers and subtowers
| Default Data
| Collection
| Workspace Data Collection tower or subtower Autotask Display tower or subtower
| Distributed DVIPA DVIPA.DVROUT AUTOCT4 TEMA.DVROUT
| Connection Routing
| Distributed DVIPA Server DVIPA.DVTAD AUTOCT2 TEMA.DVTAD
| Health
| Distributed DVIPA Targets DVIPA.DVTAD AUTOCT2 TEMA.DVTAD
| DVIPA Connections DVIPA.DVCONN AUTOCT3 TEMA.DVCONN
| DVIPA Definition and DVIPA AUTOCT1 TEMA.DVDEF
| Status
| DVIPA Sysplex Distributors DVIPA.DVTAD AUTOCT2 TEMA.DVTAD
| HiperSockets Configuration DISCOVERY.INTERFACES. AUTOCT5 TEMA.HIPERSOCKETS
| and Status HIPERSOCKETS
| Inactive TCPIP Connection TCPIPCOLLECT.TCPCONN and AUTODC3 TEMA.CONINACT
| Data TEMA.CONINACT
| NetView Applications DISCOVERY AUTOCT7 TEMA
| NetView Tasks TEMA.HEALTH AUTODC1 TEMA.HEALTH
| OSA Channels and Ports DISCOVERY.INTERFACES AUTOCT5 TEMA.OSA
| Session Data TEMA.SESSACT AUTODC4 TEMA.SESSACT
| Stack Configuration and DISCOVERY AUTOAON TEMA.SYSPLEX
| Status
| TCPIP Connection Data TEMA.CONNACT AUTODC2 TEMA.CONNACT
| Telnet Server Configuration DISCOVERY.TELNET AUTOCT6 TEMA.TELNET
| and Status
| VIPA Routes DVIPA.DVROUT AUTOCT4 TEMA.DVROUT
|
v Verify that the related towers and subtowers are enabled for the workspace in
the CNMSTYLE member.
v Verify that the ROWSxxxx value (where xxxx represents the workspace)
specified in the CNMSTYLE member for the display tower or subtower is not
zero.
v Verify that the AUTOTASK associated with the data collector is active and
running data collection commands.
| v For workspaces that have a different data collector than display tower or
| subtower, issue the COLLCTL LISTINFO command and inspect the status of the
| data collector. If the data collector is inactive, issue COLLCTL START with the
| appropriate value.
v For workspaces that have the same data collector and display tower or
subtower, issue the NACTL LISTINFO command and inspect the status of the
data collectors. If the data collector is inactive, issue NACTL START with the
appropriate value.
v Look for message BNH881I in the NetView log. If this message is present, use
the NetView for z/OS message help facility to find more information on the
cause of the failure.
v Check the NetView log for errors on the data collection autotasks. See Table 167
above.
| v VIPA Route and distributed DVIPA connection routing workspaces will contain
| no data unless you are running z/OS V1R11 Communications Server or later.
| v Do these steps if one of the DVIPA workspaces has no data:
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS Enterprise Management Agent 523
| See the NetView for z/OS Enterprise Management Agent online help for
| information on the filters that are defined for the product-provided workspaces.
| See the IBM Tivoli Monitoring: User’s Guide for information on defining and
| customizing workspaces and views.
| v Administrators control which users can access workspaces on a per-workspace
| basis. If an operator is not seeing data for a particular workspace, this operator
| might not have been permitted to this type of data. Refer to the IBM Tivoli
| Monitoring: Administrator’s Guide for more information about permitting and
| restricting operator access to workspaces.
You receive this message if a query is specified that returns a large number of rows
of data, causing an out-of-data condition. The message source, KNATCI, varies
depending on the workspace that fails. To resolve this problem, change the LIMIT
and MINIMUM values. You can also change the parameters on the RKLVIN DD
statement or the EXEC PARM field in the JCL. Additional information on this
methodology can be found in the IBM Tivoli NetView for z/OS Installation:
Configuring the Tivoli NetView for z/OS Enterprise Management Agent.
If you installed the workspaces for products not installed in your environment,
links to these products are valid destinations for dynamic cross-product links. To
prevent the inclusion of misleading links, install only the help files, workspaces,
and situations for products that you have installed.
Note: It is not likely that all OMEGAMON XE monitoring agents will be running
on all z/OS systems being monitored. In such cases, the KFWITM081E
message does not necessarily indicate a problem. For example, if you are
monitoring two z/OS systems and only one of the z/OS systems is running
DB2®, you will most likely have the OMEGAMON XE for Mainframe
Networks monitoring agent running on both systems but the OMEGAMON
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS Enterprise Management Agent 525
v Turn on PPI trace by using the NetView for z/OS TRACEPPI command. For
more information on this or any other NetView for z/OS commands, consult the
IBM Tivoli NetView for z/OS Command Reference Volume 1 (A-N) or else use the
online command Help facility.
If a cross-product link is missing from the link list, contact your system
administrator to verify these items:
v Verify that your user ID is authorized to access the target product.
v Verify that the target workspaces on the product are installed. OMEGAMON XE
| product help files, workspaces, and situations are installed using the IBM Tivoli
| OMEGAMON Data Files for z/OS DVD.
| If the V4.1 or V4.2 of the target workspace is modified (for example, to accept link
| parameters to limit the data displayed) you might notice different behavior when
| you migrate the target product from V3.1 to V4.1 or V4.2. For example, the V3.1 of
| the DB2 thread ID workspace does not filter the data. As a result, if you link to the
| V3.1 of the DB2 thread ID workspace, all threads are displayed. This same
| workspace was modified in V4.1 to accept link parameters to display data for a
| specific thread ID. When you update the OMEGAMON XE for DB2 product to
| V4.1 or V4.2, the data is now filtered to display data for a specific thread ID.
Security problems
If you encounter problems with take action security for the z/OS product that uses
the Tivoli Management Services infrastructure, first ensure that you have
configured the NetView agent or the Tivoli Enterprise Monitoring Server to
support this function. For additional information on the take action security, see
the IBM Tivoli NetView for z/OS Installation: Configuring the Tivoli NetView for z/OS
Enterprise Management Agent.
If you see one of the following symptoms, the NetView CNMLINK data set was
not concatenated as part of the Tivoli enterprise management server or NetView
agent RKANMODL DD statement in the startup procedure:
v RC=17 in the Action Status window after a command is issued
Chapter 29. Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS Enterprise Management Agent 527
v The following message in RKLVLOG: NetView interface module unavailable:
CNMCNETV
If you see any of the following symptoms, the NetView APSERV command is not
running:
v This message in RKLVLOG: NetView PPI send buffer rejected: 26
v RC=9 in the Action Status window after a take action command is issued
v Message KRAIRA002 in RKLVLOG, as in this example:
KRAIRA002, Executed <D A,L> with status 9, Producer(Automation Command)
In RKLVLOG, if you see the NetView PPI send buffer rejected: 24 message, the
NetView for z/OS subsystem interface is not active.
Copy the KNADOC, KNAATR, and KNACAT files from the &rhilev.
&rte.RKANDATV data set to the equivalent data set where the column headings are
in the table views for NetView agent workspaces. Recycle the Tivoli Enterprise
Monitoring Services to pick up the changes.
Using NetView online message and command help for the NetView
agent
v When NACMD terminates, it issues message BNH805I message accompanied by
message DWO050I. Message DWO050I should not be issued on normal
termination. Use the online Help facility to find more information on the failure.
v When data collection stops due to any abnormal cause, BNH881I messages are
logged to the NetView log. Message DWO050I can provide additional details.
Note: The NACMD command is the function designated for all REXX files
associated with the NetView agent. The following values are valid
for the common global variables:
YES or ON
This value shows the entry and exit, the commands to be
issued, and the command responses, if any. This includes
any commands issued indirectly.
04/23/07 NTV77 14:21:29 | CNMETACI 0 120 9512 04/23/07 14:15:37 R 14:21:29 | CNMTRACE
4C8481A3816EF0F461F2F361F0F740F1F47AF2F17AF2F907C3
D5D4C5E3C1C3C9404040404040404040F040404040404040F1
F2F0404040404040F9F5F1F24040F0F461F2F361F0F74040F1
F47AF1F57A
14:21:29 | CNMTRACE NACMD CNME8202 Data (hex):
F3F7404040404040D907F0F0F0F107F0F0F0F107E2E8E2C1C4
D4C9D5074C618481A3816E
Note: Use the online Help facility to find more information on the NACTL
command
Note: Use the online Help facility to find more information on the NACMD
command
Problem Scenario: A Tivoli Enterprise Portal operator notices that data is not being
updated on NetView Health workspace for a long time. The default value for data
collection being used is 30 seconds. To debug this, the operator issues the
command NACTL LISTINFO, which gives the following output:
| 14:52:05 * NACTL LISTINFO
| 14:52:05 C BNH892I DISPLAY DATA COLLECTION STATISTICS
| 14:52:05 | --------------------------------------------------------------------------------
| 14:52:05 | Tower Name : NetView Health
| 14:52:05 | Status : Active , next data collection starts in 4 seconds
| 14:52:05 | Average Time : < 1 Seconds
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 531
| 14:52:05 | Maximum Time : < 1 Seconds
| 14:52:05 | Number Of Iterations : 17102
| 14:52:05 | --------------------------------------------------------------------------------
| 14:52:05 | Tower Name : Active TCP/IP Connections
| 14:52:05 | Status : Active , next data collection starts in 586 seconds
| 14:52:05 | Average Time : < 1 Seconds
| 14:52:05 | Maximum Time : 5 Seconds
| 14:52:05 | Number Of Iterations : 571
| 14:52:05 | --------------------------------------------------------------------------------
| 14:52:05 | Tower Name : Inactive TCP/IP Connections
| 14:52:05 | Status : Active , next data collection starts in 1451 seconds
| 14:52:05 | Average Time : < 1 Seconds
| 14:52:05 | Maximum Time : < 1 Seconds
| 14:52:05 | Number Of Iterations : 143
| 14:52:05 | --------------------------------------------------------------------------------
| 14:52:05 | Tower Name : Active Sessions
| 14:52:05 | Status : Active , next data collection starts in 572 seconds
| 14:52:05 | Average Time : < 1 Seconds
| 14:52:05 | Maximum Time : 3 Seconds
| 14:52:05 | Number Of Iterations : 571
| 14:52:05 | --------------------------------------------------------------------------------
This section provides an overview of service information that you must collect
about the NetView agent and instructions for setting traces and collecting logs for
your own use and to forward to IBM Software Support. These topics are covered:
v “Problem determination flow for the NetView agent”
v “Determining if the problem was caused by the NetView agent” on page 533
v “Understanding and using RAS1 logs” on page 546
v “Capturing z/OS logs to send to software support” on page 546
IBM Software Support uses the information captured by trace logging to trace a
problem to its source or to determine why an error occurred. The default
configuration for trace logging, such as the level of trace logging, depends on the
source of the trace logging. Trace logging is always enabled.
Attention: There is CPU and I/O overhead associated with detailed RAS1 tracing
that might degrade performance of the monitoring agent. You must restore RAS1
tracing to the minimal KBB_RAS1=ERROR after problem diagnosis is completed.
In any problem scenario, all documentation should be gathered at the time of the
error. What appears to be a client problem could very well be a server problem,
especially in the scenario where data is not showing up at the client. Below are
guidelines for collecting the correct documentation for any problems reported.
As you collect logs, create an exact description of the problem. For reproducible
problems, document the exact navigation path that produced the error. Screen
prints might also help in the problem determination.
In your problem report, try to use the correct terminology when describing the
problem (for example, workspaces, views, navigators, events, and links). Consistent
use of the terminology will help IBM Software Support to understand the problem
quickly.
The sections that follow discuss types of problems that you might see and how to
capture information needed to diagnose those problems.
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 533
| Table 168. Log locations for Tivoli Enterprise Portal desktop client
| Com-
| ponent Windows UNIX-based systems
|| Tivoli v install_dir\CNP\kcjerror.log install_dir/logs/
| Enterprise hostname_PC_timestamp.log
|| Portal
v install_dir\CNP\kcjras1
|| desktop When launched via Java Web Start: where:
| client
| %USERPROFILE%\Application Data\ install_dir
| IBM\Java|Deployment\log Specifies the directory where
| \javawsnnnnn.trace Tivoli Enterprise Portal Server
| was installed.
| where nnnnn is a unique, randomly
| generated numeric suffix to support hostname
| generational logs; that is, the last Specifies the name of the
| generated log will not be overlaid by system hosting the product
|| the most current execution of Tivoli
PC Specifies the product code, cq
|| Enterprise Portal using Java Web
for the Tivoli Enterprise Portal
|| Start. This is different from the Tivoli
Server.
| Enterprise Portal Browser client,
| which has a fixed name and is timestamp
| overlaid with each execution cycle. A decimal representation of
| the time at which the process
| was started.
| When launched via Jave Web Start:
| %{user.home}/.java/deploymnet/
| log/javawsnnnnn.trace
| where nnnnn is a unique, randomly
| generated numeric suffix to support
| generational logs; that is, the last
| generated log will not be overlaid by
| the most current execution of Tivoli
| Enterprise Portal using Java Web Start.
| This is different from the Tivoli
| Enterprise Portal Browser client, which
| has a fixed name and is overlaid with
| each execution cycle.
|
v If the Tivoli Enterprise Portal browser client is being used, then collect this log:
C:\Documents and Settings\Administrator\Application Data\
IBM\Java\Deployment\log\plugin1.4.2.trace
The plugin1.4.2.trace file contains the RAS1 tracing for the Tivoli Enterprise
Portal browser client and any Java exceptions. The Tivoli Enterprise Portal client
logs contain environmental information, such as the version and build level of
the Tivoli Enterprise Portal client. The log also contains the host and port of the
Tivoli Enterprise Monitoring Server that the client is connecting to.
v The Tivoli Enterprise Portal Server log might also be useful. It can be found in
one of the locations in Table 169 on page 535:
In addition, collect the Tivoli Enterprise Monitoring Server log. While this problem
might be reported as a Tivoli Enterprise Portal client problem, the client might be
having difficulties because of a server failure.
v For the location of logs for a Tivoli Enterprise Monitoring Server on z/OS, see
“Problems reported as Tivoli Enterprise Portal Server problems” on page 536.
v Table 170 on page 536 shows the location of logs for a Tivoli Enterprise
Monitoring Server logs on distributed platforms:
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 535
Table 170. Log locations for Tivoli Enterprise Monitoring Server on distributed platforms
Com-
ponent Windows-based UNIX-based
Tivoli \install_dir\logs\ install_dir/logs/
Enterprise hostname_PC_HEXtimestampnn.log hostname_PC_timestamp.log
Monitoring
Server where: where:
install_dir install_dir
Specifies the directory where Specifies the directory where
Tivoli Enterprise Monitoring Tivoli Enterprise Monitoring
Server was installed. Server was installed.
hostname hostname
Specifies the name of the Specifies the name of the
system hosting the product system hosting the product
PC Specifies the product code, PC Specifies the product code,
ms for the Tivoli Enterprise ms for the Tivoli Enterprise
Monitoring Server. Refer to Monitoring Server. Refer to
the product code appendix the product code appendix
of IBM Tivoli Monitoring: of IBM Tivoli Monitoring:
Problem Determination Guide Problem Determination Guide
for a complete list of for a complete list of
product codes for distributed product codes for distributed
components. components.
HEXtimestamp timestamp
A hexadecimal A decimal representation of
representation of the time at the time at which the
which the process was process was started.
started
nn Represents the circular
sequence in which logs are
rotated. Ranges from 1-5, by
default, though the first is
always retained, since it
includes configuration
parameters.
Log files and trace information are provided in a common fashion across z/OS
monitoring agents, including the NetView agent, and the z/OS components of the
Tivoli Management Services. Table 171 explains the location of log and trace files
for the NetView agent and Tivoli Management Services z/OS components. See
Chapter 29, “Troubleshooting and Initial Diagnosis for the Tivoli NetView for z/OS
Enterprise Management Agent,” on page 519 for typical problems with the
NetView agent.
| Table 171. Locations of log and trace information for z/OS components
| Header Header
| The NetView agent RKLVLOG for the monitoring agent started task is the single most helpful
| piece of service information for the NetView agent. The RKLVLOG (R =
| runtime, KLV = the prefix associated with IBM Tivoli Enterprise Monitoring
| Services or ITMS:Engine) is the sysout data set or spool file that contains log
| and trace messages. Instructions on how to save the contents of this log to a
| dataset are provided under “Capturing z/OS logs to send to software
| support” on page 546.
| These additional zSeries® log files (if available) are also useful:
| v The RKLVSNAP sysout data set or spool file contains formatted dump
| output.
| v The RKPDLOG sysout data set or spool file contains the information and
| error messages related to the handling of persistent data stores.
| Tivoli Enterprise Monitoring Server Because the Tivoli Enterprise Monitoring Server on z/OS runs under
| on z/OS TMS:Engine just as an OMEGAMON XE monitoring agent on z/OS does, all
| logging under TMS:Engine is handled the same way; that is, log and trace
| data are written to RKLVLOGs and RKPDLOGs.
| ETE ETE is a base component and does not have its own RKLVLOG. This
| component writes messages to the IBM System Display and Search Facility
| (SDSF) Job Log. The User Response section of various ETE messages requests
| that you collect systems information and dumps before contacting IBM
| Software Support. How to collect this information for ETE is documented in
| the Tivoli OMEGAMON and IBM Tivoli Management Services on z/OS:
| End-to-End Response Time Feature reference.
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 537
| Table 171. Locations of log and trace information for z/OS components (continued)
| Header Header
| IBM Tivoli Management Services: TMS:Engine is a collection of basic operating system and communication
| Engine (TMS:Engine) service routines built specifically for z/OS. All address spaces used by the
| OMEGASMON XE monitoring agent on z/OS load and use the services of
| TMS:Engine.
For locations of log files for all the components of Tivoli Management Services and
information about enabling tracing for distributed components, refer to IBM Tivoli
Monitoring: Problem Determination Guide.
RAS Messages
Messages are written to the console for some problems that occur in the NetView
agent communication layer. These messages are also written to the RKLVLOG. The
messages are DWO746I, CNM217I, and CNM273I. Additionally, DWO050E
messages might be written to the log.
RAS Trace
The NVEMACMD TRACE command enables and disables the NetView agent
communication layer trace. There are three different trace types:
BFR The BFR trace entries can have one or two lines. The first line always starts
with BUFFER. Depending on the return code, the buffer itself is displayed
on the second line.
BUFFER action module Return Code: retcode
buffer
BUFFER SEND CNMIRAPP Return Code: 00000000
<*DONE*>
where
v action is SEND or RECEIVE
v module is the module doing the send or receive
538 Troubleshooting Guide
v retcode is the return code
v buffer is the actual send or receive buffer
MOD
ENTRY
Defines module or function entry.
Example of module entry:
ENTRY module
ENTRY CNMIRAPP
where
v module is the module name that has been entered
v function is the function within the module that has been entered
EXIT Defines module or function exit and shows the exit return code
Example of module exit:
EXIT module Return Code: retcode
EXIT CNMIRAPP Return Code: 00000000
where
v module is the module name that has been entered
v function is the function within the module that has been entered
v retcode is the module or function exit return code
DEBUG
Provides internal diagnostics for IBM Software Support
DEBUG text
DEBUG The ICB Address is x'1894B6A8'
v text is a string containing diagnostic information
To enable the EMA communication layer trace during EMA initialization,
code KNA_COMMTRC=types in RKANPARU member KNAENV. Valid
types are BFR, MOD, DEBUG, or ALL. An example of this specification is
KNA_COMMTRC=ALL
| To obtain the level or tracing required to have these TCP/IP initialization messages
| echoed to the log, the string KDC_DEBUG=Y must be added to either the KDSENV
| member or the KNAENV member of RKANPARU. Place the KDC_DEBUG
| environment variable statement immediately after the KDC_FAMILIES
| environment variable. You cannot dynamically alter KDC_DEBUG tracing.
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 539
| Enterprise Monitoring Server and Tivoli Enterprise Portal Server) during
| TCP/IP initialization is recorded, including data packages sent and
| received. If KDC_DEBUG=Y is active in the environment during
| initialization of TCP/IP services for this address space, you can confirm
| successful initialization of TCP/IP by looking for one of the following
| messages in RKLVLOG. If KDC_DEBUG is set to Y and none of these
| messages appear in RKLVLOG, then initialization of the TCP/IP service
| failed:
| "KDEI1_OpenTransportProvider") Transport opened: socket/ip.tcp
| "KDEI1_OpenTransportProvider") Transport opened: socket/ip.pipe
| "KDEI1_OpenTransportProvider") Transport opened: socket/ip.udp
| N The data flow between the monitoring agent and Tivoli Management
| Services components during TCP/IP initialization is not recorded. This is
| the default and the recommended setting for normal operation.
The NetView agent might not support all of the filters and classes defined in the
syntax shown in “Syntax for RAS1 traces.”
Again, be aware that RAS1 tracing log files can grow very large with the wrong
amount of filtering. There is no log management function or feature, so be careful
with the levels of tracing that you specify. You may want to run error tracing for
all components and then any additional levels depending on diagnostic needs.
The basic syntax of the RAS1 trace commands for error tracing is as follows:
KBB_RAS1= global_class
(COMP: component_type) (ENTRY: entry_point)
(UNIT: unit_name, class)
Where:
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 541
position inside the parentheses, the class is narrowed in scope to apply
only to the unit_name specified. The following are possible values. Valid
abbreviations are in parentheses.
v ERROR (ER): returns severe error messages only (this is the default for
most applications).
v STATE (ST): records the condition or current setting of flags and
variables within the process. If state tracing is enabled, you can see the
current state of particular variables or flags as the process is running.
v FLOW (FL): causes a message to be generated at an entry or exit point
of a function.
v DETAIL (DE): produces a detailed, verbose level of tracing.
v INPUT (IN): records data that is created in the execution of a particular
API, function, or process.
v ALL: causes all available messages to be recorded, a combination of all
the other forms of tracing.
Notes:
1. The default setting for all components is KBB_RAS1=ERROR, meaning that only
error tracing is enabled.
2. You can specify any combination of UNIT, COMP, and ENTRY keywords. None
of these keyword is required. However, the RAS1 value you set with the global
class will apply to all components.
Some examples of RAS1 trace syntax follow.
The unit values ST and ERR indicate that you will be collecting state and error
information for the agent framework component (KRA).
This type of agent trace is used only if you are trying to debug a specific problem,
because it greatly increases the number of messages generated by agent. With this
type of trace, messages will include a detailed dump of all rows of agent data that
have passed filtering, which includes attribute names and values, request names,
table names, and collection interval. Remember to disable this resource-intensive
form of tracing immediately after you have completed your trace.
Example 2 – Tracing proxy controller and distributed agent issues: From the
Tivoli Enterprise Monitoring Server, to trace proxy controller and Tivoli Enterprise
Monitoring Server distributed agent issues, issue this command:
KBB_RAS1=ERROR (COMP:KUX ST ER) (UNIT:KRA ALL) (UNIT:KDS FL)
In this example:
v KUX is a component identifier provided to you by a representative of IBM
Software Support so that you can collect state and error information about this
subcomponent.
v KRA is the unit name for the agent framework component. All trace information
about this component is being captured.
v KDS is the Tivoli Enterprise Monitoring Server component and the flow (FL) of
entry or exit points through this component will be documented with records
written to RKLVLOG.
Setting trace levels by editing RKANPARU: One of the simplest ways to set
trace levels for the NetView agent is to edit the RKANPARU(KppENV) member.
The text in bold is an example of what an IBM service representative might ask
you to add to this member.
| File Edit Edit_Settings Menu Utilities Compilers Test Help
| sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
| EDIT KAN.V5R4.NVNET.RKANPARU(KNAENV) - 01.72 Columns 00001 00072
| Command ===> Scroll ===> CSR
| ****** ***************************** Top of Data ******************************
| ==MSG> -Warning- The UNDO command is not available until you change
| ==MSG> your edit profile using the command RECOVERY ON.
| 000001 KDE_TRANSPORT=\
| 000002 SNA.PIPE PORT:135 USE:N\
| 000003 IP6.PIPE PORT:1918 USE:N\
| 000004 IP6.UDP PORT:1918 USE:N\
| 000005 IP.SPIPE PORT:3660 USE:N\
| 000006 IP6.SPIPE PORT:3660 USE:N\
| 000007 IP.PIPE PORT:1918 EPHEMERAL:Y\
| 000008 IP.UDP PORT:1918
| 000009 KBB_RAS1=ERROR (UNIT:KNATN ALL) (UNIT:KNAIRAFT ALL)
| 000010 CT_CMSLIST=\
| 000011 IP.PIPE:x.xx.xxx.xx;\
| 000012 IP.UDP:x.xx.xxx.xx;
| 000013 CTIRA_STANDALONE=N
| 000014 KNA_PPISND=CNMEMATX
| 000015 KNA_PPIRCV=CNMEMARX
| 000016 CTIRA_IP_PORT=0
| 000017 LANG=en_US.ibm-037
| ****** **************************** Bottom of Data ******************************
Setting trace levels dynamically using IBM Tivoli Monitoring Service Console:
You can also use the IBM Tivoli Monitoring Service Console to set trace levels for
the NetView agent, as well as for a Tivoli Enterprise Monitoring Server on z/OS or
a distributed system.
The IBM Tivoli Monitoring Service Console enables you to read logs and turn on
traces for remote product diagnostics and configuration. The IBM Tivoli
Monitoring Service Console is uniquely identified by its service point name. All
IBM Tivoli Monitoring Service Consoles for a host are linked and presented on the
IBM Tivoli Monitoring Service Index for that host. Point a browser to the HTTP
port 1920 on a specific host (for example, http://goby:1920) to launch the IBM
Tivoli Monitoring Service Index. You can also launch the service console by
connecting via the https protocol and port 3661. You can perform operations on a
specific IBM Tivoli Monitoring process by selecting the IBM Tivoli Monitoring
Service Console associated with a service point name.
Starting the IBM Tivoli Monitoring Service Console: Use the following procedure to
start the IBM Tivoli Monitoring Service Console.
1. Start Internet Explorer V5 or higher.
2. In the Address field, type the URL for the Tivoli Enterprise Portal browser
client installed on your Web server. The URL for the Tivoli Monitoring Services
Web server is
http://hostname:1920
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 543
Where hostname specifies the computer where the Tivoli Enterprise Portal
Server was installed. If the IBM Tivoli Monitoring Service Console is not
displayed, a system administrator might have blocked access to it. Refer to the
IBM Tivoli Monitoring: Problem Determination Guide for information about
blocking access to the IBM Tivoli Monitoring Service Console. A window
similar to the one shown in Figure 75 is displayed.
3. Click the IBM Tivoli Monitoring Service Console link associated with the
desired process (service point name).
4. When the login window opens, click OK.
In secure environments, you need a valid user ID and password to proceed. Upon
successful login, the IBM Tivoli Monitoring Service Console opens with three areas:
v Header
v Command Results
v Command Field
You can now issue IBM Tivoli Monitoring Service Console commands in the
command input area. For a list of available commands, type a question mark (?)
and click Submit.
The IBM Tivoli Monitoring Service Console supports the following commands,
most of which are useful for problem determination:
The setting of KDC_DEBUG can be restored to its original value using the
following:
CONFIG KDC_DEBUG=N
ctbld The ctbld command is used to determine the maintenance level of the
product.
http Displays HTTP server management
kdcstat
Displays the status of KDC (RPC Service) component
kdestat
Displays the status of the KDE (Transport Service) component
ras1 Manage RAS1 (Reliability, Availability, and Serviceability). This command
is paired with one of the following subcommands:
v dumpcvt: Display KBBRA_cvt_t
v log: Display RAS1 log capture buffer
v list: List the RAS1 filters
v interpret: Interpret the control string
v ctbld: Display the resident CTBLD data
v units: Display the registered compilation units
The RAS1 (with no operands) command can be used to view the current
ITMS:Engine log capture buffer. When operands are supplied with the
RAS1 command, the operands are assumed to be keywords applicable to
the KBB_RAS1 environment variable.
The RAS1 command is especially useful for dynamically enabling and
disabling RAS1 traces. Many times you cannot recycle the agent in order to
start tracing. The RAS1 command can be used to alter KBB_RAS1 tracing
parms dynamically without the need to recycle the product. For example,
to enable the standard IRA traces, the following Service Console command
can be used:
RAS1 'error (unit:kpx all) (unit:kra all)'
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 545
After this trace is captured, the IRA trace can be disabled with the
following Service Console command: RAS1 ’error (unit:kpx error) (unit:kra
error)’. This has the effect of restoring the RAS1 logging level from ALL to
ERROR for units kpx and kra.
res1 Displays the status of RES1 Logical Resource Manager.
Most messages with IDs are documented in the problem determination guides for
each monitoring agent. You can also determine the meaning of a message by
entering the message number into an Internet search engine. The information that
follows help you interpret the messages and status lines in a z/OS log.
Where:
PPP Is the component prefix.
xx Is the component code (for example, NS Node Status).
mmm Is the module name (for example mdg/mgr for Model/Manager).
– Initial command line settings
v Component summary, including the following:
– The name of the module
– Information about where the library was loaded from
– The date and time the module was compiled
– The version (if this detail was specified)
v Formatted output, including entry and exit points and text strings. Entry and
exit points show flow into and out of a given function. The exit shows the
return code, if applicable. The text depends on the kind of trace specified. Here
is an example:
(00D41 F9C-1{99%}:KV4MAIN.CPP,953,"MainWnd::MainWnd") Entry
(00D41 FD3-1{99%}:KV4MAIN.CPP,959,"MainWnd::MainWnd") Exit
Time,Thread,{%stack avail},pgm_name,Line#,function,text
As noted earlier, not all functions are RAS1 enabled, and trace level might
exclude some paths. Be careful with granularity.
| Note: This method works only with JES2. It does not work with JES3.
Follow these instructions to use SDSF to capture (in this example) the RKLVLOG
associated with any running task in your z/OS monitoring agent.
1. From ISPF, select the SDSF option using the =s.st 2 option (for RKLVLOG;
sometimes these options are different).
2. Enter the following on the command line:
st taskname
Where taskname is the name of the procedure whose log you are trying to
display and capture. For example, entering st cansna on the command line
would enable you to see the NetView agent job.
3. From the SDSF screen, enter ? next to the name of the started task to display a
list of the output files like the following. For example the output files for the
sample cansn3 task noted above would look like this:
JESMSGLG JES2
JESJCL JES2
JESYSMSG JES2
RKLVLOG CANSNA
RKLVSNAP CANSNA
RKPDLOG CANSNA
4. To print the RKLVLOG for this job to a dataset, type an s next to the
RKLVLOG output file. Then, on the command line of SDSF, type:
print d
Press Enter. The d means that the file should be printed to a dataset.
5. This action causes a panel similar to this one in Figure 76 to be displayed:
On this panel, type the dataset name and characteristics for the file you are
printing and press Enter.
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 547
6. You are returned to the RKLVLOG output file. On the command line, specify
the number of lines you want to print by entering a range that would include
the entire file, such as:
print 1 99999999
Then press Enter. A message in the upper right corner of the panel tells you
how many lines have been printed.
7. Type print close on the SDSF command line to close the file. The log is now
saved in the dataset that was specified in Step 5 on page 547.
For more information about SDSF commands, see z/OS SDSF Operation and
Customization (SA22-7670).
, FCB= fcb , FORM= form NO , MAXLINES= maxlines
HOLD=
YES
, UCS= ucs , USER= user , WTRNAME= wtrname
Where:
SWITCH
Is the keyword that dynamically allocates a new RKLVLOG file using the
current values, begins recording on the new file, and closes the current
RKLVLOG file, releasing it for processing by JES.
class Is the one-character JES SYSOUT class. CLASS=A is the ITMS:Engine
startup value.
copies Is the copy count. The valid range is 1-254. COPIES=1 is the startup value.
Note: Unlike the other values, MAXLINES takes effect immediately. If the
new MAXLINES value is less than the number of lines that have
already been written to the current RKLVLOG, a switch is
immediately performed.
ucs Specifies the 1 to 4 character UCS name to be used. UCS=() is the startup
value.
user Is the 1-8 character user ID to which the SYSOUT is to be spooled. Ignored
if DEST is blanks. USER=() is the startup value.
wtrname
Is the 1-8 character external writer name to be used. WTRNAME=() is the
startup value.
User Notes:
1. The TLVLOG command performs up to three functions, depending on the
keywords that are specified. Assuming that you selected all three functions,
they would be performed in the following order:
a. Updates the dynamic allocation values. With the exception of MAXLINES,
these values are used when the next dynamic allocation is performed.
Values are updated whenever they are coded on the command.
b. Lists the current dynamic allocation values. This is always done.
c. Switches RKLVLOGs. This is done only when SWITCH is specified on the
command.
Note: You can update values and request a switch with the same command;
the values are updated first, then the switch is performed.
2. RKLVLOGs might be automatically closed after a certain number of records
have been written to them, similar to the MVS SYSLOG processing. Refer to the
MAXLINES keyword for more information.
3. To set up an automatic RKLVLOG switch whenever the ITMS:Engine address
space is started, add the following command to your RKANCMD startup
CLIST:
TLVLOG MAXLINES=nnn
Chapter 30. Diagnostic tools for the Tivoli NetView for z/OS Enterprise Management Agent 549
messages. Issue the command FLUSH TLVLOG to force the current data
management buffer to be written. Do not use the TLVLOG SWITCH to spin off the
current RKLVLOG for this purpose, as it will unnecessarily fragment the
messages recorded in RKLVLOG.
7. Unless you explicitly set a non-zero MAXLINES value, RKLVLOG will never
automatically switch.
8. If any error occurs when writing to RKLVLOG, ITMS:Engine will issue a
message and disable RKLVLOG recording. However, messages will still be
written to VIEWLOG and all active operator interfaces. Depending on the error,
you might be able to restart RKLVLOG by issuing a switch request.
Flushing the log buffers: After a TLVLOG is switched, issuing an echo command
can flush the log buffers and ensure that new messages are written to the new
RKLVLOG. The ECHO command echoes any text entered back to the screen. The
syntax of the ECHO command is shown below:
ECHO
string
Where:
string Is a character string to be echoed back to the operator screen where the
ECHO command was entered.
User Notes:
1. Use ECHO to verify that the ITMS:Engine operator facility is functioning
properly and to force all buffered messages to the log.
2. Even after an ECHO, log output might not be visible in JES3 systems. This is
apparently a result of the way JES3 manages spool buffers.
3. Enclosing string in single quotes is not necessary unless you want to preserve
leading blanks.
For TRACE, see also “NetView Trace” on page 99, “GMFHS Trace” on page 214,
“The RODM Internal Trace” on page 282, and “SNA Topology Manager Traces” on
page 400.
Table 172. Summary of NetView Diagnostic Commands
Command Description
BROWSE Enables you to scan the network log or members of a partitioned data set. The
member or network log can be on a local or remote NetView program.
DEFAULTS MSGMODID Sets whether the module identification information in DSI799I is logged for certain
error conditions.
DEFAULTS STORDUMP Specifies the maximum number of times a storage dump is taken if storage overlay or
control block overwrite is detected.
DFILTER Displays the recording or viewing filters that are currently in effect.
DISCONID Displays MVS console names or IDs used by the NetView program.
DSIDIAGG Monitors and reports storage discrepancies.
FIND Locates specific information while browsing a data set and a member.
GMFHS LISTINIT Produces a formatted display of the GMFHS initialization parameters.
GMFHS SHOW Produces a report with an entry for a specified network management gateway (NMG)
or domain, or all NMGs or domains known to GMFHS.
GMFHS STATUS Produces a summary report showing the status of the GMFHS job.
GMFHS TASK Displays a GMFHS subtask status report.
GMFHS TRACE Controls the level and content of the tracing performed by GMFHS tasks.
LIST DEFAULTS Lists the current DEFAULTS settings and the number of dumps that have been taken
for storage overlay or control block overwrite conditions (DMPTAKEN).
LIST PRIORITY Lists all NetView tasks and their priorities.
LIST SECOPTS Displays a list of the security options, their current values, the date and time of the
last update, and either the last operator ID to update the option or INITIALIZATION
if the option has not been dynamically changed using the NetView REFRESH or
DEFAULTS commands.
LIST STATUS=AMLUSESS Displays all VTAM-LU sessions.
LIST STATUS=CNMSESS Displays all active communication network management (CNM) data sessions with
your NetView program and the status of these sessions.
LIST STATUS=NNT Displays all the NNT (NetView-NetView task) sessions.
LIST STATUS=OPS Displays all the operator terminals known in this domain.
LIST STATUS=PROFILES Displays a list of profiles known in this domain.
LIST STATUS=SPANS Displays a list of all the spans defined in the NetView span table.
Purpose
The RID DSIMSX command, which is in the NetView ESTAE/ESTAI routine,
suspends a task at the point of an abend. You can trap abends for one task by
issuing a RID command from another task.
Parameters
The RID DSIMSX syntax options are defined in the following list:
ID DSIMSX (identifies this as the abend ESTAE trap).
TASK
Name of the task that is being monitored for abends. This cannot be the name
of the task entering the RID command.
Usage
To use the RID DSIMSX command to stop a task:
1. Logon as an operator, for example, OPER4.
2. As OPER4, enter RID TASK=AUTO1,ID=DSIMSX.
3. Any abend or task AUTO1 will now be reported to OPER4. The task will
remain suspended until OPER4 enters RID TASK=AUTO1,CONTINUE, at
which time the abend (for example, DUMP) will proceed.
The RID command initiates monitoring of a task to trap system and user abends,
including program checks. Note the amount of what-was-running data in the
output. You can use the CSECT and OFFSET data to tell you where in a listing of
the program to look for the problem.
CLIST function commands can be used in CLISTs to display more data areas from
the registers in the RID display output. You can even write new CLISTs on TSO
and run them while the diagnosis is being done.
Restrictions
Do not use the RID DSIMSX command to stop a task when you want NetView
internal trace or dump data. It is intended to be used to debug code that is being
developed on test systems. It can be helpful during the recreation of a problem
where an adequate dump already exists.
Examples
Example: Output Generated by RID DSIMSX: The following is an example of
output generated by RID DSIMSX if OPER4 Issues EXCMD AUTO1,RESET
IMMED:
* NTV98 RID TASK=AUTO1,ID=DSIMSX
- NTV98 CNM986I RID FUNCTION 'STEP' COMPLETED FOR TASK AUTO1
*ACTIVE*
TASKURPT MAN=suffix
DSN=dsname LOGTSTAT taskid
LOG=dsname LOGOFF
MENU ABEND
HELP STOP/UNCOND
CLOSE/NORMAL
CLOSE/STOP
CLOSE/IMMED
CLOSE/ABEND
(
CYL nnnn
NEWFIRST
NOWINDOW
PREFIX prf
TAKE nnn
Purpose
TASKURPT is a REXX procedure that generates a report using the task resource
data in the System Management Facility (SMF) log. Task utilization data is
normally written to the SMF log when a task ends. You can display CPU, storage,
message queuing and I/O utilization data from an active or archived SMF log.
Your output can be filtered by taskid or LU name. Your output is limited to the
most recent number of records. The default output limit is 1000. An example is
when an operator logs off.
Parameters
The following list describes the parameters for TASKURPT:
*ACTIVE*
Displays data from the currently active SMF log.
MAN=suffix
Displays resource data from SYS1.MANsuffix where SYS1.MAN is the default
prefix value and suffix is a letter such as “X”.
DSN=dsname
Displays resource data that you created elsewhere to dsname using the
IFASMFDP system utility.
LOG=dsname
Displays resource data from the SMF log named dsname.
MENU
Displays a list of SMF log data sets You can tab to any of them and display the
resource data.
HELP
Provides command help online. This is easier to view using WINDOW
TASKURPT HELP.
The following options are separated from the options above and require left
parenthesis as the separator character. Do not use a right parentheses at the end of
the command.
NOWINDOW
Produces output as messages instead of displaying them in a window. Use
this for PIPE automation.
TAKE nnn
Limits the display to the nnn most recent records for the applicable filters.
nnn is decimal. The default is 1000.
PREFIX prf
Is used only in combination with MAN= to specify the SMF log data set
name. The default is ″SYS1.MAN″. See also MAN=.
NEWFIRST
Is used to order the SMF records, so that the most recent are at the top of
the screen.
CYL nnnn
Defines the size of the temporary VIO file. The file is allocated using nnnn
as the primary allocation and nnnn as the secondary. The default for nnnn
is 10.
Examples
Example: TASKURPT Command: Following are some examples of the
TASKURPT command:
TASKURPT MENU (CYL 100
TASKURPT *ACTIVE* (TAKE 100 NEWFIRST
TASKURPT LOG=SYS1.MANX (TAKE 50
TASKURPT DSN=ARC1.MANX (TAKE 500
TASKURPT ABEND (TAKE 50
TASKURPT MENU LOGOFF OPER6
TASKURPT MAN=Q (NOWINDOW TAKE 10 PREFIX ARCHIVE.MAN
-Ruler-0005|0010|0015|0020|0025|0030|0035|0040|0045|0050|0055|0060|
1 | Operator LU/Task Domain
2 |Date Time Event Name Name Name
3 |-------- ----------- ------------ -------- -------- --------
4 |2009.138 16:28:47.40 LOGOFF DSITRACE DSITRACE NTV98
5 |2009.138 16:28:47.80 LOGOFF BNJDSE36 BNJDSE36 NTV98
6 |2009.138 16:28:48.11 LOGOFF AAUTSKLP AAUTSKLP NTV98
7 |2009.138 16:28:49.14 LOGOFF DSISVRT DSISVRT NTV98
8 |2009.138 16:28:49.80 LOGOFF NTV98BRW NTV98BRW NTV98
9 |2009.138 16:28:50.12 LOGOFF DSILCOPR DSILCOPR NTV98
10 |2009.138 16:28:50.20 LOGOFF NTV98LUC NTV98LUC NTV98
11 |2009.138 16:28:50.33 LOGOFF DSIAMLUT DSIAMLUT NTV98
12 |2009.138 16:28:50.73 LOGOFF ALIASAPL ALIASAPL NTV98
13 |2009.138 16:28:52.07 LOGOFF AAUTCNMI AAUTCNMI NTV98
14 |2009.138 16:28:52.51 LOGOFF NTV98VMT NTV98VMT NTV98
15 |2009.138 16:28:52.58 LOGOFF DSIQSD4A DSIQSD4A NTV98
16 |2009.138 16:28:52.90 LOGOFF DSIQRV4B DSIQRV4B NTV98
17 |2009.138 16:28:53.21 LOGOFF DSIQRV4C DSIQRV4C NTV98
18 |2009.138 16:28:53.50 LOGOFF VPDTASK VPDTASK NTV98
19 |2009.138 16:28:54.09 LOGOFF DSIQSD4B DSIQSD4B NTV98
20 |2009.138 16:28:54.38 LOGOFF DSIQSD4C DSIQSD4C NTV98
21 |2009.138 16:28:54.68 LOGOFF CNM01QSD CNM01QSD NTV98
22 |2009.138 16:28:55.01 LOGOFF DSIQRV4A DSIQRV4A NTV98
23 |2009.138 16:28:55.56 LOGOFF DSICRTR DSICRTR NTV98
24 |2009.138 16:28:56.29 LOGOFF DSIKREM DSIKREM NTV98
25 |2009.138 16:28:56.51 LOGOFF BNJDSERV BNJDSERV NTV98
26 |2009.138 16:29:05.35 LOGTSTAT MAINTASK SYSOP NTV98
27 |2009.138 16:29:05.35 LOGTSTAT NTV98PPT NTV98PPT NTV98
28 |2009.138 16:29:05.35 LOGTSTAT DSIMONIT DSIMONIT NTV98
29 |2009.138 16:29:05.35 LOGTSTAT DSITIMMT DSITIMMT NTV98
30 |2009.138 16:29:05.35 LOGTSTAT DSIDCBMT DSIDCBMT NTV98
31 |2009.138 16:29:05.35 LOGTSTAT DSIHLLMT DSIHLLMT NTV98
32 |2009.138 16:29:05.35 LOGTSTAT DSISTMMT DSISTMMT NTV98
33 |2009.138 16:29:05.35 LOGTSTAT DSIWTOMT DSIWTOMT NTV98
34 |2009.138 16:29:05.35 LOGTSTAT DSIACBMT DSIACBMT NTV98
35 |2009.138 16:29:05.35 LOGTSTAT DSILOGMT DSILOGMT NTV98
36 |2009.138 16:29:05.35 LOGTSTAT OPER3 OPER3 NTV98
37 |2009.138 16:29:05.38 LOGTSTAT MAINTASK SYSOP NTV98
38 |2009.138 16:29:05.41 LOGTSTAT MAINTASK SYSOP NTV98
39 |2009.138 16:29:05.50 LOGTSTAT MAINTASK SYSOP NTV98
40 |2009.138 16:29:05.52 LOGTSTAT MAINTASK SYSOP NTV98
41 |2009.138 16:29:05.52 LOGTSTAT NTV98PPT NTV98PPT NTV98
42 |2009.138 16:29:05.52 LOGTSTAT DSIMONIT DSIMONIT NTV98
-Ruler-0065|0070|0075|0080|0085|0090|0095|0100|0105
1 | Maximum Session Used CPU
2 | CPU% seconds seconds
3 | ------- ----------------- -----------------
4 | 0.51 994.933358 0.059714
5 | 0.30 994.606067 0.039716
6 | 1.02 994.499822 0.131732
7 | 0.39 996.640239 0.040786
8 | 0.03 996.726691 0.004718
9 | 0.28 970.865641 0.017055
10 | 0.12 996.591360 0.019000
11 | 0.13 997.132887 0.024090
12 | 0.13 997.642773 0.025757
13 | 0.07 998.886562 0.014841
14 | 0.75 999.431784 0.134403
15 | 0.07 998.927449 0.006311
16 | 0.03 999.228930 0.003820
17 | 0.03 999.521794 0.003156
18 | 0.06 999.856236 0.008128
19 | 0.06 1000.437805 0.006264
20 | 0.07 1000.732072 0.005631
21 | 0.07 1001.033959 0.006353
22 | 0.03 1001.360682 0.006341
23 | 0.07 1002.385841 0.012224
24 | 0.35 1003.200227 0.039140
25 | 0.43 1003.325160 0.046776
26 | 0.35 7793.971973 2.839490
27 | 0.14 7691.891829 1.778795
28 | 0.03 7781.594283 1.780333
29 | 0.00 7781.469443 0.001038
30 | 0.00 7781.332946 0.094697
-Ruler-|0110|0115|0120|0125|0130|0135|0140|0145|0150|0155|01
1 | Penalty Average Average Maximum DSIGET
2 | seconds CPU% Penalty Kbytes K/min.
3 | ----------------- ------- ------- ------- ----------
4 | 0.000000 0.00 0.00 73 6
5 | 0.000000 0.00 0.00 89 6
6 | 0.000000 0.01 0.00 1524 101
7 | 0.951643 0.00 0.09 213 13
8 | 0.637185 0.00 0.06 16 0
9 | 0.000000 0.00 0.00 98 16
10 | 0.000000 0.00 0.00 94 6
11 | 0.000000 0.00 0.00 73 5
12 | 0.575414 0.00 0.05 102 16
13 | 1.801088 0.00 0.18 295 19
14 | 1.966722 0.01 0.19 81 29
15 | 2.228434 0.00 0.22 59 3
16 | 2.499864 0.00 0.25 37 2
17 | 2.793850 0.00 0.27 37 2
18 | 3.089860 0.00 0.30 74 5
19 | 3.672162 0.00 0.36 59 3
20 | 3.966305 0.00 0.39 59 3
21 | 4.241932 0.00 0.42 59 3
22 | 4.536043 0.00 0.45 37 2
23 | 5.321668 0.00 0.53 85 6
24 | 5.829898 0.00 0.58 73 6
25 | 5.811196 0.00 0.57 85 10
26 | 50.986242 0.03 0.65 1033 106
27 | 37.159961 0.02 0.48 168 17
28 | 0.000000 0.02 0.00 4 0
29 | 0.000000 0.00 0.00 3 0
30 | 0.000000 0.00 0.00 3 0
31 | 0.000000 0.00 0.00 3 0
32 | 0.000000 0.00 0.00 4 0
33 | 0.000000 0.00 0.00 3 0
34 | 0.000000 0.00 0.00 5 0
35 | 0.000000 0.00 0.00 3 0
36 | 136.000000 1.94 70.15 98 100
37 | 50.986242 0.03 0.65 1033 106
38 | 50.986242 0.03 0.65 1033 106
39 | 51.294330 0.03 0.65 1033 106
40 | 51.294330 0.03 0.65 0 106
41 | 37.159961 0.02 0.48 168 17
42 | 0.000000 0.02 0.00 4 0
43 | 0.000000 0.00 0.00 3 0
44 | 0.000000 0.00 0.00 3 0
-Ruler-60|0165|0170|0175|0180|0185|0190|0195|0200|0205|0210|02
1 | DSIFRE 24-GET 24-FRE MaxQin Que In
2 | K/min. K/min. K/min. K/min. K/min.
3 | ---------- ---------- ---------- ---------- ----------
4 | 5 3 3 0 0
5 | 5 3 3 0 0
6 | 70 3 61 21 0
7 | 12 3 10 2 0
8 | 0 0 0 0 0
9 | 16 10 10 6 0
10 | 5 3 3 0 0
11 | 5 3 3 12 0
12 | 15 10 10 0 0
13 | 18 3 15 7 0
14 | 25 18 18 1 0
15 | 3 1 1 0 0
16 | 1 1 1 0 0
17 | 1 1 1 0 0
18 | 4 3 3 0 0
19 | 3 1 1 0 0
20 | 3 1 1 0 0
21 | 3 1 1 0 0
22 | 1 1 1 0 0
23 | 5 3 3 0 0
24 | 5 3 3 0 0
25 | 7 5 5 17 0
26 | 100 62 61 0 0
27 | 18 5 5 26 3
28 | 0 0 0 0 0
29 | 0 0 0 0 0
30 | 0 0 0 0 0
-Ruler-15|0220|0225|0230|0235|0240|0245|0250|0255|0260|0265|0270|0275|028
1 | MaxQout Que Out In Cnt Out Cnt I/O Cnt Max I/O
2 | K/min. K/min. Messages Messages I/O Cnt IOs/min.
3 | ---------- ---------- ---------- ---------- ---------- ----------
4 | 20 0 0 8 32 2
5 | 21 0 0 6 30 347
6 | 44 0 11 20 57 371
7 | 21 0 1 6 31 47
8 | 13 0 0 9 0 0
9 | 2 0 5 1 122 1462
10 | 13 0 0 3 41 479
11 | 13 0 20 37 43 503
12 | 30 0 0 8 53 215
13 | 13 0 3 6 18 203
14 | 51 2 2 385 861 7743
15 | 14 0 0 5 3 23
16 | 6 0 0 2 0 0
17 | 6 0 0 2 0 0
18 | 14 0 0 3 20 227
19 | 14 0 0 5 3 11
20 | 14 0 0 5 3 23
21 | 14 0 0 5 3 23
22 | 6 0 0 2 0 0
23 | 14 0 0 4 41 479
24 | 21 0 0 6 27 311
25 | 42 0 2 10 155 1007
26 | 150 2 0 447 11686 90
27 | 10 0 3686 666 1547 12
28 | 8 0 0 22 0 0
29 | 0 0 0 0 0 0
30 | 0 0 0 0 0 0
31 | 0 0 0 0 0 0
32 | 7 0 0 42 0 0
33 | 0 0 0 0 0 0
34 | 0 0 0 0 0 0
35 | 0 0 0 0 0 0
36 | 2 0 4 1 112 827
37 | 150 2 0 447 11686 90
38 | 150 2 0 447 11686 90
39 | 150 2 0 448 11686 90
40 | 2 2 0 448 11686 90
-Ruler-0|0285|0290|0295|0300|0305|0310|
1 | I/Orate MQI Pen
2 | IOs/min. Seconds
3 | ---------- -----------------
4 | 2 0.000000
5 | 1 0.000000
6 | 3 0.000000
7 | 1 0.000000
8 | 0 0.000000
9 | 7 0.000000
10 | 2 0.000000
11 | 2 0.000000
12 | 3 0.000000
13 | 1 0.000000
14 | 53 0.000000
15 | 0 0.000000
16 | 0 0.000000
17 | 0 0.000000
18 | 1 0.000000
19 | 0 0.000000
20 | 0 0.000000
Usage
Following is the description of TASKURPT output columns:
Date The date the record was recorded in SMF record format
Time The time the record was recorded in SMF record format
Event The reason the data was recorded
Operator Name
The task name or operator ID (TVBOPID)
LU/Task Name
The task name or terminal name connected to the task (TVBLUNAM)
Domain Name
The NetView domain name in which the task ran
Maximum CPU
The maximum measured CPU during a 10-second interval since the task
began or since the last LOGSTAT RESETMAX command
Session Seconds
The elapsed time the task has run
Used CPU Seconds
The amount of CPU time charged to this task by MVS
Penalty Seconds
The number of seconds this task has waited because of MAXMQIN,
AVLSLOW, SLOWSTG, MAXCPU, MAXMQOUT, or MAXIO penalties
Average CPU
The percentage of one CPU this task has used. The ratio of Used CPU to
Session Seconds.
Average Penalty
The percentage of elapsed time this task has waited for penalties. The ratio
of Penalty Seconds to Session Seconds.
Maximum Kbytes
The largest recorded usage of storage for this task since the task was
started or since the last LOGSTAT RESETMAX command.
DSIGET K/Min
The average rate (for the life of the task) at which storage was obtained by
DSIGET in KB per minute.
DSIFRE K/Min
The average rate (for the life of the task) at which storage was released by
DSIFRE in KB per minute
24-GET K/Min
The average rate (for the life of the task) at which storage was obtained by
DSIGET.
Return codes
The return code for TASKURPT is zero (0), meaning the command completed,
successfully.
SUMMARY
DSISTRLS
HELP
BLOCKS
DETAIL
CELLHIST
SHOWSTOR
SHOWMQS
SHOWMQS ADDR=xxxxxxxx
SHOWMQS NAME=nnnnnnnn
Purpose
DSISTRLS is a storage list command. It can provide a variety of NetView storage
usage data through specific request keywords.
The output gives a picture of DSIGET storage locations amid a sea of empty space,
load modules, and storage obtained with GETMAIN. It can give information about
fragmentation or storage that is building up because of coding errors. The output
also provides summary statistics retrieved by DSISTRLS SUMMARY.
DSISTRLS DETAIL provides detail about DSIGET storage allocation. It lists every
individual DSIGET storage allocation currently in use, in order of address. This
request uses a large amount CPU time and needs memory for the amount of data
produced.
DSISTRLS SHOWSTOR provides storage usage details on a task and totals basis.
Only storage managed by DSIGET is shown. This is useful for tracking individual
task storage problems. Detail is at the individual byte level, so that small losses can
be detected. Data is organized spreadsheet fashion.
Parameters
The DSISTRLS <SUMMARY> command provides summary statistics about
DSIGET/DSIFRE storage services.
The following list describes the fields in the DSISTRLS <SUMMARY> output:
DSIGET storage map
This output is suppressed. See DSISTRLS BLOCKS or DSISTRLS DETAIL.
Above 24-bit storage
This is the amount of DSIGET storage in use above address X’00FFFFFF’.
Mapped below 24-bit storage
This is the amount of DSIGET storage in use below address X’00FFFFFF’
(includes the percentage of DSIGET currently allocated below the line). This
value is computed by counting the storage mapped by NetView DSIGET
storage integrity (built in) function.
Counted below 24-bit storage
This is a second count of storage, kept in an accumulator (not mapped). It is a
second opinion about 24-bit DSIGET usage.
Maximum ever
This is the highest value ever recorded for the counted below 24-bit storage. It
is a figure-of-merit of the total demand for 24-bit storage. By comparing the
24-bit storage reported by the RESOURCE command to the
counted-below-24-bit value, you can estimate how much storage is not being
monitored by DSIGET. This will give you an estimate of how much to scale up
the reported high water mark as a safety factor in estimating total demand.
Total of all storage
This is the sum of the 24-bit and above 24-bit storage managed by DSIGET.
Storage accounting
This is the amount of 31-bit storage used to map the DSIGET requests.
Accounting storage cost
This is the amount of storage used to ensure the integrity of DSIGET. This
percentage is lowest when storage usage is high.
The following list contains a description of the fields in the DSISTRLS SHOWSTOR
output:
TASK Q Current Pooled
Sum of all tasks current pooled, listed after TOTALS.
TASK Q Maximum Pooled
Sum of all tasks maximum pooled, listed after TOTALS.
TASK Q Current Non Pool
Sum of all tasks current non-pool, listed after TOTALS.
TASK Q Maximum Non Pool
Sum of all tasks maximum non-pool, listed after TOTALS.
In the following example, the GLOBAL Total Current number is larger than PART
SUM (see the following example), by the amount of NetView overhead in unused
pooled cells and storage management control blocks. The output has been edited
for space and the totals are not accurate.
Restrictions
DSISTRLS BLOCKS output can be lengthy and can consume CPU time and
storage.
Examples
Example: Output Generated by DSISTRLS<SUMMARY>:
NCCF Tivoli NetView NTV98 OPER4 01/10/09 09:38:05
* NTV98 DSISTRLS
' NTV98
DSISTRLS DSIGET Storage Map
Start End Length Decimal
-------- -------- -------- ----------
Above 24-bit storage: 1200600
Mapped below 24-bit storage: 68560 (5.40%)
Counted below 24-bit Storage: 68560 Maximum Ever: 75288
Total of all storage: 1269160
Storage Accounting : 16640 Grand Total: 1285800
1.29% = Accounting Storage Cost
DSISTRLS End of storage map
ON
DSI24TRC
HELP
OFF
Purpose
The DSI24TRC command enables you to limit the NetView internal trace for
storage to 24-bit storage requests. It also displays the current trace options.
Parameters
DSI24TRC HELP
DSI24TRC HELP displays the syntax of DSI24TRC and the current trace
options.
DSI24TRC ON
DSI24TRC ON, filters DSIGMN/DSIFMN to trace only 24-bit mode storage
requests and displays the current trace options.
DSI24TRC OFF
DSI24TRC OFF sets DSIGMN/DSIFMN to trace all storage requests and
displays the current trace options.
Examples
Example: Output Generated by DSI24TRC:
NCCF Tivoli NetView NTV98 OPER4 01/11/09 15:17:21
NTV98 DSI24TRC
NTV98 DSI24TRCS NetView trace active
NTV98 DSI24TRC7 MODE=GTF
NTV98 DSI24TRC8 TASK=ALL
NTV98 DSI24TRCF OPT=QUE
NTV98 DSI24TRCG OPT=PSS
NTV98 DSI24TRCH OPT=DISP
NTV98 DSI24TRCQ OPT=STOR
NTV98 DSI24TRCL OPT=MOD
NTV98 DSI24TRCM OPT=UEXIT
NTV98 DSI24TRCK Enable trace any mode storage
NTV98 DSI24TRC
NTV98 DSI24TRCS NetView trace active
NTV98 DSI24TRC6 MODE=EXT
NTV98 DSI24TRCN TASK=OPER4
NTV98 DSI24TRCF OPT=QUE
NTV98 DSI24TRCG OPT=PSS
NTV98 DSI24TRCH OPT=DISP
NTV98 DSI24TRCQ OPT=STOR
NTV98 DSI24TRCL OPT=MOD
NTV98 DSI24TRCM OPT=UEXIT
NTV98 DSI24TRCK Enable trace any mode storage
NTV98 DSI24TRC ON
NTV98 DSI24TRCS NetView trace active
NTV98 DSI24TRC5 MODE=INT, SIZE=250 PAGES (1000K)
NTV98 DSI24TRC8 TASK=ALL
NTV98 DSI24TRCF OPT=QUE
NTV98 DSI24TRCG OPT=PSS
NTV98 DSI24TRCH OPT=DISP
NTV98 DSI24TRCQ OPT=STOR
NTV98 DSI24TRCJ Enable trace 24-bit storage only
NTV98 DSI24TRC ON
NTV98 DSI24TRCS NetView trace active
NTV98 DSI24TRC5 MODE=INT, size=250 PAGES (1000K)
NTV98 DSI24TRC8 TASK=ALL
NTV98 DSI24TRCF OPT=QUE
NTV98 DSI24TRCG OPT=PSS
NTV98 DSI24TRCH OPT=DISP
NTV98 DSI24TRCQ OPT=STOR
NTV98 DSI24TRCJ Enable trace 24-bit storage only
* NTV98 DSI24TRC
- NTV98 DSI24TRC4 NetView trace inactive
- NTV98 DSI24TRCK Enable trace any mode storage
DSIMODQY hexaddr1
hexaddr2
Purpose
The DSIMODQY command lists load modules and control sections located at the
address (hexaddr1) or in the range (hexaddr1 through hexaddr2). NetView
displays the result from high memory to low, regardless of the order in which the
addresses are entered. This command provides the location of modules at given
memory locations; complements the DISPMOD function. DISPMOD provides the
location of a specific module.
Usage
If a large address range is specified, this module uses a large amount of CPU time.
It can be cancelled using the RESET command.
Examples
Example: Output Generated by DSIMODQY:
| DSIMODQY: CSECTS IN RANGE HIGH: 0000FFFF TO LOW: 00000000
| CSECTADR: CSECTNME COMPDATE PTFLEVEL LOADNAME LOADADDR LOADLEN
| 0000FB78: DSIDRS 09.156 NV54°Ö}* DSIMNTEX 00008200 0000BE00
| 0000F748: DSIDPRS 09.093 NV54°Ö}* DSIMNTEX 00008200 0000BE00
| 0000E918: DSIDOS 09.174 NV54°Ö}* DSIMNTEX 00008200 0000BE00
| 0000BCE0: DSIDOPS 09.093 NV54 °Ö DSIMNTEX 00008200 0000BE00
| 0000A720: DSIDCLS 09.093 NV54 °Ö DSIMNTEX 00008200 0000BE00
| 00009468: DSICMDLD 09.093 NV54&\}* DSIMNTEX 00008200 0000BE00
| 00008200: NV54 10000 -------- DSIMNTEX 00008200 0000BE00
| 00008118: DSIZVLSR -------- -------- DSIZVLSR 00008118 000000E8
| 00007550: ABCDEFGH 01234 -------- DSIEBCDC 00007550 00000600
| DSIMODQY: ENDED
| caller_id
DSIGTVBA ptrvarname
oper_id typevarname
lu_name
number
|
| Purpose
| The DSIGTVBA command returns address and type information about the Task
| Vector Block (TVB) belonging to a NetView task. For information and offsets in the
| TVB, see the DSITVB macro in SCNMMAC1.
| Parameters
| The DSIGTVBA syntax options are defined in the following list:
| ptrvarname
| This is the name of a variable into which DSIGTVBA places the EBCDIC
| (readable) hexadecimal address of the target TVB control block. Note that this
| parameter is the name of a variable and will usually be inside the quotation
| marks when the command is invoked from a REXX procedure. The value
| returned is always 8 character EBCDIC with leading zeros as needed. It is
| suitable for use with the REXX STORAGE function or with the DSIGC2VR
| command.
| caller_id
| This is the operator ID of the task that is making the request. This is the
| default.
| oper_id
| This is the user name (OPID) for which a TVB address is required.
| lu_name
| This is the LU or terminal name associated with the task. This value is listed as
| the TERM value in the LIST taskname command. For autotasks, the value is the
| same as OPID.
| number
| This indicates that the TVB you want is the one with that number in the
| internal chain of NetView TVBs. Note that virtual OSTs (VOSTs) are not found
| on this chain. Your PPT task is always the value of 1 and your maintask is
| always the value of 0.
| typevarname
| This is the name of a variable into which DSIGTVBA places a token indicating
| the task type. The value returned can be one of the following values:
| AUT autotask
| DAU distributed autotask
| HCT hardcopy task
| IOT internet operator
| MNT main task
| NNT NCCF to NCCF task
Appendix A. Diagnostic Command Summary 579
| OPT optional task
| OST normal (VTAM) operator
| PPT PPT task
| VOST virtual OST
| Note: A VOST can be found only by specifying operid (begins with DSI#). The
| MNT can be found only by specifying number.
| Return codes
| 0 Success
| 100 Required input parameters are missing
| 104 Conversion error
| 112 OPID or LU name not found
| 116 Number specified is too large
| 120 Dictionary write error
| Usage
| The storage address is returned as 8 characters of hexadecimal data.
| Examples
| Example: Displaying information about a TVB:
| /*TEST: DSIGTVBA example */
| 'DSIGTVBA TVBPTR ' arg(1) 'WHATtype'
| IF RC=0 THEN
| DO;
| say 'tvbptr='tvbptr
| say 'type='whatType
| luOFF = d2x(x2d(tvbptr) + 60)
| say 'luname='storage(luOFF,8)
| IDOFF = d2x(x2d(tvbptr) + 68)
| say 'opid ='storage(IDOFF,8)
| END;
| ELSE
| SAY 'RC was' RC
| For an inactive TVB, the first byte of OPID will be either X'00' or X'40'.
Purpose
DSIGV2VR retrieves data from the storage defined by the address, offset, and
length values. The data is converted to the character representation appropriate for
the character (CHR), hexadecimal (HEX), decimal (DEC), or binary (BIT) option.
The result is placed in the CLIST or REXX variable named varname.
Parameters
The DSIGV2VR syntax options are defined in the following list:
Address
Must be specified as a hexadecimal value.
Offset
Must be specified as a hexadecimal value.
Length
Must be specified as a hexadecimal value.
Varname
Will be resolved to the character string that results from CLIST substitution
with an ampersand (&) appended.
CHR
For this option, the data is moved, as it is, into the variable.
HEX
For this option, each byte of data is expanded to 2 characters in the range 0–9
and A–F.
DEC
For this option, the data is stored as a decimal number. The source data must
be in the range of 1–4 bytes in length. Lengths of 1 and 3 denote unsigned
decimal values. Lengths of 2 and 4 are considered to be signed values.
BIT
For this option, each byte of data is expanded to 8 characters of either 1 or 0,
denoting the binary value of the data.
Return codes
8 Abend (recovered) accessing the data specified
100 Address parameter had greater than 8 characters
104 Address parameter had incorrect hexadecimal digits
108 Length parameter had incorrect hexadecimal digits
108 Dictionary update failure
112 Length parameter had incorrect hexadecimal digits
Purpose
DSIGADHX adds two literal hexadecimal values and stores in CLIST variable
named varname. This is useful for adding offset and address values together for
use with DSISHWVR.
Parameters
The DSIGADHX syntax options are defined in the following list:
Hexval1
Must be specified as hexadecimal values.
Hexval2
Must be specified as hexadecimal values.
Varname
Will be resolved to whatever character string results after CLIST substitution
with an additional ampersand (&) appended.
MINUS keyword
Is used for subtraction and must be placed after varname. A minus (-)
character can be used, but it conflicts with the NetView CLIST language
continuation of a line function (not a problem in REXX).
Return codes
8 An abend (recovered) occurred accessing the data specified.
100 The hexval1 parameter contained greater than 8 characters.
104 The hexval1 parameter contained hexadecimal digits that are not valid.
108 The dictionary update failed.
120 The required parameters are missing.
124 The hexval2 parameter contained more than 8 characters
128 The hexval2 parameter contained hexadecimal digits that are not valid.
136 The hexadecimal data was longer than 127.
144 A hexadecimal data conversion error occurred.
Purpose
DSISHWVR displays dump format data in hexadecimal and character on the
screen.
Parameters
The following parameters apply:
v The DSISHWVR command must be run in a command procedure.
v The DSISHWVR syntax options are defined in the following list:
address
Must be specified as hexadecimal values.
length
Must be specified as hexadecimal values.
Return codes
This command issues diagnostic messages for input conditions that are not valid. If
the storage is not addressable, the display is either truncated or not produced.
Recovery logic is used in all systems to avoid abends and program checks.
The described recovery is intended for use with NetView commands, such as
modem configuration, which cannot be interrupted during screen input, and to
enable action to be taken if an operator leaves a terminal unattended with the
panel lock blocking messages.
The following fields are in a special table located by the address in MVTCPTPT.
MVTCPAWT (Word value)
The time in 1.048576 second units that any task can wait while not
accepting messages before an abend user 97 occurs. A decimal value of:
57 is 1 minute.
572 is 10 minutes.
3433 is one hour.
Other values can be computed and used. NetView adds the word value to
the first 4 bytes of the system clock at the time the task issues an
internal-to-NetView DSIWAT to determine the expiration time.
MVTCPASB (Word value)
The number of bytes of storage for which TVBGUSTR can increase while
not accepting messages and before an abend user 97 occurs.
MVTCPAOB (Byte of bits)
OI MVTCPAOB,X’80’ Will indicate that an abend user 97 will occur if the
task is posted to end, and the task is not waiting on the terminate ECB,
TVBTECB.
The recovery occurs only if the fields are set to nonzero values and a NetView
product module has issued DSIWAT. The equivalent function is not available using
the assembler DSIWAT macro.
This section contains the request/response unit (RU) flow diagrams for the
following tasks or occurrences:
v Logging on to an operator station
v Starting the hardcopy device
v Starting cross-domain sessions (VTAM-VTAM)
v Starting a cross-domain session to support session monitor conversations
v Starting an operator terminal access facility (TAF) session
Figure 77 on page 588 is a diagram of the RU flow for an operator station logon.
+RSP(BIND)
ACTLU
+RSP(ACTLU)
NOTIFY
+RSP(NOTIFY)
UNBIND
(BIND forthcoming)
+RSP(UNBIND)
CINIT
+RSP(CINIT)
BIND
+RSP(BIND)
NetView screen
and DSI020I sent
to operator
terminal
Figure 78 on page 589 is a diagram of the RU flow that occurs when you start a
hardcopy device.
DSI033I sent
to operator
station
BIND
+RSP
('session starting')
+RSP(BIND)
SDT
+RSP(SDT)
'session starting'
DSI033I sent
to hardcopy
device +RSP(message)
Figure 79 on page 590 is a diagram of the RU flow that occurs when you start a
cross-domain session (VTAM to VTAM).
+RSP(BIND)
SDT
+RSP(SDT)
UNBIND HOLD
+RSP(UNBIND)
BIND
+RSP(BIND)
SDT
+RSP(SDT)
DATA(XTH+OPID, ETC)
ROUTE domainid,
OPID, etc. +RSP(DATA)
DATA(XTH+'READY')
+RSP(DATA)
DSI020I message sent
to operator station
Figure 80 on page 591 is a diagram of the RU flow that occurs when you start a
cross-domain session that supports session monitor conversations for continuous
sessions.
Initiate
CDINIT
+RSP
+RSP
CDCINIT
CINIT
+RSP
+RSP
BIND
+RSP
SESSST
Session Monitor
is notified of Completion
allocation Information CDSESSST
completion
Send Data +RSP
Request
FMH5. DATA
Allocation Session Monitor
Notification is notified of
conversation
Receive
Request
Figure 80. RU Flow Diagram for Starting a Cross-Domain Session to support session monitor conversations for
Continuous or Persistent Sessions
Figure 81 on page 592 is a diagram of the RU flow that occurs when an operator
TAF session is started.
OP Data
OP Data
Appl Data
(No WSF)
Read Buffer
Command
Screen Contents
Escape Screen
Escape Option 2
NCCF Screen
Appl Data
Appl Data
RTRNSESS
Saved Screen
Contents
OP Data
OP Data
Session
Continues
Control Blocks
This section describes NetView control blocks and related fields.
4 TVBNEXT
8 TVBTIB
C TVBTCB
1C TVBEXMSG
4 TIBTVB 4 TIBTVB
8 TIBACB 8 TIBACB
The following list describes the fields that are found in the DSIMVT control block:
Field Description
MVTDQT
Pointer to domain qualification table.
MVTDDT
Pointer to domain definition table.
MVTSCT
Pointer to system command table. A system command entry (SCE) is built
for each CMDDEF definition.
MVTTVB
Pointer to a chain of task vector blocks (TVBs). The number of TVBs equals
the maximum number of tasks (for example, OSTs, HCTs, NNTs) for this
instance of the command facility.
MVTIND
One-byte indicator flag:
( 1... ....)
MVTINIT- Command facility initialization in progress
( .1.. ....)
MVTTERM- Command facility termination in progress
MVTACB
Pointer to the main task access method control block (ACB).
MVTTVBM
Pointer to the main task TVB.
The following list describes the fields that are found in the DSITVB control block.
Field Description
The following list describes the fields that are found in the DSITIB control block.
Field Description
TIBTVB
Pointer to the associated TVB.
TIBACB
Pointer to VTAM ACB that contains session initialization information.
MVT
X'0' 'F1'
DSITVB
'48' MVTTVB
X'0' 'F2'
8 TVBTIB
DSITIB
X'0' 'F3'
AAUTSCT
'6C' TIBNLDMP
X'0' AAUTGLOB
18 AAUTGLOB X'0'
AAUTGLOB
8 GLBFLAGS X'0' GLBKPRTM
20
30 GLBRTMP 5 GLBRTDEF
AAUTSTRR 34 GLBMSTP
2C
38 GLBPCTP E GLBBROUND
88 GLBS1D3 16 GLBRFLG
34
8C GLBS1D4
90 GLBS1D5
94 GLBDP1U
98 GLBPSES X'0' MSTNAME
4 MSTNOENT (*)
9C GLBBUFSZ 10 MSTENTRY
AO GLBBUSZ EPLUPNAM
0
B7 GLBNTBUF
B8 GLBNSBUF
1C ESLUPNAM
CC BLGAMNAM
38 MSTEPCLS
D4 GLBLUNAM
40 MSTEERN
41 MSTEVRN
E4 GLBNETNA
EC 42 MSTETPN
GLBNLDMD
F0 43 MSTEFLAG
GLBSAPUN X'0' PCTNAME
F8 GLBSSCP 8 PCTNOENT
100 GLBSSCPA
C PCTMEM
112 GLBAMVER
14 PCTOPER
113 GLBAMREL
124 1C PCTOPDOM
GLBKMSTP(1)
128 24 PCTTIME (*)
GLBKCTP
30 PCTENTRY
AAUTSTRR X'0' PCTEPCLS
8 PCTEOBJT
X'24' STRRSTAT A PCTEOBJP
C PCTEBNDS
14 PCTEDEF
AAUTSTAT
15 PCTEFLAG
X'0' STATNAME X'0' KCTNAME
40 STATASBCOUNT 8 KCTNDENT
64 STATSSCPSSCP C KCTMEM
6C STATSSCPPU 14 KCTOPER
74 STATSSCPLU 1C KCTOPDOM
7C STATLULU 24 KCTTIME (*)
84 STATRECORDQUE 30 KCTENTRY
(* ) The line indicates the second X'0' KCTEKCLS
segment in an array and the two 8 KCTESAW
data segments are contiguous. 9 KCTEDASD
A KCTEPIUS
(1) This points to a data structure
which has the same mapping as
one pointed to by GLBMSTP.
The following list describes the fields in the RTM initialization parameter table
(pointed to by GLBRTMP):
Field Description
GLBKPRTM
RTM keep wrap count (4 bytes)
GLBRTDEF
Default RTM definition (1 byte)
GLBBOUND
Default RTM bucket boundaries array of 2 byte fields (8 bytes)
The following list describes the fields found in the MST entry structure:
Field Description
EPLUPNAM
Primary session end point name (8 bytes)
ESLUPNAM
Secondary session end point name (8 bytes)
MSTEPCLS
Performance/keep class name of this entry (8 bytes)
MSTEERN
Session ER number (1 byte)
MSTEVRN
Session VR number (1 byte)
The following list describes the fields found in the PCT entry structure:
Field Description
PCTEPCLS
Performance class name (8 bytes).
PCTEOBJT
Objective response time. The default is 0 (2 bytes).
PCTEOBJP
Objective percent. The default is 0 (1 byte).
PCTEBNDS
Array of 2-byte entries of bucket boundaries. The default is 10, 20, 50, or
100 (8 bytes).
PCTEDEF
Response time monitor (RTM) definition. The default is X'F' (1 byte).
PCTEFLAG
Control indicators (1 byte):
( 1... ....)
Display RTM locally allowed. The default is 0.
( .111 1111)
Reserved.
The following list describes the fields in the KCT entry structure:
Field Description
KCTEKCLS
Keep class name (8 bytes).
KCTESAW
Session awareness filter (1 byte):
1 = Discard
2 = Keep
KCTEDASD
VSAM recording filter (1 byte):
X'00' = Never record
X'04' = Record normal end
X'08' = Record if abnormal unbind occurs
X'10' = Record if bind failure occurs
X'20' = Record if initial failure occurs
X'40' = Record if trace data exists
X'80' = Record if RTM data exists
X'C0' = Record if session has trace or RTM data
X'38' = Record if session ends abnormally
X'FF' = Always record
KCTEPIUS
PIU keep count (2 bytes).
X'8' TVBTIB
DSITIB
X'30C' TIBINT1
BNJTDIR
X'8' DIRDWCP
X'C' DIRDRCP
X'10' DIRAFTP
X'14' DIRESFTP
X'18' DIROFTP
X'1C' DIRVIEWP
X'20' DIRDSTF
X'24' DIRCDSXP
X'2C' DIRTBLP BNJTDSTF
BNJTCDSX
X'16' DSTFLG
X'8' CDSXPTR (1) DSTFLAG1
X'C' CDSXAVAL (1) DSTFLAG2
X'10' CDSXPTR (2) DSTFLAG3
X'14' CDSXAVAL (2) DSTBLFLG
DSTBRFLG
X’0’
DSIGIT
BD8 MVTGIT
X’0’
CNMDMCT
48 GITVNCA
X’0’ MCTVCH CNMDRDAT
8 MCTRCATP
X’0’ RDATNAME
18 MCTMVTP X’1A’ RDATYPE
1C MCTPVTP X’20’ RDATHINP
24 MCTIMTP X’48’ RDATSTAT
28 MCTRDATP
X’49’ RDATCTL2
6B MCTVMID
84 MCTTPNDP
88 MCTTPECB
8C MCTVSECB
90 MCTVRECB
B8 MCTRCAUP
BA MCTMONIT
BB MCTNMON
9C MCTMSGSP
The following list describes the fields that are found in the CNMDMCT control
block:
Field Description
MCTVCH
Control block header (4 bytes)
MCTRCATP
Pointer to TVB for CNMTARCA task (4 bytes)
MCTMVTP
Pointer to DSIMVT (4 bytes)
MCTPVTP
Pointer to CNMDPVT (4 bytes)
MCTIMTP
Pointer to CNMDIMT (4 bytes)
MCTRDATP
Pointer to RDAT (4 bytes)
MCTVMID
STATMON main task name, for example CNM01VMT (8 bytes).
MCTTPNDP
Pointer to SPO TPEND routine (4 bytes)
MCTTPECB
Pointer to SPO TPEND ECB (4 bytes)
MCTVSECB
SPO VTAM send ECB (4 bytes)
MCTVRECB
SPO VTAM receive ECB (4 bytes)
RECFMS Header
Bytes 00 through 13 are consistent for RECFMS record formats 00 through 05.
These bytes form the RECFMS header for each RECFMS record.
Table 175. RECFMS Header
Bytes Bits Description
00-02 Network services header: X'410384'
03-07 MS header
08-11 0-11 Block ID code
08-11 12-13 ID number in binary
12-13 Reserved
RECFMS 00
The RECFMS 00 record is created when an unsolicited alert is sent to the
mainframe server. See “RECFMS Header” for bytes 00 to 13, the RECFMS header.
Bytes Description
17 Reserved
18 User action code. The hardware monitor
uses this, along with the block number,
to locate the following information:
v Alert description on alert displays
v Event description on alert displays
v Proper recommended action display
v Proper detail display
19 Reserved
After these fields, one or more RECFM4s can be appended. Text Vector, Detail
Qualifier Vector, and Name List Vector RECFM4s are retired (supported only for
PUs not at the current level of SNA).
Bytes Description
00 Vector length in binary
01 Vector type X'0D'
02-n Detail qualifiers; this information is shown on the hardware monitor event
detail panel.
Resource
from n Type: Meaning
to end Acronym
m+1 to m+4 ADAP Adapter
m+1 to m+4 ALA Alternative line attachment
m+1 to m+4 ALS Adjacent link stations
m+1 to m+4 BRDG LAN bridge
m+1 to BSC Binary synchronous link
m+4
m+1 to m+4 CBUS CSMA/CD bus
m+1 to m+4 CHAN Channel
m+1 to m+4 COMC Communication controller
m+1 to m+4 CPU Host processor
m+1 CTF Customer transaction facility
to m+4
m+1 to m+4 CTRL Controller
Null Vector
Bytes Description
00 X'00' (zero
length) indicates the end of RECFM4s.
Bytes Description
14-15 Binary counter showing the number of times the secondary SDLC station has
received an SDLC TEST command with or without a valid frame check
sequence (FCS).
16-17 Binary counter showing the number of times the secondary SDLC station has
received an SDLC TEST command with a valid FCS and has transmitted an
SDLC test response.
RECFMS 02
RECFMS 02 records contain a summary of error statistics generated by certain
resources.
Bytes Description
15-16 Reserved
17-18 Binary counter showing product-detected hardware errors (internal)
19-20 Binary counter showing communication adapter errors (internal or external)
21-22 Binary counter showing SNA negative responses
RECFMS 03
RECFMS 03 records show error statistics generated by certain remote
(link-attached) SNA resources. The exact contents of the statistical counters
depends on the device type. The RECFMS 03 records can contain counter sets.
Bytes Description
17 Reserved
RECFMS 04
RECFMS 04 records are used for all communications between a financial system
controller and the 4700 Support Facility.
Bytes Description
14-n PU and LU dependent data
Loop Status
Bytes Description
14 Response type (X'10')
15 Reserved
16 Number of loops being reported
Bytes Description
16-n Loop error or response time data; dependent on value specified in statistics
type (byte 15, bits 6-7)
Loop Errors
The entry that follows the last entry for a loop has the extended statistical counter
ID set to X'FFFF'.
Bytes Description
16 Number of loops that have reportable data
17 Loop ID (loop number indicated by binary value)
18 Value of loop basic counter 2
The maximum size of an RU, including the header, is 256 bytes. Loop extended
counters can overflow into additional RUs.
Bytes 16-n Response Time Entries for Each Workstation Being Measured
16 Workstation ID: binary number of the workstation that is the source of interval
timer data.
17 Number of timers: binary value that indicates the number of interval timer
entries that follow.
Bytes 18-n Timer Data: 13-byte Entry with 01 Timer Number Included
02-04 Minimum response time; bytes 2 and 3 are seconds in the range of 0 to 65535,
and byte 4 represents the fractional portion of a second.
05-07 Maximum response time; bytes 5 and 6 are seconds in the range of 0 to 65535,
and byte 7 represents the fractional portion of a second.
08-11 Cumulative elapsed time; bytes 8, 9, and 10 are seconds in the range of 0 to
16777215, and byte 11 represents the fractional portion of a second.
12-13 Number of intervals; a binary value representing the number of intervals
totaled in cumulative elapsed time (bytes 8-11).
A data item (such as a set of statistical counters) is not split between RUs.
Bytes Description
16 Number of bytes of counter data + 1
See “RECFMS Header” on page 609 for bytes 00 to 13, the RECFMS header.
This data provides you with hexadecimal data that can be interpreted to describe
the hardware, microcode, or programming levels of SNA controllers.
The following products provide release level data to the NetView program:
v System/38
v 3104 Display Terminal
v 3174 Subsystem Control Unit
v 3274/6 Control Unit
v 3720 Communication Controller
v 3725 Communication Controller
v 3776/7 Communication Terminal
v 7426 Terminal Interface Unit
v 8775 Display Terminal
You can find 3174 configuration information in “RECFMS 05, 3174 Configuration
Information” on page 622.
IBM System/38
Bytes Value Description
14 X'02' Constant X'02' to identify bytes 15-18
15 X'02' Planar level number
16 X'02' SCA-ROS card level
17 X'02' Periodic EC level
18 X'02' OU level
IBM 3104
Bytes Description
14 Part number of chip 1
18 Part number of chip 2
22 Part number of chip 3
26 Part number of chip 4
30 Part number of chip 5
Succeeding REQMS 05 requests retrieve the second type of response, while the
continuation byte indicates more data. If the continuation byte indicates no further
data, the first type of response is sent at the next request from the mainframe
server.
This pattern of response to REQMS 05 requests continues while the physical unit is
active.
See “RECFMS Header” on page 609 for bytes 00 to 13, the RECFMS header.
Bytes Description
17 Release level
18 Suffix level
19-21 Maintenance level
Bytes Description
23 Reserved
24 Reserved
Bytes Description
26 Reserved
34 Channel Configuration
Bytes Value Adapter Question
Information
34 X'00' Burst size 002 225 = 0
34 X'10' Burst size 004 225 = 1
34 X'20' Burst size 008 225 = 2
34 X'30' Burst size 016 225 = 3
35 Channel Configuration
Bytes Value Adapter Question
Attention Value
35 X'0A' (SNA) 10 to 99 223
-X'63' milliseconds
36 Channel
Adapter Configuration
Bytes Value Support of Question
Command Retry
36 X'01' Command retry 222 = 1
Bytes Description
38, 39 Reserved
40, 41 Control unit model number
42 Reserved for host-attach mode
Bytes Description
47-53 Reserved
Bytes Description
55 Reserved
Bytes Description
65-76 Reserved
Bytes Description
100-117 Reserved
Bytes Description
128-140 Reserved
Bytes Description
156-173 Reserved
Bytes Description
178 Reserved
179 Reserved
Bytes Description
181 Reserved
Bytes Description
192-207 RPQ parameter list
208-223 Reserved
Bytes Description
226-233 RTM time boundary
IBM 3274
3274 configurations C and D, besides providing EC level information, also provide
a complete copy of the configuration table.
Bytes Description
15-30 Installed patch ID values
31 Number of RPQs installed on 3274
32 Reserved
33 RPQ 1 ID
38 RPQ 2 ID
Bytes Description
16 Feature level (see bytes 127, 128)
17 System level (see bytes 129, 130)
18 Language level (see bytes 131, 132)
Bytes Description
20 Channel address (for models 1A, 21A, 31A, and 41A)
Bytes Description
24 BSC polling address
25 BSC or SDLC selection address
Bytes Description
29 Control storage base
30-33 Control storage addition
Bytes Description
40 Total category B terminals
41 Total category A terminals
42 Total all terminals
Bytes Description
44 EBCDIC BSC control unit ID
45 Language type
46 Extended function store response
47 'A' DCB count
48 Total DCB count
49 Print authorization matrix entry count
Bytes Description
51 Extended DCB count
Bytes Description
54 X.21 switched retry timing response
55-56 Validation number
57-75 Reserved
Bytes Description
77 Number of redialing attempts allowed
Bytes Description
79 Reserved
Bytes Description
81-96 Patch ID values
97 Number of RPQs installed
98 Reserved
Bytes Description
104-108 EC level of second RPQ installed
109-113 EC level of third RPQ installed
114 Feature diskette expected suffix
115 System diskette expected suffix
116 Language diskette expected suffix
117 Reserved
118-120 PU ID number
Bytes Description
123-126 Reserved
127 Feature diskette level
128 Feature diskette suffix
129 System diskette level
Bytes Bits Value 133-134 EC and Suffix Levels of First RPQ Installed
133-134 0 X'0' EC and suffix levels
133-134 0 X'1' The following conditions apply:
X'8100' Configuration level A
X'8200' Configuration level B
X'8400' Configuration level C
X'8800' Configuration level T
X'C000' Configuration level D or above
Bytes Description
135-136 EC and suffix levels of second RPQ installed (same conditions as bytes 99-103)
137-138 EC and suffix levels of third RPQ installed (same conditions as bytes 99-103)
Bytes Description
141-152 Reserved
Bytes Description
154-158 Reserved
159 EC level for load diskette
160 Suffix level for load diskette
161-165 ID for 3290 RPQ
166 First port with multiple interactive screen capability
167 Number of ports with two LTERM addresses
168 Number of ports with three LTERM addresses
Bytes Description
175 Reserved
176 Number of primary local device defined on 3274
177 Number of logical terminal extensions
Bytes Description
180-190 Reserved
191-206 RPQ parameter list
207-225 Category A port assignment table (32 possible ports)
226-270 Reserved
IBM 3276:
Bytes Description
14 Implementation-defined data describing hardware, microcode, and program
levels. 3276s have 48 fields. Each field is 4 bytes in length, is an unsigned
packed decimal, and contains a ROS chip 7-digit part number.
Bytes Description
14 6-digit current EC level of installed microcode plus a 2-digit patch level
Bytes Description
14-23 Microcode level
24 Customer program type and level
25-70 Customer identification
71-74 3720
75-76 01/02/11/12
77-84 Machine serial number
Bytes Description
85 Microcode historical data
v Last microcode fix (MCF)
Applied: MCF ID (8 characters) application date (3 characters)
v Number of most recently applied patches (binary)
v Most recently applied patches: 16 entries, each entry contains
Patch ID (8 characters)
Status:
X'01'APPLIED
X'02' NONAPPLIED
X'04' IN PROGRESS
X'08' BAD CHECKSUM
IBM 4701
Bytes Description
14 6-digit current EC level of installed microcode plus a 2-digit patch level
IBM 7426
Bytes Description
14 8-digit load module EC number (EBCDIC)
22 ROS Module-0 Chip-1 P/N (packed decimal)
26 ROS Module-0 Chip-2 P/N (packed decimal)
30 ROS Module-1 Chip-1 P/N (packed decimal)
34 ROS Module-1 Chip-2 P/N (packed decimal)
38 ROS Module-2 Chip-1 P/N (packed decimal)
42 ROS Module-2 Chip-2 P/N (packed decimal)
46 ROS Module-3 Chip-1 P/N (packed decimal)
50 ROS Module-3 Chip-2 P/N (packed decimal)
54 ROS Module-4 Chip-1 P/N (packed decimal)
58 ROS Module-4 Chip-2 P/N (packed decimal)
62 ROS Module-5 Chip-1 P/N (packed decimal)
66 ROS Module-5 Chip-2 P/N (packed decimal)
70 ROS Module-6 Chip-1 P/N (packed decimal)
74 ROS Module-6 Chip-2 P/N (packed decimal)
78 ROS Module-7 Chip-1 P/N (packed decimal)
82 ROS Module-7 Chip-2 P/N (packed decimal)
Bytes Description
86 SDLC station address
87 Downstream load data set name
95 Loop carrier speed and loop data
speed (Mod. 1)
Bytes Description
98-101 Reserved
Bytes Description
105-109 Reserved
Bytes Description
113-127 Reserved
Bytes Description
129-165 Reserved
IBM 8775
Bytes Description
14 8-digit hardware part number of the ROS module located at X'8000' and shown
in the format 4421XXXC where XXX is a variable
| DSINDEF provides the VTAM node control application input record containing the
| run parameters to the NetView status monitor task. DSINDEF is built by the
| CNMNDEF (CNMSJ007) job, and resides on the DSIPARM data set.
Each record in DSINDEF is 80 bytes long. Each record provides information on:
v Major nodes of the network
v Minor nodes of the network
v Comments
The records in DSINDEF must adhere to a hierarchy in which minor nodes follow
major nodes; for example, an NCP name followed by a LINE, followed by PUs,
and then LUs.
Note: The status monitor accepts data created by CNMNDEF but does not support
any logic to verify this data. Therefore, take care when modifying or
viewing this data to maintain the correct values for the entries specified in
DSINDEF.
Fields not in the MDB are set to zero (0). These fields show how the WTO SVC was issued, not to what the message
is about.
IBM may have patents or pending patent applications covering subject matter
described in this document. The furnishing of this document does not give you
any license to these patents. You can send license inquiries, in writing, to:
For license inquiries regarding double-byte (DBCS) information, contact the IBM
Intellectual Property Department in your country or send inquiries, in writing, to:
The following paragraph does not apply to the United Kingdom or any other
country where such provisions are inconsistent with local law:
Any references in this information to non-IBM Web sites are provided for
convenience only and do not in any manner serve as an endorsement of those Web
sites. The materials at those Web sites are not part of the materials for this IBM
product and use of those Web sites is at your own risk.
Licensees of this program who wish to have information about it for the purpose
of enabling: (i) the exchange of information between independently created
programs and other programs (including this one) and (ii) the mutual use of the
information which has been exchanged, should contact:
IBM Corporation
2Z4A/101
11400 Burnet Road
Austin, TX 78758 U.S.A.
The licensed program described in this document and all licensed material
available for it are provided by IBM under terms of the IBM Customer Agreement,
IBM International Program License Agreement or any equivalent agreement
between us.
Programming Interfaces
This publication documents information that is NOT intended to be used as
Programming Interfaces of Tivoli NetView for z/OS.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of
International Business Machines Corporation in the United States, other countries,
or both. If these and other IBM trademarked terms are marked on their first
occurrence in this information with a trademark symbol (® or ™), these symbols
indicate U.S. registered or common law trademarks owned by IBM at the time this
information was published. Such trademarks may also be registered or common
law trademarks in other countries. A current list of IBM trademarks is available on
the Web at ″Copyright and trademark information″ at http://www.ibm.com/legal/
copytrade.shtml .
Java and all Java-based trademarks and logos are trademarks or registered
trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other
countries.
Notices 663
664 Troubleshooting Guide
Index
A aggregate correlated objects with same object
problem scenario and resolution 204
abend aggregate correlated objects, information displayed changes
E/AS task 468 problem scenario and resolution 206
Event/Automation Service task 468 aggregate resource status incorrect 347
abend (abnormal end) AIFR (automation internal function request) 653
0C1 233 AIX problems
0C4 234 server window disappears
0C8 236 problem scenario and resolution 193
301 55 topology server does not complete initialization
9C5 (reason code 0) 237 problem scenario and resolution 193
9C5 (reason code 33) 237 alert
A78 received at NetView termination 53 no change in status 178
A78 received at task termination 53 not in hardware monitor history panel, NetView
classifying a problem management console 178
GMFHS 10 not in history at workstation, NetView management
NetView 9 console 177
RODM 10 alert adapter services, IP trace 492
SNA topology manager 11 alert and alert history problems and resolutions
diagnostic procedure 24 problems and resolutions 177
documenting procedure 24 alert problems
first failure data capture trace 146 alert not in hardware monitor history panel 178
FLBTOPO (task) 314 alert not in history at workstation 177
keyword (ABEND) 9 alerts are not converted to Tivoli Enterprise Console
NetView events 477
301 55 alerts are not forwarded 474
A78 received at NetView termination 53 alerts do not change status 178
A78 received at task termination 53 continuously forwarded 477
first failure data capture trace 146 incorrectly cached 478
RID function 56 alert/message adapter, event receiver 496
subtask 52 allocate storage request, NetView trace 144
U0258 54 AON (Automated Operations Network), worksheet 447
U0268 54 API 239
U0269 54 application
reporting procedure 24 RODM
RID (remote interactive debug) function 56 user API does not return from EKGWAIT 239
RODM application, RODM
0C1 233 failure 230
0C4 234 loop 238
0C8 236 array, allocate storage request, NetView trace 145
9C5 (reason code 0) 237 asynchronous method loop, RODM 238
9C5 (reason code 33) 237 attribute error, RODM 183
abends when SNA topology manager starting 237 Automated Operations Network (AON) 447
SNA topology manager 315 automation
abend error condition 316 not driven when expected 63
abends after initialization 315 not occurring correctly 62
abends during initialization 315 unexpectedly driven 62
FLBTOPO (task) 314 automation internal function request (AIFR) 653
subtask 52
U0258 54
U0268 54
U0269 54 B
ABEND keyword 9 BIND trace record, IP services 117
abend problems blank status history 330
NetView management console 166 BNH160I message 67
abnormal reaction from RODM 240 BNH161I 69
ACCEPT trace record, IP services 117 BNH162I 69
accessibility xxi BNH163I 69
activity (VY) trace record 136 books
aggregate correlated objects see publications xvii
cannot navigate between object and contained resources
problem scenario and resolution 207
Index 667
diagnostic tool, NetView program (continued) distributed mainframe server (continued)
PPI trace facility (continued) no data recorded 66
locating oldest record 152 solicited data not recorded 66
locating table 151 DOC keyword 11
trace anchor block 148 documentation
trace record 149 keyword (DOC) 11
trace table 148 documentation problem
service aid command documenting procedure 31
DSI24TRC command 575 reporting procedure 31
DSIGADHX command 583 documentation problems
DSIGTVBA command 579 NetView management console 168
DSIGV2VR command 581 documenting a problem
DSIMODQY command 578 abend (abnormal end)
DSISHWVR command 584 determining which task abended 27
DSISTRLS command 568 documenting procedure 24
RID DSIMSX command 556 dump of a NetView abend 25
TASKURPT command 558 dump of an FLBTOPO abend 30
WAIT time-out and storage limit 585 out-of-storage problem causing abend 28
SMF record 38 subtype 2 task utilization data 99 reporting procedure 24
TASKURPT 99 short-of-storage abend 28
trace collecting problem data 19
See trace, NetView program documentation
diagnostic tool, RODM documenting procedure 31
dump utility reporting procedure 31
class index report 290 documentation needed
class listing report 287 APAR (authorized program analysis report) 19
coding control parameter 285 CNMSTYLE 20
description 284 component ID number 19
dumping dataspace allocated by RODM 284 FMID (function monitor ID) 19
informational message 293 IHSERROR.LOG 20
invoking 285 NetView trace 20
object index report 291 network log 19, 20
object listing report 290 PTF (program temporary fix) 19
statistical report 292 status monitor preprocessor job output 20
load function error listing 295 system log 19, 20
log entry trace 19
See log entry, RODM incorrect output
message documenting procedure 32
See message, RODM reporting procedure 32
return code loop
See return code, RODM documenting procedure 33
trace, internal 282 reporting procedure 33
diagnostic tool, SNA topology manager performance
IPCS (interactive problem control system) 363 documenting procedure 35
log entry reporting procedure 35
See log entry, SNA topology manager wait
message documenting procedure 37
See message, SNA topology manager reporting procedure 37
NetView internal trace 363 DSI124I 56
network log 363 DSI24TRC command 575
recovery from trace error 427 DSI625I 54
system log 363 DSIAMII 191
TASKMON 363 DSIGADHX command 583
TASKUTIL 363 DSIGET/DSIFRE trace record 113
TOPOSNA LISTxxxx request 427 DSIGTVBA command 579
trace DSIGV2VR command 581
See trace, SNA topology manager DSIMODQY command 578
VTAM CMIP trace 427 DSINDEF form 649
diagnostic tools DSIPSS trace record 114
IP Management 161 DSISHWVR command 584
NetView for z/OS Enterprise Management Agent 529 DSISTRLS command 568
NetView management console 209 DSIWAT/DSIPOS/DISPATCH trace record 113
directory names, notation xxiii dump utility, RODM
Distributed DVIPA class index report 290
Connection Routing 522 class listing report 287
distributed mainframe server coding control parameter 285
error not recorded 65 description 284
Index 669
FLBSYSD initialization file 309 GIVESOCKET trace record, IP services 123
FLBTOPO task abend 314 GMFHS
FLC070E duplicate resource
RODM Errors 436 problem scenario and resolution 191
FLC076E error received during configuration initialization 185
RODM Errors 436 problems
flows information required for TSC 169
multiple Init_Accept received problems and resolutions 185
problem scenario and resolution 188 status solicitation failure 185
flows, request unit 587 GMFHS (Graphic monitor facility host subsystem)
force INACT (FINA) trace record 136 problem determination 439
formatter, RODM log GMFHS (Graphic Monitor Facility host subsystem)
customizing 247 diagnosing a problem 175
description 247 list of problem scenarios 175
invoking 249 problem background worksheet 169
message 250 trace
return code 250 See trace, GMFHS
free storage request, NetView trace 145 GTF (generalized trace facility)
FREEADDRINFO TCxx trace record, IP services 119 event ID
SNA topology manager 401
format ID
G SNA topology manager 401
record format
gateway
data format, SNA topology manager 401
COS gateway error 182
example, SNA topology manager 402
PPI gateway error 183
header, SNA topology manager 401
general information
return code
problem worksheet
SNA topology manager 373
NetView management console 165
generalized trace facility
See GTF (generalized trace facility)
GETADDRINFO TCxx output trace record, IP services 119 H
GETADDRINFO TCxx trace record, IP services 119 hang problems
GETCLIENTID TAxx trace record, IP services 131 NetView management console 167
GETCLIENTID TCxx trace record, IP services 120 hardware monitor
GETHOSTBYADDR trace record, IP services 120 control block
GETHOSTBYNAME trace record, IP services 120 BNJTCDSX 605
GETHOSTID TAxx trace record, IP services 132 BNJTDIR 604
GETHOSTID TCxx trace record, IP services 121 BNJTDSTF 605
GETHOSTNAME TAxx trace record, IP services 132 DSITIB 604
GETHOSTNAME TCxx trace record, IP services 121 hardware monitor history panel, NetView management
GETNAMEINFO TCxx output trace record, IP services 122 console
GETNAMEINFO TCxx trace record, IP services 121 alert not listed in 178
GETPEERNAME trace record, IP services 122 high-level language (HLL)
GETSOCKNAME trace record, IP services 122 first failure data capture trace 146
GETSOCKOPT TAxx trace record, IP services 132 RID (remote interactive debug) function 56
GETSOCKOPT TCxx Option Value Mapping 1, IP hints, problem determination 435, 505
services 132 DVIPA management
GETSOCKOPT TCxx Option Value Mapping 2, IP A command returns incomplete data 157
services 133 Connection Routing data is incomplete in the EMA
GETSOCKOPT TCxx Option Value Mapping 3, IP workspace 158
services 133 no configuration changes received 156
GETSOCKOPT TCxx Option Value Mapping 4, IP No data is returned from a DVIPA 3270 command 157
services 133 no DVIPA statistics recorded 157
GETSOCKOPT TCxx Option Value Mapping 5, IP no SNMP traps received 156
services 133 IP Management 155
GETSOCKOPT TCxx Option Value Mapping 6, IP Sysplex Topology 160
services 134 Tivoli NetView for z/OS Enterprise Management
GETSOCKOPT TCxx Option Value Mapping 7, IP Agent 519
services 134 XCF services
GETSOCKOPT TCxx trace record, IP services 123 BNH067I message 160
GETTOPO command BNH558E message 160
failures 438 BNH587I message 159
processing 439 discovery commands fail 160
tracing 438 master NetView, unexpected switch 160
GETTOPO TMERES, after reissuing no data returned 159
RODM error 8/45077 PLEXCTL command fails 159
problem scenario and resolution 437 START XCFGROUP problems 160
Index 671
IP services (continued) IPCS (interactive problem control system)
TAxx trace entries (continued) CNMIPCS TRACE verb
IOCTL trace record 134 options 79
TCPX trace record 135 CNMIPCS verb
TCxx trace entries options 78
ACCEPT trace record 117 command output
BIND trace record 117 ASID 81
CANCEL trace record 118 CPOOL 82
CLOSE trace record 118 D 82
CONNECT trace record 118 DISPLAY 83
FREEADDRINFO trace record 119 DISPMOD 84
GETADDRINFO output trace record 119 DTCB 85
GETADDRINFO trace record 119 LEVEL 85
GETCLIENTID trace record 120 LRCE 85
GETHOSTBYADDR trace record 120 MAP 86
GETHOSTBYNAME trace record 120 NLDM 87
GETHOSTID trace record 121 NPDA 89
GETHOSTNAME trace record 121 QUE 90
GETNAMEINFO output trace record 122 SAVEAREA 91
GETNAMEINFO trace record 121 STORE 91
GETPEERNAME trace record 122 SUMMARY 80
GETSOCKNAME trace record 122 TBLUSECT 93
GETSOCKOPT Option Value Mapping 1 132 TRACE 94
GETSOCKOPT Option Value Mapping 2 133 WHO 95
GETSOCKOPT Option Value Mapping 3 133 description 73
GETSOCKOPT Option Value Mapping 4 133 installation 74
GETSOCKOPT Option Value Mapping 5 133 operation 74
GETSOCKOPT Option Value Mapping 6 134 option selection example 79
GETSOCKOPT Option Value Mapping 7 134 VERBX CNMIPCS command
GETSOCKOPT trace record 123 summary 77
GIVESOCKET trace record 123
INITAPI trace record 123
INITAPIX trace record 123
IOCTL REGARG Mapping 1 134
K
keyword
IOCTL REGARG Mapping 2 134
ABEND (abnormal end) 9
IOCTL REGARG Mapping 3 134
ABENDxxx 27
IOCTL REGARG Mapping 4 135
AUTO 64
IOCTL REQARG Mapping 1 124
building a string 4
IOCTL REQARG Mapping 2 124
DOC (documentation) 11
IOCTL REQARG Mapping 3 124
INCORROUT (incorrect output) 11
IOCTL REQARG Mapping 4 125
LOOP 12
IOCTL REQARG Mapping 5 125
MINUS 583
IOCTL REQARG Mapping 6 125
MSG (message) 13
IOCTL trace record 124
NETID 651
LISTEN trace record 125
NUMILU 182
RECV trace record 125
OPERSEC 52
RECVFROM trace record 126
OPTION 116, 135
SELECT exit trace record 126
PARM (parameter) 247
SELECT trace record 126
PERFM (performance) 15
SEND trace record 126
SAF 137
SENDTO trace record 127
SAFA 137
SETSOCKOPT Option Value Mapping 1 128
STATOPT 649
SETSOCKOPT Option Value Mapping 2 128
SUPPRESS 64
SETSOCKOPT Option Value Mapping 3 128
TASK 135
SETSOCKOPT Option Value Mapping 4 129
WAIT 17
SETSOCKOPT Option Value Mapping 5 129
WQELBK 658
SETSOCKOPT Option Value Mapping 6 129
KFWITM081E 524
SETSOCKOPT Option Value Mapping 7 129
SETSOCKOPT trace record 127
SHUTDOWN trace record 129
SOCKET Trace Record 129 L
TAKESOCKET trace record 130 layout of view is incorrect
TERMAPI trace record 130 problem scenario and resolution 199
trace exit record 135 links, cross-product 524, 527
IP trace, E/AS 492 list
IP trace, Event/Automation Service 492 for tree view is incorrect
problem scenario and resolution 198
Index 673
log entry, system interface (continued) message problems (continued)
major code 22 (continued) not converted to Tivoli Enterprise Console event 481
minor code 32 (22-32) 371 not forwarded to event server 479
minor code 37 (22-37) 371 performance output problems
minor code 38 (22-38) 371 information required - reporting problems to TSC 173
minor code 39 (22-39) 371 problem classification
minor code 40 (22-40) 372 information required - reporting problems to TSC 171
minor code 47 (22-47) 372 wait classification
minor code 56 (22-56) 372 information required - reporting problems to TSC 172
log file message queueing rates
components that output data to RODM 244 diagnosing 99
RODM, components that output data to 244 message retrieval tool, LookAt xx
log, , network message, NetView management console
documenting a problem GMFHS
documentation needed 20 IHSERROR.LOG 210
log, , system message log 210
documenting a problem IHSERROR.LOG 210
documentation needed 20 message log 210
logon/bind problem with command facility 52 output logs 211
LookAt message retrieval tool xx message, NetView program
loop BNH160I 67
asynchronous method, RODM 238 BNH161I 67, 69
diagnostic procedure 33 BNH162I 67, 69
disabled 13 BNH163I 67, 69
documenting procedure 33 CNM983E 56
enabled 13 CNM998E 56
keyword (LOOP) 12 CNM999E 56
reporting procedure 33 DSI124I 56
RODM DSI625I 54
user application 238 DWO049W 57, 59
LOOP keyword 12 DWO158W 60
loop problems DWO627E 62
NetView management console 167 network log 96
looping 99 origin using MSGMODID 98
looping, alert/message adapter 496 message, RODM
lost connection with agent node 328 EKG1101E 231
lost property changes EKG1104E 239
problem scenario and resolution 191 EKG1105E 239
lost trace record, NetView 112 EKG1106E 239
LU 6.2 EKG1111I 232
cannot initiate session, NetView management console 181 EKG1112E 239
NETCONV, cannot initiate session 181 EKG1113I 239
LU not found with locate resource 331 EKG1116I 241
LUC EKG1117I 241
macro invocation 109 EKG1326D 237
receive exit 109 IEC161I 052-084 239
IEC161I 203-204 231, 232
IEC161I 227-229 239
M IEC340I 239
incorrect output in EKGPRINT data set 239
manual, problem with 31
range 14
manuals
message, SNA topology manager
see publications xvii, xxi
FLB300W 318
map, opening failures 441
FLB403I 321
MDB (message data block) 653
FLB404I 327
MENT (module entry) trace record 112
FLB405W 326
message adapter services, IP trace 492
FLB407E 328
message data block (MDB) 653
FLB408W 328
message keyword (MSG) 13
FLB409W 321
message problems
FLB420I 321
documentation problems
FLB421I 327
information required - reporting problems to TSC 173
FLB422W 326
incorrect output problems
FLB424E 328
information required - reporting problems to TSC 172
FLB425W 328
incorrectly cached 481
FLB426W 321
loop classification
FLB443I 327
information required - reporting problems to TSC 171
FLB481E 317
NetView management console 166
Index 675
network log (continued) PPI (program-to-program interface) (continued)
job output for status monitor preprocessor trace facility (continued)
documentation needed to document a problem 20 locating table 151
MSGMODID 98 trace anchor block 148
TASKMON command 96 trace record 149
TASKUTIL command 96 trace table 148
network views, unable to monitor preprocessor
problem scenario and resolution 200 documenting a problem
not enough storage, topology manager 309 documentation needed 20
notation job output for status monitor preprocessor
environment variables xxiii documentation needed to document a problem 20
path names xxiii preview image
typeface xxiii partially painted 207
nvsrvc utility printing
configuring Web server ports 505 trace record
when MODE=EXT 103
when MODE=INT 100
O problem
classifying
object index report, RODM dump utility 291
abend (abnormal end) 9
object listing report, RODM dump utility 290
building a keyword string 4
object status update failures 441
describing a problem 4
objects with same object
documentation 11
problem scenario and resolution 204
INCORROUT (incorrect output) 11
OC (operator command) trace record 135
loop 12
OMEGAMON XE data 508
message 13
online help support for E/AS 489
performance 15
online help support for Event/Automation Service 489
RETAIN database 4
online publications
search string 4
accessing xxi
wait 17
operator command (OC) trace record 135
collecting data 19
ordering publications xxi
documenting
OSA
abend 24
data collection tower or subtower 522
documentation 31
missing data 158
documentation needed 19
workspace data missing 523
INCORROUT (incorrect output) 32
OST (operator station task)
loop 33
error, GMFHS 182
performance 35
out-of-storage problem causing abend 28
wait 37
output logs, GMFHS
reporting
description 211
IBM Software Support 19
output, incorrect
IBM Software Support specialist 3
NetView management console 168
Information/Access IBM licensed program 3
searching the database for a solution 3
software support database, searching solutions 3
P worksheet
path names, notation xxiii AON 447
penalty time, amount assessed E/AS 459
diagnosing 99 Event/Automation Service 459
PERFM keyword 15 GMFHS 169
performance problem MultiSystem Manager 431
diagnostic procedure 35 NetView 45
documenting procedure 35 NetView management console 165
keyword (PERFM) 15 RODM 221
reporting procedure 35 SNA topology manager 301
performance problems 99 Tivoli NetView for z/OS Enterprise Management
NetView management console 168 Agent 513
pop-up menu in business tree Web application 501
not displayed on AIX platform problem classification
problem scenario and resolution 207 documentation problems
PPI (program-to-program interface) information required - reporting problems to TSC 173
gateway error 183 incorrect output problems
recycling 486 information required - reporting problems to TSC 172
trace facility loop problems
description 148 information required - reporting problems to TSC 171
GTF output files 152 message problems
locating oldest record 152 information required - reporting problems to TSC 171
Index 677
problem scenario, MultiSystem Manager problem scenario, NetView management console (continued)
list of problem scenarios 435 symptom (continued)
problem scenario, NetView management console view shows incorrect connectivity 201
alert time-out error
no status change 178 command 183
alert and alert history problems and resolutions 177 topology console 189
attribute error, RODM 183 hangs during sign-on 190
component or connection status unable to connect from, to topology server 190
not properly reflected on topology console 186 topology console hangs during sign-on 190
duplicate GMFHS resource 191 topology console hangs when accessing view 190
events not received from z/OS instrumentation 186 topology display subsystem view problem 208
gateway error topology server 192
COS (common operations services) 182 topology server does not complete initialization 193
GMFHS Views 195
attribute error, RODM 183 problem scenario, NetView program
cannot initiate LU 6.2 session using NETCONV 181 abend (abnormal end)
command time-out error 183 301 55
COS gateway error 182 A78 received at NetView termination 53
NETCONV, cannot initiate LU 6.2 session 181 A78 received at task termination 53
OST (operator station task) error 182 RID (remote interactive debug) function 56
PPI gateway error 183 subtask 52
service point error 183 U0258 54
GMFHS problems and resolutions 185 U0268 54
incorrect resource status 187 U0269 54
incorrect timestamps automation
if topology server is on Windows platform 193 not driven when expected 63
instrumentation (z/OS-based) 186 not occurring correctly 62
IP session 180 unexpectedly driven 62
list of problem scenarios 175 distributed mainframe server error not recorded 65
minimized window problem 191 DSIFRE message 57
multiple Init_Accept flows received DSIGET message 59
problem scenario and resolution 188 EP/local error not recorded 64
pop-up menu in business tree list of problem scenarios 51
not displayed on AIX platform 207 logon/bind problem with command facility 52
preview image partially painted 207 message
property changes lost 191 CNM983E 56
resource exists but status is not updated CNM998E 56
problem scenario and resolution 188 CNM999E 56
server windows disappear on AIX platform 193 DSI124I 56
service point error 183 DSI625I 54
status changes to resources not reflected in views DWO049W 57, 59
problem scenario and resolution 189 DWO158W 60
status problems and resolutions 187 DWO627E 62
symptom MS transport cancels 62
alert not in hardware monitor history panel 178 return code
alert not in history at workstation 177 U0258 abend code 54
alerts do not change status 178 U0268 abend code 55
attribute error, RODM 183 U0269 abend code 54
cannot navigate between object and contained RMTCMD RUNCMD response on MVS console 67
resources 207 security problem 67
command cannot initiate LU 6.2 session using solicited data not recorded 66
NETCONV 181 unsolicited remote error not recorded 65
command result unexpected 182 problem scenario, RODM
COS gateway error 182 abend (abnormal end)
missing configuration 196 0C1 233
more detail view does not exist 196 0C4 234
NETCONV, cannot initiate LU 6.2 session 181 0C8 236
no result when issued or sent 182 9C5 (reason code 0) 237
OST (operator station task) error 182 9C5 (reason code 33) 237
PPI gateway error 183 when topology manager starting 237
real resource not shown 205 abnormal reaction 240
sent with no result 182 application failure 230
service point error 183 asynchronous method loop 238
time-out error 183 checkpoint processing fails 239
topology console hangs when accessing view 190 debugging method 229
unable to open view 200 fails to complete checkpoint processing 239
view layout is incorrect 199
Index 679
problem worksheet resource, missing
NetView management console view does not contain
abend problems 166 problem scenario and resolution 202
documentation problems 168 resources
general information 165 status changes not reflected in views
incorrect output problems 168 problem scenario and resolution 189
loop problems 167 RETAIN database 4
message problems 166 return code
performance problems 168 NetView program
problem classification 166 U0258 abend code 54
problem description 166 U0268 abend code 55
system-related information 165 U0269 abend code 54
wait problems 168 RODM
problems 12, reason code 121 231
information required for TSC 12, reason code 122 232
general information 169 12, reason code 194 232
GMFHS-related information 169 12, reason code 20 230
problem classification 170 12, reason code 211 234
problem description 170 12, reason code 212 235
RODM applications information 169 12, reason code 213 235
RODM methods information 169 return code 12
system-related information 169 RODM 437
processor exception problems RID (remote interactive debug) function 56
NetView management console 166 RID DSIMSX 556
processor traps RMTCMD errors 440
problem classification RMTCMD RUNCMD response on MVS console 67
information required - reporting problems to TSC 170 RODM (Resource Object Data Manager)
property changes lost abend
problem scenario and resolution 191 See abend, RODM
publications xvii diagnosing a problem 227
accessing online xxi diagnostic tool
ordering xxi See diagnostic tool, RODM
dump utility
See dump utility, RODM
R errors 436, 437
FLC070E 436
real resource not shown
FLC076E 436
as member of correlated aggregate object
list of problem scenarios 227
problem scenario and resolution 205
log entry
reallocate storage request, NetView trace 145
See log entry, RODM
reason code 122
message
RODM 437
See message, RODM
RECFMS record format 609
problem background worksheet 221
recording failure, database 32
reason code 122 437
RECV trace record, IP services 125
return code
RECVFROM trace record, IP services 126
See return code, RODM
reissuing GETTOPO TMERES, after
return code 12 437
RODM error 8/45077
trace, internal 282
problem scenario and resolution 437
RODM applications
remote interactive debug (RID) function 56
problems
reporting a problem
information required for TSC 169
IBM Software Support 19
RODM error 8/45077
IBM Software Support specialist 3
after reissuing GETTOPO TMERES
Information/Access IBM licensed program 3
problem scenario and resolution 437
searching the database for a solution 3
RODM methods
software support database, searching solutions 3
problems
request unit (RU) flows 587
information required for TSC 169
resource
routing
duplicates for GMFHS
messages 436
problem scenario and resolution 191
RUNCMD errors 440
exists but status is not updated
problem scenario and resolution 188
icon is missing from view
problem scenario and resolution 197 S
status change processed (CE) trace record 136 SAF (security authorization facility) trace record
resource status, incorrect AUTH 138
problem scenario and resolution 187 description 137
EXTRACT 139
Index 681
symptom (continued) symptom (continued)
abend (abnormal end) (continued) list of problem scenarios (continued)
FLBTOPO task 314 NetView program 51
keyword (ABEND) 9 RODM 227
RID (remote interactive debug) function 56 SNA topology manager 307
RODM, when topology manager starting 237 sysplex 155
subtask 52 Tivoli NetView for z/OS Enterprise Management
topology manager error condition 316 Agent 519
U0258 54 Web application 505
U0268 54 logon/bind problem with command facility 52
U0269 54 loop
abnormal reaction from RODM 240 asynchronous method, RODM 238
APM keyword (LOOP) 12
cannot initiate IP session using NETCONV 180 user application, RODM 238
NETCONV, cannot initiate IP session 180 message
application failure, RODM 230 CNM983E 56
asynchronous method loop, RODM 238 CNM998E 56
automatic monitoring, topology manager 320 CNM999E 56
automation DSI124I 56
not driven when expected 63 DSI625I 54
not occurring correctly 62 DWO049W 57, 59
unexpectedly driven 62 DWO158W 60
blank status history 330 DWO627E 62
cannot find LU with locate resource 331 EKG1101E 231
cannot issue resource control command EKG1104E 239
generic commands fail 331 EKG1105E 239
cannot obtain topology data 321 EKG1106E 239
checkpoint processing fails, RODM 239 EKG1111I 232
command EKG1112E 239
cannot initiate IP session using NETCONV 180 EKG1113I 239
NETCONV, cannot initiate IP session 180 EKG1116I 241
debugging method, RODM 229 EKG1117I 241
distributed mainframe server error not recorded 65 EKG1326D 237
documentation problem FLB300W 318
keyword (DOC) 11 FLB403I 321
DSIFRE message 57 FLB404I 327
DSIGET message 59 FLB405W 326
EP/local error not recorded 64 FLB407E 328
hung, topology manager FLB408W 328
RODM checkpoint 319 FLB409W 321
incorrect output FLB420I 321
EKGPRINT data set, RODM 239 FLB421I 327
keyword (INCORROUT) 11 FLB422W 326
incorrect view FLB424E 328
aggregate resource status incorrect 347 FLB425W 328
class of node object is incorrect 335, 355 FLB426W 321
multiple subnetworks in same view 353 FLB443I 327
resource not in view 324, 335 FLB481E 317
resource status incorrect 343 FLB482E 312, 318
resource status unknown 339 FLB485E 312
unexpected resource in view 332, 356 FLB486I 319
view cannot be displayed 335, 349 FLB540I 321
view disappears 335, 349 FLB541W 326
initialization failure, topology manager FLB542E 328
cannot access RODM 312 FLB544W 321, 328
cannot access VTAM CMIP services 311 FLB584I 327
error reading customization table 309 FLB610I 327
error reading file FLBSYSD 309 FLB677E 311
not enough storage 309 FLB684E 317
severe error 310 FLB685W 321, 328
warning error 310 IEC161I 052-084 239
wrong autotask 308 IEC161I 203-204 231, 232
list of problem scenarios IEC161I 227-229 239
E/AS 467 IEC340I 239
Event/Automation Service 467 incorrect output in EKGPRINT data set, RODM 239
MultiSystem Manager 435 keyword (MSG) 13
NetView management console 175
Index 683
topology server problems trace, NetView program (continued)
incorrect timestamps if topology server is on Windows SNA topology manager
platform allocate storage request 144
problem scenario and resolution 193 array, allocate storage request 145
server does not complete initialization on AIX free storage request 145
problem scenario and resolution 193 reallocate storage request 145
server windows disappear on AIX storage request 144
problem scenario and resolution 193 starting 99
trace, GMFHS status monitor, internal trace 135
description 214 stopping 99
event manager task tracing 217 TRACE command 99
IPC task tracing 217 when to use 99
starting 214 trace, RODM
stopping 214 description 282
TRACE command 214 internal 282
using in-storage trace table 216 trace, SNA topology manager
viewing output online 215 category
trace, NetView performance consideration 16
documenting a problem GTF event ID 401
documentation needed 20 GTF format ID 401
trace, NetView program GTF record format
capture first failure data 146 data format 401
CNMTARCA module trace record 135 example 402
external trace header 401
table example 104 multi-record event 401
first failure data capture 146 internal trace buffer
GMFHS TRACE command 214 description 404
internal trace format 404
IP services 116 record header 404
status monitor 135 NetView trace
table example 102 allocate storage request 144
internal trace dataspace 108 array, allocate storage request 145
IP services, internal trace 116 free storage request 145
locating when MODE=INT 100 reallocate storage request 145
MONOPER keyword 105 storage request 144
NetView TRACE command 99 record format
PPI trace facility 4000-0002 (CENT) 406
description 148 4000-0003 (CEXT) 407
GTF output files 152 4001-0008 (LOGS) 408
locating oldest record 152 4002-0007 (MSGS) 409
locating table 151 4003-000E (CMIP) 410
trace anchor block 148 4004-0019 (RTIB) 412
trace record 149 4004-001A (RARY) 413
trace table 148 4005-0015 (RCLS) 414
printing record 4005-0016 (RON) 415
when MODE=EXT 103 4005-0017 (ROBJ) 416
when MODE=INT 100 4005-0018 (RATR) 417
record 4007-001E (UPDT) 418
constant for option byte (DSIPSS) 114 4008-0000 (GET) 419
description 107 4008-0001 (FREE) 420
DSIGET/DSIFRE 113 4009-0006 (FSM) 421
DSIPSS 114 400A-0004 (NEW) 422
DSIWAT/DSIPOS/DISPATCH 113 400A-0005 (DEL) 423
installation exit (UX) 111 400A-001B (CBEG) 424
lost 112 400A-001C (CEND) 425
LUC macro invocation 109 400A-001D (XMOG) 426
LUC receive exit 109 CBEG (400A-001B) 424
message queuing service (MQS) 110 CEND (400A-001C) 425
MODE=EXT 103 CENT (4000-0002) 406
MODE=INT 101 CEXT (4000-0003) 407
module entry (MENT) 112 CMIP (4003-000E) 410
module exit (MXIT) 112 DEL (400A-0005) 423
presentation services (DSIPSS) 114 FREE (4008-0001) 420
TIME record 116 FSM (4009-0006) 421
trace table header 108 GET (4008-0000) 419
VPDTASK trace record 111 LOGS (4001-0008) 408
SAF (security authorization facility) trace record 137 MSGS (4002-0007) 409
Index 685
Web browser (continued)
sign on panel 507
unexpected presentation 507
web pages 506
Web Services Gateway
server data 508
Web Services server 508
window
problem when minimizing
problem scenario and resolution 191
windows
for server disappear on AIX
problem scenario and resolution 193
worksheet, problem
AON 447
E/AS 459
Event/Automation Service 459
GMFHS 169
MultiSystem Manager 431
NetView management console 165
abend problems 166
documentation problems 168
general information 165
incorrect output problems 168
loop problems 167
message problems 166
performance problems 168
problem classification 166
problem description 166
system-related information 165
wait problems 168
NetView program 45
RODM 221
SNA topology manager 301
Tivoli NetView for z/OS Enterprise Management
Agent 513
Web application 501
wrong autotask, topology manager 308
X
XCF services
BNH067I message 160
BNH558E message 160
BNH587I message 159
discovery commands fail 160
master NetView, unexpected switch 160
no data returned 159
PLEXCTL command fails 159
START XCFGROUP problems 160
unable to contact enterprise system 160
unexpected switch of master NetView 160
Y
Yahoo user group, NetView xxii
Z
z/OS
instrumentation problems and resolutions 186
Printed in USA
GC27-2507-00