Beyond The ESM Administrator Guide

Beyond the ESM
Administrators Guide
Nathan Tisdale, Advanced Support Engineer
Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Introduction
Nathan Tisdale
Advanced Support Engineer
3+ years in ArcSight Technical Support

Train new support engineers
Assist in Premier Investigations and technical escalations
Advocate bug prioritization on behalf of customers
Believe in empowering ArcSight Admins
Agenda
A troubleshooting perspective:
Data flow
Oracle vs CORR-Engine
Basic log analysis

Logs, whiner messages, memory
Advanced log analysis

Exceptions, Thread Dumps, Logfu
Live monitoring
Advanced Management Console
Audience
Is this presentation for you?
ArcSight Administrator
Responsible for ensuring continuous event flow through ESM
Enough experience to be curious about Thread Dumps
This presentation is similar to

SN62: Gain Rock Star Status: ArcSight ESM Manager Administrator
Refocused to provide insight on how to identify bottlenecks with current managers
Participation
Q&A
Hallway chats are welcome
Data flow
Simple ESM deployment

ArcSight
SmartAgent
ArcSight Console
ArcSight
SmartAgent
ArcSight Manager
ArcSight Web
ArcSight
SmartAgent
ArcSight Database
6
Events insertion versus events retrieval

Event insertions
Event insertion flow events are being inserted into the database
Start-of-flow
Threads
(Normalization)
SeededJsse
Listener threads
Pre-persistor
Threads
Post-persistor
Threads
(rules engine)
Bytes read from Socket

and converted to Java
SecurityEvent Objects
Database
Event retrievals
Active channel
queries
Report queries
Different resources retrieving event data from the database

7
XCPUDMPC
Threads (Data
Monitors)
Trend queries
Symptoms of performance issues

Event data retrieval
Channels slow to load
Channels dont finish loading
Channels show Loading Event ID
Reports failing or not running
ORA-01555 or user cancelled operation
Reports based on trends are empty
Trends failing or not running
Trends getting disabled
Symptoms of performance issues

Event data insertion
Connectors caching continuously
Connector status shifting between up and
down frequently
Manager logs show one of the following
It appears the database is hung
Rejected threads
Delayed events (maybe not)
Processing stages
Event Verifier
10
Event Category
Adder
Event Asset
Resolver
Annotation
Initializer
Geo Info
Adder
Security
Event Persistor
Rules
engine
Threat Level
Handler
Data
monitors
Event Forwarder
Symptom
Making sense of Agent State
Queues filling caused by
Database performance
Disk I/O
Slow rules engine processing
Slow Data Monitor processing
Symptom
Events Cache
STM eps < P-A eps
11
Basic log analysis
13
Application logs
Manager
Oracle
<ARCSIGHT_HOME>/logs/default/*.log*
SERVER.LOG
SERVER.STD.LOG
SERVER.STATUS.LOG
SERVER.REPORT.LOG
SERVER.SQL.LOG
SERVER.LICENSE.LOG
PARTITIONMANAGER.LOG
PARTITIONARCHIVER.LOG
PARTITIONCOMPRESSER.LOG
PARTITIONSTATSUPDATER.LOG
<ORACLE_HOME>
/admin/arcsight/bdump/ALERT_<LISTENER>.LOG
/network/log/LISTENER.LOG
/network/log/SQLNET.LOG
14
CORR-Engine
/opt/arcsight/logger/current/arcsight/logger/logs/*
/opt/arcsight/logger/data/mysql/*.log*
/opt/arcsight/logger/data/pgsql/serverlog*
Log rotation
Log files are always limited in size
10MB default
Automatic log file rotation
10 files are kept, plus the current file
Can extend logging
<ARCSIGHT_HOME>/config/server.properties
Copy settings from <ARCSIGHT_HOME>/config/server.defaults.properties [DO NOT EDIT THIS FILE]
# The maximum size of the log file before it
# will be rolled over. The size is specified
# in MB (MegaByte).
log.channel.file.property.maxsize=10MB
# The maximum number of backup files to create
# for rolling over.
log.channel.file.property.maxbackupindex=10
15
Key manager logs

SERVER.STD.LOG
SERVER.LOG
Basic application log

Exceptions with detailed traces
Initialization messages
General progess messages
Event batch insert times
Garbage collector information
Critical warnings
Uncaught exceptions
Watchdog messages
Wrapper manages life cycle of manager processes
Log rotation configured via wrapper

<ARCSIGHT_HOME>/config/server.wrapper.conf
Copy settings from server.defaults.wrapper.conf
[do not edit the defaults file]
16
SERVER.STATUS.LOG
Information from Mbeans
Agent throughputs and status
Active Lists statistics
Rule and Data Monitor resource consumption
Also see manage.jsp
Other manager logs

SERVER.SQL.LOG
SERVER.REPORT.LOG
Needs to be enabled
Useful for Oracle DBA
Logs report being run

More info in SERVER.LOG
grep for [logReportInfo]
SERVER.CHANNEL.LOG
Active Channel queries
PARTITION*.LOG
SERVER.PULSE.LOG
Oracle partition management

no present with CORR-Engine
Updated every 10 seconds
SERVER.LICENSE.LOG
License compliance per 24hrs
Approaching or exceeded limit(s)
17
Other data
Manager
Oracle
Thread dump
Heap dump
Operating system
System logs
Performance data
Database sessions
RDA
AWR
lsinvetory
CORR-Engine
Session Waits
Core Dump
18
Data to collect
99% of time, bottleneck found on manager
Thread Dumps
System tables
Generate five Thread Dumps during the slowness
If reproduction to be performed
Logs
Agent logs
Manager logs
Oracle-based Manager
Alert Log
DB Sessions
CORR-E based Manager
Session Waits
mysql.log
If manager is not identified as the bottleneck

Agent stability
Network connectivity
Network latency
Save time and collect when generating TDs
19
Collecting logs
ArcSight Sendlogs
Wizard interface allows user to easily gather:
Manager logs
Agent logs
Web logs
Console logs
Oracle Alert log
Thread Dumps
Session Waits
Output from SQL
Run from manager or console
./arcsight sendlogs
20
BASH your logs

Demo script available
Get Status of Services
/sbin/service arcsight_services status
Generate Thread Dumps
./arcsight managerthreaddump
Generate Session Wait
./arcsight arcdt session-waits sp spool
Generate threaddumps.html
./arcsight threaddumps <path_to_server.std.log>
Collect Database Logs
Oracle
CORR-E
Place all data in tarball or zip file for upload
21
Whiner messages
System alerts via email
Why
Where
Subsystem failures
Database connection problem
Event insertion times high
SSL certificate expiration
Database space shortage
Running out of space
Usually event space
Sometimes system table space
Partition manager failures
Get your DBA!
22
stdout., server.std.log, server.log

Email
Console pop-up
Internal event
Memory utilization
Memory usage
Manager allocates memory in Java heap
Server.std.log
2006/02/22 23:22:51 | Memory Status: 765.6 MB Used, 1,014.0 MB Max
2006/02/22 23:22:52 | [Full GC
2006/02/22 23:22:58 | 797362K->471587K(1038336K), 5.9847261 secs]
Java heap is garbage collected
Server only allocates memory
Java VM reclaims unused memory automatically
Manager doesnt know how much garbage is in the heap
Reported memory usage includes garbage
24
Memory
Two Types of Garbage Collection (GC)
Java heap is divided into generations
Young
Tenured
Minor GC
Only collects young generation
May expand to entire heap, and become a major collection
[GC 929899K->838966K(1036928K), 0.0353791 secs]
Major GC or Full GC
Collects both young generation and tenured generation
[Full GC 932135K->542955K(1036928K), 3.9721866 secs]
25
GC pause
Stop the world GC
When GC is happening, everything else is stopped
Pause Time
[Full GC 932135K->542955K(1036928K), 3.9721866 secs]
Minor GC pause ([GC ])

Should be under 1 sec
Major GC pause ([Full GC .])

Actual time depends on hardware
Estimate: ~1 sec every 200 MB heap
26
Real memory usage the working set

Real memory usage is captured in Full GC messages
server.std.log
2006/02/22 23:22:51 | Memory Status: 765.6 MB Used, 1,014.0 MB Max
2006/02/22 23:22:52 | [Full GC
2006/02/22 23:22:58 | 797362K->471587K(1038336K), 5.9847261 secs]
Working set is defined as the memory that is in actual use and doesnt have any garbage.
Working set of the Manager can be found as above, immediately after a Full GC
27
How to choose heap size?

Recommendation for heap size is 2 x working set
Too small
Frequent full GC
Bad performance
Manager could die on OutOfMemoryError
Too large
Peak performance is good, but
Full GC takes long time to finish
Manager could get killed by Wrapper for being hung for a long time
Adjust heap size through Management Console, or by running managersetup
28
Out of memory
Server will restart on out of memory errors
Check Logfu
Check CapsManager from server.status.log to check overall memory utilization by Data
Monitors, channels, Active Lists etc.
If you see a spike
Multiple memory intensive tasks at the same time?
Increase heap size
Memory leak
Memory usage keeps growing
Increasing heap size only delays the problem
Memory leak is hard to track down
Contact support
29
Log analysis
Exceptions
Details of application errors
Java construct encapsulates some failures
Coding errors
Transient bugs
A full stack trace is included
Shows where in the code the error occurred
Not all exceptions are equal
Misclassified or not significant impact
Sometimes related to content
31
Where is the cache?

SERVER.STATUS.LOG
Agent Statuses
AgentStatuses="[|||Name|ID|Reported|Agent Time|Received by Agent Count|Received by Agent EPS|Post-Filter
Count|Post-Filter EPS|Post-Aggregation Count|Post-Aggregation EPS|Estimated Cache Size|Sent To Manager
Count|Sent To Manager EPS|Failed Connection Attempts,
archiver|3lSvf5BEBABCBfCSYubv3rw==|05/08 11:23:54|05/08 11:23:54|0|0.0|0|0.0|0|0.0|0|0|0.0|0,
Syslog|3t5MEiRcBABDmQv3t56sCJQ==|05/08 11:23:31|05/08
11:23:31|4000|62.5|3643|56.921875|3643|56.921875|120,000|3650|57.03125|807761
Total|-|-|-|35,434|582.8|33,328|549.8|33,086|546.0|120,100|88,898|1,476.2|1,335,569]
32
Delayed events
SERVER.LOG
default.com.arcsight.util.TimedRingBuffer][increment] Throwing out increment X, increment time = X,
acceptable range X - X (discarded=X)
Active channel
Gaps between Manager Receipt Time and End Time and Agent Receipt Time
Device Receipt Time
Connector Receipt Time (a.k.a. Agent Receipt Time)
Manager Receipt Time
Start Time
End Time
33
Database connectivity
SERVER.STD.LOG
Connectivity Issues
SERVER.STD.LOG
SUBSYSTEM STATUS CHANGED
Persistence Rate
should take less than 100ms
INFO | jvm 2 | 2009/05/07 20:41:58 | (02-Pre-SecurityEventPersistor330) Persisted 100 events
in 32 ms.
INFO | jvm 1 | 2009/05/08 11:20:53 | (02-Pre-SecurityEventPersistor1) Persisted 100 events
in 3,698 ms
34
Manager busy
SERVER.STD.LOG
Manager stops accepting events
INFO | jvm 1 | 2005/04/04 00:42:26 | WARNING: '1' agent requests REJECTED because the limit of '64'
agent threads was exceeded.
35
Thread Dumps
Insertion issues require Thread Dumps

Event insertions
Event insertion flow events are being inserted into the database
Start-of-flow
Threads
(Normalization)
SeededJsse
Listener threads
Pre-persistor
Threads
Post-persistor
Threads
(rules engine)
Bytes read from Socket

and converted to Java
SecurityEvent Objects
Database
Event retrievals
Active channel
queries
Report queries
Different resources retrieving event data from the database

37
XCPUDMPC
Threads (Data
Monitors)
Trend queries
A Java snaphsot
Dont restart or reboot before collecting!
Why Thread Dumps
Stack trace for each thread in the VM
Many different threads
Bottleneck area usually identifiable
Session Waits or DB Sessions needed to correlate database activity
Generating Thread Dumps

Manage.jsp | NGServer | generateThreadDumps Invoke
<ARCSIGHT_HOME>/bin/arcsight managerthreaddump
Formatting Thread Dumps

<ARCSIGHT_HOME>/bin/arcsight threaddumps > threaddumps.html
38
Servlet engine
SeededJsseListener
Read bytes from network sockets
Convert read bytes to Java Objects Security Event Batch
Place event batches into queue for Flow 1
39
Flow 1: Start
Start-of-flow
Vulnerability Scanner Reports
Place event batches into queue for Flow 2
40
Flow 2: Pre-persistor
Pre-SecurityEventPersistor
41
Remove event from batch from queue

Initialize and normalize event fields
Write to database
Put event batch in to queue for Flow 3
Flow 3: Post-persistor
Post-SecurityEventPersistor
42

Evaluate events against rules
Generate Correlation events
Put event batch in to queue for Dashboards
Content: Dashboards
XCPUDMPC-Thread
43

Evaluate events against Data Monitors
Generate Correlation events
Put event batch in to queue for garbage collection
Logfu
Why Logfu?
Logfu is not an officially supported tool
Discerning patterns
Examines server.log, server.std.log, and server.status.log
Syntax
ArcSight logfu m noplot
-m is for manager
-noplot skipps plotting on graph
Outputs logfu.html to logs/default/Logfu_<date>/
Interesting data points
Famous Last Words Why did it die
Exception Groups Quickly identify repeating exceptions
Memory Identify growth in memory consumption
Event Insertion Is the database/disk able to keep up
45
Memory patterns
46
Event throughput patterns
47
Shutdown patterns
48
Event insertion patterns
49
Use with Connectors too

Plot time per batch to identify network lag
50
Advanced Management Interface

a.k.a. manage.jsp
Status on demand
https://<HOST_NAME>:8443/arcsight/web/manage.jsp
Interesting Mbeans
Agent State Tracker
Specific and overall EPS for connectors
SessionManager
How many users are logged in
SubsystemStatus Tracker
Whiner
ActiveList Monitoring
Memory consumption
Channels
How many
Validating the SQL
52
Groups and filters
53
Mbean: RulesEngine
54
Mbean: AgentStateTracker
55
Thank you
Security for the new reality


Beyond The ESM Administrator Guide

Uploaded by

Copyright:

Available Formats

Beyond The ESM Administrator Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Beyond The ESM Administrator Guide

Uploaded by

Copyright:

Available Formats

Beyond the ESM

3+ years in ArcSight Technical Support

Basic log analysis

Advanced log analysis

This presentation is similar to

Simple ESM deployment

Events insertion versus events retrieval

Bytes read from Socket

Different resources retrieving event data from the database

Symptoms of performance issues

Symptoms of performance issues

Basic log analysis

Key manager logs

Basic application log

Log rotation configured via wrapper

Other manager logs

Logs report being run

Oracle partition management

Updated every 10 seconds

Generate five Thread Dumps during the slowness

If manager is not identified as the bottleneck

BASH your logs

stdout., server.std.log, server.log

Two Types of Garbage Collection (GC)

Java heap is divided into generations

Minor GC pause ([GC ])

Major GC pause ([Full GC .])

Real memory usage the working set

How to choose heap size?

Adjust heap size through Management Console, or by running managersetup

Where is the cache?

Insertion issues require Thread Dumps

Bytes read from Socket

Different resources retrieving event data from the database

Generating Thread Dumps

Formatting Thread Dumps

Remove event from batch from queue

Remove event from batch from queue

Remove event from batch from queue

Event throughput patterns

Event insertion patterns

Use with Connectors too

Advanced Management Interface

Groups and filters

Security for the new reality

You might also like