Elasticsearch and The Elk Stack For Monitoring and Data Analysis

ElasticSearch and the ELK stack for monitoring and
data analysis
Clemens Düpmeier (KIT / IAI)
Institute of Applied Computer Science (IAI)
KIT – University of Baden-Württemberg

and National Research Center of the Helmholtz Alliance www.kit.edu
Overview
Introduction to the ELK stack
Use Cases
Summary
2 09/09/2015 Research group “ Web based Information Systems“ Institute for Applied Computer Science (IAI)
Elasticsearch ELK Software Stack
ELK consists of three open source software

products provided by the company “Elastic”
(formerly Elasticsearch)
E => Elasticsearch
(Highly scalable search index server)
L => Logstash
(Tool for the collection, enrichment, filtering and
forwarding of data, e.g. log data)
K => Kibana
(Tool for the exploration and visualization of data)
Logstash
Log Kibana
Logstash
Open source software to collect, transform, filter and forward

data (e.g. log data) from input sources to output sources (e.g.
Elasticsearch)
Implemented in JRuby and runs on a JVM (Java Virtual Machine)
Simple message based architecture
Extendable by plugins (e.g. input, output, filter plugins)
Configuration
Multiple inputs of
different types
Conditionally
filter and
transform data;
some common
formats are
already known
Forward to
multiple outputs
5 09/09/2015 Institute for Applied Computer Science (IAI)
Console output processing apache log files
Run logstash with: bin/logstash -f logstash.conf
Configuration for parsing syslog messages
Input filter receives messages directly

from tcp and udp ports
Filter splits messages and adds fields

Console output processing syslog messages
Run logstash with: bin/logstash -f logstash.conf
Input Plugins
file -> for processing files

tcp, udp, unix -> reading directly from network sockets
http -> for processing HTTP POST requests
http_poller -> for polling HTTP services as input sources
imap -> accessing and processing imap mail
Different input plugins to access MOM (message queues)
Rabbitmq, stomp, …
Different plugins for accessing database systems
jdbc, elasticsearch, …
Plugins to read data from system log services and from
command line
syslog, eventlog, pipe, exec
And more
Lumberjack plugin + Logstash forwarder
The “Logstash forwarder” application allows to forward

input from one “data source” host to another host for
processing
The “Lumberjack input plugin” can then be configured to
consume the messages of the “Logstash forwarder”
Transfer can be secured by “security certificate” and
encrypted transmission
Output plugins
stdout, pipe, exec -> show output on console, feed to command

file -> store output in file
email -> send output as email
tcp, udp, websocket -> send output over network connections
http -> send output as HTTP request
Different plugins for sending output to database systems, index
server or cloud storage
elasticsearch, solr_http, mongodb, google_bigquery, google_cloud_storage,
opentsdb
Different output plugins to send output to MOM (message queues)
Rabbitmq, stomp, …
Different output plugins for forwarding messages to metrics
applications
graphite, graphtastic, ganglic, metriccatcher
Multiple node writes
The Elasticsearch output plugin can write to multiple nodes

It will distribute output objects to different nodes (“load
balancing”)
A Logstash instance can also be part of a Elasticsearch
cluster and write data through the cluster protocol
Filter plugins
grok -> parse and structure arbitrary text: best generic option to
interpret text as (semi-)structured objects
Filter for parsing different data formats

csv, json, kv (key-valued paired messages), xml, …
multiline -> collapse multiline messages to one logstash event
split -> split multiline messages into several logstash events
aggregate -> aggregate several separate message lines into one
Logstash event
mutate -> perform mutations of fields (rename, remove, replace,
modify)
dns -> lookup DNS entry for IP address
geoip -> find geolocation of IP address
And more
grok usage example
Input: 55.3.244.1 GET /index.html 15824 0.043

grok filter
filter {
grok { match => { "message" => "%{IP:client} %
{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes}
%{NUMBER:duration}" }
}
Then the output will contain fields like:

client: 55.3.244.1
method: GET
request: /index.html
bytes: 15824
duration: 0.043
Scaling and high availability

Elasticsearch
Server environment for storing large scale structured index

entries and query them
Written in Java
Based on Apache Lucene
Uses Lucene for index creation and management
Document-oriented (structured) index entries which can (but must
not) be associated with a schema
Combines “full text”-oriented search options for text fields with more
precise search options for other types of fields, like date + time
fields, geolocation fields, etc.
Near real-time search and analysis capabilities
Provides Restful API as JSON over HTTP
Scalability of Elasticsearch
Elasticsearch can run as one integrated application on

multiple nodes of a cluster
Indexes are stored in Lucene instances called “Shards”
which can be distributed over several nodes
There a two types of “Shards”
Primary Shards
Replica
Replicas of “Primary Shards” provide
Failure tolerance and therefore protect data
Make queries (search faster) faster
Indexing data with Elasticsearch
Send JSON documents to server, e.g. use REST API

No schema necessary => ElasticSearch determines type of
attributes
But’s possible to explicitly specify schema, i.e. types for attributes
Like string, byte, short, integer, long, float, double, boolean, date
Analysis of text attributes for “full text”-oriented search
Word extraction, reduction of words to their base form (stemming)
Stop words
Support for multiple languages
Automatically generate identifier for data sets or specify
them while indexing
Indexing data using the REST API
PUT request inserts the JSON payload into the index with name “megacorp” as
object of type “employee”
Schema for type can be explicitly defined (at time of index creation or
automatically determined)
Text field (e.g. “about”) will be analyzed if analyzers are configured for that field
Request URL specifies the identifier “1” for the index entry
Retrieval of a index entry
GET /megacorp/employee/1
A “GET” REST API call with “/megacorp/employee/1” will

retrieve the entry with id 1 as JSON object
Simple Query
GET /megacorp/employee/_search
GET request with

“_search” at the end of
the URL performs query
Search results are
returned in JSON
response as “hits” array
Further metadata
specifies count of
search results (“total”)
and max_score

Simple Query with search string
GET /megacorp/employee/_search?q=last_name:Smith

More complex queries with Query DSL
Query DSL is a JSON language for more complex queries

Will be send as payload with the search request
Match clause has same semantics as in simple query
More complex queries with Query DSL
Consist of a query
and a filter part
Query part
matches all entries
with last_name
“smith” (2)
Filter will then only
select entries
which fulfill the
range filter (1)
“age”: {“gt” : 30 }
Some query possibilities
Combined search on different attributes and different

indices
Many possibilities for full-text search on attribute values
Exact, non-exact, proximity (phrases), partial match
Support well-known logical operators
(And / or, …)
Range queries (i.e. date ranges)
…
Control relevance and ranking of search results, sort them
Boost relevance while indexing
Boost or ignore relevance while querying
Different possibilities to sort search results otherwise
More advanced features
Multi-tenant
Spatial data queries
Search suggestions
Real time aggregation of search data
Statistical calculations (sums, mean value, max, min, …)
Faceting
By using terms
Statistical calculations
Classification ( Grouping by using ranges
Filter rules
By geographical distance
…
Kibana
Web-based application for exploring and visualizing data
Modern Browser-based interface (HTML5 + JavaScript)
Ships with its own web server for easy setup
Seamless integration with Elasticsearch
Configure Kibana
After installation first configure Kibana to access
Elasticsearch server(s)
Should be done by editing the Kibana config file
Then use web UI to configure indexes to use
Discover data
Create a visualization
Different types of visualizations
Combine visualizations to a Dashboard
USE CASES
Some use cases of the ELK stack
Log data management and analysis

Monitor systems and / or applications and notify operators
about critical events
Collect and analyze other (mass) data

i.e. business data for business analytics
Energy management data or event data from smart grids
Environmental data
Use the ELK stack for search driven access to mass data
in web based information systems
Many different types of logs

Application logs
Operating system logs
Network traffic logs from routers, etc.
Different goals for analysis

Detect errors at runtime or while testing applications
Find and analyze security threats
Aggregate statistical data / metrics
Problems of log data analysis
No centralization
Log data could be everywhere
on different servers and different places within the same server
Accessibility Problems
Logs can be difficult to find
Access to server / device is often difficult for analyst
High expertise for accessing logs on different platforms necessary
Logs can be big and therefore difficult to copy
SSH access and grep on logs doesn’t scale or reach
No Consistency
Structure of log entries is different for each app, system, or device
Specific knowledge is necessary for interpreting different log types
Variation in formats makes it challenging to search
Many different types of time formats
The ELK stack provides solutions
Logstash allows to collect all log entries at a central place (e.g. Elasticsearch)
End users don’t need to know where the log files are located
Big log files will be transferred continuously in smaller chunks
Log file entries can be transformed into harmonized event objects

Easy access for end users via Browser based interfaces (e.g. Kibana)
Elasticsearch / Kibana provide advanced functionality for analyzing and visualizing the log data
Monitoring
The ELK stack also provides good solutions for monitoring

data and alerting users
Logstash can check conditions on log file entries and even aggregated
metrics
And conditionally sent notification events to certain output plugins if
monitoring criteria are met
E.g. forward notification event to email output plugin for notifying user (e.g.
operators) about the condition
Forwarding notification event to a dedicated monitoring application
Elasticsearch in combination with Watcher (another product of Elastic)
Can instrument arbitrary Elasticsearch queries to produce alerts and
notifications
These queries can be run at certain time intervals
When the watch condition happens, actions can be taken (sent an email or
forwarding an event to another system)
Log analysis examples from the Internet
Logging and analyzing network traffic

http://www.networkassassin.com/elk-stack-for-network-ope
rations-reloaded/
How to Use ELK to Monitor Performance

http://logz.io/blog/elk-monitor-platform-performance/
How Blueliv Uses the Elastic Stack to Combat Cyber
Threats
https://www.elastic.co/blog/how-blueliv-uses-the-elastic-sta
ck-to-combat-cyber-threats
Centralized System and Docker Logging with ELK Stack

http://www.javacodegeeks.com/2015/05/centralized-system
39
-and-docker-logging-with-elk-stack.html
09/09/2015 Research group “ Web based Information Systems“ Institute for Applied Computer Science (IAI)
Summary
The ELK stack is easy to use and has many use cases
Monitor systems and / or applications and notify operators about
critical events
Collect and analyze other (mass) data
Providing access to big data in large scale web applications
Thereby solving many problems with these types of use

cases compared to “hand-made”-solutions
Because of its service orientation and cluster readiness it

fits nicely into bigger service (microservice) oriented
applications
BACKUP SLIDES
Search based web applications
Use search engine technology
as key element for data access
(e.g. ElasticSearch)
Available data is a mixture

of
unstructured
semi-structured
structured
information coming from

different sources
Use Logstash and similar

technologies for
aggregation,
normalization and
classification of data
And a natural language Search based environmental

approach for data access based on
ElasticSearch Queries
information portal
The openTA portal: www.openta.net
DfG project with partners

(IAI, ITAS, KIT library)
from KIT
Web portal for the Network

Technology Assessment
(NTA)
Aggregates and provides
information about
members, organizations
news and events
scientific publications
Provides services and web
technology (widgets) to use Homepage of openTA with
aggregated news feed and calendar
the aggregated information remotely
Search driven architecture (example openTA)
User
(User Management, CMS, Social)

Data Services
User
Web UI
Portal
Data and Service Access

Standard Functionalities
OpenTA Calendar
NTA
Organization News Service
Publication
NTA Service
Portal Service
Organization
APIs
NTA Search Engine

Organization (Aggregation,
Analysis)
NTA
Organization RSS / Atom
Structured indices
ICalendar
Most information will be ingested
Pub.-Form.
using service interfaces and crawling.
Data Import / Crawler
Example: www.openta.net
Prototype: Search with „Energie –Rohracher“

Elasticsearch and The Elk Stack For Monitoring and Data Analysis

Uploaded by

Copyright:

Available Formats

Elasticsearch and The Elk Stack For Monitoring and Data Analysis

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Elasticsearch and The Elk Stack For Monitoring and Data Analysis

Uploaded by

Copyright:

Available Formats

ElasticSearch and the ELK stack for monitoring and

Institute of Applied Computer Science (IAI)

KIT – University of Baden-Württemberg

Introduction to the ELK stack

ELK consists of three open source software

Open source software to collect, transform, filter and forward

Input filter receives messages directly

Filter splits messages and adds fields

7 09/09/2015 Institute for Applied Computer Science (IAI)

file -> for processing files

The “Logstash forwarder” application allows to forward

stdout, pipe, exec -> show output on console, feed to command

The Elasticsearch output plugin can write to multiple nodes

Filter for parsing different data formats

Input: 55.3.244.1 GET /index.html 15824 0.043

Then the output will contain fields like:

15 09/09/2015 Institute for Applied Computer Science (IAI)

Server environment for storing large scale structured index

Elasticsearch can run as one integrated application on

Send JSON documents to server, e.g. use REST API

A “GET” REST API call with “/megacorp/employee/1” will

GET request with

21 09/09/2015 Institute for Applied Computer Science (IAI)

22 09/09/2015 Institute for Applied Computer Science (IAI)

Query DSL is a JSON language for more complex queries

Combined search on different attributes and different

Log data management and analysis

Collect and analyze other (mass) data

Many different types of logs

Different goals for analysis

Log file entries can be transformed into harmonized event objects

The ELK stack also provides good solutions for monitoring

Logging and analyzing network traffic

How to Use ELK to Monitor Performance

Centralized System and Docker Logging with ELK Stack

Thereby solving many problems with these types of use

Because of its service orientation and cluster readiness it

Available data is a mixture

information coming from

Use Logstash and similar

And a natural language Search based environmental

DfG project with partners

Web portal for the Network

(User Management, CMS, Social)

Data and Service Access

NTA Search Engine

You might also like