Kibana Course Overview

In this course, you will be reading about all the ingredients required for a quick hands-on on
Kibana. By the end of the course, you will be aware of the following things:

 Setting up Kibana
 Getting Started
 Discover
 Visualize
 Dashboard
 Timelion
 Dev Tools
 Management

Introducing Kibana
The audience reading this course must have had an experience with Elasticsearch. In
Elasticsearch, we were restricted to queries and results in JSON. There was no option of
visualizing the data in the form of graphs and charts.

Kibana is a member of the ELK stack family, which is designed to visualize as well as
explore the relations between the data.

Advantages of Using Kibana

We would like to highlight some of the advantages and new possibilities concealed within
Kibana:

 Capable of creating beautiful histograms, line graphs, pie charts, sunbursts within

no time from the data.
 You can visualize geospatial data.
 You can perform advanced time series analysis on your Elasticsearch data!
 Unconceal the hidden relationships in data.
 Detect the anomalies hidden as well as the hidden properties influencing them.
 Send the URL of your visualization easily.
 A good alternative to the command prompt for operating Elasticsearch.
 No more curl commands for fetching data.

Installing Kibana - Ubuntu

Here are the steps for installing Kibana in your Ubuntu system:
Step 1:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Step 2:

sudo apt-get install apt-transport-https

Step 3:

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a

/etc/apt/sources.list.d/elastic-5.x.list

Step 4:

sudo apt-get update && sudo apt-get install kibana

Step 5:

sudo -i service kibana start

Installing Kibana - Windows

Here are the steps for installing Kibana on Windows:

Step 1:

Here is a link to .zip Windows archive for Kibana v5.5.2 :

https://artifacts.elastic.co/downloads/kibana/kibana-5.5.2-windows-x86.zip

Step 2: Unzip the downloaded file which will result in the creation of a folder called kibana-
5.5.2-windows-x86. Access it in the terminal, for example:

CD c:\kibana-5.5.2-windows-x86

Step 3:

Run Kibana on the command line:

.\bin\kibana

Configuring Kibana
You can find the configuration for Kibana in the file kibana.yml. The default location to run
Kibana is:
localhost:5601

You can open Kibana in your web browser with the following link:

http://localhost:5601/app/kibana

You will also have to link Kibana to Elasticsearch installed on your machine to access
documents. Search for the setting elasticsearch.url in the kibana.yml file and make sure
that it is set to http://localhost:9200.

Connecting with Elasticsearch

When you open Kibana initially in the browser, you will be prompted to define an index
pattern. Hence, specify an appropriate index pattern that matches the name of one or more
of Indices.

Did you remember the bank index that we used in the Elasticsearch course? You can set the
index pattern name as "bank" for now to experiment with the data. We will be using it as our
data-set in this course.

Finally, click on "Create" to add the index pattern that you specified. You can add/change the
index pattern anytime under Management > Index Patterns.

Getting Started with Kibana

In this section, you will read the following topics that will make you ready for quick hands-on
experience:

 Defining the Index Pattern

 Loading the sample data set
 Exploring tools/section of Kibana User Interface.

Index Pattern
As we mentioned, to navigate through the data, the first thing you need to do is defining the
index pattern.

In addition to defining the complete name of an index in the pattern, you can also take help of
the wildcards that will search for all the index following the pattern. For example,

Setting up the pattern as ba* will search for all the similar pattern index such as
banking, ball, and balancesheet.
Loading Sample Data Set
In case you have not loaded the bank data-set in Elasticsearch, here is the link for the same:

https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip

Once you import the data in Elasticsearch (please refer to the "Elasticsearch - search raptors"
course), you can set the index pattern as "bank".

Quick Fact

Users willing to try Kibana on the online playground may

try the one present on elastic's official website. This demo
comes with sample data. Hence, upcoming hands-on in
this course will be based on it.
Exploring Kibana User Interface
In the Kibana user interface, you can find the following things on the left side as a list:

 Discover
 Visualize
 Dashboard
 Timelion
 Management
 Dev Tools

Unleashing the Data

As soon as you click discover, you can view the complete documents within the index
appearing as a list. You can view the list of fields on the left side. Since we have a bank as
our dataset, the fields appearing should be account_number, age, _id, etc.

On further clicking the fields, you can view the details of top 5 values based on the
document in the table.

The Search Bar

On the top of the UI screen, you can find the search bar where you can write queries that will
return relevant results in the same way it returned in Elasticsearch.

A sample query you can run is:

account_number:<500

It will return all the documents having field name "account_number" less than
500. Above is the screenshot for the same.

You can specify basic simple strings as well as you can use Apache query syntax for
searching a query.

New / Save / Open / Share

You can find these options with respective functions in the top right corner:

 New - Start a new search.

 Save - Save performed search.
 Open - Open a saved search.
 Share - Share the search with URL / Snapshot.

In upcoming topics, you will learn about several pages like discover, and visualize in detail.

Discover
You can access discover page in Kibana by clicking "Discover" just below the Kibana logoat
the top left corner. Here are the important points regarding discover page in Kibana:

 It is used mainly to explore the data.

 Every document within all the index that matches the index pattern can be
accessed here.
 Search queries can be specified here. You can filter the search results to view
document data.
 The number of document results matching the query is also shown.

If your Index does not contain a time field, you cannot set the time filter field name. This filter
is used to filter events with the global time filter. We will discuss this functionality at a later
stage.

Searching Data
Upon submitting a search request in the search bar, the histogram, field list, and document
tables are automatically updated to show search results, and the total number of hits is shown
in the toolbar.

The listing is done in reverse chronological order, i.e., newest document is shown first.

In the search bar, you can search as simply as:

Edenburg

It will return the result:

city:Edenburg account_number:397 balance:37,418 firstname:Leonard lastname:Gray age:36

gender:F address:840 Morgan Avenue employer:Recritube email:leonardgray@recritube.com
state:AL _id:397 _type:account _index:bank _score:4.615

Regarding the usage of Lucene query syntax, please refer this link.

Using Lucene Query Syntax

Here is a guide for searching data with Lucene query syntax:

Searching for a value in a specific field can be done by:

balance:49222

A range of value can be searched like:

account_number:[1 TO 499]

Boolean operators like AND, OR, and NOT can also be used in the search:

account_number:[1 TO 100] AND (city:Yardville OR city:Shaft)

Saving and Opening Saved Search

Save option in the Kibana toolbar enables you to reload the search into the "Discover" page
so that it can be used for further visualization.

Saving a search will save the search query with the index pattern used at the time of
search.

To save a search:

Kibana toolbar > Save > Enter a name > Click Save .

You can find saved searches in:

Management > Kibana > Saved Objects

To open a saved search:

Kibana toolbar > Open > Select the search

Filtering Search Results

As you can guess, filtering the search results means sorting only those documents that
contain a particular value.

Filter may be:

 Positive- Include documents containing a particular value.

 Negative - Exclude documents containing a particular value.

You can add filters in Kibana as follows:

 Click on field name that you want to filter. It will display the top 5 values of the field by
default.

A sample of filtering is given in the image above.

Adding Positive and Negative Filter

To add a positive filter:

 Select the field name to be filtered from the list.

 Now click the positive filter button aside to the value that you want to be filtered.
 Similarly, you may click the negative filter button aside to the value that you want to be
blacklisted.

You can click add button adjacent to the field name to add values other than top 5 values.

Manually Adding a Filter

To manually add a filter, follow these steps:

 Click upon "Add Filter" present on the left side below the search bar.
 In the popup, you can set a filter as given in the example below:

age > is > 27 > save

You can also click "Edit Query DSL" if you want to customize the query further.

Visualize
Visualize page is the most important page of Kibana as it helps in creating the different types
of visualization corresponding to the data present in Elasticsearch.

 Visualize page is responsible for creating the visualization of data present in the

indices.
 These visualizations are based on the queries.
 In this topic, you will learn:

a. Bucket and metric aggregations.

b. Creating a visualization.

c. Creating several charts.

Above is a screenshot of visualize page in Kibana.

Bucket Aggregations
As you know, the aggregations are a collection of data which is stored in buckets. In bucket
aggregation, buckets are created to store various documents. Different kinds of bucket
aggregations are:

 Date Histogram - Performed on date / time value is automatically extracted from

documents. It puts in all the document that matches the criteria of the bucket whose
value is within the interval.
 Histogram - Done in the numeric field. Interval with numeric values can be defined.
For example, you can create a dynamic bucket with an interval of 10 for the set of
documents that contain a numeric field between 1-100.
 Range - Performed to specify a range size with each range size representing a bucket.
Aggregation on numeric as well as date/time fields can be performed.
 Date Range - Used to specify range size in date format. Similar to other, each range
size will represent a bucket. For example, from now-1M/M to now.
 IPV4 Range - Used to specify a range in IP format. For example, 192.168.1.1 to
192.168.1.100.
 Terms - Used to create buckets that are based on the values of the field. It is very
similar to GROUP BY of SQL.
 Filters - It is used to specify a set of filters for data.
  Significant Terms - It shows the result of experimental significant
termsaggregation.
 Geohash - Used to create buckets which are based on the geo_point fields.

Metric Aggregations
As the name suggests, metric aggregation is used for computing metrics on documents. It is
used after creating a bucket aggregation. Then, the metric aggregation can be used to
calculate the value of each bucket. Hence, it provides single value per bucket.

Types of metric aggregations are:

 Count - This aggregation is used to return total documents present in each bucket as a
value.
 Sum - It will return the sum of numeric fields present in each bucket.
 Average - It will return in average value of numeric fields present in each bucket.
 Min - It will return minimum value from numeric fields present in each bucket.
 Max - It will return maximum value from numeric fields present in each bucket.
 Unique count - This aggregation is used to count the number of unique values
corresponding to a field stored in the bucket.
 Percentile - This aggregation is used to calculate percentile upon numeric values
corresponding to a field stored in the bucket.
 Percentile ranks - Used to calculate single or multiple percentile ranks.

Creating a Visualization
To create a visualization:

Click Visualize in side navigation  - Create new visualization button - Select visualization
type
The types of visualization we will discuss are:

 Basic Charts
 Data
 Maps

Create Vertical Bar Chart

Let us create our first visualization of the most common scenario that you must have did
during school - "Visualizing age group of interval 5 vs total count" :

 Go to Visualize > Create a visualization.
  Select Vertical Bar under Basic charts.
 Select the index bank.
 Under metrics > Y-axis, select count. You can also give a customized name under Custom
Label.
 Go to buckets > Select buckets type > X-axis and select Range in Aggregation.
 Select age under Field.
 Enter range under From-To text fields i.e. 0-5, 5-10, 10-15, 15-20....till 70-75.
 Finally, click Apply changes (Play button). You will get the chart instantly as shown in the
above picture.

Please save the visualizations (by selecting save on top right corner and giving a
name to it) as we will require it during Dashboard creation.

Create a Pie Chart

We will create a pie chart for the same case i.e. count of unique persons within each age
range i.e. 5-10, 10-15, ... :

 Go to Visualize > Create a visualization.
 Select Pie under Basic charts.
 Select the index bank.
 Under metrics > Slice Size, select count. You can also give a customized name under
Custom Label.
 Go to buckets > Select buckets type > Split Slices and select Range in Aggregation.
 Select age under Field.
 Enter range under From-To text fields i.e. 0-5, 5-10, 10-15, 15-20....till 70-75.
 Finally, click on Apply changes (Play button). You will get the pie chart instantly as shown in
the above picture.

Create an Area Chart

Let us think of a situation where you want to plot a chart that displays the age count of the
people from most popular state!

Here are the steps for the same:

 Navigate to Visualize > Create a visualization > Area > Select the Index Bank.
 Select count aggregation in Y-axis metrics.
  Select buckets > select buckets type > Split Chart.
 Select Terms in Aggregation and select state.keyword in field.
 Select order as descending and size as 1 so that only the topmost populous
state is selected.
 Add another sub-bucket and select X-axis. Select terms in sub aggregation.
 Select age in field and order by term.
 Select order as ascending and size as 50.
 You can select line mode as "smoothed" in Metrics and Axes > line mode.

Above picture is the resultant graph.

Create a Line Chart

Here we will discuss a famous and common situation where you need a plot that
shows  each account number with corresponding balance .

Here are the steps:

 Navigate to Visualize > Create a visualization > Line > Select the Index Bank.
 Under Metrics > Y-Axis select Sum as the type of aggregation and select
balance in the field.
 Select Bucket type as X-Axis and select Terms in Aggregation .
 Select Account number in the field and Order by Term .
 Select Ascending in the order and size as 50 (increasing the size will make the plot
complex).
 Finally, click the Apply changes button (play button).

Above is the screenshot of the plot.

Create a Heat Map

Here is an example to create a heatmap in Kibana:

 Select Visualize > Create a Visualization > Heatmap > Select Index Bank.

 Select buckets > Select Bucket type > X-Axis.
 Select terms in Aggregation, state.keyword in field, metric: Count in Order
By, Descending in Order and 10 in size.
 Now, in Add Sub-buckets, select Y-Axis.
 Select Sub Aggregation as range and in field, select age. In from-to fields,
specify age ranges. In our case, we took 20-25, 25-30, 30-35,....,45-50.
  Click on apply changes (play) button. The heatmap chart specifying top 10 states on
X-axis and age range on Y-axis with each cell representing the count of people will be
plotted.
 In Options, you can select color schema as "Yellow to Red".

Above is the resultant heatmap chart.

Create a Gauge
Gauge indicates the status of metric about a threshold value. Suppose you would like to
view the average age of all the bank account holders. Here are the steps for the same:

 Select visualize > Create new visualization > Data > Gauge > Select Index (bank
in our case).
 In metric > Aggregation > select average > select age in field.
 Click on apply changes button. You can customize the theme in Options.

Above is the screenshot of the resultant plot.

Create a Coordinate Map (1/3)

We will discuss an interesting scenario here. We can plot latitude and longitude on a map
with the help of "geo_point" field. Suppose you got data of coordinates (latitude, longitude)of
recent earthquakes all over the world that happened in past 24 hours. You want to point these
coordinates on the world map. The data (from https://earthquake.usgs.gov) is as follow:

7km S of Breznik, Bulgaria - 42.670°N 22.899°E - M 3.2

100km NNE of Kemeduran, Indonesia - 6.099°S 112.950°E - M 4.6
2km SSW of Wako, Japan - 35.772°N 139.615°E - M 4.6
25km SW of Boca de Yuma, Dominican Republic - 18.222°N 68.791°W - M 4.3

and few more...

We will insert these data in Elasticsearch with the help of console in Kibana Dev tools (You
can also use regular Elasticsearch for this). Also, we will have to map the location field to
"geo_point". Here is the procedure for same (next card).

Create a Coordinate Map (2/3)

Create the index with mapping:

PUT earthquakes
{
"mappings": {
"coordinates": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}

Now put data in the index as:

PUT earthquakes/coordinates/1
{
"place": "7km S of Breznik, Bulgaria",
"magnitude": "3.2",
"location": "42.670,22.899"
}

Put rest of the data similarly. In the case of 68.791°W, use -68.791.

Now we will create a new index pattern named earthquakes in Kibana.

Create a Coordinate Map (3/3)

Finally, go to visualize > create a new visualization > coordinate map > select index
earthquakes.

Select bucket type as Geo Coordinates, let Geohash be in aggregation and select location in
the field. Finally, click on apply changes. You can customize the view in options.

Above is the resultant plot.

Dashboard
In this section, you will learn about Dashboard page of Kibana with the visualizations that we
created. This single page acts as a display for more than one visualizations together.
They can be prioritized in any way, resized, replaced and removed. It makes visualizing
data more easy and fast.

Some benefits of using it:

 It is very handy from a business point of view.

 No coding needed to add/ remove visualizations.
 Filters can be created by clicking upon visualization.

Above is the home page for Dashboard in Kibana.

Creating a Dashboard
We will create a dashboard with the visualizations that we saved. To create a dashboard:

 Select Dashboard > Create a Dashboard.

 Select Add. It will list all the saved visualizations. Select some of them. In our case, we
selected gauge, heatmap, and coordinate map. Selecting these will reflect all these
visualizations together on the screen.

You will find multiple options like drag and drop, enlarge, remove, etc. Get familiar with these
options by playing around. You can save a dashboard with save option in top right
corner.

Above is a resultant dashboard.

Sharing Saved Dashboard

You can share your dashboard in a number of ways. Here are the steps:

 Open the saved Dashboard and select Share on top right corner.

 On clicking, you will be provided following options to share the dashboard:

1. Share saved dashboard

2. Share Snapshot

Each of these two options will be having two options:

 Embedded iframe (is required to add the dashboard to the HTML source)

 Link (These links are shareable and can be used by your colleague to view the dashboard)

Note that all the clients to whom the stuff are shared should have access to Kibana to
view them (visualizations, dashboards, etc.).
Embedding Dashboard in Web Page
Let us take the sample Embedded iframe link of the saved dashboard :

<iframe src="http://localhost:5601/app/kibana#/dashboard/02e0f270-9941-11e7-a327-
a93fe119ec90?embed=true&_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue
%3A0)%2Ctime%3A(from%3A'2017-09-14T11%3A34%3A35.544Z'%2Cmode%3Aabsolute%2Cto%3A'2017-09-
14T11%3A49%3A35.544Z'))" height="600" width="800"></iframe>

It can be inserted within our HTML code as:

<html>
<head>
<title>
Test
</title>
</head>
<body>
<iframe src="http://localhost:5601/app/kibana#/dashboard/02e0f270-9941-11e7-a327-
a93fe119ec90?embed=true&_g=(refreshInterval%3A(display%3AOff%2Cpause%3A!f%2Cvalue
%3A0)%2Ctime%3A(from%3A'2017-09-14T11%3A34%3A35.544Z'%2Cmode%3Aabsolute%2Cto%3A'2017-09-
14T11%3A49%3A35.544Z'))" height="600" width="800"></iframe>
</body>
</html>

Above is the resultant screenshot of the web page.

Quick Fact

You can find sample dashboards for a better

understanding of the official elastic's Kibana Playground.
Timelion
Yes! You read it right, "Timelion"!

Timelion, pronounced as Timeline, is a time series data visualizer that makes it possible to
visualize independent data sources as a unity. Some of the use cases are:

 Unique users viewing pages over time.

 Difference in traffic volume.
 Cumulative sum of all search requests.
Above is a timelion page. Please refer to elastic's official Kibana timelion playground for better
understanding.

More on Kibana Timelion

Here is a video on Timelion that will help you get started with the basics.
If you have trouble playing this video. Please download the root certificate and proxy certificate and install.
No transcript is available for this video.

Dev Tools
Dev tools page is used by developers to interact with the data in Kibana more in the
form of query console fashion.

In Dev tools, the console plugin allows you to interact with the REST API of Elasticsearch. It
has two portions, one is the Editor part and other is the Response part displaying response
to the request. While typing a query, you will get related API suggestions from which you can
select the required one, thus saving the time.

Here is a command to try :

GET /_search
{
"query": {
"match_all": {}
}
}

Above screenshot is the Dev tools page with a sample query and response.

Management
The management page of Kibana is used for:

 Performing Runtime Configuration.

 Configuring Index Pattern.
 Configuring objects such as visualizations, searches, and dashboards.

Above is the screenshot of Kibana's Management page.

Creating Index Pattern

Index pattern is required to identify one or more indices of Elasticsearch that is to be explored
with Kibana. All the indices that match the specified pattern can be used in Kibana. Here is an
example pattern:

earth-*

This pattern will find all the index such as earthatmosphere, earthwater, earthquakes, etc.

Here is the procedure to create an index pattern:

1. Go to Management > Kibana > Index Patterns.

2. Click on Create Index Pattern and give an index name or a pattern.
3. Select Index contains time-based events and select index field containing timestamp (if any).
4. Click Create to create the Index pattern.

You can also set an Index as the default index, refresh the field list or delete it.

Above is a sample screenshot once the Index / Index pattern is specified.

In addition to these, we have an advanced settings page where you can directly control
Kibana's behavior. This section should be handled with utmost care as it can easily damage
and break the application if handled incorrectly.

Kibana Summary
In this course, you studied the basic ingredients required to get started with Kibana. You read
about the tools like Discover, Visualize, Dashboard, Timelion, etc. These tools are very
useful while analyzing an extensive data and helps in revealing hidden relationships within as
well as among the data.

QUIZ :
1. Kibana is an independent tool nothing to do with ELK stack. F
2. Configuration file for Kibana is named as ________. KIBANA.YML
3. Which of the following is an appropriate query to be typed in search bar for getting all documents in
which “state” equals “TX”? STATE.TX
4. Is it necessary to use curl command while querying in Kibana? F
5. What is Kibana?A VISUALIZATION TOOL
6. You can only set the Index pattern once.F
7. Default location to run Kibana is _________. 5601
8. What can be created with the help of Kibana? ALL
9. If Elastic search is not visible via Kibana, what could be the problem? BOTH
10. In bucket aggregation, buckets are created to store _____. DOCUMENTS
11. Visualize page is responsible for __________. ALL
12. Date histogram is performed on date/time values that are automatically extracted from documents.
T
13. Geohash is used to create buckets based upon ________.  geo_point fields.
14. chronological order MEANS
15. Which of the following is not a metric aggregation? LOG
16. Can Boolean operators like AND, OR etc. can be used in Lucene Query Syntax? YES
17. Where can you access discover page in Kibana? TOP LEFT SIDE
18. Upon searching, the listing is done in chronological order which means ___________. NEWEST
DOCUMENT FIRST
19. Discover is used for ________. BOTH
20. Filters may be ___________. BOTH
21. What will Max Aggregation do? Return Max value from numeric fields present in each bucket
22. Can you share a dashboard as we shared a visualization? YES
23. Percentile ranks are used to calculate single or multiple percentile ranks.T
24. Which of the following is an appropriate query to be typed in search bar for getting all documents in
which “state” equals “TX”? STATE:TX
25. What are the steps to create a coordinate map? Visualize > Create a new visualization > Coordinate
map > Select relevant index.
26. Can we save the visualizations we create in Kibana? YES
27. A dashboard can be embedded in a HTML page. T
28. You can only set the Index pattern once.F
29. Why is Dashboard used for? All of these
30. Timelion is a ______. TIME SERIES DATA VISUALIZER
31. What is the function of Gauge? It indicates the status of metric in reference to a threshold value.
32. We can show multiple dashboards in visualization page – F
33. How can we insert a dashboard into a HTML page - With Embedded iframe