C H A P T E R 10

Introduction to
Dashboards and Alerts
The Splunk platform provides a rich eco-system for developing visualizations and
dashboards. Using dashboards, you can readily display useful information that can
be used by different types of audiences. For example, you can create a dashboard
for a NOC (Network Operations Center) that displays a service's health. You can
make another dashboard for the software development team that displays the
application resource utilization, such as CPU and Memory. Using the Dashboard
editor, you can easily put together relevant visualizations as panels in a dashboard
within minutes. In addition, the dashboard can be made interactive by providing
users a form to provide input. For example, a user can choose an option from a drop-
down list. Further, the dashboard can contain drill-down actions that can, among
other things, take a user to another dashboard or an external site.

Using the Splunk platform you can also create Alerts. Alerts are saved
searches with a threshold component attached to it. When a threshold is breached,
you can have Splunk trigger an action, such as emailing someone or paging the on-
call engineer. For example, you can set up an alert to page the on-call engineer if a
service becomes unavailable. In addition to emailing and paging, you can use Splunk
to trigger varieties of actions, such as posting the alert data to an external website or
posting a message in a chat room. You can also create your own custom action by
using Splunk's custom action framework.

Let us begin by creating a basic dashboard in the next section.

CHAPTER

Creating a Basic Dashboard

We will use Splunk's tutorial data for creating a dashboard. You can download the
Splunk tutorial data from the following link:

https://docs.splunk.com/Documentation/Splunk/latest/SearchTutorial/
Systemrequirements#Download_the_tutorial_data_files

Download tutorialdata.zip and Prices.csv.zip to your local drive. Upload them

to your Splunk instance. For detailed instructions on uploading the turorialdata.zip,
refer to Introduction to the Splunk Platform chapter from the book. For detailed
instructions on uploading the Prices.csv.zip, refer to the Using Lookups chapter from
the book.

We will start by creating a basic dashboard and add functionality throughout

this chapter. Let's start by building a web server health dashboard. We'll call our
fictious company Acme Corp.

Adding the First Panel

A dashboard is made up of panels that display information. More specifically. they

display search results. The simplest way to add a panel to a dashboard is to first
write the SPL query, and simply add the results to a dashboard panel. Let's start by
writing the SPL.

index=main sourcetype=access_combined_wcookie
| timechart span=1h count AS Total_Requests,
count(eval(status LIKE "5%")) AS Server_Errors

2
 CHAPTER

The SPL query plots a timechart of total number of requests coming into the
web server and the number of requests failing with 500 series HTTP codes. First, it
filters the retrieved data by the sourcetype (access_combined_wcookie). It then uses
timechart command with a span of 1 hour to calculate the total number of
requests within that span, and the number of requests where the status is 5 followed
by any characters (5%). This will match status like 500,501,502 and so on. Make
sure to set the time picker to All Time.

The result will be as shown in figure 10-1.

Figure 10-1. Running a search before adding to a dashboard

Navigate to the Visualization tab to reveal the graph that Splunk has come up
with. In the visualization tab, you can choose the type of visualization you like. For
example, you may choose an area chart instead of a line chart. See figure 10-2.

3
CHAPTER

Figure 10-2. Choosing the visualization type

Also, based on the visualization you choose, you can configure varieties of
formatting options. For example, you can choose a custom X axis title instead of the
default _time. See figure 10-3.

Figure 10-3. Customizing formatting options of a visualization

Once you have the visualization displayed the way you want, in order to save
this visualization as a dashboard panel, select Save As at the top right and select
Dashboard Panel. See figure 10-4.

4
 CHAPTER

Figure 10-4. Saving a visualization as a dashboard panel

In the resulting Save As Dashboard Panel screen, enter the following details:

Dashboard: New

Dashboard Title: Acme Corp Web Server Health

Description: Acme Corp's Web server statistics dashboard

Panel Title: Web Server Traffic Over Time

Panel Content: Line Chart

Click Save.

See figure 10-5.

5
CHAPTER

Figure 10-5. Saving a visualization as a dashboard panel

Tip By default, the dashboard gets created with Private permission. If you need to
grant access to other users of Splunk, select Shared in App. You can also change
the permissions at any time after you create the dashboard.

6
 CHAPTER

After you click Save, you can view the dashboard by clicking View Dashboard
on the resulting screen. See figure 10-6.

Figure 10-6. Notification of successful creation of the dashboard panel

Congratulations. You have just created a dashboard. Upon clicking on View

Dashboard, you will be taken to the dashboard. See figure 10-7.

7
CHAPTER

Figure 10-7. A newly created dashboard with one panel

Let us continue building our dashboard by adding two more panels. First,
we'll add a timechart that shows the average and 95th percentile response times. The
response time of a request is logged as the last field in an access log event. In
Splunk's tutorial data that we are using, this field is extracted with the name other. In
the event below, the highlighted field is the response time in milliseconds (159).

91.205.189.15 - - [25/Mar/2020:18:22:16] "GET /oldlink?

itemId=EST-14&JSESSIONID=SD6SL7FF7ADFF53113 HTTP 1.1" 200
1665 "http://www.buttercupgames.com/oldlink?itemId=EST-
14" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.46
Safari/536.5" 159

Let us first build our SPL query.

index=main sourcetype=access_combined_wcookie
| timechart span=1h avg(other) AS Average_Response_Time,
perc95(other) AS 95th_Precentile_Response_Time

The line chart visualization of the result of the SPL is shown in in figure 10-8.

8
 CHAPTER

Figure 10-8. A timechart showing the response times using the web access log

Add the visualization to the dashboard, by clicking on Save As > Dashboard

Panel. This time choose Existing dashboard by selecting Acme Corp Web Server
Health. Provide a panel title and click Save. See figure 10-9.

9
CHAPTER

Figure 10-9. Choosing an existing dashboard to add panels

Let us add another panel that shows the breakdown of HTTP status codes.
We'll use a pie chart this time. Here is the SPL query:

index=main sourcetype=access_combined_wcookie
| stats count by status

The SPL simply uses the stats command to break down a count of the events
by status. After the search completes, choose Pie chart from the visualization, and
save the visualization to the same dashboard (Acme Corp Web Server Health). Use
the panel title HTTP Status Code Breakdown. See Figure 10-10.

10
 CHAPTER

Figure 10-10. Selecting Pie Chart as visualization.

Our now dashboard now has three panels and looks like figure 10-11.

11
CHAPTER

Figure 10-11. Web Server health dashboard with 3 panels.

Tip You can always launch the dashboard by navigating to Dashboards menu in the
App bar.

Our dashboard is shaping up nicely. But you may have noticed that the
panels stack up in a row, one underneath the other. This is the default behavior. You
can edit the dashboard by dragging them to different positions. For example, you can
drag the pie chart to be at the same level as the Response time line chart. In order to
do this, click on the Edit button at the top right. See Figure 10-12.

Figure 10-12. Launching the Edit interface of a dashboard.

Once in the edit interface, you can drag a panel using its handle (denoted by
the two dotted lines at the top of the panel). See figure 10-13.

12
 CHAPTER

Figure 10-13. Re positioning panels in a dashboard

Make sure you save the dashboard by clicking on the Save button at the top
right. The completed dashboard now looks like figure 10-14.

13
CHAPTER

Figure 10-14. A complete basic dashboard

You have just created a basic dashboard and performed some minimal
editing as well. Great job! In the next section, let's look at various ways of adding
data to the panels.

Generating Dashboard Content

We saw how the search results can be used to create panels in a dashboard. Using
search results is one of the basic ways to add content to a dashboard. The panels
created by using search results in this manner use inline searches. There are other
ways to power a dashboard panel. These ways are mentioned below:

1. Using saved searches (reports)

2. Using prebuilt panels

3. Using Pivots

4. Using global base searches.

14
 CHAPTER

I will briefly discuss each of these methods.

Using Saved Searches (Reports)

Instead of using inline searches, you can create a panel using a saved search.
Saved searches are also known as reports. If the reports are scheduled, the panel
will display the results from the latest run of the report. Using scheduled searches to
power the panels is one of the best ways of improving dashboard performance. This
is because the dashboard simply loads the results from the last run of the scheduled
search, instead of re-running the search. You can add a report as dashboard panel
by using Add Panel and choosing an available report. See figure 10-15.

Figure 10-15. Using reports to add contents to dashboard panel

You can also add a report to a dashboard panel from the report. Simply click
on Add to Dashboard button at the top right.
15
CHAPTER

Caution Reports can be configured to run as either owner or the user who is running
the report. Scheduled reports always run as owner. When a dashboard is loaded
that has panels backed by reports, the concurrent search limit of the owner will be
impacted.

Using Prebuilt Panels

An inline panel can be converted to a pre-built panel. A pre-built panel can be used
in any other dashboard to quickly add visualizations to that dashboard. To convert an
inline panel to pre-built panel, simply edit the dashboard and use the menu Convert
to Prebuilt Panel in the panel edit menu (click on the gear icon to bring up the panel
edit menu). See figure 10-16.

Figure 10-16. Converting an inline panel to prebuilt panel

To use a prebuilt panel in a dashboard, simply use the Add panels menu and
choose from the available prebuilt panels. See figure 10-17.

16
 CHAPTER

Figure 10-17. Adding a prebuilt panel to a dashboard

Note Only the available pre-built panels are shown under Add Prebuilt Panel menu

Using Pivots

A pivot refers to a visualization such as a chart or table that you create using Pivot
Editor. Pivots are used as a quick way to create visualizations using data models
without having to write SPL queries. See figure 10-18.

17
CHAPTER

Figure 10-18. Adding a pivot to a dashboard

We will not discuss Pivots in detail in this book as it is beyond the scope.

Using Global Base Searches

You can define a SPL query as global base search within a dashboard. Using the
results of the base search, you can populate other panels within the dashboard with
any desired post-processing. This is especially useful if you have a dashboard with
many panels performing similar searches. You can save resources by defining a
base search and having the panels in the dashboard use a post-process search to
further modify the search results.

You would define the base search using a search id and reference that
search id in the child panels using the base attribute.

In the next section, let's add interactivity to your dashboards by introducing

input controls.

18
 CHAPTER

Using Forms to Process User Input

You can make your dashboards interactive by adding input panels in a form. For
example, you can have the user choose the time frame for the searches in the
dashboard panels. As another example, you can allow the user to select an option
from a list of options in a drop-down box. You can configure dashboard panels to
respond to the user input. The Splunk platform achieves this by the use of tokens.

Tokens are similar to variables in high-level programming languages. They

can be set to a value based on a user action and retrieved by dashboard panels. We
will consider two examples in this section. First, we'll add a time input to allow the
user to choose a time frame. Next, we'll add a drop-down list to allow the user to
choose an option.

Adding Time Input

For this example, let's create a new dashboard. Using Splunk tutorial data and
prices.csv lookup table, we'll create a Business Analytics dashboard. First, we'll
display a single value visualization that shows the total revenue in dollars. The total
revenue is obtained by summing the sale price of all purchased products. We can
derive this by the following SPL query:

index=main sourcetype=access_combined_wcookie
action=purchase
| lookup prices.csv productId OUTPUTNEW sale_price
| stats sum(sale_price) AS total
| eval Total_Revenue = "$" . tostring(total,"commas")
| fields – total

The SPL query retrieves the events where action is purchase and looks up
prices.csv to bring in the sale price. The stats command sums up the sale prices and
the eval command prints the dollar amount using $ sign and commas for readability.
The result is shown below:

Total_Revenue
19
CHAPTER

-------------
$45,558.28

We'll add this result as a single value visualization in a new dashboard. First,
display the result using single value visualization. See figure 10-19.

Figure 10-19. Choosing Single Value visualization

As before, select Save As > Dashboard Panel and create a new dashboard.
See figure 10-20.

20
 CHAPTER

Figure 10-20. Creating a new dashboard from single value visualization

The dashboard is created and looks like figure 10-21.

21
CHAPTER

Figure 10-21. Newly created dashboard with single value visualization.

Let us enable dark theme by editing the dashboard and selecting the Dark
Theme toggler at the top middle. You need to reload the dashboard for the change to
take effect. The dashboard now looks like figure 10-22.

Figure 10-22. Dashboard with dark theme

Let us add another panel next to the Total Revenue panel. We will show the
top 5 products based on the revenue. The SPL query is as follows:

index=main sourcetype=access_combined_wcookie
action=purchase
| lookup prices.csv productId OUTPUTNEW
product_name,sale_price
22
 CHAPTER

| stats sum(sale_price) AS Revenue by product_name

| sort 5 -Revenue
| eval Revenue = "$" . tostring(Revenue,"commas")

Splunk produces the following output:

product_name Revenue
------------------------------
Dream Crusher $5,647.74
Manganiello Bros. $5,572.77
World of Cheese $5,497.25
Mediocre Kingdoms $5,317.34
SIM Cubicle $4,638.27

Instead of choosing a visualization, simply use the table output in the

statistics tab and save the table as dashboard panel. Choose the existing Acme Corp
Business Analytics Dashboard. The dashboard now looks like figure 10-24

Figure 10-23. Dashboard with two panels

Now let us add the time picker input. This will let a user choose a time frame.
Edit the dashboard and choose Add Input > Time. See figure 10-24.

23
CHAPTER

Figure 10-24. Selecting Time Input

A new Time Input panel is added to the top right. Click on the pencil icon at
the top to edit the Time Input. Enter the following details:

Label: Select Time Range

Search on Change: Select the check box

Token: tok_time

Click Apply. See figure 10-25.

24
 CHAPTER

Figure 10-25. Editing Time Input

The Label is the text that the user sees in the user interface. Provide a
meaningful name for this. The option Search on Change dynamically re-runs all the
panels that use this Time Input when a time-range is selected. The Token is the id
used to reference the selected time in searches that make use of this input. This is
how we pass the value of the time the user selected to other panels in the search. To
reference a token in a search, use $token_name$. For example, to refer to the Time
selected by the user using this input, you will use $tok_time$.

Tip Always provide a meaningful name to the token so that it is easy to reference in
searches. Avoid using unclear names such as field1.

25
CHAPTER

Now that you've saved the Time Input, we'll need tie in this input to all the
panels that make use of this Time. For this, we'll need to edit the search of each
panel. While still in the Edit interface, click on the search icon of the Total Revenue in
Dollars panel. See figure 10-26.

Figure 10-26. Editing the search of a panel in the dashboard

In the resulting Edit window, select Shared Time Picker (tok_time) for the
Time Range. If you recall, this is the token name that we used in the Time Input.
Click Apply. See figure 10-27.

26
 CHAPTER

Figure 10-27. Editing the search of a panel to update the Time Range

Repeat the same process for the panel Top Products Based on Revenue.
Click on Save at the top right to save the dashboard. Now, our dashboard looks like
figure 10-28.

27
CHAPTER

Figure 10-28. Dashboard with Time Input

From now onwards, you can choose the time frame you want for the panels
to use. When you select a time range, the panels automatically reload the data for
the time frame selected. See figure 10-28.

28
 CHAPTER

Figure 10-28. Selecting the time range using Time Input

Great going. In the next section, let us add a drop-down input so that user
can choose an option from the drop-down list. This can be a great way to add
interactivity to your dashboards.

Adding a Drop Down Input

A drop-down input lets a user make a selection from the list of options. When an
option is selected, you can capture the value of the selected option in a token. You
can configure the searches in other panels of the dashboard to make use of the
token value by referring to the token value using $token_name$ notation.

For example, let us build a panel that shows the trend of the revenue by
product. We will use timechart command. The SPL query is as follows:

index=main sourcetype=access_combined_wcookie
action=purchase
| lookup prices.csv productId OUTPUTNEW
product_name,sale_price
| timechart span=6h limit=30 sum(sale_price) AS Revenue
by product_name

The SPL query retrieves events with the source type

access_combined_wcookie and further filters with the condition action=purchase. It
then looks up prices.csv to bring in product_name and sale_price fields based on
productId field. Finally, it uses timechart command to sum the sale_price every 6
hours. It uses the limit option to set the value to 20 in order to cover all the products.
By default, timechart will only show 10 values and combine all other values in to
OTHER. Splunk produces the visualization as shown in figure 10-29.

29
CHAPTER

Figure 10-29. Plotting time chart of revenue by product

We will add this visualization into the dashboard with a drop-down input that
lets a user select a product to view the trend. The drop-down list should have all the
products available. The panel should automatically update based on the selection.
As before, update the X axis title to Time using the format menu. Then simply click
on Save As > Dashboard Panel. Provide the following values:

Dashboard: Select the existing dashboard Acme Corp Business Analytics

Dashboard

Panel Title: Revenue Trend by Product

Figure 10-30 shows the Save As Dashboard panel screen.

30
 CHAPTER

Figure 10-30. Saving Revenue by Product as a dashboard panel

Our dashboard now looks like figure10-31.

31
CHAPTER

Figure 10-31. Dashboard with three panels

Now let us proceed to add the drop-down list. Edit the dashboard and select
Add Input > Drop-down. A new drop-down input appears at the top of the dashboard.
Drag this input to the Revenue Trend by Product panel. See figure 10-32.

32
 CHAPTER

Figure 10-32. Adding a new drop-down input.

Click on the Pencil icon to edit the drop-down. Provide the following details:

Label: Choose Product

Search on Change: Select the check box

Token: tok_product

Scroll down to find the Static options and Dynamic options section.

Static Options

With static options, you can provide a list of options that you manually type in. This
can be useful if you already know which options to provide. In our example, we will
provide one static option named ALL that will choose all products. See figure 10-33.

33
CHAPTER

Figure 10-33. Adding static options in drop-down input.

The Name is the label that will show up in the drop-down list as option to
select. The Value is the value that will be assigned to the token. In this case, the
value * will be assigned to the token.

Dynamic Options

34
 CHAPTER

While static options are useful, in most circumstances you would want to dynamically
generate the drop-down options. You can do this by using the results of the search
as the drop-down options. First, you need to come up with the search that lists the
options you want to show in the drop-down. In our example, we'll use the following
search:

| inputlookup prices.csv
| table product_name

The SPL query retrieves the content of the lookup table prices.csv by using
the inputlookup command. It then filters the data using the table command. The
partial output is shown below:

product_name
-------------------
Mediocre Kingdoms
Dream Crusher
Final Sequel
World of Cheese
World of Cheese Tee
Puppies vs. Zombies
...
...

In the Dynamic Options section, enter the SPL query in the Search String
field. In the Field For Label field, enter product_name. This is the field returned from
the search results to use as the drop-down option label. In the Field For Value field,
enter product_name again. This is the field returned from the search results to use
as drop-down option value. This is the value that gets passed to the token. Click
Apply. See figure 10-34.

35
CHAPTER

Figure 10-34. Adding dynamic options in drop-down input.

Now, you can verify the contents of the drop-down list by clicking on the
Select drop-down. See figure 10-35.

36
 CHAPTER

Figure 10-35. Verifying the drop-down list options.

Once you verify the options, edit the drop-down again to select a default
value. In our example, let's use the static option ALL as the default value. This is the
option that will show by default when the dashboard loads. See figure 10-36.

37
CHAPTER

Figure 10-38. Selecting the default option in a drop-down input.

Now all that is left is to update the target panel (Revenue Trend by Product)
to use the token.

38
 CHAPTER

Updating the Target Panel to Use the Token

Merely creating the drop-down input does not enable the panels to make use of it.
We need to update the search that is powering the panel. Click on the Search icon
on the Revenue Trend by Product panel and edit the search. Edit the search string
by adding a new filter criterion as follows:

index=main sourcetype=access_combined_wcookie
action=purchase
| lookup prices.csv productId OUTPUTNEW
product_name,sale_price
| search product_name = “$tok_product$”
| timechart span=6h limit=20 sum(sale_price) AS Revenue
by product_name

As you can see, we are adding a new search filter by restricting the output to
the selected product. The string $tok_product$ represents the value of the token
tok_product. Note the double quotes surrounding the token. This is required to
ensure that the token values with spaces in them will be surrounded by double
quotes in the resulting search string. Also select the Time Range to Shared Time
Picker (tok_time). See figure 10-39.

39
CHAPTER

Figure 10-39. Utilizing the token value in search string

Click Apply and Click Save to save the dashboard. Reload the dashboard.

Note Always reload the dashboard when you make changes to it. Many features of
dashboard visualization take effect only when it is reloaded. You can simply refresh
the browser window to reload the dashboard.

Our dashboard now looks like figure 10-40.

40
 CHAPTER

Figure 10-40. Dashboard with drop-down input

Note that since we chose ALL as the default value, all products are shown.
Go ahead and select any of the products from the drop-down list. The panel will
instantly reload the data for the product you chose. See figure 10-41.

Figure 10-41. Selecting an option from drop-down input

41
CHAPTER

If the data does not load as expected, you can review the search generated
by the panel by clicking on the search icon to open the search in a new window. See
figure 10-42.

Figure 10-41. Reviewing the search generated by a panel

Nice job. You have learned how to add interactivity to your dashboards by
using drop-down input. In similar fashion, you can add other inputs such as radio
buttons and multi-select. In the next section, let's learn how to add drilldown
capabilities to your dashboards.

Creating Drilldowns
Adding drilldown capability to your dashboards can greatly enhance the interactivity
and user experience. Using drilldowns, you can click on an area of a panel to display
other detailed relevant data. You can perform the following actions using drilldowns:

42
 CHAPTER

1. Run a secondary search

2. Open another dashboard or report

3. Launch an external website (custom url)

4. Update values of tokens that can produce changes in other panels in

the same dashboard

When a drilldown is invoked by clicking on an area of a panel, Spunk makes

many predefined tokens available for the target. In this way, values from source
panel (where the user clicks) can be passed to target panel (which is invoked as a
result of drilling down).

Tokens Available for Drilldown

The tokens have slightly different meaning depending on the type of source panel.
There are two major categories of panels: Charts and Tables. The following section
describes the tokens available in each category.

Chart

Here are the predefined tokens and their descriptions in a chart visualization.

$click.name$

This token will carry the X-axis field or category name for the clicked location. If the
chart is timechart, this token will carry the name "_time"

$click.value$

This token will carry the X-axis field or category value.

43
CHAPTER

$click.name2$

This token will carry the Y-axis field or category name for the clicked location. For
example, if you have multiple series plotted in a timechart, this token will carry the
name of the series that was clicked.

$click.value2$

This token will carry the Y-axis field or category value.

$row.<fieldname>$

By specifying the field name, you can access any Y axis field value corresponding to
the location in X axis. This token gives more fine-grained control over what to send to
the target panel.

Now let's take a look at what these tokens mean in tables.

Table

Here are the predefined tokens and their descriptions in a table visualization.

$click.name$

This token will carry the left most field name (the column name).

$click.value$

This token will carry the left most field value (column) in the clicked row.

$click.name2$

This token will carry the field name of the clicked cell.

$click.value2$

This token will carry the field value of the clicked cell.

44
 CHAPTER

$row.<fieldname>$

This token lets us access any field value (column) from the clicked row. For example,
to get to the product name of any row, use $row.product_name$.

Now, let us build our dashboard to demonstrate some of the drilldown

capabilities.

Drilldown to a Secondary Search

In this drilldown, you can invoke a secondary search based on where the user
clicked on the source panel. In order to enable drilldown, you would edit the
dashboard and click on the Edit Drilldown menu under the More actions button (the
three vertical dots). See figure 10-42.

Figure 10-42. Invoking the drilldown editor

45
CHAPTER

For this example, let us invoke a secondary search that shows all the
purchases for the product that user clicked. In the Drilldown Editor, select Link to
search. See figure 10-43.

Figure 10-43. Linking to a search in Drilldown Editor

Choose Custom and include the following search string:

index=main sourcetype=access_combined_wcookie
action=purchase

46
 CHAPTER

| lookup prices.csv productId OUTPUTNEW

product_name,sale_price
| search product_name = "$row.product_name$"
| table _time,clientip,productId,product_name,sale_price

The SPL query retrieves all purchase events and filters them using the
product_name field. It utilizes $row.product_name$ token. This token will carry the
value of the product name from the row clicked by the user. See figure 10-44.

47
CHAPTER

Figure 10-44. Configuring a custom search string in Drilldown Editor

48
 CHAPTER

Click Apply and save the dashboard. Notice how the dashboard now shows
the table contents in blue color, indicating that they can be clicked. Upon clicking on
any row within the table, you will be taken to a secondary search page where all
purchase events for the product name are displayed. See figure 10-45.

Figure 10-45. Drilldown page that shows details of product purchases

Now, let us build another drilldown that will create contextual changes within
the dashboard.

Showing and Hiding Panels

When a user clicks on a visualization, you can set or unset a token. By referencing
this token elsewhere in the dashboard, you can make dynamic changes to the
dashboard. For example, when the user clicks on the Total Revenue in Dollars single
49
CHAPTER

value visualization, you can show a daily revenue panel by product name in a
stacked column chart. By default, we won't display this panel. Only when a user
clicks on the Total Revenue, this panel should display. And when the user clicks on
the newly displayed panel, it should be configured to disappear. In order to hide and
display panels, we will use the depends attribute of a panel. Also, in order to
configure this functionality, you must edit the Simple XML code of the dashboard.

About Simple XML

Simple XML is the markup Splunk uses to configure and layout dashboards. All
dashboards can be modified using the Simple XML code via the Simple XML Code
Editor. For most configurations, Drilldown Editor is sufficient which provides a GUI to
configure drilldown. However, for configuring conditional drilldowns and contextual
changes (like the one we are working on), we will need to edit the Simple XML code.

Configuring Drilldown for Contextual Changes

First, we’ll come up with the visualization for the target panel. Let’s use the following
SPL query:

index=main sourcetype=access_combined_wcookie
action=purchase
| lookup prices.csv productId OUTPUTNEW
product_name,sale_price
| timechart span=1d sum(sale_price) AS Revenue by
product_name

50
 CHAPTER

The SPL query retrieves all purchase events, calculates the daily revenue
and plots them in a timechart. Using stacked mode of column chart format option,
the following visualization is produced. See figure 10-46.

Figure 10-46. Creating a stacked column chart

Add this visualization to our dashboard and drag it up to place under the Total
Revenue. See figure 10-47.

51
CHAPTER

Figure 10-47. Dashboard with stacked column chart

Now, let us edit the Simple XML code. First, we'll work on the source panel.
In order to invoke the Simple XML code editor, Edit the dashboard and click on the
Source button. See figure 10-48.

52
 CHAPTER

Figure 10-47. Invoking the Simple XML code editor

In Simple XML code editor, locate the panel with title Total Revenue in
Dollars. Add the following drilldown element under the option element.

...
...
<option name="useColors">1</option>
<option name="useThousandSeparators">0</option>
<drilldown>
<set token="showDetailsPanel">true</set>
</drilldown>
</single>
</panel>

The Simple XML code will set the token showDetailsPanel to be true. We will
use this token in the target panel.

53
CHAPTER

In addition, locate the option drilldown and set it to all (by default, it is set to
none). See the code below:

...
...
<option name="colorMode">none</option>
<option name="drilldown">all</option>
<option name="numberPrecision">0</option>

Next, locate the panel with title Daily Revenue by Product add the depends
attribute in the panel.

...
...
<row>
<panel depends="$showDetailsPanel$">
<title>Daily Revenue by Product</title>
<chart>

The Simple XML code will make this panel display in the dashboard only
when the token showDetailsPanel is set. To make the panel disappear when the
user clicks anywhere in the panel, add a drilldown section as shown below:

...
...
<option name="trellis.size">medium</option>
<drilldown>
<unset token="showDetailsPanel"></unset>
</drilldown>
</chart>

54
 CHAPTER

The Simple XML code unsets the token upon a user click. This will make the
panel disappear because it depends on showDetailsPanel token to be set.

In addition, locate the option charting.drilldown and set it to all (by default, it is
set to none). See the code below:

<option
name="charting.chart.style">shiny</option>
<option name="charting.drilldown">all</option>
<option
name="charting.layout.splitSeries">0</option>

Save the code and reload the dashboard. Initially the dashboard looks like
figure 10-48.

Figure 10-48. Initial loading of a dashboard with hidden panel

55
CHAPTER

Upon clicking on the Total Revenue in Dollars dollar amount, the details
panel shows up under it. See figure 10-49.

Figure 10-49. Upon drilling down, the hidden panel is displayed

When you click anywhere in the new Daily Revenue by Product panel, the
panel disappears.

Now that you have learned how to configure drilldowns to make your
dashboard interactive, let us move on to the next section where we'll learn about
configuring alerts.

Creating Alerts
Splunk alerts provide a way to trigger an action by monitoring your machine data. It
can be extremely valuable to prevent an outage to your service. For example, the on-
call engineer can be paged when the memory utilization of your application goes

56
 CHAPTER

above a threshold. Splunk can be configured to take varieties of actions when the
threshold is breached. In addition, you can create your own custom action.

Alerts are essentially scheduled searches that contain a threshold and action
to take when the threshold is breached. To demonstrate alerts, let us build an alert
using Splunk tutorial data. We will create an alert that emails the support group when
the number of HTTP Server errors (500 series errors) goes beyond a certain
threshold.

Creating the Search

First, we need to come up with the SPL query that we'll use to monitor the 500 series
errors. We'll use the following SPL query:

index=main sourcetype=access_combined_wcookie status = 5*

| stats count

The SPL query retrieves the events that have status field starting with the
number 5. This will cover all 500 series errors such as 500,501,502 and so on. When
the timeframe is set to All Time, Splunk produces the following output:

count
-----
2165

In practice, you will use more appropriate time frame such as 60 minutes.
This means that the output of the SPL will show the number of 500 series errors in
the past 1 hour. If you need faster alert time frame, you can go to 30 or 15 minutes.

Once the results are loaded, click on Save As > Alert at the top right. See
figure 10-50.

57
CHAPTER

Figure 10-50. Saving a search as Alert

In the Save As Alert screen, provide the following details:

Title: Acme Web Server HTTP 500 Errors

Description: This alert will trigger if the number of HTTP 500 errors (as found
in the access logs) go above 1000 over the past 60 minutes.

Alert Type: Scheduled

For Scheduling, I recommend using cron Schedule instead of choosing a time

from the drop down. With cron scheduling, you will have the maximum flexibility in
choosing the time. To run this alert at 2-minute past ever hour, we will use the
expression 2 * * * *.

Time Range: Choose a meaningful range. For this example, I've chosen All
Time because we are using Splunk tutorial data. In practice, you will rarely choose
All Time. Since we are scheduling this alert to run on hourly basis, a time range of 60
minutes makes sense.

Caution In large production environments, selecting All Time can have severe
performance impact.

Trigger Condition: Instead of choosing the number of results as the trigger

condition, I recommend choosing custom and provide a search string to determine if
the alert should be triggered. This search will be run on the results of the Alert's

58
 CHAPTER

search (can also be referred as base search). For this example, since the output of
the search shows the count of the number of 500 errors, I can set the search string
as follows:

search count > 1000

With the above search, the alert will trigger if the count field has value of
more than 1000.

Trigger: Once

See figure 10-51 that shows the information we entered so far.

59
CHAPTER

Figure 10-51. Configuring options while saving an alert

Now you must choose an action that must be taken when the alert is
triggered. Splunk provides many actions to choose from. The most common one is
the Email action. Instead, you can send the payload of the alert to a specific URL
using the webhook action. For this example, let's use Email action. See figure 10-52.

60
 CHAPTER

Figure 10-52. Selecting Email action for an alert.

Caution Your Splunk Server must be configured with valid SMTP configuration in
order to send emails. Contact your Splunk Administrator for assistance.

61
CHAPTER

In the Send Email screen, you can provide additional details to construct the
email. Note that you can include a PDF of the alert results which may be useful. You
can also choose to include the search string used by the alert. See figure 10-53.

Figure 10-53. Configuring Email action for an alert.

Click Save. Congratulations. You have successfully created an Alert in

Splunk.

62
 CHAPTER

Tip Always include detailed description in the alert message. For example, you can
include a description of the Alert, where to find additional information and an on-call
phone number, if applicable.

In addition to using pre-built actions, you can also create your own custom
action by using Splunk custom action framework. You can find more details about
custom actions at
https://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModAlertsIntro .

That brings us to the end of this chapter. Here are the key takeaways from
this chapter.

Key Takeaways
In this chapter, you've learned how to create interactive Splunk dashboards and
Alerts. These are extremely useful knowledge objects that will add immense value to
your organization. Splunk dashboards can be a great way to share information with
others. Dashboards are made up of panels which are powered by search results.
Splunk can also trigger alerts based on search results and a threshold you specify.
Let's take a look at the key takeaways from this chapter.

1. A Splunk dashboard can be created by simply running a search and

choosing Save As > Dashboard panel in the search interface.

2. A Dashboard panel can be powered by in-line search, saved search

(report), scheduled search, pre-built panel or a pivot.

3. For best performance, use a scheduled search to power the

dashboard panels. The data displayed will be the result of the latest
run of the scheduled search.

4. Using Dashboard editor, you can drag the panels to create the layout
you desire.
63
CHAPTER

5. The visualizations can be formatted using Dashboard editor or using

Simple XML code.

6. You can add interactivity to your dashboards by adding input panels

such as Time picker and drop-down input

7. Tokens are similar to variables in high-level languages that you can set
and retrieve in Splunk dashboards.

8. You can configure drilldown to launch a new secondary search, a

dashboard, an external URL, or contextual changes within the
dashboard.

9. You can configure Splunk alerts to trigger based on search results and
a threshold set by you.

10. Splunk provides many pre-built actions that can be executed upon
triggering of an alert. This includes emailing and sending the alert
payload to an external website.

Great going! By learning how to create Splunk dashboards and alerts, you
can make the best use of the Splunk platform to suit your needs. We have just
scratched the surface on these capabilities. Please refer to Splunk documentation at
docs.splunk.com to learn more about them.

64