Splunk Essentials

Hands-On Labs
1. Downloading Splunk
1.1. Splunk Portal Registration
By registering and creating an account on splunk.com you are able to benefit from downloading for free the Splunk software
and also other applications and add-ons that you are able to benefit from right out of the box.

Follow the steps outlined below in order to register you on the www.splunk.com website and login in with your new Splunk
username.

Step 1 & 2. Navigate to www.splunk.com with your favorite browser and & sign-up.
1

Step 3 & 4. Fill in the required fields that has the * in front of them and after you have completed the mandatory questions
press the submit button.

Page 2
3

Page 3
Step 5 & 6. After you click submit, you receive a confirmation email from Splunk which you will have to click on the link located
in the body of the email in order to activate your account.

Step 7 & 8. Wait until your email is validated and then you can log-in.

Page 4
 1.2. Let’s get some free goodies
Step 1 & 2. Click on the free Splunk link to download the software – Splunk Enterprise

Step 3 & 4. Choose the platform that you want to download and install the splunk-7.3.0-657388c7a488-Linux-x86_64.tgz
package (if required login with the credentials that you used to register with above).

Page 5
 4

Step 5 & 6. As a next step we will NOT download the Splunk tgz package but we will COPY the wget link which we will then
paste into the virtual instance’s terminal windows to automatically download it locally

Note: For now paste it into your notepad and keep it until it is needed in the steps below.

Page 6
Splunk main Installation & Configuration

1.3. Installing Splunk Instance >Splunk SH

For the training purposes we have already pre-configured the virtual machines in order for you to install Splunk and start
working right away.

Step 1 & 2. Launch your SSH client and login to the CentOS operating system with the details that have been provided to you
and when prompted login with the username/password assigned to you.

Enter username/password
2
assigned to you.

Note: Your User has sudo privileges

Step 3. Download the rpm package using wget and install it (use the url command you copied before in section 1.2/step 6).

$ cd /tmp
$ sudo adduser splunk

Page 7
$ wget -O splunk-7.3.0-657388c7a488-Linux-x86_64.tgz
'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.3.0&produ
ct=splunk&filename=splunk-7.3.0-657388c7a488-Linux-x86_64.tgz&wget=true'| use the link you copied before
$ sudo tar-zxvf /tmp/splunk-7.3.0-657388c7a488-Linux-x86_64.tgz –C /opt | install the splunk software
$ sudo chown -R splunk:splunk /opt/splunk
$ /opt/splunk/bin/splunk start | start the splunk daemon & accept license agreement
$ sudo /opt/splunk/bin/splunk enable boot-start -systemd-managed 0 -user splunk | start the splunk daemon on every
reboot

### Optional Configuration for LAB purpose only

nano /opt/splunk/etc/system/local/server.conf
[diskUsage]
minFreeSpace = 500

1.4. Splunk initial configuration >Splunk SH

Step 1. Navigate with your web browser to the IP address of the instance you just installed as shown below (use your public IP).

http://<amazon_publc_ip_address>:8000

Step 2. Log into the splunk web console and change the default changeme password.

Step 3-7. Enable https for the splunk web console

Page 8
 5

Note: You will have to restart the web server for the settings to take effect. Also it is strongly recommended that you replace
the default certificates with those provided by the client.

As seen below the URL has changed to HTTPS from HTTP.

Page 9
Step 8-10 – Set Search Preferences – Configure Search Default time to All Time
8

10

Step 11 – Change the 24Hrs to All Time

11

Note – Its note recommended to change Default Search time to All Time in production. This setting is being configured for ease
of use in Lab/educational environments.

2. Indexing and Representing

2.1. File Based Inputs

2.1.1. Indexing web access logs >Splunk SH

In this exercise, you will index the file access.log (apache access logs),

Step 1. In Splunk Web, navigate to the Search & Reporting app.

Page 10
 1

NOTE: This step is only necessary for this lab exercise to easily locate your configuration stanza. By selecting the Search &
Reporting app, you are changing the app context. The app context determines where the specific configuration files get saved.

Step 2 & 3. Click Settings > Data inputs > Add data

Select Upload data

Page 11
 3

Time Stamp was correctly recognized

Page 12
Step 6. Configure the correct input settings in order to insure that the data gets correctly ingested and with the correct
parameters; when finished press Review.

Input Settings
App Context Search & Reporting
Host field Identifier Constant value
Host field Value www1
Index Default

Step 7. Check to ensure that the data has been correctly indexed, in order to do so, go to the Search page.

Page 13
Step 10. Click on the selected field names to see the assigned host and source type.

Page 14
 2.1.2. Searching & Reporting for web access logs >Splunk SH
Working with Tables
Step 1. Type in index=* and press enter, you should see all the logs that your instance is receiving. [Do you want to know of a
different way to search all indexes?]

Step 2. Try looking only for apache access logs (Hint: access_combined).

Page 15
Step 3. Search for a Page Not Found status code (Do you know what a Page Not Found request code is?) -
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

Step 4. Next step will be to represent in a tabular format the following fields: _time, clientip, status.

Spaces between searches represent a

4 logical AND

index=* sourcetype=access_combined status=404 | table _time, clientip, status

First part is the Search & Second part after the | is

the additional representation of the data

Step 5. To represent the columns in a more “meaningful” way, we will rename the fields (Hint: Pay attention to the | symbol).

index=* sourcetype=access_combined status=404

| table _time,clientip,status
| rename clientip as “Client_IP”, status as “Status_Code”

Note: To start a new line press Shift+Enter

Working with Fields

Step 1. When searching sometimes you want to only get back specific fields from the events, this helps the analysts with looking
at specific data and also speeding up the search (test it and see!).

index=* sourcetype=access_combined status=200

| fields _time,clientip,status

Note: Let’s get the time the job needed in order to finish go to Job > Inspect Job. / Search for This search has completed and has
returned 4 results by scanning 4 events in 0.083seconds.

Step 2. You can also exclude specific fields from searches, for example I want to exclude the action field.

index=* sourcetype=access_combined status=200

| fields - action

Page 16
Before excluding fields

After excluding fields

As you can see no more “action” field.

*Inspect the jobs before and after

Tip: Try also excluding _raw, do you see any differences?

Working with stats and graphs

Step 1. Search for all apache logs

index=* sourcetype=access_combined

Tip: The more specific you are in your searches the faster they run (i.e. specifying the exact index).

Page 17
Step 2. Get all the data about the methods being used (Hint: keyword method) and the client IP (Hint: clientip) and represent
them in a table (as done above).

index=* sourcetype=access_combined | table clientip, method

Do you see something below that catches your eye? Something repetitive?

Step 3. We need to find a way so we can get a summary about each clientip and the method they have used to login to our
website; The key here are statistical functions, more precisely a command called stats.

index=* sourcetype=access_combined | stats count by method

Step 5. The above would look a lot nicer in a more graphical representation such as a pie chart (don’t you agree?)

Click on the visualization tab.

Choose the Pie chart representation

Page 18
 Nice Pie Chart with a few simple steps.

Step 4. How about we start creating a nice dashboard with our web data?

Save the search as a “Dashboard Panel”

Fill in the fields as shown in the

picture (the rest leave as default)

*Don’t forget to save ☺

Note: Do not view the dashboard; we will add one table before seeing what we have accomplished.

Page 19
Step 4. But this is not what we are looking for; we also wanted to see the clientip, so we will modify our search to include just
that!

index=* sourcetype=access_combined | stats count by clientip,method

Note: Click on the statistics tab to go view results.

Task: Try on your own and rename both the clientip and method fields to “Client IP” and “HTTP Method”.

Step 5. The next step will be to sort based on the new field that stats has created, which is called count.

index=* sourcetype=access_combined | stats count by clientip,method | sort count

Note: This will put the lowest number on top, in order to reverse the order you will have to put -count

Sorting by ascending order.

| sort by count

Page 20
Sorting by descending order.

| sort by -count

Step 6. Save the above as a dashboard under the existing one we created named “Apache Logs” with the title “Client IP Access
Details”.

Your Dashboard should look like the below picture.

Try pressing on edit > edit panels

and moving the table in parallel
with the pie chart (drag and
drop).

Working with top and rare

Step 1. Provide to you the top IPs generating successful connections (status=200) on your website and what percent of the total
does it represent.

index=* sourcetype=access_combined status=200 | top clientip

Page 21
Step 2. How would you create a search for the rare values? (Hint: keyword rare)

I will pass by and look at the command you have typed.

Working with filtering options - <where>

Step 1. If you remember in section 6.1 (working with tables), we showed a search query where you are able to find successful
web access logins for your website (using sourcetype=access_combined).

What I want from you is to create a search that will show me the number of successful logins that are greater than five (5) and
present it by clientip and sort it in a descending manner. In order to write this query use the following hints: [sourcetype, stats,
action, clientip, where, sort].

index=* sourcetype=access_combined status=200 1 Forming the search

Step 2. Next step will be to count the number of failed logins by src.

index=* sourcetype=access_combined status=200 Performing the count

2 calculations
| stats count by clientip

Step 3. Show only the IPs that have successfully connected more than five (5) times.

index=* sourcetype=access_combined status=200 Show results greater

| stats count by clientip 3 than five (5)
| where count > 5

Step 4. Sort the results in a descending manner.

index=* sourcetype=access_combined status=200

| stats count by clientip Sort the data in a
| where count > 5 4 descending order
| sort -count

Extra Points: Rename the fields into something more interesting.

Step 5. Wouldn’t this data look better in a bar format?

Page 22
 a

Have a look at the format options

and modify some settings to make
it look nicer.

Step 6. As a last step, you will have a report and name it “Top Access IPs”

Page 23
 a

@ Reports have a look under which

application it was created. Do you know why?

c *Open the Report and view it – Check out

the time picker

Working with iplocation & Geostats

Splunk has a GeoIP database embedded in the core product, thus you are able to create tables and stats based on countries.

Step 1. Let’s take into consideration the apache access logs and perform a search again, this time focusing on the Client IP which
in our case are the external client IPs hitting our website.

index=* sourcetype=access_combined | iplocation clientip

Page 24
 Have a look at the fields that are generated.
1 Task: Create some nice searches based on the new
interesting fields (use what you have learnt so far).

Step 2. Wouldn’t this type of data be best represented on a map? (Hint: keyword geostats) – Represent the method data on the
map by Country.

index=* sourcetype=access_combined First Part is retrieving the GeoIP information

a in order to use it in our next query.
| iplocation clientip

index=* sourcetype=access_combined Represent the data by Country (remember

| iplocation clientip this is a field generated by iplocation).
b
| geostats count by Country
Geostats works in the same way as stats
does.
Step 3. Visualize the above information on a map in two simple steps.

Nice Geo Maps within two minutes, hover

over the circles.
c
Play around and create some more complex
queries.

Page 25
 2.2. Indexing syslog events from a Linux OS (no UF)

2.2.1. Linux syslog configuration >Linux Machine

Step 1. SSH into the Linux/Unix machine that has been assigned to you and follow the commands are they are shown below.

$ssh splunk@linux_machine
$sudo vim /etc/rsyslog.conf

Add the following lines in the end of the file and save the file.

#Sending syslog data to Splunk

authpriv.* @<splunk_sh_private_ip>:5514

Note: Replace the IP with the one of your own Splunk instance; keep in mind the Private IP

$sudo service rsyslog restart

2.2.2. syslog Input configuration >Splunk SH

In this section we will cover how to enable the receiver on the Splunk instance in order to be listening for incoming data.

Step 1. Go to Settings > Data Inputs in order to configure your input type.

Step 2. Choose Add New next to UDP in order to create a UDP listener.

Page 26
Step 3. Choose the UDP option, enter port 514 and input the IP that you will be sending the data from, and press Next.

Make sure you put the Linux Private IP (not

Public)

Step 4. Configure the correct input settings; here you will choose for sourcetype linux_secure (under operating Systems), the
Searching & Reporting application as App Context and Method IP. Lastly, you will choose the Default Index; Press Next.

Step 5. Review the Settings in the existing page and press Submit. Do you see an error? [why?] Go back to Select Source and
modify the port from 514 to 5514; Press Next until done.

Page 27
 5

Step 2. Go to the Search Application (you may need to generate some login data in the Linux machine).

Step 3. Search for index=* sourcetype=linux_secure and press [Enter].

source port is udp:5514 &

sourcetype linux_secure

Step 4. Is something missing from these events?

Do you see any fields with source

4
IPs? or username?

Page 28
Step 5. By Downloading the UNIX & Linux add-on, it will help us to generate the interesting fields – Go to App > Find More Apps

Step 6. Search for the “Splunk Add-on for Unix and Linux” (just type in unix in the search box) and press search.

Step 7. Enter your Splunk credentials that you created in section 1 and proceed with the download and installation.

Credentials you created in the first

section are entered here.

Don’t forget to accept the Terms and

Conditions

Page 29
Step 8. Restart Splunk in order for the changes to take effect and login again.

Step 9. Search for the linux access logs again and see the difference in the interesting fields.

We can now see user and other interesting

fields (schema on the fly!)

Page 30
 2.3. Indexing a scripted input on a Linux OS (with UF)

2.3.1. Installing and configuring a Linux UF >Linux Machine

In this section we will cover how you will install a Universal Forwarder on our Linux machine and how we are able to bring in
data from a scripted input, which in our case will be the ps command.

Step 1. You would normally download the Splunk UF from the Splunk website (www.splunk.com), but for the sake of not
wasting time, we have done that for you, and have placed it under /home/splunk/software.

$cd /home/splunk/software
$sudo rpm -i splunkforwarder-6.3.2-aaff59bb082c-linux-2.6-x86_64.rpm

Step 2. The next step after installing splunk is starting it up the daemon.

$/opt/splunkforwarder/bin/splunk start | start the splunk daemon and accept the license
$sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk | start the splunk daemon on every start-up
$/opt/splunkforwarder/bin/splunk add forward-server <private_SH_IP>:9997 -auth admin:changeme | forward all data to
SH
$/opt/splunkforwarder/bin/splunk restart

Note: Do not forget to replace the IP with the Private IP of your Splunk SH.

Step 3. Copy the Unix/Linux TA located under /opt/software to /opt/splunkforwarder/etc/apps.

$cp -r /home/splunk/software/Splunk_TA_nix /opt/splunkforwarder/etc/apps

$cd /opt/splunkforwarder/etc/apps/Splunk_TA_nix
$mkdir local
$cp /opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local
$vim /opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf

Find the line that starts with [script://./bin/ps.sh] and modify the following:

disabled = 1 → disabled = 0

$/opt/splunkforwarder/bin/splunk restart

2.3.2. Configuring Receiving on Splunk >Splunk SH

Step 1-3. On the Splunk instance enable receiving on port 9997.

Page 31
 2

2.3.3. Searching and reporting on our scripted data > Splunk SH

In this section we will discuss how you can bring in data that is in tabular format from the forwarder and extract those fields and
manipulate them. In our case we have brought in data from the ps command, but the same exercise can be performed for any
type of data in Linux/Unix that has output in a table format.

Step 1. For this exercise we will be using the sourcetype ps that we indexed in section 3.3.1.

index=* sourcetype=ps

Minimizing the search time can help you

a more accurate understanding of what
1 is occurring on your system now.

Fields Parsed from the ps

command

Page 32
Step 2. Manipulating data coming in with a tabular format can be done with the multikv command.

index=* sourcetype=ps
| multikv fields USER, cpu_load_percent, ELAPSED, COMMAND, ARGS
| table USER, cpu_load_percent, ELAPSED, COMMAND, ARGS
| sort -cpu_load_percent

Task: Modify your search and rename USER as “User”, CPUTIME as “CPU Time”, COMMAND as “Command Used”, and ARGS as
“Argument”.

2.4. Windows UF installation and Event monitoring

2.4.1. Windows UF Installation > Windows

In this section we will cover how you will install a Universal Forwarder on our Windows machine and how we are able to bring in
data from Application, System, Security.

Connect via RDP with the Public IP (If you have a MAC use the Microsoft Remote Desktop).

Note: For the ease of this exercise we have already downloaded the UF and have placed it under c:\software\

Step 1. Locate the software under c:\Software\ and double on the splunkforwarder-6.3.2-aaff59bb082c-x64-release to initiate
the installation.

Step 2. Check the appropriate settings as shown below & choose “Customize Options”.

Page 33
Step 3. Choose the installation path (unless you have explicit requirements leave default) and press Next.

Step 4. For our training purposes leave the settings empty and press Next.

Step 5. Choose the “Local System” as installation method.

Page 34
Step 6. For this lab we will monitor Application, Security & System Logs.

Step 7. Tell the installer to install the Windows TA together with the forwarder (Do you remember what a TA is?).

Step 8. Skip this step and press Next. [would you like to know what a deployment server does?]

Leave Empty & Press Next

Page 35
Page 36
Step 9. Input the IP and port of your assinged instance (or in Splunk terminology an indexer).

Enter Private IP of SH

9
Step 10. Install & Check the installation directory.

10

Step 11. Navigate to C:\Program Files\SplunkUniversalForwarder

Note: All Splunk log files are stored under C:\Program Files\SplunkUniversalForwarder\var\log\splunk.

Page 37
 2.5. Searching & Representing with Windows > Splunk
Step 0. Do you see any errors under Messages? Before you begin indexing data from the windows UF you will first have to
create an index that Splunk is expecting “wineventlog”.

Leave the rest empty or

default options

Page 38
Step 1. Having some good searching fun – Type in index=* and press enter, you should see all the logs that your instance is
receiving. [Do you want to know of a different way to search all indexes?]

Step 2. Try looking only for Security Windows Event Logs (Hint: Sourcetype=WinEventLog:Security).

Page 39
Step 3. Search for Windows Logons (Hint: EventCode=4624) – Do you know what these codes mean?
(https://www.ultimatewindowssecurity.com/securitylog/encyclopedia/Default.aspx).

Step 4. Next step will be to represent in a tabular format the following fields: src, Account_Name, dest.
Spaces between searches represent a
4
logical AND

index=* sourcetype="WinEventLog:Security" EventCode=4624 | table src, Account_Name, app, dest

First part is the Search & Second part after the | is

the additional representation of the data

Step 5. Represent the fields in a more “meaningful” way, thus we will rename the fields (Hint: Pay attention to the | symbol).

index=* sourcetype="WinEventLog:Security" EventCode=4624

| table src, Account_Name, app, dest
| rename src as Source, Account_Name as "Account Name", app as "Login Type", dest as "Logon Source"

Note: To start a new line press Shift+Enter

Step 6. We are not interested in seeing any account names that end with a $ sign, thus we will filter those out.

index=* sourcetype="WinEventLog:Security" EventCode=4624 NOT Account_Name="*$"

| table src, Account_Name, app, dest
| rename src as Source, Account_Name as "Account Name", app as "Login Type", dest as "Logon Source"

Page 40
Step 7. Identify multiple failed login attempts by a single username (modify the search and try from a single source IP).

index=* sourcetype="WinEventLog:Security" EventCode=4625 NOT Account_Name="*$"

| stats count by Account_Name
|where count > 5

Task:
a. Add the data to a dashboard named “Windows Failed Logins”.
b. Create a Bar chart with the top 10 failed login IPs (hint: keyword top or maybe a sub command within sort) –
docs.splunk.com
c. Sort this data in a descending manner

Page 41