Scribd
Upload
423 views

12.2.1.5 Lab - Convert Data Into A Universal Format

The document contains log files and directory listings from a Security Onion network security monitoring system. It shows logs from applications like Bro, Snort and Elsa as well as system logs. The files are stored under directories like /nsm, /var/log/nsm and within application-specific subdirectories. Commands like ls, cd and cat are used to view the files and directories.

Uploaded by

Anna Shin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
423 views

12.2.1.5 Lab - Convert Data Into A Universal Format

The document contains log files and directory listings from a Security Onion network security monitoring system. It shows logs from applications like Bro, Snort and Elsa as well as system logs. The files are stored under directories like /nsm, /var/log/nsm and within application-specific subdirectories. Commands like ls, cd and cat are used to view the files and directories.

Uploaded by

Anna Shin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 11

Log entries

log file

normalization
2|Z|1219071600|AF|0
3|N|1219158000|AF|89
4|N|1220799600|AS|12
1|Z|1220886000|AS|67
5|N|1220972400|EU|23
6|R|1221058800|OC|89

o
o
o

[analyst@secOps ~]$ cd ./lab.support.files/


[analyst@secOps lab.support.files]$ ls -l
total 580
-rw-r--r-- 1 analyst analyst 649 Jun 28 18:34 apache_in_epoch.log
-rw-r--r-- 1 analyst analyst 126 Jun 28 11:13 applicationX_in_epoch.log
drwxr-xr-x 4 analyst analyst 4096 Aug 7 15:29 attack_scripts
-rw-r--r-- 1 analyst analyst 102 Jul 20 09:37 confidential.txt
<output omitted>
[analyst@secOps lab.support.files]$

[analyst@secOps lab.support.files]$ awk 'BEGIN


{FS=OFS="|"}{$3=strftime("%c",$3)} {print}' applicationX_in_epoch.log
2|Z|Mon 18 Aug 2008 11:00:00 AM EDT|AF|0
3|N|Tue 19 Aug 2008 11:00:00 AM EDT|AF|89
4|N|Sun 07 Sep 2008 11:00:00 AM EDT|AS|12
1|Z|Mon 08 Sep 2008 11:00:00 AM EDT|AS|67
5|N|Tue 09 Sep 2008 11:00:00 AM EDT|EU|23
6|R|Wed 10 Sep 2008 11:00:00 AM EDT|OC|89
||Wed 31 Dec 1969 07:00:00 PM EST
[analyst@secOps lab.support.files]$

[analyst@secOps lab.support.files]$ nano applicationX_in_epoch.log


[analyst@secOps lab.support.files]$ awk 'BEGIN
{FS=OFS="|"}{$3=strftime("%c",$3)} {print}' applicationX_in_epoch.log >
applicationX_in_human.log
[analyst@secOps lab.support.files]$

[analyst@secOps lab.support.files]$ cat applicationX_in_human.log


2|Z|Mon 18 Aug 2008 11:00:00 AM EDT|AF|0
3|N|Tue 19 Aug 2008 11:00:00 AM EDT|AF|89
4|N|Sun 07 Sep 2008 11:00:00 AM EDT|AS|12
1|Z|Mon 08 Sep 2008 11:00:00 AM EDT|AS|67
5|N|Tue 09 Sep 2008 11:00:00 AM EDT|EU|23
6|R|Wed 10 Sep 2008 11:00:00 AM EDT|OC|89
[analyst@secOps lab.support.files]$

[analyst@secOps lab.support.files]$ cat apache_in_epoch.log


198.51.100.213 - - [1219071600] "GET
/twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.ConfigurationVariables
HTTP/1.1" 401 12846
198.51.100.213 - - [1219158000] "GET
/twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.1" 200 4523
198.51.100.213 - - [1220799600] "GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
198.51.100.213 - - [1220886000] "GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200
7352
198.51.100.213 - - [1220972400] "GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200
5253
198.51.100.213 - - [1221058800] "GET
/twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore&m1=1.12&m2=1.12 HTTP/1.1"
200 11382

o
o
o
o

o
o

[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS="


"}{$4=strftime("%c",$4)} {print}'
/home/analyst/lab.support.files/apache_in_epoch.log

[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS=" "}


{gsub(/\[|\]/,"",$4)}{print}{$4=strftime("%c",$4)}{print}'
apache_in_epoch.log
o

[analyst@secOps lab.support.files]$ awk 'BEGIN {FS=OFS="


"}{gsub(/\[|\]/,"",$4)}{print}{$4=strftime("%c",$4)}{print}'
apache_in_epoch.log
analyst@SecOnion:~/Desktop$ cd /nsm/elsa/data/elsa/log
analyst@SecOnion:/nsm/elsa/data/elsa/log$

analyst@SecOnion:/nsm/elsa/data/elsa/log$ ls -l
total 99112
total 169528
-rw-rw---- 1 www-data sphinxsearch 56629174 Aug 18 14:15 node.log
-rw-rw---- 1 www-data sphinxsearch 6547557 Aug 3 07:34 node.log.1.gz
-rw-rw---- 1 www-data sphinxsearch 7014600 Jul 17 07:34 node.log.2.gz
-rw-rw---- 1 www-data sphinxsearch 6102122 Jul 13 07:34 node.log.3.gz
-rw-rw---- 1 www-data sphinxsearch 4655874 Jul 8 07:35 node.log.4.gz
-rw-rw---- 1 www-data sphinxsearch 6523029 Aug 18 14:15 query.log
-rw-rw---- 1 www-data sphinxsearch 53479942 Aug 18 14:15 searchd.log
-rw-rw---- 1 www-data sphinxsearch 32613665 Aug 18 14:15 web.log
analyst@SecOnion:/nsm/elsa/data/elsa/log$

analyst@SecOnion:/nsm/elsa/data/elsa/log$ cd /nsm/bro/logs/current
analyst@SecOnion:/nsm/logs/current$

analyst@SecOnion:/nsm/bro/logs/current$ ls -l
total 100
-rw-rw-r-- 1 sguil sguil 368 Aug 18 14:02 capture_loss.log
-rw-rw-r-- 1 sguil sguil 46031 Aug 18 14:16 communication.log
-rw-rw-r-- 1 sguil sguil 2133 Aug 18 14:03 conn.log
-rw-rw-r-- 1 sguil sguil 2028 Aug 18 14:16 stats.log
-rw-rw-r-- 1 sguil sguil 40 Aug 18 14:00 stderr.log
-rw-rw-r-- 1 sguil sguil 188 Aug 18 13:46 stdout.log
analyst@SecOnion:/nsm/bro/logs/current$

analyst@SecOnion:/nsm/bro/logs/current$ cd /nsm/sensor_data
analyst@SecOnion:/nsm/sensor_data$

analyst@SecOnion:/nsm/sensor_data$ ls -l
total 16
drwxrwxr-x 7 sguil sguil 4096 Jun 19 23:16 seconion-eth0
drwxrwxr-x 7 sguil sguil 4096 Jun 19 23:16 seconion-eth1
drwxrwxr-x 7 sguil sguil 4096 Jun 19 23:16 seconion-eth2
drwxrwxr-x 5 sguil sguil 4096 Jun 19 23:08 seconion-eth3
analyst@SecOnion:/nsm/sensor_data$

analyst@SecOnion:/nsm/sensor_data$ ls -l seconion-eth0/
total 52
drwxrwxr-x 2 sguil sguil 4096 Jun 19 23:09 argus
drwxrwxr-x 10 sguil sguil 4096 Jul 7 12:09 dailylogs
drwxrwxr-x 2 sguil sguil 4096 Jun 19 23:08 portscans
drwxrwxr-x 2 sguil sguil 4096 Jun 19 23:08 sancp
drwxr-xr-x 2 sguil sguil 4096 Jul 7 12:12 snort-1
-rw-r--r-- 1 sguil sguil 27566 Jul 7 12:12 snort-1.stats
-rw-r--r-- 1 root root 0 Jun 19 23:08 snort.stats
analyst@SecOnion:/nsm/sensor_data$
analyst@SecOnion:/nsm/sensor_data$ cd /var/log/nsm/
analyst@SecOnion:/var/log/nsm$ ls -l
total 8364
-rw-r--r-- 1 sguil sguil 4 Aug 18 14:56 eth0-packets.log
-rw-r--r-- 1 sguil sguil 4 Aug 18 14:56 eth1-packets.log
-rw-r--r-- 1 sguil sguil 4 Aug 18 14:56 eth2-packets.log
-rw-r--r-- 1 sguil sguil 182 Aug 18 13:46 ossec_agent.log
-rw-r--r-- 1 sguil sguil 202 Jul 11 12:02 ossec_agent.log.20170711120202
-rw-r--r-- 1 sguil sguil 202 Jul 13 12:02 ossec_agent.log.20170713120201
-rw-r--r-- 1 sguil sguil 202 Jul 14 12:02 ossec_agent.log.20170714120201
-rw-r--r-- 1 sguil sguil 202 Jul 15 12:02 ossec_agent.log.20170715120202
-rw-r--r-- 1 sguil sguil 249 Jul 16 12:02 ossec_agent.log.20170716120201
-rw-r--r-- 1 sguil sguil 202 Jul 17 12:02 ossec_agent.log.20170717120202
-rw-r--r-- 1 sguil sguil 202 Jul 28 12:02 ossec_agent.log.20170728120202
-rw-r--r-- 1 sguil sguil 202 Aug 2 12:02 ossec_agent.log.20170802120201
-rw-r--r-- 1 sguil sguil 202 Aug 3 12:02 ossec_agent.log.20170803120202
-rw-r--r-- 1 sguil sguil 202 Aug 4 12:02 ossec_agent.log.20170804120201
-rw-r--r-- 1 sguil sguil 42002 Aug 4 07:33 pulledpork.log
drwxr-xr-x 2 sguil sguil 4096 Aug 18 13:46 seconion-eth0
drwxr-xr-x 2 sguil sguil 4096 Aug 18 13:47 seconion-eth1
drwxr-xr-x 2 sguil sguil 4096 Aug 18 13:47 seconion-eth2
drwxr-xr-x 2 sguil sguil 4096 Jun 19 23:08 securityonion
-rw-r--r-- 1 sguil sguil 1647 Jun 19 23:09 securityonion-elsa-config.log
-rw-r--r-- 1 sguil sguil 7708106 Aug 18 14:56 sensor-clean.log
-rw-r--r-- 1 sguil sguil 1603 Aug 4 00:00 sensor-newday-argus.log
-rw-r--r-- 1 sguil sguil 1603 Aug 4 00:00 sensor-newday-http-agent.log
-rw-r--r-- 1 sguil sguil 8875 Aug 4 00:00 sensor-newday-pcap.log
-rw-r--r-- 1 sguil sguil 53163 Aug 4 05:01 sguil-db-purge.log
-rw-r--r-- 1 sguil sguil 369738 Aug 4 07:33 sid_changes.log
-rw-r--r-- 1 sguil sguil 22598 Aug 8 01:35 so-bro-cron.log
drwxrwxr-x 2 sguil securityonion 4096 Jun 19 23:09 so-elsa
-rw------- 1 sguil sguil 7535 Jun 19 23:09 sosetup.log
-rw-r--r-- 1 sguil sguil 14046 Jun 19 23:09 sosetup_salt_call.log
-rw-r--r-- 1 sguil sguil 63208 Jun 19 23:09 sphinx_initialization.log
-rw-r--r-- 1 sguil sguil 81 Aug 18 14:55 squert-ip2c-5min.log
-rw-r--r-- 1 sguil sguil 1079 Jul 16 06:26 squert-ip2c.log
-rw-r--r-- 1 sguil sguil 125964 Aug 18 14:54 watchdog.log
analyst@SecOnion:/var/log/nsm$

You might also like