Industrial Control System Internal Network Security

Monitoring with Open-Source Tools

Author: Charles Carroll, cwcarroll@outlook.com

Advisor: Dr. Tim Proffitt

Accepted: 03/29/2024

Abstract

Security vendors have made many advances in internal network security monitoring
(INSM) in recent years. Numerous vendors have developed specialized platforms that
provide industrial control system (ICS) security teams with detailed network visibility.
However, the cost of these platforms is prohibitive to smaller electric distribution
providers and cooperatives. This scenario leaves these smaller but critical systems
vulnerable to cybersecurity threats. While IT teams regularly monitor security, ICS
systems have lagged in adopting centralized security monitoring. However, ICS security
teams could safely leverage many well-established open-source tools in their
environment to provide essential network security monitoring and asset visibility.

This research will demonstrate how INSM monitoring could be performed by ICS
security teams using a single monitoring platform with multiple open-source tools,
including Zabbix, OpenSearch, and Suricata, and deployed in Docker containers. These
services provide asset inventory, intrusion detection, and server health monitoring with
minimal hardware requirements. This research proves that while essential monitoring is
possible, it is critical to understand the network protocols and device limitations to build
a comprehensive monitoring strategy.
 Performing Internal Network Security Monitoring with Open-Source Tools 2

1. Introduction
In January 2023, the Federal Energy Regulatory Commission (FERC) issued
Order No. 887. This order directed the North American Reliability Corporation (NERC)
to expand the scope of required network monitoring beyond the electronic security
perimeter (ESP) and into the local area network (LAN) or internal network security
monitoring (INSM) (North American Reliability Corporation, 2024). While this
requirement will initially apply only to high-impact and medium-impact systems with
external routable connectivity (ERC), many low-impact and medium-impact systems will
evaluate their ability to implement similar controls. Larger entities, such as those the
requirement initially applies to, can budget for specialized monitoring systems such as
Dragos, Nozomi, or Microsoft Defender for IoT. However, these solutions will likely be
cost-prohibitive for smaller entities such as electric cooperatives and distribution
providers. Consequently, they will continue to operate with little or no network
monitoring in their critical infrastructure.

The lack of asset and network visibility in some of the most critical infrastructure
in the United States is troubling. While network monitoring is commonplace in IT
systems, the electric power sector has lagged significantly in implementing any internal
asset monitoring due in part to the lack of standardization in the environments (Sanchez,
2022). Rather than relying on regulation to force these smaller entities to enact INSM via
compliance or continuing to react after an impact on their system, it could be beneficial to
leverage existing open-source packages that one can deploy as a single monitoring sensor
into the network for this task. This research aims to build a functional, easily deployable
INSM system utilizing existing open-source tools to provide basic network traffic
inspection and asset inventory monitoring. As Jason Christopher mentions in his recent
white paper, while IT enterprise networks have focused heavily on network and asset
visibility but have not typically extended into the ICS networks, this information is
critical in the ability to defend a system and to respond in the case of an incident
(Christopher, 2023). The combination of open-source tools will comprise a platform to
provide a basic network intrusion detection system (IDS) and asset monitoring to
facilitate visibility into LAN assets and traffic.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 3

2. Research Method
For the research, the INSM platform will be configured to monitor lab devices
and test scenarios will be performed to validate the system's baseline and security
monitoring. The primary focus of the testing will be on field devices such as relays and
meters. These device types typically lack security monitoring because they do not
provide typical monitoring interfaces. The research will endeavor to validate the
effectiveness of the monitoring solution and describe novel methods of critical data
retrieval and monitoring.

2.1. Lab Environment

The INSM platform should perform two primary functions. First is basic asset
inventory and monitoring; the platform should be able to discover assets and enrich them
with asset information. Second, the platform should provide a basic network IDS
function. Both functions should require the least computing resources possible and be
easily deployable. A small lab with representative equipment and interfaces commonly
used in a substation environment was developed to test the toolset's ability to meet these
requirements.

2.1.1. Lab Equipment and Interfaces

Most of the equipment found in a modern substation are intelligent electronic
devices (IEDs). These are typically embedded platforms that perform the system's critical
monitoring, control, and logic calculations. Embedded platforms do not expose the
underlying operating system to the user. They are challenging to monitor as they cannot
support agent-based monitoring. It is also typical that control systems use commercial-
off-the-shelf (COTS) operating systems for interfacing with IEDs. These are typically
Microsoft Windows or Linux platforms that would host engineering software, human-
machine interface (HMI) software, and web browsers. The lab comprises several
representative ICS devices that are network-connected via a single RUGGEDCOM
Ethernet switch. To ensure the completeness of the overall solution, the testing
encompasses monitoring the network infrastructure and typical COTS devices.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 4

Each of these devices presents various interfaces for interaction with the user.
This experiment utilizes several interfaces representing a typical substation environment;
details are provided in Table 2.1.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 5

Device Interfaces Details

SEL-3555 REST API The SEL RTAC is a multifunctional platform that can act as a
remote terminal unit (RTU), logic processor, and protocol
converter (Schweitzer Engineering Laboratories, 2024). It is a
very flexible platform that supports many protocols and offers
numerous interfaces. The experiment will utilize the built-in
REST API.

SEL-751 TELNET The SEL-751 is a feed protection relay (Schweitzer Engineering

Laboratories, 2024).

SEL-735 TELNET The SEL-735 is a high-accuracy power quality meter

(Schweitzer Engineering Laboratories, 2024).

Siemens SNMP The RSG2100 is an industrially hardened, fully managed

RUGGEDCOM Ethernet switch (Siemens, 2024).
RSG2100 These switches provide the Ethernet network for the system
under test and a port mirror to the monitoring platform.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 6

SEL-3360 Agent-Based The SEL-3360 is a substation-hardened computing platform that

Compact Monitoring can run various operating systems (Schweitzer Engineering
Rugged Laboratories, 2024)
Computer This application uses this hardware as an engineering
(Engineering workstation utilizing Microsoft Windows Server 2016.
Workstation)

SEL-3355 Agent-Based The SEL-3355 is a substation-hardened computing platform that

Rugged Server Monitoring can run various operating systems (Schweitzer Engineering
Laboratories, 2024).

This hardware is used as the application's monitoring platform

and uses the Ubuntu 22.04 operating system with Docker
software.

Table 2.1: Devices and Interfaces

A Caldera virtual machine is deployed to the network to provide an automated

platform to test the IDS capabilities of the INSM platform. Caldera is a research project
at MITRE that provides an adversary emulation platform built on the MITRE ATT&CK
framework (MITRE, 2024).

SEL

1 SEL-3355 2
INSM Platform 1
Caldera Virtual Server
Mirror Port

1 2 3 4 5 9 11
RUGGEDCOM SWITCH

SEL

SEL
SEL-735 SEL
SEL-751 SEL-3360 1
Engineering
1 1 Workstation
SEL
RTAC
Meter Feeeder 1
Relay
Automation Controller

Figure 2.1: Test Lab Diagram

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 7

2.2. Monitoring Platform

Beyond the essential asset monitoring and IDS function, the INSM platform must
be easy to deploy. The INSM platform is built in Docker and from a single docker-
compose file (there are other supporting Docker files and configuration files, but the
system is built from a single docker-compose file). Details on docker-compose and
associated files and scripts referenced in this research can be found in the GitHub
repository: https://github.com/basswrench/INSM. Docker is a container technology that
allows building applications into a single package isolated from the operating
environment (Docker, 2024). This method ensures the platform can be deployed with
minimum changes and repeated once developed. One of the critical aspects of the
development is ensuring that data persist through the build-up and tear-down of the
Docker containers; this is achieved through various Docker volume mounts, as noted in
the compose file.

2.2.1. Applications
The two primary applications that make up the INSM monitoring stack are Zabbix
and Suricata. Zabbix is an open-source, all-in-one monitoring solution advertised as
capable of monitoring anything (Zabbix, 2024). Zabbix provides baseline server health
information, centralized asset information, and essential network monitoring for this use
case. The Zabbix images used for this experiment are pulled directly from the Docker
hub using the official Zabbix images (Zabbix, 2024). Zabbix is ideal for this solution
because it provides a wide range of interface compatibility required for the diverse
systems under test.

Suricata is a well-known IDS platform maintained by the Open Information

Security Foundation (OISF) and is an integral part of numerous other network monitoring
platforms (Suricata, 2024). Suricata is limited in support of industrial protocols but does
include parsers for ENIP/CIP, DNP3, and MODBUS. The INSM platform will use the
docker-Suricata container built and maintained by Jason Ish, available on github.com
(Ish, 2024). Suricata is the platform used for this experiment because of its simple
interface, flexibility in writing custom rules, and ease of deployment.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 8

Several other applications are required to support the base functionality of Zabbix
and Suricata. The complete list of applications is detailed in Table 2.2.

Application Docker Image Details

Source

OpenSearch opensearch:1.0.1 OpenSearch and the OpenSearch dashboard are derived from
Elasticsearch and Kibana but are open-source (OpenSearch
Project, 2024). OpenSearch is the search engine daemon.

OpenSearch OpenSearch- This application provides a graphical user interface for

dashboards:1.0.1
Dashboards viewing and searching data in the OpenSearch platform
(OpenSearch project, 2024)

Fluent Bit fluent-bit The fluent-bit container is a lightweight log processor that
sends data to OpenSearch (Fluentd project, 2024). Log
directories to ship to OpenSearch are mounted as volumes in
the fluent-bit container.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 9

MySQL mysql:8.1 MySQL, a popular database platform, provides the backend

database for the Zabbix server (MySQL, 2024). This
application uses version 8.1, which is the latest supported by
Zabbix

Zabbix Web Zabbix-web- This application provides the frontend web interface for
Interface nginx- Zabbix servers that use MySQL backend; the web interface is
mysql:alpine-6.4- built on Nginx (Zabbix, 2024).
latest

Zabbix Agent Zabbix- The Zabbix agent is installed on a target to monitor (Zabbix,
agent:alpine-6.4- 2024). In this experiment, the agent is deployed as a container
latest to monitor the Zabbix server instance and installed locally on
the EWS.

Zabbix Server Zabbix-server- This is the primary Zabbix process; this image is built on
MySQL:alpine- Alpine Linux with support for the MySQL backend (Zabbix,
6.4-latest 2024).

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 10

Suricata suricata:latest Suricata is deployed in a stand-alone docker container that

attaches to the host network stack. Jason Ish developed and
maintained this container image (Ish, 2024).

Table 2.2: INSM Applications

3. Test Scenarios
The following test scenarios will test the basic system functionalities and identify
limitations. The asset monitoring section will focus on the ability to retrieve critical
operational data and enrichment of that data. The attack simulation portion will focus on
the platform's ability to identify basic attack patterns based on MITRE ATT&CK and
prevalent threat groups. The test will also identify any specific tuning or modifications to
the platform components required for successful detections.

3.1. Asset Inventory Monitoring

While there are numerous asset monitoring solutions for the IT space, most of the
solutions for ICS are vendor-specific. For this use case, a solution must provide detailed
asset inventory information about IT-type equipment (server health, disk usage, CPU)
and OT equipment such as relays and controllers. Each device in the lab uses a different
interface type based on what the device supports and would likely be included/found in a
production environment. An important distinction between the devices in the following
sections is that while the controllers, network equipment, and computers all support
typical SYSLOG implementation for security event logging, the relays and meters or
IEDs do not.

3.1.1. Relays and Meters

TELNET is the primary ethernet-based interface for many relays and meters
manufactured by Schweitzer Engineering Laboratories (SEL). Although TELNET is not
typically present in the IT network, it is still used in most control systems. Zabbix has a

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 11

built-in TELNET agent for performing simple checks; however, this method would not
work for this application because the TELNET server needs to provide a login prompt.
The SEL relay and meter used in the test do not provide a prompt; when a client
successfully connects to the server, the client is presented with a blank screen, as shown
below.

Figure 3.1: Connecting to the server

Upon a successful connection, the client can run a limited number of

informational commands but must authenticate to a high permission-level group to make
configuration changes or control actions (Schweitzer Engineering Laboratories, 2024).
The primary information to retrieve for this test was the ID command. This command
prints the details of the relay to the TELNET session and would be critical baseline
information for our asset records (see Figure 3.2.).

Figure 3.2: ID Command

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 12

Because the built-in Zabbix TELNET agent would not work for this application,
two options remain to move this data into Zabbix. The first is to use the "External
Checks" item to allow the Zabbix server to call an external script to spawn a TELNET
session and issue the ID command to send the data back to the Zabbix server. The second
option is to run the command elsewhere, such as on the Docker server itself, and have it
use the “Zabbix-sender” to send the data to the “Zabbix-trapper” for ingest into the
platform. Both methods were experimented with, but the second method was ultimately
successful. Data is imported into Zabbix using the bash script on the server, and the
“expect” command is combined with a regular expression to automate the data retrieval
and package the data in the format required by Zabbix.

Figure 3.3: ID Bash Script

This script will run as a Cron job on the server and then send the data to Zabbix.
The screenshot below demonstrates the successful import of data into Zabbix.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 13

Figure 3.4: ID Retrieved by Zabbix

Using this method, anything accessible from the TELNET interface of the
connected devices can be retrieved and viewed within the Zabbix container.

Another critical missing piece of information from a monitoring perspective is the

lack of SYSLOG to provide security alerts to a central server. However, engineers can
configure these particular IEDs with host event logging called sequential event records
(SER), typically used to track system events. Additionally, engineers can also configure
specific SERs to identify device configuration changes and security events such as
incorrect passwords and privileged access. To test this, the relay settings file is modified
to record the SALARM SER.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 14

Figure 3.5: IED SALARM Configuration

It should be noted that the SALARM word bit can indicate several events; it is not
explicit, meaning that an invalid password will trigger the same event that a setting
change will. However, as seen in Figure 3.6 below, a default SER is also logged for
settings changes for this relay model.

Figure 3.6: Relay SER Logging

The next step is to move this data into the monitoring platform. One potential
method is to retrieve this data as a log file and have Zabbix monitor it. The other option is
to bring this data directly into OpenSearch as SYSLOG. To test this, a simple Python
script was created to access this device from a read-only telnet session and convert the
SER to SYSLOG. With this method, the SERs are automatically retrieved and indexed
into our OpenSearch instance.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 15

Figure 3.7: SALARM indexed to OpenSearch

3.1.2. Controllers and RTUs

Controllers and RTUs differ significantly in interface support; some vendors
provide a standard Linux SSH interface, while others require specialized software. The
device in this experiment is a SEL-3555 RTAC. The RTAC exposes an API to query for
many device and system parameters.

Figure 3.8: RTAC API Reference

For this test, the administrator configures the Zabbix server to use the built-in
HTTP agent to perform an API query asking for the device "Id" information. The
response from this request will be JSON formatted and include information such as the
device serial number, BIOS version, and feature set. To do this, a web interface on the
host is specified for the Zabbix server to query.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 16

Figure 3.9: RTAC Zabbix Interface Configuration

Then, the administrator creates an item in Zabbix using basic authentication to

perform the query.

Figure 3.10: Zabbix API Configuration

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 17

It can now be verified that Zabbix is successfully receiving the RTAC asset data.

Figure 3.1l: RTAC Asset Data

3.1.3. Network Switches

The approach to network monitoring from Zabbix depends on the switch vendor
used. Zabbix offers templates for over three hundred network devices and the ability to
deploy network-based Zabbix agents (Zabbix, 2024). However, the switch used in the lab
is a legacy Siemens RUGGEDCOM RSG 2100, for which no template is available.

Zabbix provides a built-in SNMP agent used in the experiment to monitor the lab
switch by configuring the SNMP agent in the RUGGEDCOM host and the SNMP item in
the Zabbix server.

Figure 3.12: Zabbix SNMP Configuration

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 18

The INSM platform uses a generic SNMP template provided by Zabbix to

normalize the retrieved information.

Figure 3.13: RSG 2100 SNMP Monitoring in Zabbix

3.1.4. Engineering Workstations and Servers

The value of the Zabbix platform is the most obvious for this monitoring scenario.
Zabbix has numerous pre-built templates for monitoring most operating systems and
server platforms, and the instructions for doing this are well-documented (Zabbix, 2024).
The existing templates and the ability to use agent-based monitoring provide a rich
contextualized data set.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 19

Figure 3.14: Server Monitoring in Zabbix

The template is also pre-configured with graphical representations of the data.

This data can all be viewed per host or mapped into the dashboard, as seen in Figure 3.15.

Figure 3.15: Server Monitoring Dashboard

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 20

3.2. Attack Simulations

The second part of the experiment is to evaluate the Suricata application in the lab
environment and determine if the level of network monitoring is sufficient to provide an
alert of malicious activity. The research utilized a Caldera virtual machine deployed in
the lab for the IDS test. The Caldera agent is deployed from the server to the engineering
workstation. The test in the following groups follows the attack patterns of named
adversary groups; however, since the Caldera platform does not include all ICS adversary
groups, the experiments will focus on the tactics these groups are known to use. Each
attack scenario will performed twice, and the experiment will analyze whether or not
each detection is successful using the INSM platform.

3.2.1. An Environment Aware Approach

A security operator's advantage in an ICS environment, especially in the electric
power sector, is that these networks are very static. Typically, once commissioned, these
networks will rarely change. This default network posture allows monitoring these
networks to be precisely tuned to alert on unexpected ports and services. This type of
monitoring would be cumbersome to manage in very dynamic networks, but this can be
easily accomplished in the typical substation environment.

Since many ICS devices use unencrypted protocols for management, it is possible
to easily create a few custom rules that alert specific network behavior. For instance, the
SEL-735 meter and SEL-751A relay have two access levels and corresponding access
groups for this use case. The access group "acc" provides interactive read access, and the
access group "2ac" provides access to make device setting changes. From a network or
security monitoring perspective, it is critical that attempted "2ac" access and
authenticated "2ac" should be logged. Because the devices use TELNET as their network
protocol for interactive access, writing Suricata rules to alert them to this is trivial.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 21

Figure 3.16: IED Privileged User Login

As demonstrated in Figure 3.16, when the user attempts to elevate to "2ac," they
enter that into the terminal, and when they are successful, they receive a banner stating,
"Level 2." Suricata can capture this activity by creating a few custom rules to alert people
about this attempted and eventual successful access.

alert tcp any 23 -> any any (msg: "PRIVILEGED ACCESS TO IED"; content:
"Level 2"; nocase; sid:1000001; rev:1;)

alert tcp any any -> any 23 (msg:"2AC ATTEMPTED"; content:"2ac"; nocase;
sid:1000002; rev:1;)

The first rule will alert when the "Level 2" banner is returned from the TELNET server,
and the second rule will alert when the "2ac" command is sent to the server. The
successful alerts in OpenSearch are shown below.

Figure 3.17: Suricata Privilege Elevation Alerts

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 22

Alerts can be further expanded by creating a ruleset based on the known traffic
patterns in the network. For instance, because it is known that a device should only be
using a specific port and protocol for communication, it is logical that the platform
should generate an alert for any deviation from that pattern. In Suricata, accomplishing
this baseline alerting involves creating a list of variables representing the network's host
profile and alerting any deviation. For the following tests, Suricata is configured with
some basic rules to support this approach.

alert ip 192.168.0.60 any -> ![192.168.0.54, 192.168.0.56] !23 (msg: "Alert

unexpected traffic from EWS"; sid:1000004; rev:1;)

alert ip 192.168.0.53 any -> ![192.168.0.54, 192.168.0.56] ![23, 502] (msg: "Alert
on non-Telnet and non-Modbus traffic from RTU"; sid:1000003; rev:1;)

This approach is similar in theory to a deny-by-default network design, which

operates on the logic that network traffic should be blocked if a data channel is not
explicitly needed. Any communication outside the baseline is a valid first indicator of
potential malicious activity for monitoring a static system. The drawback to this design is
that as devices and protocols are added to the network, the IDS must be tuned
accordingly.

3.2.2. 3.2.1 Caldera Lateral Movement Attack Simulation

An operation was created in Caldera to perform lateral movement to simulate an
adversary exploiting remote access systems and using living-off-the-land tactics for
lateral access within the environment.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 23

Figure 3.18: Caldera Operation

Caldera ran several tests successfully. However, as shown in Figure 3.19 below,
Suricata generated no alerts about this activity.

Figure 3.19: Suricata Monitoring

However, the alert is immediately triggered when Caldera performs the attack again
using the modified ruleset.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 24

Figure 3.20: Suricata Alerts from Modified Ruleset

This platform can further alert on lateral movement by leveraging the SYSLOG records
while the Python SER to SYSLOG converter monitors SER data and sends it to
OpenSearch.

Figure 3.21: SER Monitoring in OpenSearch

3.2.3. 3.2.1 Caldera Worm Attack Simulation

For this scenario, the adversary profile of a worm was configured in Caldera.

Figure 3.22: Caldera Operation

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 25

This attack used the EWS to enumerate the network using built-in Windows tools
and attempted to download and run Mimikatz to dump credentials. As shown in the
figures below, Caldera invoked a command to download the tool from Git Hub.

Figure 3.23: EWS Exploit

Figure 3.24: Attack Command Details

Suricata logged the activity and alerted on a possible policy violation.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 26

Figure 3.25: Caldera Worm Attack Alerts

3.2.4. 3.2.1 Caldera Malware Attack Simulation

For this attack scenario, Caldera installed Procdump, performed binary execution,
and installed potentially unwanted software (PUP).

Figure 3.26: Caldera Malware Attack

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 27

Figure 3.27: Attack Artifact Visible on EWS

As with previous tests, the network traffic is visible and recorded on the sensor;
but, as with the first scenario, the standard Suricata ruleset did not trigger an alert, but the
traffic allow-listing approach did. However, Windows Defender detected the malicious
activity.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 28

Figure 3.28: Windows Defender Alerts

This further illustrates that when implementing INSM, one must consider all
monitoring cases; leveraging one monitoring technique is insufficient.

3.3. Test Result Overview

3.3.1. Quantitative Analysis
The preceding test demonstrated that it is possible to monitor various devices and
interfaces with a single network monitoring platform comprised of the preselected tools.
Even though the devices monitored lack standard interfaces in IT systems, these
preselected software packages could retrieve meaningful data, which users can expand to
a wide range of devices.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 29

As expected, the field devices are the most difficult to obtain data from; however,
monitoring them from a network standpoint is significantly more straightforward due to
the limited data flow and network protocols supported. The one less-than-ideal
component is the TELNET interface; pulling this data directly from Zabbix is preferred
rather than running a script outside the platform and sending it in via traps. The current
TELNET agent implementation is a known drawback in the platform, and there is an
open feature request for this ability (Franquesa, 2023). The following tables contain the
summarized results of the testing.

Asset Monitoring
Device Default INSM Test Result Notes
Monitoring Component Pass/Fail/
Capability Partial
None Zabbix Server Zabbix was Partial This is listed
Active able to as partial due
Checks retrieve to the limited
critical amount of
device asset
information information
through the that can be
Relay use of retrieved.
and custom
Meters scripts.
None Zabbix Server Zabbix can Pass
Active retrieve both
Checks asset data
and
configuration
monitoring
Controll data through
ers and the built-in
RTUs API
SNMP Zabbix SNMP Zabbix can Pass
pull critical
asset and
Network device health
Switche data using
s SNMP
Agent-Based Zabbix Agent The Zabbix Pass
Monitoring agent can
provide
Servers complete

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 30

platform
visibility for
servers.
Table 3.1: Asset Monitoring Analysis

Security Monitoring
Device Default INSM Test Result Notes
Monitoring Component Pass/Fail
Capability /Partial
SER stored SER to Zabbix was Partial The Suricata can
locally SYSLOG able to monitor some
custom script retrieve ICS protocols,
and IDS critical and the
device monitoring can be
information enriched by
through the developing
use of custom scripts.
custom
scripts.
SYSLOG
and IDS
Relay logs sent to
and OpenSearc
Meters h
SYSLOG SYSLOG and SYSLOG Pass IDS was most
Suricata IDS and IDS effective when
Control logs sent to configured with
lers and OpenSearc custom rulesets.
RTUs h
SYSLOG SYSLOG and SYSLOG Pass IDS was most
Networ Suricata IDS and IDS effective when
k logs sent to configured with
Switch OpenSearc custom rulesets.
es h
Windows SYSLOG and SYSLOG Pass IDS was most
Event Logs Suricata IDS and IDS effective when
logs sent to configured with
OpenSearc custom rulesets.
Servers h
Table 3.2: Security Monitoring Analysis

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 31

3.3.2. Qualitative Analysis

Although the essential function of the system is proven, the deployment and
maintenance of this platform requires more nuanced skill than would be typically
available for the smaller organizations mentioned as a potential use case at the outset of
this testing. The Docker component, Zabbix software, Suricata software, and supporting
visualization tools require in-depth knowledge and troubleshooting ability. This makes it
more likely that they would opt for a vendor-supplied package that was easier to deploy
and maintain. Security platforms such as Dragos, Nozomi, and others consistently add
new features to enrich their alerting based on the vendors used in the monitored
environments. However, the platform is well suited for security teams that provide
monitoring services and have staff dedicated to updating and tailoring the INSM platform
for the particular scope for which they are responsible.

4. Recommendations and Implications

While this experiment did prove the primary function of the INSM platform and
its ability to monitor the network and network assets successfully, there are some critical
implications to moving this to a production environment. Additionally, the platform's
value proposition could be augmented by adding some missing features and tailoring the
Zabbix platform to normalize the retrieved data.

4.1. Recommendations for Practice

This experiment focused on the ability of the INSM platform to bring in asset data
and perform network monitoring. Administrators should implement appropriate security
controls before deploying this toolset in production. For example, in this test
environment, the platform did not use TLS between attached nodes; administrators
should implement a certificate authority and host certificates in production.

4.2. Implications for Future Research

Several takeaways immediately come to mind while contemplating improving the
INSM platform from a usability and functionality standpoint. Users immediately realize
the value of the Zabbix platform for devices with agents and supported templates in terms

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 32

of usability. However, because most industrial devices used for testing do not support
agents, the platform is limited in what it can retrieve. Nevertheless, by creating templates
that correspond on a per-device level, administrators could normalize the data retrieved
through active checks to make that more accessible for the user. Secondly, the inability to
use the TELNET without a standard response interface was unfortunate, and although a
workaround was found, it still needs to be improved from a monitoring standpoint.

One consistent problem with using open-source tools in industrial environments is

more protocol support. Most network monitoring vendors for ICS-specific networks
develop custom protocol parsers. However, the open-source community has not kept
pace. One prominent example is IEC 61850, a widely used standard in the power system
and often used for time-critical applications.

Lastly, CIP-007 R1 specifies that all ports, protocols, and services be identified
per asset and evidence provided (North American Electric Reliability Corporation, 2024).
Typically, the compliance team would record evidence using either an NMAP scan or
Netstat output, but it would be beneficial to audit this persistently. Zabbix can monitor an
individual TCP port and discover devices based on TCP sweep; however, it does not
record all open ports found. Administrators could accomplish this by using the built-in
network discovery tool in the RTAC and then querying the API for that data. However,
pulling that information directly from Zabbix by having Zabbix spawn an NMAP scan
and retrieve the data would be cleaner.

5. Conclusion
Although security improvements in recent years have been made, a gap exists in
network visibility and asset management for many ICS systems. This gap is most notable
in comparatively smaller organizations with limited budgets. With all the open-source
tools available focusing on IT systems, this research intended to understand how some of
these open-source software solutions could be tailored to the needs of ICS. While the
experiment was ultimately successful in retrieving device information and essential
network monitoring, it also highlighted the amount of effort required in even a minimal

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 33

lab environment, given the heterogeneous nature of the devices, and further emphasized
the value proposition posed by ICS-focused vendor platforms.

This work should encourage ICS security practitioners to reevaluate the available
toolsets and that perhaps tools previously considered as "IT only" may benefit ICS
environments. It is possible to achieve visibility into the ICS system using open-source
tools; however, it may take some customization to achieve the level of visibility required.

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 34

References
Christopher, J. (2023, June 22). Breaking IT/OT Silos with ICS/OT Visibilty. Retrieved

from sans.org: https://www.sans.org/white-papers/breaking-it-ot-silos-ics-ot-

visibility/

Django Software Foundation. (2024, 02 19). Data Prepper. Retrieved from

opensearch.org: https://opensearch.org/docs/latest/data-prepper/

docker. (2024, 02 18). Use containers to Build, Share, and Run your applications.

Retrieved from docker.com: https://www.docker.com/resources/what-container/

Dragos. (2024, 02 25). electrum. Retrieved from dragos.com:

https://www.dragos.com/threat/electrum/

Dragos. (2024, 25 02). kostovite. Retrieved from dragos.com:

https://www.dragos.com/threat/kostovite/

Dragos. (2024, 02 25). talonite. Retrieved from dragos.com:

https://www.dragos.com/threat/talonite/

Fluentd project. (2024, 02 18). fluent/fluent-bit. Retrieved from hub.docker.com:

https://hub.docker.com/r/fluent/fluent-bit

Franquesa, M. (2023, 12 04). Zabbix Feature Request. Retrieved from zabbix.com:

https://support.zabbix.com/browse/ZBXNEXT-5837

Ish, J. (2024, 02 18). docker-suricata. Retrieved from github.com:

https://github.com/jasonish/docker-suricata

MITRE. (2024, 18 02). CALDERA. Retrieved from caldera.mitre.org:

https://caldera.mitre.org/

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 35

MySQL. (2024, 02 18). mysql. Retrieved from hub.docker.com:

https://hub.docker.com/_/mysql

North American Electric Reliability Corporation. (2024, 02 26). CIP-007-6 - Cyber

Security - Systems Security Management. Retrieved from nerc.com:

https://www.nerc.com/pa/Stand/Reliability%20Standards/CIP-007-6.pdf

North American Reliability Corporation. (2024, 02 18). Project 2023-03 Internal

Network Security Monitoring (INSM). Retrieved from nerc.com:

https://www.nerc.com/pa/Stand/Pages/Project-2023-03-INSM.aspx

OpenSearch Project. (2024, 02 19). opensearch. Retrieved from hub.docker.com:

https://hub.docker.com/r/opensearchproject/opensearch

OpenSearch project. (2024, 02 19). opensearch-dashboards. Retrieved from

hub.docker.com: https://hub.docker.com/r/opensearchproject/opensearch-

dashboards

Sanchez, M. (2022, 09 28). Network Security Monitoring for BES Cyber Systems.

Retrieved from www.computer.org: https://www.computer.org/publications/tech-

news/community-voices/network-security-monitoring-bes-cyber-systems

Schweitzer Engineering Laboratories. (2024, 02 25). 751A. Retrieved from selinc.com:

https://selinc.com/products/751A/docs/

Schweitzer Engineering Laboratories. (2024, 02 18). SEL Real-Time Automation

Controller (RTAC). Retrieved from selinc.com:

https://selinc.com/products/RTAC/

Schweitzer Engineering Laboratories. (2024, 02 18). SEL-3355. Retrieved from

selinc.com: https://selinc.com/products/3355/

Author Name, email@address

 Performing Internal Network Security Monitoring with Open-Source Tools 36

Schweitzer Engineering Laboratories. (2024, 02 18). SEL-3360. Retrieved from

selinc.com: https://selinc.com/products/3360/

Schweitzer Engineering Laboratories. (2024, 02 18). SEL-735. Retrieved from

selinc.com: https://selinc.com/products/735/

Schweitzer Engineering Laboratories. (2024, 02 18). SEL-751. Retrieved from

selinc.com: https://selinc.com/products/751/

Siemens. (2024, 02 18). RUGGEDCOM RSG2100. Retrieved from siemens.com:

https://support.industry.siemens.com/cs/products/6gk6021-0as2.-..../ruggedcom-

rsg2100?pid=303165&mlfb=6GK6021-0AS2.-....&mfn=ps&lc=en-WW

Suricata. (2024, 02 18). Suricata. Retrieved from suricata.io: https://suricata.io/

Zabbix. (2024, 02 25). network monitoring. Retrieved from zabbix.com:

https://www.zabbix.com/network_monitoring

Zabbix. (2024, 02 18). Zabbix. Retrieved from zabbix.com: https://www.zabbix.com/

Zabbix. (2024, 02 18). Zabbix SIA. Retrieved from docker hub:

https://hub.docker.com/u/zabbix

Zabbix. (2024, 02 25). zabbix.com. Retrieved from server monitoring:

https://www.zabbix.com/server_monitoring

Zabbix. (2024, 02 19). zabbix-agent. Retrieved from hub.docker.com:

https://hub.docker.com/r/zabbix/zabbix-agent

Zabbix. (2024, 02 19). zabbix-server-mysql. Retrieved from hub.docker.com:

https://hub.docker.com/r/zabbix/zabbix-server-mysql

Zabbix. (2024, 02 19). zabbix-web-nginx. Retrieved from hub.docker.com:

https://hub.docker.com/r/zabbix/zabbix-web-nginx-mysql

Author Name, email@address