Data analytics for Cyber Security

Chapter Five
Anomaly Detection in Cyber security
School of Information Technology and Engineering
Addis Ababa Institute of Technology
Addis Ababa University
Dec 2024

sD
by senait Desalegn
 Content
 Understanding of Anomaly Detection
 Types of Anomalies
 Importance of Anomaly detection in cyber security
 Methods of Anomaly Detection
 Challenges in detecting Anomaly
 Anomaly Detection steps
 Examples of Anomaly Detection
 Tools for Anomaly Detection

sD
 What is anomaly detection?

Anomaly detection also referred to as outlier detection plays a crucial role in

cyber security. By leveraging advanced technologies such as machine learning
and artificial intelligence, anomaly detection systems can recognize deviations
from normal behavior and events within a network or system, swiftly
identifying unusual patterns or points that may indicate a potential threat or a
cyber attack that is underway.
Integrating anomaly detection into a comprehensive cyber security strategy
enhances an organization's ability to protect sensitive data and systems from
D

malicious attacks, proactively address threats, and maintain the integrity of

critical information and systems.
 Fundamentals of anomaly detection

Anomaly detection is the process of analyzing a dataset and

identifying single occurrences or patterns that deviate
significantly from baseline activity.
In the context of cyber security, these anomalies, or outliers,
can often be an early warning sign of a malicious event, such as
a data breach, cyber attack or system failure. By identifying
these anomalies sooner, organizations can potentially contain
the security risk, thereby minimizing damages and expediting
recovery.
 Types of Anomalies

There are three main types of anomalies detectable by advanced anomaly

detection systems:
Point anomalies: A point anomaly is when an individual data point
significantly deviates from the rest of the data set and the so-called “norm”.
An example of a point anomaly may be a sudden spike in network traffic.
Contextual anomalies: A contextual anomaly is an individual data point that
differs from the rest of a data set, but only within a specific context. For
example, if a user logs into a system during non-business hours or from an IP
address that does not match their geographic location, that may be a
contextual anomaly.
 Types of Anomalies

Collective anomalies: A collective anomaly is when a group of related data

points collectively deviate from the expected pattern, even though individual
data points may fall within normal and acceptable use. For example, a sudden
surge in network traffic from a variety of IP addresses may indicate a coordinated
attack and would be an example of a collective anomaly.
 Importance of anomaly detection in cyber
security
Most breaches have warning signs. The question is: Does your organization have the
right tools to detect and act on those signals?
Anomaly detection systems play a vital role in helping an organization maintain a
strong security posture.
 Data Security: Anomaly detection helps detect fraudulent activities and security
breaches in real-time, safeguarding sensitive data.
 Cost Savings: Identifying anomalies in industrial processes can prevent equipment
failures, reducing maintenance costs and downtime.
 Importance of anomaly detection in cyber
security

 Improved Decision-Making: By spotting anomalies in financial or market data,

businesses can make informed decisions to mitigate risks and seize opportunities.
 Enhanced Customer Experience: Anomaly detection can identify unusual
customer behavior, enabling businesses to provide better service and personalized
experiences.
 Early Disease Detection: In healthcare, anomaly detection in patient data can lead
to early disease diagnosis and timely interventions.
 Efficient Resource Allocation: It helps organizations allocate resources more
efficiently by identifying inefficiencies in operations.
 Fundamental methods of anomaly detection

Anomaly detection can be approached using various techniques, but the five
fundamental methods are:
 Statistical methods
 Machine learning-based methods
 Rule-based methods
 Density-based methods
 Time series methods
 Fundamental methods of anomaly detection

1. Statistical methods
Statistical methods are among the most straightforward and commonly used
approaches for anomaly detection. These methods assume that the normal data
follows a certain statistical distribution, such as Gaussian (normal) distribution. Data
points that fall significantly outside the expected range are flagged as anomalies.
 Z-Score: This method uses the standard deviation to determine how many standard
deviations a data point is away from the mean. Points that exceed a specified
threshold (e.g., 2 or 3 standard deviations) are considered anomalies.
 Fundamental methods of anomaly detection

• Modified Z-Score: Similar to the Z-Score method but more robust to outliers.
• Density-based anomaly detection: It relies on estimating the probability density
function of the data and identifying points with low probability as anomalies.

2. Machine learning-based methods

Machine learning techniques are increasingly popular for anomaly detection, especially
in complex and high-dimensional datasets. These methods train models on the normal
data and then use the model to identify anomalies based on deviations from what the
model learned as normal behavior.
 Fundamental methods of anomaly detection

 Unsupervised learning: Algorithms like K-Means clustering or Autoencoders are

commonly used for unsupervised anomaly detection, where the model identifies
data points that do not fit well within any cluster or do not reconstruct well using the
auto encoder.
 Supervised learning: In supervised anomaly detection, the model is trained on
labeled data, where anomalies are marked. The model then learns to distinguish
between normal and anomalous patterns.
 Semi-supervised learning: This approach uses a combination of labeled normal data
and unlabeled data to train the model, making it more practical for real-world
scenarios where obtaining labeled anomaly data can be challenging.
 Fundamental methods of anomaly detection

3. Rule-based methods
Rule-based methods rely on defining explicit rules or thresholds to identify anomalies.
These rules are often based on domain knowledge or expert input. If data points violate
these rules, they are flagged as anomalies.
 Domain knowledge rules: Experts in a specific domain can define rules based on
their understanding of what constitutes normal or abnormal behavior.
 Business rules: In certain cases, business rules can be defined based on specific
business requirements or constraints, and data points deviating from these rules are
considered anomalies.
 Fundamental methods of anomaly detection

4. Density-based methods
Density-based methods focus on estimating the data density and identifying regions of
low density as anomalies.
These methods are particularly useful for detecting local anomalies. Some density-
based anomaly detection methods include:
 DBSCAN (Density-based Spatial Clustering of Applications with Noise): Clusters data
points based on density and identifies outliers as points that do not belong to any
cluster.
 LOF (Local Outlier Factor): Measures the local density around each data point and
identifies points with significantly lower densities as anomalies.
 Fundamental methods of anomaly detection

5. Time series methods

Time series data poses unique challenges for anomaly detection due to its temporal
nature. Time series anomaly detection methods consider temporal patterns and
changes in data over time.
Some time series anomaly detection methods include:
 Seasonal Decomposition of Time Series (STL): Decomposes time series into seasonal,
trend, and residual components to identify anomalies in the residual component.
 ARIMA (Auto Regressive Integrated Moving Average): A model for time series
forecasting that can be used to detect anomalies based on forecast errors.
 Challenges faced in detecting anomaly

Anomaly detection, while highly beneficial, also presents several challenges that can
impact its effectiveness. Understanding these challenges is essential for
organizations aiming to implement or improve their anomaly detection systems.
Some of the key challenges include:
1. High false positive rates
One of the most significant challenges in anomaly detection is distinguishing
between true anomalies and false alarms. High false positive rates can lead to
unnecessary alerts, causing organizations to waste resources investigating normal
variations in data as potential threats or issues.
 Challenges faced in detecting an anomaly

2. Data quality and availability

The effectiveness of anomaly detection is heavily dependent on the quality and
completeness of the data. Incomplete, inconsistent, or noisy data can lead to
inaccurate detection of anomalies, either by missing real issues or flagging non-
issues as problems.
3. Dynamic data and changing patterns
In many real-world scenarios, data patterns can change over time due to evolving
trends, behaviors, or environmental factors. Anomaly detection systems must be
able to adapt to these changes to remain effective, which can be a complex task.
 Challenges faced in detecting an anomaly

4. Defining anomaly
Establishing what constitutes normal behavior or patterns within a dataset is a
fundamental challenge. In many cases, there is no clear definition of “normal,” and it
can vary significantly across different contexts or environments.
5. Scalability and performance
As datasets grow in size and complexity, maintaining the performance and scalability
of anomaly detection systems becomes challenging. Processing large volumes of
data in real-time requires significant computational resources and efficient
algorithms.
6. Domain-specific challenges
Each industry or application may present unique challenges for anomaly detection.
For example, in healthcare, patient data can vary widely, making it difficult to
 Challenges faced in detecting an anomaly

7. Integration with existing systems

Effectively integrating anomaly detection into existing systems and processes can be
challenging. It often requires a thorough understanding of the current infrastructure
and careful planning to ensure compatibility and minimal disruption.
8. Cost and resource constraints
Implementing and maintaining an effective anomaly detection system can be
resource-intensive, requiring skilled personnel, advanced technology, and ongoing
maintenance. This can be a significant hurdle, especially for smaller organizations.
9. Interpretation of results
The results of anomaly detection need to be interpretable and actionable.
Understanding the context and implications of detected anomalies is crucial for
taking appropriate actions.
 How does anomaly detection work?

Anomaly detection is a process used to identify unusual patterns or observations in

data that do not conform to expected behavior. These anomalies can indicate critical
incidents, such as fraud, structural defects, system failures, or other significant
issues. Understanding how anomaly detection works involves several key
components and methodologies:
1. Data collection and preprocessing
The first step is gathering and preparing the data. This can involve cleaning the data,
handling missing values, normalizing data scales, and selecting relevant features. The
quality and relevance of the data directly impact the effectiveness of the anomaly
detection process.
 How does anomaly detection work?

2. Establishing a baseline of normalcy

Anomaly detection systems need a baseline or model of what constitutes normal
behavior in the dataset. This model can be established using historical data,
statistical measures, or machine learning algorithms. The goal is to define a
boundary of normal behavior, against which new data can be compared.
3. Choosing the right algorithm
Various algorithms can be used for anomaly detection, and the choice depends on
the nature of the data and the specific application.
 How does anomaly detection work?

4. Anomaly detection models

There are several types of models used for anomaly detection, such as:
Point anomalies: Identifying single data points that are significantly different from
the rest.
Contextual anomalies: Detecting anomalies that are context-specific (e.g., a sudden
spike in energy usage on a normally low-usage day).
Collective anomalies: Finding collections of related data points that, together,
indicate an anomaly (e.g., a sequence of transactions that are suspicious when taken
together).
 How does anomaly detection work?

5. Training and model fitting

The chosen algorithm is trained on the dataset to learn the patterns of normal
behavior. In supervised learning, the model is trained on a labeled dataset, whereas,
in unsupervised learning, the model tries to fit itself to the data without predefined
labels.
6. Anomaly detection and validation
Once the model is trained, it can then be used to detect anomalies in new data.
Detected anomalies are often subjected to further validation or investigation to
determine their significance or cause.
 How does anomaly detection work?

7. Feedback loop
Anomaly detection systems often include a feedback mechanism. When an anomaly
is identified and investigated, the outcome can be fed back into the system to
improve its accuracy and adapt to changing data patterns.
8. Continuous monitoring and updating
Anomaly detection is typically an ongoing process, with continuous monitoring of
new data and periodic updates to the model to reflect new patterns or changes in
the environment.
By effectively implementing these steps, anomaly detection systems can provide
critical insights and early warnings of potential issues, supporting timely decision-
making and intervention.
 Top 10 examples of anomaly detection

Anomaly detection serves various important purposes across different industries and
applications. Some of the key examples of anomaly detection include:
1. Fraud detection
In finance and cyber security, anomaly detection is used to identify unusual patterns
of transactions or network activities that could indicate potential fraudulent activities
or cyber attacks. By detecting anomalies in real-time, organizations can take
immediate action to prevent financial losses and protect sensitive data.
2. Network intrusion detection
Anomaly detection is employed in network security to identify unauthorized access
attempts, unusual traffic patterns, and potential security breaches. It helps network
administrators to quickly respond to threats and safeguard their systems and data.
 Top 10 examples of anomaly detection

Anomaly detection serves various important purposes across different industries and
applications. Some of the key examples of anomaly detection include:
3. Manufacturing quality control
In manufacturing processes, anomaly detection is used to identify defective products
or equipment malfunctions. By detecting anomalies early, manufacturers can take
corrective actions to maintain product quality and prevent wastage
4. Healthcare monitoring
Anomaly detection in healthcare can be used to identify abnormal patient
conditions, such as irregular heart rhythms, unusual physiological parameters, or
potential medical errors. Early detection of anomalies can lead to timely
interventions and improved patient outcomes.
 Top 10 examples of anomaly detection

5. Predictive maintenance
In industries like aviation, transportation, and manufacturing, anomaly detection is
used for predictive maintenance. By detecting anomalies in sensor data from
machines or equipment, organizations can schedule maintenance tasks proactively,
minimizing downtime and reducing maintenance costs.
6. Traffic monitoring
Anomaly detection is utilized in traffic management systems to identify traffic
incidents, congestion, or accidents on roads. This information helps authorities
respond promptly, manage traffic flow, and optimize transportation routes.
 Top 10 examples of anomaly detection

7.Environmental monitoring
Anomaly detection is used in environmental monitoring to identify abnormal events
or changes in environmental factors, such as air quality, water levels, or seismic
activity. Early detection of anomalies can help in disaster management and
environmental protection.
8. Retail and E-commerce
Anomaly detection is applied in retail to detect unusual shopping patterns, customer
behavior, or inventory discrepancies. Retailers can use this information for inventory
management, pricing strategies, and personalized customer experiences.
 Top 10 examples of anomaly detection

9. Insurance claim analysis

In the insurance industry, anomaly detection can identify suspicious or potentially
fraudulent insurance claims, helping insurance companies prevent fraudulent
payouts and reduce losses.
10. Energy management
Anomaly detection is used in energy consumption data to identify anomalies that
may indicate energy wastage or equipment malfunction. Organizations can then take
steps to optimize energy usage and reduce costs.
 Tools for Anomaly Detection

Scikit-Learn: A popular Python library that offers a wide range of machine learning
algorithms for anomaly detection.
TensorFlow and PyTorch: These deep learning frameworks provide tools to build
custom anomaly detection models.
ELK Stack (Elasticsearch, Logstash, Kibana): This stack is widely used for real-time
log and event data analysis, making it valuable for anomaly detection in IT
operations.
Microsoft Azure Anomaly Detector: A cloud-based service that simplifies anomaly
detection with pre-built models.
Google Cloud AI Platform: Offers machine learning tools and infrastructure for
building custom anomaly detection solutions.
Thank You