ANOMALY DETECTION IN SURVEILLANCE

CAMERA

Capstone Project Report

End-Semester Evaluation

Submitted by:

(101603005) Abhijeet Singh

(101603013) Abhishek Modi
(101611001) Abhishek Garg
(101611044) Rajan Goyalsc
BE Third Year, COE

CPG No. 104

Under the Mentorship of

Dr Shatrughan Modi
Lecturer

Computer Science and Engineering Department

Thapar Institute of Engineering and Technology, Patiala
JAN-MAY 2019
 ABSTRACT

In this project, we develop a system for detecting anomalies in surveillance footage. An anomaly
refers to a deviation from the normal routine, for example: a person abandoning a bag at a crowded
or public location, or a computer being stolen from a lab. For efficient large-scale monitoring, it is
critical to design a system based on intelligent video analysis. The system is based on a simple
mathematical model, and works efficiently at resolutions at which most CCTV cameras function.
The framework is based on a Background Subtraction technique, with pre-processing involving a
dual-time background subtraction algorithm, which maintains two backgrounds, one updated with
a relatively high frequency, and the other at a low frequency. Results show that the system is fast,
and robust in both moderately and sparsely populated areas. As the system does not use expensive
filters and statistical calculations, it is less intensive computationally, and at the same time delivers
good detection results.
 DECLARATION

We hereby declare that the design principles and working prototype model of the project entitled
Anomaly detection in surveillance camera is an authentic record of our own work carried out in
the Computer Science and Engineering Department, TIET, Patiala, under the guidance of Dr
Shatrughan Modi during 6th semester (2019).

Date:

Roll No. Name Signature

101603005 Abhijeet Singh Chahal

101603013 Abhishek Modi

101611001 Abhishek Garg

101611044 Rajan Goyal

Counter Signed By:

Faculty Mentor:
Dr. Shatrughan Modi
Lecturer
CSED,
TIET, Patiala
 ACKNOWLEDGEMENT

We would like to express our thanks to our mentor Dr Shatrughan Modi. She has been of great
help in our venture, and an indispensable resource of technical knowledge. He is truly an amazing
mentor to have.
We are also thankful to Dr Maninder Singh, Head, Computer Science and Engineering
Department, entire faculty and staff of Computer Science and Engineering Department, and also
our friends who devoted their valuable time and helped us in all possible ways towards successful
completion of this project. We thank all those who have contributed either directly or indirectly
towards this project.
Lastly, we would also like to thank our families for their unyielding love and encouragement. They
always wanted the best for us and we admire their determination and sacrifice.

Date:

Roll No. Name Signature

101603005 Abhijeet Singh Chahal

101603013 Abhishek Modi

101611001 Abhishek Garg

101611044 Rajan Goyal

TABLE OF CONTENTS
ABSTRACT 2

DECLARATION 3

ACKNOWLEDGEMENT 4

LIST OF TABLES 7

LIST OF FIGURES 8

LIST OF ABBREVIATIONS 9

CHAPTER 1- INTRODUCTION 10
1.1 Project Overview 10
1.1.1 Technical terminology 10
1.1.2 Problem Statement 10
1.1.3 Goal 11
1.1.4 Solution 11
1.2 Need Analysis 11
1.3 Research Gaps 12
1.4 Problem Definition and Scope 14
1.5 Assumptions and Constraints 14
1.6 Approved Objectives 14
1.7 Methodology Used 15
1.8 Project Outcomes and Deliverables 15
1.9 Novelty 15

CHAPTER 2- REQUIREMENT ANALYSIS Error! Bookmark not defined.

2.1 Literature Survey Error! Bookmark not defined.
2.1.1 Theory Associated With Problem Area Error! Bookmark not defined.
2.1.2 Existing System and Solutions Error! Bookmark not defined.
2.1.3 Research Findings for Existing Literature Error! Bookmark not defined.
2.1.4 Problem Identified Error! Bookmark not defined.
2.1.5 Survey of Tools and Technologies Used Error! Bookmark not defined.
2.2 Standards Error! Bookmark not defined.
2.3 Software Requirement Specification Error! Bookmark not defined.
2.3.1 Introduction Error! Bookmark not defined.
2.3.1.1 Purpose Error! Bookmark not defined.
2.3.1.2 Intended Audience and Reading Suggestions Error! Bookmark not defined.
2.3.1.3 Project Scope Error! Bookmark not defined.
2.3.2 Overall Description Error! Bookmark not defined.
2.3.2.1 Product Perspective Error! Bookmark not defined.
2.3.2.2 Product Features Error! Bookmark not defined.
2.3.3 External Interface Requirements Error! Bookmark not defined.
2.3.3.1 User Interfaces Error! Bookmark not defined.
2.3.3.2 Hardware Interfaces Error! Bookmark not defined.
2.3.3.3 Software Interfaces Error! Bookmark not defined.
2.3.4 Other Non-functional Requirements Error! Bookmark not defined.
2.3.4.1 Performance requirements Error! Bookmark not defined.
2.3.4.2 Safety Requirements Error! Bookmark not defined.
2.3.4.3 Security Requirements Error! Bookmark not defined.
2.4 Cost Analysis Error! Bookmark not defined.
2.5 Risk Analysis Error! Bookmark not defined.
2.5.1 Risks to consider Error! Bookmark not defined.
2.5.2 Impact of the risk Error! Bookmark not defined.
2.5.3 Unpredictable Factors Error! Bookmark not defined.
2.5.4 Predictable Factors Error! Bookmark not defined.

CHAPTER 3- METHODOLOGY ADOPTED Error! Bookmark not defined.

3.1 Investigative Techniques 17
3.2 Proposed Solution 17
 3.3 Work Breakdown Structure 18
3.4 Tools and Technologies Used 18

CHAPTER 4- DESIGN SPECIFICATIONS Error! Bookmark not defined.

4.1 System Architecture 19
4.1.1 Component Diagram 19
4.1.2 MVC Architecture 19
4.1.3 Tier Architecture 19
4.2 Design Level diagrams 20
4.2.1 Use case diagram 20
4.2.2 Use Case template 21
4.2.3 Data Design diagram Error! Bookmark not defined.
4.2.4 Class diagram Error! Bookmark not defined.
4.2.5 Context diagram Error! Bookmark not defined.
4.2.6 Sequence diagram Error! Bookmark not defined.
4.2.7 State chart diagram Error! Bookmark not defined.
4.2.8 Activity diagram Error! Bookmark not defined.
4.3 User Interface diagrams Error! Bookmark not defined.
4.4 System Screenshots 25

CHAPTER 6- CONCLUSIONS AND FUTURE DIRECTIONS Error! Bookmark not defined.

6.1 Conclusions Error! Bookmark not defined.

REFERENCES Error! Bookmark not defined.

LIST OF TABLES

Table No. Table Caption Page No.

Table 1.1 Assumptions and Constraints 15

Table 2.1 Research Findings for Existing Literature 18

Table 2.2 Risk Impacts 24

Table 3.1 Investigative Techniques 25

Table 4.1 Use Case Template 31

LIST OF FIGURES

Figure No. Figure Caption Page No.

Figure 3.1 Work Breakdown Structure 27

Figure 4.1 Component Diagram 28

Figure 4.2 MVC Architecture 29

Figure 4.3 Tier Architecture 29

Figure 4.4 Use Case Diagram 30

Figure 4.5 Data Design Diagram 32

Figure 4.6 Class Diagram 32

Figure 4.7 Context Diagram 33

Figure 4.8 Sequence Diagram 33

Figure 4.9 State Chart Diagram 34

Figure 4.10 Activity Diagram 35

Figure 4.11 Interface Design 36

LIST OF ABBREVIATIONS

MMI Man Machine Interface

SMS Short Message Service

ADL Activities of daily living

ACM Acceleration Cubic-product-root Magnitude

AVCM Angular Velocity Cubic-product-root Magnitude

kNN k-nearest neighbors algorithm

GPS Global Positioning System

MVC Model, View, Controller

GPU Graphical Processing Unit

HMM Hidden Markov Model

SVM Support Vector Machine

IMU Inertial Measurement Unit

SMLR Sparse Multinomial Logistic Regression

 CHAPTER 1- INTRODUCTION

1.1 Project Overview

In this project, we develop a system for detecting anomalies in surveillance footage.

Imagine a system that takes video from surveillance cameras, understands when something
abnormal occurs and relays this information to the relevant user. This would free up the
human resources that are now being used to monitor video from these cameras, able
cameras that are not being used actively to join in without extra cost and gain much more
than a post incident forensic tool. In this project the current solutions available for
abnormality detection will be reviewed, a method for abnormality detection will be
designed, implemented and tested. Results show that the system is fast, and robust in both
moderately and sparsely populated areas. As the system does not use expensive filters and
statistical calculations, it is less intensive computationally, and at the same time delivers
good detection results.

1.1.1 Technical terminology

Surveillance videos are able to capture a variety of realistic anomalies. In this paper,
we propose to learn anomalies by exploiting both normal and anomalous videos.
To avoid annotating the anomalous segments or clips in training videos, which is
very time consuming, we propose to learn anomaly through the deep multiple
instance ranking framework by leveraging weakly labelled training videos ,i.e. the
training labels (anomalous or normal) are at video-level instead of clip-level. In our
approach, we consider normal and anomalous videos as bags and video segments
as instances in multiple instance learning (MIL), and automatically learn a deep
anomaly ranking model that predicts high anomaly scores for anomalous video
segments

1.1.2 Problem Statement

The main requirement of anomaly detection through a surveillance camera is

recording the real-time data with high accuracy and create alert during anomaly.
Sometimes, when the camera is not working properly, it may send wrong values
which will generate a false alert. The issue tackled by this project is the location
of abandoned objects in a single colour camera’s video feed quickly, efficiently,
and reliably.
 The system must subscribe to certain specifications. The system should work online
in real time, must stay active all the time, and thus should work with varying light
conditions.

If these specifications are not met, then it might be too late to take proper action, as
a bomb may explode, or a shoplifter can escape. The algorithm used by us runs in
real time on modern inexpensive computer systems. It should have no problem
running on old hardware, if the video resolution is QVGA (320*240 pixel
resolution), as is the case with most security CCTV cameras

To summarize, the main problems in anomaly detection through a camera are:

● Accuracy of dataset

● Fast & real-time response

● Secure and functional network

● Working on modern inexpensive systems

● Varying light Conditions

1.1.3 Goal

The goal is to build an application that can detect the anomaly with high accuracy
and to achieve the fastest possible response time of the Anomaly detection algorithm.

1.1.4 Solution

We are planning to use CCTV cameras connected with computer. Footage from
CCTV with be sent to application. After that, the data will be sent as input parameters
of anomaly detection algorithm. Algorithm will compute the degree of anomaly and
will then classify the anomaly. If the result is beyond threshold limit, an alert will be
generated on application to check if it is false alert or not.

1.2 Need Analysis

There have been an increasing number of terrorist attacks in crowded public places such
as train stations, subways, airports, malls, etc. Security staff and police forces rely on
video surveillance systems; however, it is labour intensive, and there are limits of human
effectiveness in monitoring and analysing crowded scenes, particularly when multiple
cameras are involved. It is rather unfortunate if there is no one to monitor the feed, or if
negligence leads to an attack. Automated surveillance systems have been used for mostly
 offline video analysis after an event has occurred, much research is underway on online
analysis and prediction of events. We can detect anomalies and suspicious behaviours in
the video feed online, abandoned objects particularly. Many algorithms proposed to deal
with the challenge of abandoned object detection depend on complex probabilistic
mathematics, resulting in the need for large amount of resources, and their failure to work
fast in real time. Other difficulties in detecting abandoned objects are occlusion by people
in a crowd, and the background subtraction lacking dynamically updating features or
being computationally intensive. Successive frames, the system keeps a lookout for the
bag positions, whose presence in anomaly in footage is not limited to detection of static
objects. It can be extended to detecting a fight, a person jumping a fence or an accident.

When an abandoned bag is detected, the system analyses its database to determine the
most likely bag position(s), where the position is defined as the location where the bag
into the scene is left unattended. Thro or disappearance from the scene defines the status
of the bag, and decides the appropriate course of action. Automatic threat detection
systems can assist security personnel by providing better situational awareness, enabling
them to respond to critical situations more efficiently.

1.3 Research Gaps

Loss of consciousness is one of the leading problems faced by the user who monitor
surveillance camera’s feed. To prevent this from happening, a constant supervision of the
application is needed. Hence an alternate surveillance method is required. Anomaly require
immediate attention and the only way one can establish that is by designing a sensing and
alerting method.
According to a survey done by Michael D. Breitenstein, Helmut Grabner and Luc Van
Gool [1], They focus on situations where tracking and object detection is difficult due to
the objects of interest being small, the frame-rate quite low (time-lapse like footage) and
image quality also low. Therefore the data is treated as a stream of separate images focusing
on the spatial information in each image. The basic idea is that everything not observed in
the past is unusual. They discriminate between unusual scenes (never seen before) and rare
scenes (similar scene seen before, but not very often). A model is learned using the previous
observed data. When a new image is captured a similarity measure is calculated that
compares it with the learned model. Different similarity measures can be used, they use
normalized cross correlation. Since the representation of the image has a high
dimensionality most points are equally distant to a query point. Therefore concept of
meaningful nearest neighbours is employed. To do this they store distribution of the
similarities. If it is unlikely that the similarity score of a new image is drawn from this
distribution, then the image is rated as unusual. No temporal information is represented
with this method. There might be a problem with the testing being too much of anecdotal
nature. The results are not thoroughly documented with ROC or confusion matrix.
R. Schuster, R. M¨orzinger, W. Haas, H. Grabner and Luc Van Gool [2] proposed As with
Hunting Nessie they focus on low frame-rate data where optical flow or other methods
cannot be used. They utilise the HOG descriptor to describe each region and a database
classification system where each entry is a HOG descriptor. When a descriptor is
introduced into the classifier the distance from the input descriptor to the nearest entry in
the database is measured. If the nearest distance is greater than the inner model distance of
the database classifier the two nearest entries are merged and a new is created with the
newest input. There is an upper limit to how many entries there can be in the database, to
keep the computational requirements to low. This is regulated by a constant which can be
set differently. The method achieve 20 fps. The method works quite well for detecting out
of place objects and runs in real-time. There is an unsupervised learning algorithm which
is optimal and it is data-driven so no manually thresholds have to be set.
Chen Change Loy, Tao Xiang and Shaogang Gong [3] in its research, Optical flow is
computed for each pixel in each region using the Lucas-Kanade methods. The motion in
each region is quantised into four directions and the motion in the next frames are summed
up to make a four-bin histogram containing the motion of 50 frames. This will then be
converted to a codebook of 16 possible combinations ranging from no significant motion
in either of the 4 directions to significant motion in all of the four directions. The classifier
is a naive Bayes classifier and a threshold is used for determining if a frame is abnormal or
not.
Vijay Mahadevan, Weixin Li, Viral Bhalodia and Nuno Vasconcelos. [4] Developed
method combines both spatial and temporal anomaly detection. There is attaining and a
testing phase, so they do not use on-line updating of the normal model. For the temporal
anomalies an MDT is computed on each frame in the training set and in the running phase
the computed MDTs are compared by using log likelihood for the cluster that is closest to
the MDT. This will produce the temporal anomaly map. For the spatial anomalies saliency
method is used. This is a measure of how much the region stands out of its surroundings.
An MDT is computed using an approximation, since it would otherwise be too
computationally heavy. The spatial anomaly map is computed by using KL divergence
between the MDT of the training-set and the input frame. The two anomaly maps are joined
by adding them together and the anomaly detection is done using a simple threshold. In the
testing phase it takes 25s for each frame to be processed, which makes this method far from
real-time.
All the above research papers provided us an insight of the techniques and the algorithms
being currently being used in this research area. We, hence, deduced the need of a portable,
low cost and environment independent system which can alert the user as soon as the
anomaly is detected to prevent.
1.4 Problem Definition and Scope

The main problem in detection of anomaly is the generation of false alarm. It can have
various reasons like the application is not working properly, the application crashes off
from the user accidentally and some more.
A CCTV camera will be attached to computer through wires or Wi-Fi. Footages from the
camera will be sent to application for the detection of anomaly. This will be done with the
help of a Anomaly detection algorithm which will classify the results obtained after
computing the recorded data into normal or anomalous. An alert will be generated on the
application with a 10 seconds window to respond, failing which the security will be alerted
through alarm system.

1.5 Assumptions and Constraints

TABLE 1.1: Assumptions and Constraints

S. No. Assumptions and Constraints

1 The surveillance system requires a fraction of seconds to detect an anomaly, so any

anomaly taking less time than the threshold time will not be detected.

2 A backup power supply should be maintained 24×7 to keep a check on the

operational efficiency of the surveillance system

3 The camera must be super sensitive to low light conditions too.

4 The sensor used may or may not have a higher surviving capacity, thus,
malfunctioning of it can occur, so we need to keep a check on the sensor condition
timely.

5 The hardware must contain GPU that efficiently process the algorithms and image
processing.

6 For the alarm to get to the active state, a proper trained algorithm must be used
to measure the level of anomaly. If the level is greater than the threshold value,
then only the alarm will go the active state.
1.6 Approved Objectives
1. To successfully detect the anomaly
2. To successfully alert the user
3. To ensure that device is recording valid values and in every possible light condition
c on modern inexpensive systems.
4. To optimize the time complexity of our anomaly detection algorithm
5. To classify the normal and anomalous videos

1.7 Methodology Used

The proposed methodology uses a machine learning approach for the anomaly detection.
1. Data Collection
Data is recorded by using surveillance cameras, attached with detection computer
system. Live data is then transmitted to the server by the hardware using the wires or
Wi-Fi.
2. Data Processing
Data is processed through the anomaly detection algorithm which uses machine
learning to classify the data and based on the results, it generates an alert when required.
3. Fall Detection and Alert
If an anomaly is detected, an alert will be sent to the concerned person, i.e., first to the
user, to check if it’s the false alarm or not. In case of true alarm, alert will be sent to
security personnel.

1.8 Project Outcomes and Deliverables

● Anomaly Detection algorithm.
● A software that classify normal and anomalous videos
● Alert generation after anomaly detection on application.
● Window based application for the user and its security system.
● Storage for fall detection algorithm.

1.9 Novelty
 CHAPTER 3- METHODOLOGY ADOPTED

3.1 Investigative Techniques

TABLE 3.1: Investigative Techniques

S. Investigative Project Investigative Techniques Investigative Projects Examples

No. Techniques Description

1 Descriptive An investigation in which scientific questions Projects based on designing completely new
are investigated and observations of system models, concepts, algorithms
phenomenon are recorded and catalogued. etc.

2 Comparative Investigations where observations are made Comparison Based Projects (Algorithm
that compare two objects or phenomenon. based, System based etc.)

3 Experimental An organized investigation that includes a Image Processing based Projects etc.
control group and is designed to test the
hypothesis, includes independent and
dependent variables

4 Analytical An investigation in which a proposed method Image Processing based Projects etc.
is analysed to calculate its effectiveness.

5 Case Study An investigation in which the problem Software Engineering projects, quality
statement is studied in real time Assurance projects, etc.

6 Hypothesis An investigation in which a hypothesis is Image processing and mathematical

proposed to solve the problem statement and projects.
then proved

7 Questionnaire Survey An investigation in which a survey is circulated Comparison Based Projects (Algorithm
to compare results based on feedback either based, System based etc.)
from a targeted audience or from a variable
audience.

8 Literature Review An investigation in which a comprehensive All types of projects.

study of the existing projects is done.

3.2 Proposed Solution

3.3 Work Breakdown Structure

FIGURE 3.1: Work Breakdown Structure

3.4 Tools and Technologies Used

 CHAPTER 4- DESIGN SPECIFICATIONS

4.1 System Architecture

4.1.1 Component Diagram

FIGURE 4.1: Component Diagram

4.1.2 MVC Architecture

FIGURE 4.2: MVC Architecture

4.1.3 Tier Architecture

FIGURE 4.3: Tier Architecture

4.2 Design Level diagrams

4.2.1 Use case diagram

FIGURE 4.4: Use Case Diagram

4.2.2 Use Case template
TABLE 4.1: Use Case Template

Name
Login
Description This use-case depicts that if you need to use our software you need
proper user-id and password to use it for security reasons.
1. Software is uploaded on hardware.
Assumptions 2. Software should be configured properly.
3. User and Caregivers are comfortable with using software.
Trigger Users is provided user id and password for login.

Primary Actor User

Secondary Actor -

Interests This use-case is created to ensure their safety.

1. Software is built on hardware.

Preconditions 2. User have registered and have their login id and password.

1. User starts the software and enters the correct user-id and
Normal Scenario password to monitor screen displays.

Registration of user is done during software uploading.

2.
Extensions 1. If user forgets password then they can verify themselves
by forgot password option.

Alternate Scenario -

Success end conditions:

1. Login id and password verified successfully.
2. If user forgets password they can verify themselves by
forgot password option.

Post-conditions Failure end conditions:

1. Login id and password is not verified.

2. User is not verified by forgot password option.

Minimal Guarantee:
1. User is verified by customer care.

Special Requirements -

Frequency of Occurrence Login use-case is used only one time while using the software to
open homepage.

Issues -
TABLE 4.2: Use Case Template

Name Anomaly Acknowledged

The Anomaly acknowledgement in diagram depicts that when our

Description device tracks the movements of people and if any event which seem
to be odd it alerts the user so they can take action accordingly.

1. Hardware should be ON all the time.

2. Sufficient environment condition for camera.
Assumptions 3. Hardware must be compatible with software.
4. User have little knowledge to use software.
Trigger It is trigger when any unexpected event occurs.

Primary Actor User (Security head/Local Retailer)

Secondary Actor -

Stakeholders and Interests People owning shops, gyms, malls etc. who want security in their
system.

Precondition 1. All camera/cameras must be connected to hardware.

2. The user should be logged in.
3. Hardware contain GPU.
1. On occurrence of Anomaly in frame software detects it and
Normal Scenario alert is given to user.
2. On false detection user can ignore alert.

1. On true anomaly detection user informs the security handler/

Extensions attendant.
2. Attendant takes the action according to the situation.
Alternate Scenario -

Success end conditions:

1. Anomaly detected successfully.
2. Right action taken by the security
Post-conditions Failure end conditions:
1. False anomaly alert message.

Minimal Guarantee:
1. Data is recorded correctly and is then classified to
Detect the anomaly.
Performance:
1. The anomaly should be detected in real time.
Special Requirements 2. Power consumption should be less
User Interface:
1. It should be very simple for the user.
2. It should take correct decision.
Frequency of Occurrence When anomaly is detected at that time it will give alert message.

Issues False anomaly generated by the device.

4.2.3 Data Design diagram

FIGURE 4.5: Data Design Diagram

4.2.4 Class diagram

FIGURE 4.6: Class Diagram

4.2.5 Context diagram

FIGURE 4.7: Context Diagram

4.2.6 Sequence diagram

FIGURE 4.8: Sequence Diagram

4.2.7 State chart diagram

FIGURE 4.9: State Chart Diagram

4.2.8 Activity diagram

FIGURE 4.10: Activity Diagram

4.3 User Interface diagrams

FIGURE 4.11: Interface Design

4.4 System Screenshots

 REFERENCES

[1] Michael D. Breitenstein, Helmut Grabner and Luc Van Gool. “Hunting Nessie – Real-Time
Abnormality Detection from Webcams." IEEE 12th International Conference on Computer Vision
Workshops (ICCV Workshops) 118.18 (2009): 2537-2543.

[2] R. Schuster, R. M¨orzinger, W. Haas, H. Grabner and Luc Van Gool, etc. al " Real-time
detection of unusual regions in image streams." Proceedings of the international conference on
Multimedia 39.10 (2010): 105002.

[3] Chen Change Loy, Tao Xiang and Shaogang Gong, et al. “Stream-based active unusual event
detection” Procedia Computer Science 141 (2008): 358-365.

[4] Vijay Mahadevan, Weixin Li, Viral Bhalodia and Nuno Vasconcelos. “Anomaly Detection in
Crowded Scenes." Instrumentation Science & Technology 44.4 (2010): 333-342.