Abnormal Activity Detection in ATM

Using Deep learning

Abstract

Abnormal human action recognition from digital videos is a hot topic in the field of
computer vision. It has a pretty assortment of applications in a myriad of fields such as
video surveillance, abnormal human-computer interaction, visual information retrieval,
and unmanned driving. With the exponential growth of surveillance data on the Internet
in recent years, how to implement effective and efficient analysis of the video data is
extremely crucial. Traditional machine learning methods that only extract computable
features have limitations and do not suit massive visual data, meanwhile deep learning
methods, especially convolutional neural networks, have gained great attainments in this
field.

The goal of abnormal human action recognition is to classify patterns so as to

understand abnormal human actions from visual data and export corresponding tags. In
addition to spatial correlation existing in 2D images, abnormal human actions in a video
own the correlation in temporal domain. Due to complexity of abnormal human actions,
changes of perspectives, background noises, and lighting conditions will affect the
recognition.

In order to solve these thorny problems, we propose CNN+LSTM, HMDB-51

dataset is employed to test these algorithms and gain the best results.

Keywords: Abnormal human action recognition, convolutional neural network, deep

learning, LSTM,

I
 Table of Contents

Abstract..............................................................................................................................I
Table of Contents..............................................................................................................II

Chapter 1 Introduction.........................................................................................................

1.1...................................... Background and Motivations

1.2..................................................... Objective

Chapter 2 Literature Review...............................................................................................

2.1................................................... Introduction

2.2................................................. Deep Learning

Chapter 3 Methodology.......................................................................................................

3.1...................................................... Design

3.2.................................................. CNN+LSTM

3.2.1................................ Convolutional Neural Network

I
 3.2.2................................... Long Short-Term Memory

Chapter 4 Expected Results.................................................................................................

4.1........................Data Collection and Experimental Environment

Chapter 5 Analysis and Discussions...................................................................................

5.1...................................................... Analysis

5.1.1.................................................. Dataset

5.1.2....................................... Experimental Design

5.1.3.................Action Recognition Based on CNN+LSTM Model

References...........................................................................................................................

I
Chapter 1
INTRODUCTION

Firstly, the background and motivation of this report are explicated. In this report, we expound

the development of deep learning methods and machine vision algorithms in the field of

artificial intelligence as well as a wide spectrum of applications for abnormal human action

recognition.

1.1 Background and Motivations

Abnormal human action recognition from videos is based on the analysis of a sequence of
video frames by using computers, so as to automatically find abnormal human actions
without manual operations (Popoola & Wang, 2012). In the era of the Internet with mobile
phones, people’s daily lives have been surrounded by access control of building gates,
traffic sensors, security cameras, and many others. The ubiquitous cameras enable
everyone's actions in public to be monitored, identification of abnormal human actions in
surveillance videos has tremendous significance in the field of cybersecurity (Bruno,
Mastrogiovanni, Sgorbissa, Vernazza & Zaccaria, 2013). In addition, analysis and
understanding of abnormal human actions in digital videos encapsulate multiple interesting
research topics such as object detection (Burgos-Artizzu, Dollár, Lin, Anderson & Perona,
2012), semantic segmentation, motion analysis, etc. Hence, abnormal human action
analysis has a broad spectrum of applications including intelligent surveillance, intelligent
care.

Traditional methods in machine learning for abnormal human action recognition

extracted visual features primarily based on abnormal human observations (Aggarwal &
Xia, 2014). It is subject to a great amount of abnormal human experience and background
knowledge. Most of these algorithms only performed well based on the exact dataset for a
specific experiment. At present, there are a huge number of digital video footages available
on the Internet, like YouTube. It is impossible to satisfy the demand to annotate all tags
5
and extract the features only based on our abnormal human labor. We need machines to
learn how to extract the characteristics of the abnormal human action.

Fortunately, the surge of deep learning methods in recent years has provided a
solution. Deep learning algorithms generate feature maps based on artificial neural
networks (Xiao, Xu & Wan, 2016). Deep learning has remarkable achievements in the
field of computer vision, natural language processing and robotics. However, the
applications of deep learning to abnormal human motion recognition are still in its early
stages, as it has only emerged in the last few years. Meanwhile, abnormal human motion is
relatively complex, the relevant motion analysis is affected by various determinants such
as chaotic background, lighting conditions, image acquisition, and insufficient pattern
classes, etc. (Baccouche, Mamalet, Wolf, Garcia & Baskurt, 2011). Deep learning thus has
a large room for developing abnormal human action recognition. Additionally, deep
learning has extremely vital value to implement self-learning and transfer learning.

In this Internet era, we are experiencing exponential growth of various types of data
(Schmidhuber, 2015), a typical example is the huge volume of digital videos generated
from CCTV (i.e., Closed-Circuit Television) systems. This has provided the opportunity
for developing applications of deep learning methods.

Moreover, high-performance computing (HPC) platform is the engine, which drives

the development of deep learning (Shuai Zhang, Yao, Sun & Tay, 2019). Deep learning
models require a large number of data samples. The previous HPC hardware was not
enough to be applied for training hundreds of layers of deep learning models. In 2011,
Google DeepMind accommodated 1,000 machines and 16,000 Graphics Processing Units
(CPU) to train a deep learning model with up to a billion neurons (Holcomb, Porter, Ault,
Mao & Wang, 2018). Now, with only a few GPUs, we accomplish the same attainments of
calculations and iterations. Therefore, the rapid development of GPUs, supercomputers,
cloud computing, and other HPC platforms makes deep learning as real reality (Chang, Fu,
Hu & Marcus, 2016). The strong computing power is conducive to the rapid
implementations and validations of deep learning algorithm. The accumulated experience
imposes us to amend the models, and further advance the accuracy of the model.
6
 The structural innovation of artificial neural networks is efficient and promising.
Throughout continuous optimization of deep learning methods, visual objects on an image
are evidently and distinctly identified. In the field of biometrics, complex environment
such as orientation, lighting, angle of view, resolution will influence the recognition of
abnormal human faces and actions. This requires that deep learning models have strong
generalization ability for further optimization. An appropriate number of neural network
layers will improve the performance of the deep learning model (LeCun, Bengio & Hinton,
2015).

Computer vision is a machine vision in which computers are able to assist our abnormal

human beings to recognize, track, and understand visual objects (Szeliski, 2010). It has been

one of the hotspots in the field of artificial intelligence. Evidences show that 80%~85%

abnormal human perception, learning, cognition activities are mediated by using our visual

system (Parker, 2010). In artificial intelligence, the acquisition of information is particularly

important. Machine vision is the main way to obtain information. Computer acquisition and

processing of visual information is an important step to realize artificial intelligence.Video

footages have been taken into account in big data. At present, 70%~80% traffic data is from

videos, which are thought as the big visual data (Koresh & Deva, 2019). The massive videos

bring us new challenges in storage, transmission, processing, management, and other aspects as

well as great opportunities, of course. These visual data was collected from a better resource as

a dataset. The continuous expansion and updating of the data set has made a valuable

contribution to future experiments. Thus, letting computers analyze and understand the visual

data has become a shortcut path to observe and understand our natural world (Beijbom, 2012).

Deep learning has a powerful learning representation ability, which enables machine
vision to perform better than abnormal human visual system in visual object recognition.
Hence, computer vision can gain more visual information from digital images than our
abnormal human visual system (Grauman & Leibe, 2011). In ImageNet Large Scale Visual
7
Recognition Challenge 2016 (ILSVRC2016), the error rate of image recognition was
nearly 2.9%, far lower than the error 5.1% of our abnormal human visual system
(Russakovsky et al., 2015). The rapid progress of computer vision and the swift
development of deep learning not only broaden the applications of our vision, but also
deepen the research work related to deep learning (Junior, Musse & Jung, 2010).

Video retrieval and analysis are based on mobile phones, digital cameras, tablets, and
other portable devices, which instantly shoot and transmit video footages in a convenient
and speedy way. It makes the volume of visual data on the Internet growing at an
exponential rate. By the end of 2015, YouTube had more than 500 video uploadings per
minute. A bulk of these videos are currently annotated with textual tags labelled by
uploaders (Mühling et al., 2017). However, manual annotation is obviously tiresome and
inefficiency. The subjectivity of these labels results in bias and leads to low accuracy of
pattern classification and visual information retrieval (Dong & Li, 2018).

Therefore, abnormal human action recognition is necessitated to analyze the video

content and automatically adjust the annotations, which is able to effectively lessen the
diversity of abnormal human subjectivity and successfully promote the accuracy of video
retrieval. In addition, abnormal human action recognition is able to automatically purge
inappropriate videos that are not suitable for dissemination on the Internet.

In surveillance monitoring, indoor or outdoor surveillance cameras around the world

constantly generate massive visual data (Shao, Cai & Wang, 2017). The uninterruptedly
surveillance data from the networked surveillance cameras extremely relies on manually
grouping. Spotting abnormal and suspicious behaviors, such as handbag or wallet abandon
in real time, have become extremely tough from a mass of video footages. When
surveillance staff is distracted, hazardous behaviors will not be found in time so as to take
effective measures, which endangers public security and safety. Therefore, the abnormal
action recognition assists security staff to fulfil the real-time monitoring work, which gains
the tradeoff between costs and work efficiency (Guo, Luo & Yong, 2015).

Intelligent surveillance based on abnormal human action recognition is able to

8
automatically monitor abnormal and hazardous events as well as risky and perilous
abnormal human actions from the elderly (such as tumble, falls, etc.). Surveillance alarms
will be sent out timely and accurately. This will cut down the delay of medical treatment
(Chen, 2010). Similarly, intelligent surveillance is able to accomplish real-time assistance
for patients, children, and the disabled through monitoring.

1.2 Objective
In the field of deep learning, an efficient and automatic method is urgently needed to
classify and identify abnormal human actions in a large amount of video data. Although
deep learning method for image recognition has made a wealth of research results, the time
series analysis is still in the process of continuous improvement in the aspect of video
recognition. There is still a room to improve the accuracy of abnormal human motion
recognition. Hence, the abnormal human action for ATM security.

9
Chapter 2

Literature Review

Machine vision is a hot topic at present. In this chapter, we introduce the state-of-the-art

methods in deep learning and machine vision. Similarly, this chapter summaries the past work

in the field of abnormal human behavior analysis. The new methods for abnormal human action

recognition are put forward.

2.1 Introduction
In recent years, a great deal of breakthroughs have been attained in the field of machine
vision and deep learning. A plenty of methods for abnormal human action recognition
based on deep learning have been explored and exploited (Pleshkova, Bekyarski &
Zahariev, 2019). Compared with machine learning methods for abnormal human action
recognition, deep learn approaches do not require a specific type of abnormal human
experience and domain knowledge. Instead, abnormal human actions in a video are
identified directly in the end-to-end way (Zhang, Quan & Ren, 2016). According to feature
extraction methods, the approaches are grouped into two categories, e.g., abnormal human
action recognition based on skeletons, abnormal human action recognition based on feature
maps. Among the deep learning methods, spatiotemporal networks and the Two-Stream
networks are the prominent ones in abnormal human action recognition. In these methods,
CNN and RNN are most popular (Baisware, Sayankar & Hood, 2019).

A multimodal learning approach was proposed for the recognition and classification of
abnormal human actions. Multiple deep neural networks are used for different modal
information to dig out the multimodal characteristics of abnormal human action (Khan,
Rahmani, Shah & Bennamoun, 2018). This is beneficial to the in-depth study of deep
neural network. In 2017, 3D convolutional neural network (3D-CNN) and two-way long
short-term memory network (ConvLSTM) were proposed which fulfil abnormal human
action recognition by using Support vector Machine (SVM) classifier from multimodal and
10
spatiotemporal information (Jing, Ye, Yang & Tian, 2017). A deep dynamic neural network
(DDNN) was designed to implement action recognition from input data under multimodal
framework, which extracts spatiotemporal features from RGB (color map) and RGB-D
images (Wu et al., 2016). A scene-flow dynamic model was deployed to extract visual
features from RGB and depth images, which were imported for training by using CNN
networks (Wang et al., 2018). A 3D CNN network was taken into account to learn high-
level features from the original images and calculated the position and angle of bone joint
information. The two features were fused by using SVM for abnormal human action
classification (Li, 2017). In 2018, CNN and RNN were integrated together to cope with the
spatiotemporal information of abnormal human actions and achieved promising results.The
effect of different frame sequence correctly classified test data up to 96.66% (Russo,
Filonenko & Jo, 2018).

Training deep learning models needs a huge number of datasets. In this digital era,
everybody including adults or children is allowed to upload videos and photos to the
Internet for sharing. This favorite in social networks thus has led to an explosion of video
and image dataset. A vast number of images and videos provide the resources for resolving
the problems in computer vision and deep learning (Morota, Ventura, Silva, Koyama &
Fernando, 2018). For example, the HMDB-51 dataset and UCF101 dataset, both were
collected from the Internet. The HMDB-51 is collected by Brown university in 2011. In
computer vision, abnormal human action recognition plays an important role in our daily
life, which is applied to video surveillance, robotics and unmanned vehicles, medical
service, anomaly analysis, abnormal human- computer interaction, and other aspects
(Baisware, Sayankar and Hood, 2019). These applications make us have extensive and
important significance in the study of abnormal human motion recognition.

Specifically, the results of abnormal human action recognition provide evidence for
video surveillance, but also accommodate clues for lessening the criminals by using deep
learning methods at present (Räty, 2010). In terms of medical services, abnormal action of
patients can be identified through motions and then reported to doctors and nurses.
Otherwise, In addition, the abnormal human action recognition also has the ability to

11
analyze the patient's movement to assist the diagnosis and rehabilitation (Li et al., 2017). In
the aspect of analysis of abnormal abnormal human actions, the actions in the given videos
are labelled, archived, classified, and predicted. If there is a large deviation between the
actual action and the predicted action, the possibility of abnormal action is predicted so as
to prevent the occurrences of incidents. This has taken great effects in monitoring the
elderly and children as well as the disabled and patients in hospitals (Roh, Heo & Whang,
2019).

In terms of abnormal human-computer interaction, due to the demand for

continuously ameliorating our life quality, people are enjoying more and more convenient
services. The demand for life services has spurred the rise of the robotics (Rautaray &
Agrawal, 2015). In order to improve robots serves, abnormal human action recognition can
help robots understand abnormal human behavior. The deep learning algorithms can
effectively identify the motion information and translate it into machine-readable language.
This has a great help in uplifting the understanding ability of robots. Thus, computers can
also plan and respond interactively to abnormal human behavior.

Compared with manual operation, deep learning has the advantages of high efficiency
and unlimited working time in abnormal human action recognition. It plays a pivotal role
in intelligent identification which saves a lot of manpower and other resources. In robotics,
if a deep learning model has been trained very well, the communications between
abnormal human and computers (HCI) will be accurate, facile, and fluency (Rodríguez-
Moreno et al., 2020).

2.2 Deep Learning

Deep learning is fundamentally based on Multilayer Perceptron (MLP) or deep neural
networks. The most effective multilayer neural network is CNN. At present, CNN
performs well in text, image, audio, and video signal processing (Strong, 2016). In terms of
fundamentals, deep learning mimics the working mechanism of our abnormal human
brains, especially CNN models. From a statistical point of view, deep learning as a neural
12
work is to predict the class label of a given sample with a probability. Training a model
from the given dataset and testing the trained model to predict class labels of new samples
require that the test samples and the training samples have the similar pattern classes which
have been labelled in the homogeneous datasets with the predefined tags (Sadoughi et al.,
2018).

AI is defined as a computer to emulate our abnormal human intelligence. Machine

learning enables computers to execute programs of various algorithms. Deep learning is a
subfield of machine learning which enables computers to continuously learn from the
training dataset as well as abnormal human experience. In the past decades, deep learning
and machine vision have made a real reality (Wang, Zhang & Wei, 2019). The deep neural
networks and graphics processing unit (GPU) or neural processing unit (NPU) have
powerful capability for intensive computing. This provides a powerful hardware
infrastructure for the development of deep learning.

Compared to traditional machine learning methods, there are a vast number of

advantages of deep learning approaches. Firstly, noisy information is able to be filtered
out, deep learning algorithms classify patterns through training dataset, export class labels
for solving the classification problems in our real world. Moreover, deep learning
algorithms detect and recognize computable events based on valuable patterns, especially
time series analysis. The unstructured or structured data is conveniently classified by using
deep belief methods (DBM) and CNN. Although artificial neural networks emulate the
mechanism of our abnormal human brains, deep learning models need time to gradually
learn how to solve problems in the end-to-end way (LeCun, Bengio & Hinton, 2015) which
means the imported data and its features will be independent on the design of neural
networks.

Deep learning also encounters challenges, e.g., a large amount of data as a prerequisite
is imported to deep learning models for model training, successful pattern classification by
using deep neural networks is very tough to be collected. The overfitting in pattern
classifications will dilute the model effectiveness. A well-trained deep neural network

13
should not only carry out pattern classification, but also outperform others for a newly
assigned task, namely, transfer learning (Voulodimos, Doulamis, Doulamis &
Protopapadakis, 2018). With the rise of deep learning methods in machine vision, deep
neural networks are extremely suitable for empirical applications, eventually serve the
community in practical ways (Liu et al., 2017).

In the near future, more deep learning methods will be exploited in the areas such as
remote diagnosis, intelligent surveillance, autonomous vehicles, robotics, and smart
agriculture (Freeman, 2012). Because computing power is soaring at present, deep learning
methods will have a long-standing impact on the field of computational vision. In addition,
deep learning will greatly boost the methods in machine vision and digital image
processing. Undoubtedly, intelligent learning will benefit from deep learning.

14
Chapter 3

Methodology

In this chapter, we firstly put forward the overall scheme and structure of abnormal human action

recognition. The methods are explained thoroughly. The parameters and details of the algorithm are

analyzed. The implementation is based on accurately recognizing abnormal human actions in given

videos. This chapter is the key one of this report.

3.1 Design

The ultimate goal of this report is to implement abnormal human action recognition from
the given videos. We sparse the footages and imported the frames as the input data. Three
deep learning methods are applied to generate feature maps for abnormal human action
recognition. Throughout network training, we recognize abnormal human actions and
finally export the class tags. As shown in Figure 3.1, the overall pipeline of this design is
shown as follows.

Featur
Input
Build Pretre e
Recog Output
video
featur atmen selecti
nition result
e t on and
netwo extrac
rk tion

Figure 3.1 The pipeline of abnormal human action recognition

The first step is to input video information. The second step is to preprocess the input
video data and parse video frames. The third step is to analyze the characteristics of each
movement and extract them. The fourth step is to build the feature network of the action
with the extracted features. The fifth step is to use the feature network to identify the
actions. Finally, we output the result.
15
3.2 CNN+LSTM
3.2.1 Convolutional Neural Network

Convolutional Neural Network (CNN) is a kind of feedforward neural network, which is

chiefly composed of input layer, convolutional layer, pooling layer, full connection layer,
and output layer. The convolutional layer and pooling layer are alternated. The activation
functions of the CNN are different with various structures (Wigington, et al. 2017).

Figure 3.2 Basic structure of CNNs

The convolutional layer of CNN contains one or more feature planes. Each feature
plane is composed of neurons with a pattern, the neurons in the same feature plane share
the weights. The shared weights are related to the convolution kernel, the reasonable
weights are obtained by optimizing the parameters of the network through model training.
By extracting local features and synthesizing them, the CNN network not only obtains
global features but also reduces the number of neuron nodes. At this point, the number of
neurons is still very large, by setting the weight of each neuron equally, the number of
network parameters will be greatly reduced.

On the first 𝑚 convolution layer, its output is 𝑦𝑚, the output of the first 𝑘 convolution

𝑘 16
kernels is 𝑦𝑚,

𝑦𝑚 = 𝛿(∑ 𝑛−1 𝑦𝑚−1 ∗ 𝑊𝑚 + 𝑏𝑚), (3.1)

𝑘 𝑦𝑖 ∈𝑀𝑘 𝑖 𝑖𝑘 𝑘

where 𝛿 (·) is activation function, 𝑀𝑘 is a layer of characteristic collection, 𝑊𝑖𝑘𝑚 is the

convolution kernel, ∗ is convolution. 𝑏𝑘𝑚 is bias or offset. The pooling layer follows the
convolutional layer to reduce the dimension of input data and accelerate the convergence
of the network training.

The other is to remove redundant features and prevent network over fitting. Each neuron in
the full connection layer is connected with all neurons in the previous layer. Throughout
the full connection layer, all local features extracted in the previous layer can be integrated
to form the overall features. Each neuron in the full connection layer operates with an
activation function, which is then transferred to the output layer.

3.2.2 Long Short-Term Memory

In recurrent neural network (RNN), it is impossible to distinguish the importance of the

information at each layer, which results in the useless information being stored in the
memory unit. However, the truly valuable information is squeezed out (Ma & Hovy, 2016)
based on RNN. Therefore, Long Short-term Memory (LSTM) was proposed. By using
memory unit and gate mechanism, the problems of gradient vanishing and gradient
exploding occurs in the training process of RNN were effectively overcome. Each unit of
LSTM network contains memory unit, input gate, forget gate and output gate.

The structure of LSTM is shown in Figure 3.3, which is applied to control the amount
of information updated by using memory unit. That is, how much information about the
current state of the network needs to be saved as internal state. 𝑓𝑡 refers to forget door,
which is used to control before how much memory unit 𝐶𝑡−1 is saved to the memory unit
𝐶𝑡. 𝑂𝑡 stands for the output, which is used to control 𝐶𝑡 namely, how much output to the
next hide status ℎ𝑡, ℎ𝑡−1 is the hidden layer at time t-1.
17
 LSTM memory unit receives the current information at time of t, external state ℎ𝑡−1
and internal state 𝐶𝑡−1 at time t-1. 𝑋𝑡 and ℎ𝑡−1 are the input. How to calculate the input
gate, forget gate, and output gate is shown as eq. (3.2),

𝑖𝑡 = (𝑊𝑖𝑥𝑡 + 𝑣𝑖ℎ𝑡−1 + 𝑏𝑖)

{𝑓𝑡 = (𝑊𝑓𝑥𝑡 + 𝑣𝑓ℎ𝑡−1 + 𝑏𝑓). (3.2)
𝑜𝑡 = (𝑊𝑜𝑥𝑡 + 𝑣𝑜ℎ𝑡−1 + 𝑏𝑜)

The memory unit 𝐶𝑡 is calculated through different gating data memory and forget
update as shown in eq. (3.3)

𝑐̃𝑡 = tanh (𝑊𝑐𝑥𝑡 + 𝑣𝑐ℎ𝑡−1 + 𝑏𝑐) . (3.3)

{ 𝑐𝑡 = 𝑐𝑡−1∗𝑓𝑡 + 𝑐̃𝑡 ∗ 𝑖𝑡

18
 The updated hidden state is calculated as

ℎ𝑡 = 𝑜𝑡 ⊗ tanh (𝑐𝑡), (3.4)

where 𝜎 is sigmoid functions, 𝑤 is weights, 𝑣 is weight, 𝑏 is biase, ∗ is the operator for dot
product.

LSTM memory unit receives 𝑋𝑡, ℎ𝑡−1, in the form of 𝜎 activation function, produces
control input respectively by using input gate (𝑖𝑡), forget gate (𝑓𝑡), the output gate (𝑂𝑡), 𝜎 is
sigmoid function. LSTM is based on the current moment, updating the memory unit and
generate the current state of the moment output ℎ𝑡, as subsequent additional input, t + 1
time and again.

In LSTM, ̃𝑡 and are used to update the internal state, if 𝑖𝑡 tends to 0, 𝑐̃𝑡 is very
small amounts of information that was saved to the internal states; on the other hand, this
method is used to decide how much the information will be saved. Forget gait 𝑓𝑡 controls
𝐶𝑡−1 and updates 𝐶𝑡. Throughout the training process, LSTM network optimizes weights
𝑤, 𝑣 and offset , makes sure that unit value 𝑐 spontaneously is adjusted in temporal domain.

Abnormal human actions contain not only spatial information but also temporal
information. CNN networks cannot fully use of the temporal information of the videos.
Although the recognition method based on 3D CNN and the method based on dual-flow
CNN take use of the timing information simultaneously, the main consideration is the
short-term action information of adjacent frames.

If temporal information between video frames is not taken into account,

misclassification easily occurs. For a simple example, “Pick up the water glass”, “Drink
water”, and “Put down the water glass” as three video frames. We expand the three frames
out of order, regardless of the sequence of the video frames. It is tough to say whether
current action is “picking up the glass” or “putting it down”. The output of LSTM is
determined by using the combined action of the current input and the previous historical
output. Temporal information is used to represent a sequence. The CNN+LSTM model of
this report is shown in Figure 3.3.

19
 Figure 3.3 The structure of CNN+LSTM network

First of all, the test video is split into frames to form an image dataset. This dataset is
employed as the input of single-channel CNN+LSTM for CNN pretraining. The training
results are stored in a feature dataset and the feature map is generated. The characteristic
dataset is thus input into the LSTM network as the input data. LSTM network is used to
train the sequence data. After model training, the fixed network parameters of CNN are
used for abnormal human action recognition.

In each video sequence, eight video frames are extracted at an interval as the input.
The spatial features are extracted from the given video frames. The spatial feature is input
into the LSTM to explore the temporal relationship among the video frame sequence. We
obtain the fixed network parameters after model training. We complete the learning and
training process of single-channel CNN+LSTM model. At the test stage, eight video frames
extracted with equivalence distance are taken as the input data of the CNN+LSTM model.
After spatial feature extraction and temporal relationship classification, the predicted
output value of LSTM is taken as the final classification result.

20
Chapter 4
Expected Results

4.1 Data Collection and Experimental Environment

In this experiment, HMDB-51 dataset was used as the training set. HMDB-51 has a total of
51 classes and 6,766 short videos. (URL:http://serre-lab.clps.brown.edu/resource/hmdb-a-
large-abnormal human-motion-database/#dataset)

The data comes from a vast range of sources, including movies, YouTube videos, etc.
The dataset contains 51 classes, each class contains more than 101 videos. HMDB-51 is
split into five groups:

 Facial movements (smile, laugh, chew, talk)

 Facial movements that are coordinated by something else (smoke, eat, drink)

 Common body movements (climb, dive, jump)

 Common body movements that are coordinated by something else (brush hair,
catch, draw sword)

 Abnormal human interaction between the body movements (hug, kiss, shake hands)

The difference between the visual data for abnormal human action recognition and the
data for general action recognition is that, in addition to the label of each clip, the data is
manually labelled:

 Visible body parts

 Camera motion

 Camera view point

 Video quality

21
  The number of individuals involved

Figure 4.1 The 51 classes in HMDB-51

In this experiment, abnormal human actions that can be accomplished by using a

single person are selected as the test video. In our experiments, few actions were selected,

This experiment was conducted based on the software platform MATLAB R2020a. In
recent years, MATLAB has been developed quickly in deep learning and computer vision,
with new features and showcases are released. For example, followed Computer Vision
toolbox, MATLAB simulink also launched its support package for computer vision. In our
experiments, MATLAB Image Processing Toolbox and Computer Vision Toolbox are
employed. The image processing toolbox includes visual features such as image and video
22
preprocessing and post processing, image analysis, spatial transformation, and color image
processing. After the preprocessing, the video can be imported into the nets for abnormal
human action recognition effectively.

23
Chapter 5

Analysis and
Discussions

In this chapter, we chiefly state the analysis of the experimental results and discuss the relevant

issues. After a large amount of data is obtained, the control variable method is employed to

compare and analyze the experiments. We analyze the data and explain the possibility of the

results. Throughout the analysis and discussion, the results of our experiments are clarified, the

shortcomings of the experiments will be identified.

5.1 Analysis

In terms of evaluating the methods or algorithms, our experimental results are analyzed.
The classes contained in the dataset, the characteristics of the dataset and related
classification are described in this chapter. We clarify the characteristics of the selected
dataset. Then, the whole method of this experiment is expounded. We evaluate the overall
experimental design. The advantages and disadvantages of this experiment are pointed out
and analyzed. Each algorithm identifies the characteristics of the action aspect. The
experimental data in Chapter 4 are used for evaluation. Finally, the whole experiment
assessment is accomplished.

5.1.1 Dataset

The HMDB-51 dataset contains 51 different classes. This dataset is ideal for evaluating
abnormal human action recognition. However, the dataset was collected in 2011. There are
plenty of video clips from digital movies. The description of abnormal human actions,
especially the characteristics of actions, is not obvious. The video collects the movements
of people from different angles or ages. For intelligent recognition, the dataset is very
limited. In order to achieve the desired outcomes, a larger dataset is required. Of course,
larger datasets mean more powerful hardware and efficient algorithms.

24
 In the HMDB-51 dataset, there are 56.3% of the videos contain a complete abnormal
human body. Only upper body videos are taken into account for 30.5%. Only the lower
half of the video accounted for 0.8%. Finally, only the head accounts for 12.3%. 59.1 % of
them were sports shots. That means the background is more complex. The background will
change with the person. The remaining 40.1% was filmed in still.

The shooting angle is different in dataset. That is the direction of the person shown in
the video. 40.8% are heads and 18.2% are tails. The left and right angles are 22.1% and
19.0%, respectively.

The clarity of the video includes good and low. “Good” means high quality video. In
17.1% of the high-quality videos, noses and eyes of abnormal human faces and the
details of small parts could be seen clearly. The majority of video quality is 62.1%. “Low”
refers to the low- quality video. This kind of videos only include the movement of the
abnormal human body, which is 20.8%.

5.1.2 Experimental Design

For the main experimental design of this experiment, the main variable is the difference of
recognition algorithm. So, the rest of the structure of the experiment is as consistent as
possible. In other words, the recognition video selected is the same. By identifying the
same video data, the advantages and disadvantages of different algorithms are compared.
In addition, the data as a training set is the same. Therefore, in this experiment, the dataset
of HMDB-51 is selected as the dataset of all algorithms. The results are even more telling.
The emphasis of this experiment is to analyze and compare various algorithms. It's
important to keep the rest of the variable’s constant.

5.1.3 Action Recognition Based on CNN+LSTM Model

The CNN-LSTM model makes up for the difficulty of the convolutional neural network in
modeling behavioral temporal relations. The abnormal human action recognition learns the

25
visual appearance information in video frames and the temporal relationship between video
frames simultaneously. Although CNN+LSTM's working principle and structure are clear,
what information the structure needs is still a more ambiguous question. So the ability to
recognize a single action is ideal. Spatial and temporal information for multimodal data
input. It is known that modeling leads to differentiated behavioral characteristics. By
effectively integrating multi-modal behavior characteristics, we can learn from each other
and obtain more discriminative behavior descriptors.
CNN is used in the algorithm to get the global description of the video level, increasing
the number of frames can significantly improve the classification performance. Five
pooling structures are proposed through comparative analysis.

Figure 5.1 Different feature pooling architectures

In Figure 5.1, the stacked convolutional layers are denoted by ‘C’. Blue, green, yellow
and orange rectangles represent max-pooling, time-domain convolutional, fully connected
and softmax layers respectively.

26
 (a) Conv pooling: Use max pooling after the final convolutional layer of the video
frames. The main advantage of this model is that the output information of the
convolutional layer can be retained by the maximum pooling operation in the time domain.
(b) Late pooling: Firstly, input the features after convolution into the two full
connection layers, and then pool the output. The parameters of all convolutional layers and
full connection layers are shared. Different from others, this method directly pools the
inter- frame features of higher level.

(c) Slow pooling: Use small time windows to pool the features of the frame level. The
two-step pooling strategy is adopted, max pooling is used to pool the 10 frame features.
The step is 5 (pooling here can also be regarded as the convolution kernel with size 10
convolving the one-dimensional characteristics of the input with step 5. After each maximum
pooling layer is the full connection layer with shared parameters. The second step is to maximize
pooling on all the output of the full connection layer. The characteristic of this method is to
combine the local time features before pooling the inter-frame advanced features.

(d) Local pooling: Similar to slow pooling, this method is piecewise pooling of frame-
level features after the final convolutional layer, except that there is only one pooling stage.
After pooling, we connect two full connection layers and share parameters. Finally, we
input all output into a large softmax layer. Reducing a pooling layer can avoid the potential
loss of timing characteristics.
(e) Time-domain convolution layer is added before the pooling layer for this method.
The convolutional layer is composed of 256 3 × 3 convolution kernels with a stride length
of 5. This method can capture the local relationship between frames through a small-time
window.
The Conv-pooling method works best and outperforms all other feature pooling
architectures. The late-pooling method is the worst. Relevant experiments show that it is
important to retain the original spatiotemporal information before pooling. In this method,
the frame features are directly input to the full connection layer, and the spatial and
temporal information is lost to obtain the high-level features. The effect of the Time-
Domain Convolutional method is also poor, which indicates that the effect of the
convolutional layer in the time domain is not as good as max pooling.

27
 References
Aggarwal, J. K., & Xia, L. (2014). Abnormal human activity recognition from 3D data: A
review.
Pattern Recognition Letters, 48, 70-80
Agrawal, P., Girshick, R., & Malik, J. (2014). Analyzing the performance of multilayer
neural networks for object recognition. In European Conference on Computer
Vision
An, N., Yan, W. (2021) Multitarget tracking using Siamese neural networks. ACM
Transactions on Multimedia Computing, Communications and Applications
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., & Baskurt, A. (2011). Sequential deep
learning for abnormal human action recognition. In International Workshop on
Abnormal human Behavior Understanding
Baisware, A., Sayankar, B., & Hood, S. (2019). Review on recent advances in abnormal
human action recognition in video data. In International Conference on Emerging
Trends in Engineering and Technology-Signal and Information Processing
(ICETET-SIP-19) (pp. 1-5)
Barros, P., Parisi, G. I., Jirak, D., & Wermter, S. (2014). Real-time gesture recognition
using a abnormal humanoid robot with a deep neural architecture. In IEEE-RAS
International Conference on Abnormal humanoid Robots
Beijbom, O. (2012). Domain adaptations for computer vision applications. arXiv preprint
arXiv:1211.4860
Bertinetto, L., Valmadre, J., Henriques, J. F., Vedaldi, A., & Torr, P. H. (2016). Fully-
convolutional siamese networks for object tracking. In European Conference on
Computer Vision
Bruno, B., Mastrogiovanni, F., Sgorbissa, A., Vernazza, T., & Zaccaria, R. (2013). Analysis
of abnormal human behavior recognition algorithms based on acceleration data. In
IEEE International Conference on Robotics and Automation
Burgos-Artizzu, X. P., Dollár, P., Lin, D., Anderson, D. J., & Perona, P. (2012). Social
behavior recognition in continuous video. In IEEE Conference on Computer Vision
and Pattern Recognition
Chang, H. S., Fu, M. C., Hu, J., & Marcus, S. I. (2016). Google deep mind's AlphaGo.
OR/MS Today, 43(5), 24-29
Chen, C., Liu, M.-Y., Tuzel, O., & Xiao, J. (2016). R-CNN for small object detection. In
Asian Conference on Computer Vision
Chen, Y. (2010). Study of moving object detection in intelligent video surveillance system.
In International Conference on Computer Engineering and Technology
Chen, Y., Li, W., Sakaridis, C., Dai, D., & Van Gool, L. (2018). Domain adaptive Faster
R- CNN for object detection in the wild. In IEEE Conference on Computer Vision
and Pattern Recognition
Cui, W., Yan, W. (2016) A scheme for face recognition in complex environments. Int. J.
Digit.
Crime Forensics 8(1): 26-36
Dašić, P., Dašić, J., & Crvenković, B. (2017). Improving patient safety in hospitals through
28
 usage of cloud supported video surveillance. Macedonian Journal of Medical
Sciences, 5(2), 101
Dong, Y., & Li, J. (2018). Video retrieval based on deep convolutional neural network. In
International Conference on Multimedia Systems and Signal Processing
Du, Y., Fu, Y., & Wang, L. (2015). Skeleton based action recognition with convolutional
neural network. In Asian Conference on Pattern Recognition (ACPR)
E-Martín, Y., R-Moreno, M. D., & Smith, D. E. (2015). A fast goal recognition technique
based on interaction estimates. In International Conference on Artificial
Intelligence
Fan, Y., Lu, X., Li, D., & Liu, Y. (2016). Video-based emotion recognition using CNN-
RNN and C3D hybrid networks. In ACM International Conference on Multimodal
Interaction
Fernández-Llatas, C., Benedi, J.-M., García-Gómez, J. M., & Traver, V. (2013). Process
mining for individualized behavior modeling using wireless tracking in nursing
homes. Sensors, 13(11), 15434-15451
Freeman, H. (2012). Machine Vision: Algorithms, Architectures, and Systems. Elsevier.
Girshick, R. (2015). Fast R-CNN. In IEEE International Conference on Computer vision
Gopi, E., Lakshmanan, N., Gokul, T., & KumaraGanesh, S. (2006). Digital image forgery
detection using artificial neural network and auto regressive coefficients. In
Canadian Conference on Electrical and Computer Engineering
Grauman, K., & Leibe, B. (2011). Visual object recognition. Synreport Lectures on
Artificial Intelligence and Machine Learning, 5(2), 1-181
Gu, D., Nguyen, M., Yan, W. (2016) Cross models for twin recognition. Int. J. Digit. Crime
Forensics 8(4): 26-36
Gu, Q., Yang, J., Yan, W., Klette, R. (2017) Integrated multi-scale event verification in an
augmented foreground motion space. In PSIVT (pp.488-500)
Guo, S., Luo, H., & Yong, L. (2015). A big data-based workers behavior observation in
China metro construction. Procedia Engineering, 123, 190-197
Holcomb, S. D., Porter, W. K., Ault, S. V., Mao, G., & Wang, J. (2018). Overview on
deepmind and its AlphaGo. In International Conference on Big Data and Education
Hsu, S. C., Huang, C. L., & Chuang, C. H. (2018). Vehicle detection using simplified
fast R-CNN. In International Workshop on Advanced Image Technology (IWAIT)
(pp. 1-3)
Isaacson, M., & Shoval, N. (2006). Application of tracking technologies to the study of
pedestrian spatial behavior. The Professional Geographer, 58(2), 172-183
Ji, H., Liu, Z., Yan, W., Klette, R. (2019) Early diagnosis of Alzheimer's disease based on
selective kernel network with spatial attention. In ACPR (2, pp.503-515)
Ji, S., Xu, W., Yang, M., & Yu, K. (2012). 3D convolutional neural networks for abnormal
human action recognition. IEEE Transactions on Pattern Analysis and Machine
Intelligence, 35(1), 221-231
Jiang, H., & Learned-Miller, E. (2017). Face detection with the Faster R-CNN. In IEEE
International Conference on Automatic Face & Gesture Recognition (FG 2017)
Jing, L., Ye, Y., Yang, X., & Tian, Y. (2017). 3D convolutional neural network with multi-
model framework for action recognition. In IEEE International Conference on
Image Processing (ICIP) (pp. 1837-1841)
29
Jung, I., Son, J., Baek, M., & Han, B. (2018). Real-time MDNet. In European Conference
on Computer Vision (ECCV)

30
Junior, J. C. S. J., Musse, S. R., & Jung, C. R. (2010). Crowd analysis using computer
vision techniques. IEEE Signal Processing Magazine, 27(5), 66-77
Khan, R. Z., & Ibraheem, N. A. (2012). Hand gesture recognition: A literature review.
International Journal of Artificial Intelligence & Applications, 3(4), 161
Khan, S., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018). A guide to
convolutional neural networks for computer vision. Synreport Lectures on
Computer Vision, 8(1), 1-207
Koo, S.-y., Lim, J. G., & Kwon, D.-s. (2008). Online touch behavior recognition of hard-
cover robot using temporal decision tree classifier. IEEE International Symposium
on Robot and Abnormal human Interactive Communication
Koresh, M., & Deva, J. (2019). Computer vision based traffic sign sensing for smart
transport.
Journal of Innovative Image Processing (JIIP), 1(01), 11-19
Lan, W., Dang, J., Wang, Y., & Wang, S. (2018). Pedestrian detection based on YOLO
network model. In IEEE International Conference on Mechatronics and
Automation (ICMA)
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning.Nature, 521(7553), 436-444
Liu, C., Yan, W. (2020) Gait recognition using deep learning. Handbook of Research on
Multimedia Cyber Security (pp.214-226)
Li, C., Wu, X., Zhao, N., Cao, X., & Tang, J. (2018). Fusing two-stream convolutional
neural networks for RGB-T object tracking. Neurocomputing, 281, 78-85
Li, C., Song, D., Tong, R., & Tang, M. (2019). Illumination-aware faster R-CNN for
robust multispectral pedestrian detection. Pattern Recognition, 85, 161-171
Li, F., Zhang, Y., Yan, W., Klette, R. (2016) Adaptive and compressive target tracking
based on feature point matching. In ICPR (pp.2734-2739)
Li, H., Li, Y., & Porikli, F. (2015). DeepTrack: Learning discriminative feature
representations online for robust visual tracking. IEEE Transactions on Image
Processing, 25(4), 1834-1848
Li, H., Lin, Z., Shen, X., Brandt, J., & Hua, G. (2015). A convolutional neural network
cascade for face detection. In IEEE Conference on Computer Vision and Pattern
Recognition
Li, J. (2017). Parallel two-class 3D-CNN classifiers for video classification. In
International Symposium on Intelligent Signal Processing and Communication
Systems (ISPACS) (pp. 7-11)
Li, J., Liang, X., Shen, S., Xu, T., Feng, J., & Yan, S. (2017). Scale-aware Fast R-CNN for
pedestrian detection. IEEE Transactions on Multimedia, 20(4), 985-996
Li, R., Nguyen, M., Yan, W. (2017) Morse codes enter using finger gesture recognition. In
DICTA (pp.1-8)
Li, W., Hsieh, C., Lin, L., & Chu,W. (2012). Hand gesture recognition for post-stroke
rehabilitation using leap motion. In International Conference on Applied System
Innovation (ICASI) (pp. 386-388)
Li, Y., Zhang, H., & Shen, Q. (2017). Spectral–spatial classification of hyperspectral
imagery with 3D convolutional neural network. Remote Sensing, 9(1), 67
Li, Z., & Zhou, F. (2017). FSSD: Feature fusion single shot multibox detector.
arXiv:1712.00960
31
Approach,

Prototype approach and production approach (college funding is required, development time >1 year,
NDA sign),

 Same principle is applied for detecting normal and abnormal behavior in patients.
 HMBD-51 data set contains about 2 GB of video data for 7000 clips over 51 classes
 HMBD-51 data set is trained for 51 specific actions which are general to day to day activities,
like playing, kicking the ball, drinking, eating and more. But if we are adapting these actions to a
patient monitoring systems some of the activities might not be related like for eg fencing, push
ups and horse rides.
 If we can get a normal and abnormal dataset then it becomes very easy to classify. This can be
our custom data
 Apparently, we need a very huge set of data and a very powerful machines to train the custom
data. Also, there are no data sets that are available online.
 For prototyping purpose we can classify HMBD data sets into normal and abnormal behavior as
per our need. With this we are just using transferring our inferences of normal and abnormal
from the HMBD datasets.
 LSTM can be used to classify these actions.
 High end computer atleast with 4-16GB graphics card is needed if we are training the model by
ourselves. But for prototyping we can make use of pre trained model and in our personal
computer we can just classify the given video sequence.
 Accuracy of the outcome depends on the model that we have
◦ for custom model with custom data set – expect very high accuracy approx. 75 – 95%
◦ for HMBD model – expect moderate accuracy approx. 50-60 %
 Since its about patient monitoring we can even add fall detection/vibration detection using
sensors. With hardware interfacing.
 Body temperature, vital signs monitoring
https://www.mathworks.com/help/deeplearning/ug/classify-videos-using-deep-learning.html
https://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/#Downloads

32