Online Hand Gesture Recognition &

Classification for Deaf & Dumb

Nitesh S. Soni1, Prof. Dr. M. S. Nagmode2, Mr. R. D. Komati3
Student1, Professor2, Ass. Professor3
Department of Electronics and Telecommunication
MIT College of Engineering, Pune
{soninitesh.nitesh1, manoj.nagmode2, rdkomati3}@gmail.com

ABSTRACT: Sign language is widely used by interaction, communication disparity, education,

individuals with hearing impairment to communicate behavioural problems, mental health, and safety
with each other conveniently using hand gestures. concerns [3]. The ways in which one can interact
However, non-sign-language speakers find it very with computer are either by using devices like
difficult to communicate with those with speech or keyboard, mouse or via audio signals, while the
hearing impairment since it interpreters are not former always needs a physical contact and the latter
readily available at all times. Many countries have is prone to noise and disturbances [3].
their own sign language, such as American Sign Physical action carried by the hand, eye, or any
Language (ASL) which is mainly used in the United part of the body can be considered as gesture. Hand
States and the English-speaking part of Canada. The gestures are the most suitable and easily interpretable
proposed system helps non-sign-language speakers in for humans. Here, single handed gesture recognition
recognising gestures used in American Sign system is proposed, it uses right handed gestures, and
Language. The system described in this paper is are classified and recognised for the specific
implemented using MATLAB. In this approach, character. Static gesture recognition system proposed
firstly, the signs are captured using a webcam. The here does not require any colour code. The sign
images captured are then processed further and the recognition system proposed, recognizes the signs
features are extracted from the captured images using with great accuracy and with less overheads in
PCA. Comparison of the features is done using features and time.
Euclidean Distance with the training sets. Minimum
Euclidean distance helps to recognise the character.
2. LITERATURE SURVEY
This system will enable non-sign-language speakers
to better understand and communicate with those Many novel methods have been developed in
with impaired hearing. past few years, to facilitate communication between
the sign language users and those who can’t speak
sign language. Mrs.Neela Harish, Dr.S.Poonguzhali
[1], proposed a system which depends on flex
KEYWORDS: Sign language, Euclidean Distance, sensors, accelerometers output values such as, co-
PCA. ordinates given by accelerometer and the bending
values given by the flex sensors, for the interpretation
1. INTRODUCTION
of signs. Depending on the. Suganya R,
The use of sign language is not only limited to Dr.T.Meeradevi, [2] implemented a system using
individuals with impaired hearing or speech to Feed Forward Neural Network, for identifying hand
communicate with each other or non-sign-language gestures characteristics & to train gestures captured.
speakers and it is often considered as a prominent Dhananjai Bajpai, et al. [4], proposed a system
medium of communication. Instead of acoustically enabled to decode more than 36 gestures using 6 flex
conveyed sound patterns, sign language uses manual sensors, to recognise corresponding alphabets, or
communication to convey meaning. It combines hand corresponding features. Pre-requisite for using this
gestures, facial expressions along with movements of system to perform gesture recognition and store it in
other body parts such as eyes, legs, etc. This paper the database is not needed. Al-Ahdal and Tahir [7]
proposes a design for recognizing signs used in ASL proposed a method using data glove and EMG
and interpreting them. Some of the challenges Sensors for designing SLR. Hand muscles
experienced by speech and hard of hearing people electromyography signals are recorded, for allocating
while communicating with normal people were social words from streams of characters in continuous SLR.
Vision based systems mainly uses colour information Segmentation is done to divide image into two
for detection of hand. They use skin colour models or regions, background and the foreground containing
different colour gloves [10]. region of interest i.e. hand region. The segmented
image has the hand region with the pixel value ‘1’
3. BLOCK SCHEMATIC
and the background as the ‘0’. The image is then used
as a mask to get the hand region from the RGB image
Input Pre- Feature by multiplying the black and white image i.e. binary
Image Processi Extractio image with the original RGB image. The image is
resized to reduce size of the matrix used for the
recognition process. The images are then converted
into column matrix for feature extraction.
Sign Training
Recogniti- Classificat- & 3.3 Feature Extraction
on ion
Testing
Feature extraction is the most significant step in
recognition stage as the size of the data
3.1 Input image dimensionality is reduced in this stage. Here, features
are extracted using Principle Component Analysis
Input image through web cam is captured and it can (PCA), where we can get Eigen Values and Eigen
be used to store as dataset for training or as input Vectors. To calculate this Eigen Values and Eigen
image to recognise the character. The image is Vectors, column matrix of all the images is formed
captured and stored in any supported format specified and concatenated to form single matrix, mean of this
by the device. The resolution of the camera and the matrix is calculated and it is subtracted for
supported format of the connected device, i.e. web normalisation. The mean of the matrix is calculated
cam can be found using the command: as:
“imaqhwinfo('winvideo')” μ= ∑ n
This command provides the details such as: where, M is the number of column matrix.
AdaptorDllName: [1x81 char]
AdaptorDllVersion: '4.5 (R2013a)' Mean from each of the column vector of the database
AdaptorName: 'winvideo' Ti is subtracted as:.
DeviceIDs: {[1]} Temp = A- μ
DeviceInfo: [1x1 struct]
Using device ID we can make get the supported where, Ais column matrix & μ is the mean.
format for the images to be captured and stored by
3.4 Training and testing phase
the web cam, as we have used web cam for training
and testing of the proposed system Detection of human hand in white background will
enhance the performance of image pre-processing, in
3.2 Pre-Processing
terms of accuracy and speed. Training phase and the
Pre-processing is required on every image to enhance recognition phase are two Sign recognition phases.
the functionality of image processing. Captured
Training phase: Training phase is the first phase,
images are in the RGB format. The pixel values and
where the generation and storing of dataset is done.
the dimensionality of the captured images is very
Number of images per character stored is directly
high. As images are matrices and mathematical
proportional to the accuracy of the system. During
operations are performed on images are the
generation of the training set the images are pre-
mathematical operations on matrices. So we convert
processed. Column matrix is generated for each of
the RGB image into Gray image using “rgb2gray”
the images of dataset and concatenated matrix is
function and thus converting the Gray image into
formed. Using this matrix Eigen vector is calculated
Binary image. Image segmentation technique is used
which is used to get feature vector. This feature
on the Binary image to detect the hand region.
vectors are used further for classification process.
Testing phase: This stage is the second stage which which is used to get feature vector. This feature
requires the input gesture, which is to be recognised. vectors are used further for classification process.
Input image is normalised with the mean calculated Testing phase: This stage is the second stage which
of the dataset, and Eigen vectors are used for requires the input gesture, which is to be recognised.
projecting the input image on the dataset. Maximum Input image is normalised with the mean calculated
score using Euclidean Distance is calculated and the of the dataset, and Eigen vectors are used for
gesture is recognised to display the recognised projecting the input image on the dataset. Maximum
character to corresponding input gesture. score using Euclidean Distance is calculated and the
gesture is recognised to display the recognised
11 … 1 character to corresponding input gesture.
A= ⋮ ⋮ ⋮
1 ⋯ 4. Dataset
Where, A is the dataset column matrix, m = Size of
Single image of each character from
image & n = Number of Images in dataset.
dataset for gesture recognition is shown. Increasing
Euclidean Distance is calculated as:

= , − ,

Where, j =1,2,…,m, n=total pixels in single image

m= total number of images in the dataset

3.5 Classification

Euclidean Distance is used to classify the gestures of

hand. Feature vectors calculated for the dataset and
the Feature vector calculated for the input image are
used to classify the character related to the input
gesture. Classification of character is based on the
number of images in dataset will increase accuracy of
maximum score. Maximum score is the maximum
matching score i.e. minimum distance of input Fig.2 Dataset of few characters
gesture with the dataset image of the corresponding the recognition. For creation of dataset we used pre-
character. processing, PCA algorithm on the captured image
and thus created image is resized and saved.
3.6 Sign Recognition
5. RESULTS
Detection of human hand in white background will
enhance the performance of image pre-processing, in The input gesture is taken online as well as
terms of accuracy and speed. Training phase and the offline. The taken image is pre-processed and resized.
recognition phase are two Sign recognition phases. Principle Component Analysis is done to get the
Training phase: Training phase is the first phase, Eigen Vector and Euclidean Distance is calculated
where the generation and storing of dataset is done. between the training set and the input gesture. The
Number of images per character stored is directly accuracy table shows the percentage accuracy for
proportional to the accuracy of the system. During various character.
generation of the training set the images are pre-
processed. Column matrix is generated for each of
the images of dataset and concatenated matrix is
formed. Using this matrix Eigen vector is calculated
 directly with normal people without any knowledge
of Sign Language.
The accuracy of the system depends on the
illumination conditions. Under good illumination and
plain light coloured background we achieved nearly
98% accuracy.

REFERENCES
[1] Mrs. NeelaHarish , Dr. S .Poonguzhali, “Design and
development of hand gesture recognition system for speech
impaired people”, International Conference on Industrial
Instrumentation and Control (ICIC) College of Engineering
Pune, India. May 28-30, 2015.
[2] Suganya R, Dr.T.Meeradevi, “Design Of A Communication
Aid For Physically Challenged”,in IEEE Sponsored 2nd
International Conference On Electronics And
Fig.3 Results of character Y Communication System (Icecs 2015),2015.
[3] P Raghu Veera Chowdary, M Nagendra Babu, Thadigotla
Character Offline mode Online mode Venkata Subbareddy, Bommepalli Madhava Reddy, V
Elamaran, “Image Processing Algorithms for Gesture
Recognition using MATLAB”, IEEE International
Conference on Advanced Communication Control and
A 96% 93% Computing Technologies (ICACCCT), 2014.
[4] Meenakshi Panwar and Pawan Singh Mehra, “Hand Gesture
Recognition for Human Computer Interaction”,
International Conference on Computing, Communication
C 92% 96% and Applications (ICCCA), Dindigul, Tamilnadu, India,
Feburary 2012.
[5] Liton Chandra Paul, Abdulla Al Sumam, “Face Recognition
D 93% 97% Using Principal Component Analysis Method”, in
International Journal of Advanced Research in Computer
Engineering & Technology (IJARCET) Volume 1, Issue 9,
November 2012.
G 97% 97% [6] Adi Izhar Che Ani, Anis Diyana Rosli, Rohaiza Baharudin,
Mohd Hussaini Abbas, Mohd Firdaus Abdullah,
“Preliminary Study of Recognizing Alphabet Letter via
L 96% 94% Hand Gesture”, in Computational Science and Technology
(ICCST), 2014 International Conference, Kota Kinabalu,
India, August 2014.
[7] M. Ebrahim AI-Ahdal & Nooritawati Md Tahir," Review in
W 97% 94% Sign Language Recognition Systems" Symposium on
Computer & InfOlmatics(ISCI),pp:52-57, IEEE ,2012
Sruthi Upendran, Thamizharasi. A, “American Sign
Y 97% 96% Language Interpreter System for Deaf and Dumb
Individuals” in International Conference on Control,
Instrumentation, Communication and Computational
Table1. Accuracy Table Technologies (ICCICCT), 2014.
[8] A. Sharmila Konwar, B. Sagarika Borah, C. Dr.T.Tuithung,
“An American Sign Language Detection System using HSV
6. CONCLUSION Color Model and Edge Detection”, International Conference
on Communication and Signal Processing, April 3-5, 2014.
Hard of hearing and mute people make use of sign [9] Pravin R Futane , Rajiv v Dharaskar, “Hasta Mudra
language to communicate. Signs, such as hand anvinterpretatoin of Indian sign hand gestures”,
international conference on digital object identifier, vol.2,
gestures are used in many sign languages, for the pp:377-380, IEEE ,2011.
purpose of this project, American Sign Language has [10] Adi Izhar Che Ani, Anis Diyana Rosli, Rohaiza Baharudin,
Mohd Hussaini Abbas, Mohd Firdaus Abdullah,
been used. Principle Component Analysis technique “Preliminary Study of Recognizing Alphabet Letter via
is used for Gesture Recognition. Using PCA we Hand Gesture”, International Conference on Computational
Science and Technology (ICCST), 2014.
calculate Eigen vector and feature vector to find
Euclidean distance. Minimum Euclidean distance
leads to character recognition for the input gesture.
This project helps special persons to communicate