See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/335705845

Hand Gesture Vocalizer for Dumb and Deaf People

Article in SCITECH Nepal · September 2019

DOI: 10.3126/scitech.v14i1.25530

CITATIONS READS

4 1,163

4 authors, including:

Er. Ashish Kr. Jha

Nepal Engineering College
14 PUBLICATIONS 110 CITATIONS

SEE PROFILE

All content following this page was uploaded by Er. Ashish Kr. Jha on 19 November 2022.

The user has requested enhancement of the downloaded file.

SCITECH Nepal, Vol. 14, No. 1

Hand Gesture Vocalizer for Dumb

and Deaf People
Sanish Manandhar1, Sushana Bajracharya2, Sanjeev Karki3, Ashish Kumar Jha4
[1,2,3]
Department of Computer Science, Nepal Engineering College, Changunarayan, Bhaktapur, Nepal
1
sanish.manandhar20@gmail.com, 2sushanabajra96@gmail.com, 3sanjeevkarki77@gmail.com
4
Assistant Professor, ashishkj@nec.edu.np

Sanish Manandhar is a student

of BE computer Engineering, Abstract:
Batch 2014 at Nepal Engineering
College and recently he has The main purpose of this paper is to confer the system
received his bachelor degree that converts a given sign used by disabled person
from Pokhara University. He is into its appropriate textual, audio, and pictorial
currently working as web developer in a prominent form using components such as Arduino Mega, Flex
software company. sensors, Accelerometer, which could be understand
by a common person. A wearable glove controller is
design with flex sensors attached on each finger, which
allows the system to sense the finger movements, and a
Sushana Bajracharya is a student Gy-61 accelerometer, which are uses to sense the hand
of BE computer Engineering,
movement of the disabled person. The wearable input
Batch 2014 at Nepal Engineering
College and recently she has glove controller sends the collected input signal to the
received her bachelor degree system for processing. The system uses Random forest
from Pokhara University. She algorithm to predict the correct output to an accuracy
is currently working as web of 85% on current training model.
developer in a prominent software company.
Keywords: Random Forest, Speech synthesis, Flex
sensors, Data glove, Smart Glove.

I. Introduction
Sanjeev Karki is a student of BE
computer Engineering, Batch 2014 Humans possess the voice capability for
at Nepal Engineering College interaction and communication among each other.
and recently he has received his Unfortunately, not everybody has the capability of
bachelor degree from Pokhara speaking and hearing. Sign language used among the
University. He is currently working community of people who cannot speak or hear as the
as web developer in a prominent software company.
means of communication. Sign language is a gesture
representation that involves simultaneously combining
hand shapes, orientation and movement of the hands,
Ashish Kumar Jha received his arms or body, and facial expressions to express fluently
Master's degree in Computer with a speaker’s thoughts. The people who cannot
Science specialization in
speak makes use of the sign languages to communicate
Networking from Sharda
University in 2017. He has
with other fellow vocally impaired person and even
involved in software development with other normal people who knows the meanings of
and teaching profession since sign languages or an interpreter is needed to translate
2013, currently working as assistant professor in Nepal the meanings of sign languages to other people who can
Engineering College. His research interest includes speak and do not know the meanings of sign languages.
Internet of things, Image Processing and Pattern However, it is not always possible for an individual to
Recognition. be around all the time to interpret the sign languages

- 22 -
 Hand Gesture Vocalizer for Dumb and Deaf People

and not everybody can learn the sign languages. Thus, B. Sign Language Recognition System [3] [4]
another alternative is that we can use a computer or a Yang Quan, a Chinese student, defined a Basic
smart phone as a mediator. The computer or a smart Sign Language Recognition system that is able to
phone could take an input from the vocally impaired translate a sequence of signs into the commonly
person and give its textual as well as and audio form of used speech language and vice versa. The sign
output. language/speech bidirectional translation (from
signs to speech and speech to signs) focused on the
II. Related Work Chinese Manual Alphabet where every single sign
A Lexicon Statics Survey of Signed Language in belongs to a single letter from the alphabet. The
Nepal This thesis proposed by Hope M. Hurl but after system was composed of a camera, a video display
his field study about the variety of sign languages used in terminal (VDT), a speaker, a microphone, and
Nepal and its similarity or dissimilarity with the American a keyboard. Two kinds of data were used which
Sign Language (ASL) and British Sign Language (BSL) were; vector of hand gestures and vector of lip
through the collection of world list and interviews. The actions. In order to characterize these vectors, they
results of the world list comparisons showed that there used the Normalized Moment of Inertia (NMI)
are at least three signed languages in the country [1]. In algorithm and Hu moments. The former attempts
more recent years NpSL has been introduced to many to solve translation invariability, size invariability,
Hearing Impaired Associations have undertaken to open anti-gradation distortion, and so on and the latter is
classes for hearing impaired children in many places a set of algebraic invariants that combines regular
where the government does not yet have a school for the moments and is useful to describe the information
hearing impaired. In this course, a research conducted of the image. They do not vary under change of
in the following cities of Nepal – Kathmandu, Surkhet, size, translation, and rotation and have been widely
Jumla, Pokhara, Ghandruk, Dharan, and Rajbiraj used in Pattern Recognition and successfully
where it was discovered that there was one national proven in sign language letter recognition. As said
sign language used by all hearing impaired named the before, they combine the hand gestures recognition
NpSL. The survey further concluded that, there were 3 with the lips movement reader in order to make the
village sign language that were used in the Indian Sign system more accurate. By using a multi-features
Language (ISL), ASL and BSL and the most widely SVMs classifier trained with a linear kernel, the
used technique was the sign writing which is somewhat 30 letters from the Chinese manual alphabet
similar to imitate the object used in spoken language. recognized with average accuracy of 95.55%.

In this area, there has been a lot work done with C. Microcontroller and Sensors Based Gesture
implementation different sign language and Vocalizer [5]
technologies few work are discuss below:
Sensors Based Gesture Vocalizer describes the
design and working of a system that is useful
A. Kinetic sign language translator [2]
for dumb, deaf and blind people to communicate
This project was a result of collaboration, with each another and with the normal people
facilitated by Microsoft Research, between the as well. Gesture Vocalizer is a large-scale multi-
Chinese Academy of Sciences, Beijing Union microcontroller based system that designed to
University, and Microsoft Research Asia, in which facilitate the communication among the dumb,
all the organization has made crucial contributions. deaf and blind communities with the normal
The system understands the gestures of sign people. In the project a data glove is used which
language and converts them to spoken and written can detect almost all the movements of a hand
language and vice versa. This system captures a and microcontroller based system converts some
conversation from both sides, displays the signer specified movements into human recognizable
and renders a written and spoken translation of the voice. The data glove is equipped with two
sign language in real-time. It also takes the spoken types of sensors, which are; the bend sensors
words of non-signer and turns them into accurate, and accelerometers as tilt sensors. This system is
understandable sign language. beneficial for dumb people and their hands will

- 23 -
SCITECH Nepal, Vol. 14, No. 1

speak having worn the gesture, vocalizer data at a flat and steady position gives a resistance value
glove. This project is similar to our proposed near to 25k ohm whereas when the sensors are fully
project but it does not make use of any smart bent they give a resistance value near to 72k ohm.
phone, instead it uses external speakers and a LCD They are usually in the form of a thin strip, are very
display of outputs. comfortable to use, as they are very light in weight,
and are easily bent [8].

Figure 2: Flex sensor

Figure 1: Block diagram of Microcontroller and One side of the sensor is printed with a polymer ink
Sensors Based Gesture Vocaliser that has conductive particles embedded on it, and note
that the flex sensors gives a proper change in resistance
D. Hand Gesture Recognition Using KINECT [6] value only when it is bent away from the ink like and
This is an automatic Gesture Recognition System, provides very minimum change in resistance value
which uses Microsoft Kinect for capturing the raw when it is bent in the reverse direction.
image of the gesture. Originally developed for The flex sensor comes in various sizes and made by
gaming, the Kinect’s sensors read a user’s body various companies, and in our research, we have
position and movements and, with the help of a used 10 flex sensors of 2.2 inches each placed on five
computer, Gesture Recognition System translates fingers. Each flex sensor consists of two strips, one is
them into words. This system thus has tremendous a thin strip connected to the high voltage (that can be
potential for understanding the complex gestures 5v or 3.3v) and another is a fat strip consisting of the
and for translating the signs into spoken or written polymer ink connected to ground through a resistor and
words. It check textual character in train data and provides the resistance output as well.
display as output if present. If no match found, then
the system allows the user to store the new gesture By connecting the flex sensor with a static resistor to
and its corresponding textual form. The system create a voltage divider, it produces a variable voltage
allows the user to auto-create a dictionary. The that can read by a microcontroller’s analog-to-digital
main motive of this project is to make interaction converter [9].
possible between an unimpaired and hearing-
impaired individual. The accelerometer can use to measure the static
acceleration of gravity in tilt- sensing applications and
can used to measure dynamic acceleration resulting
III. Hardware Details: Flex Sensors from motion, shock, or vibrations. By measuring the
The flex sensors also referred as the bend sensor amount of static acceleration due to gravity, we can
is a type of sensor that measures the amount of find out the angle the device tilted at with respect
bending or deflection or flexing as the name itself to the earth and by sensing the amount of dynamic
has it. As the sensors are bent the sensors provides us acceleration, you can analyze the way the device is
with an electrical resistance value, the sensor gives so moving. There are many different ways to make an
the more the sensors are bent, more resistance value. accelerometer.
When measured with a multi meter, the flex sensor

- 24 -
 Hand Gesture Vocalizer for Dumb and Deaf People

classifying objects [13]. For example, running a

prediction over Naive Bayes, SVM and Decision
Tree and then taking vote for final consideration of
class for test object.

Similarly, in the random forest classifier, higher

number of trees in the forest results in higher
accuracy of the outputs. Random forest classifier
creates a set of decision trees from randomly
selected subset of training set and then it aggregates
the votes from different decision trees to decide
the final class of the test object.

Suppose training set is given as: [X1, X2, X3,

X4] with corresponding labels as [L1, L2, L3,
L4], random forest may create three decision trees
taking input of subset for example [14], [X1, X2,
X3], [X1, X2, X4]& [X2, X3, X4]
Figure 3: Tilt and inclination of the accelerometer
So finally, it makes predictions based on the majority
Acceleration along the axes deflects the movable mass.
of votes made from each of the decision trees. This
This displacement of moving plate (mass) unbalances
algorithm works well because a single decision
the differential capacitor, which results in sensor output.
tree may be prone to a noise, but aggregation of
Output amplitude is proportional to acceleration. 16-bit
many decision trees reduce the effect of noise gave
ADC used to get digitized output. The full-scale range
more accurate result for a different sets of input.
of acceleration is +/- 2g, +/- 4g, +/- 8g, +/- 16g. It
The random forest can apply weight concept for
measures in g (gravity force) unit. When device placed
considering the impact of result from any decision
on flat surface it will measure 0g on X and Y-axis and
tree. Tree with high error rate give low weight
+1g on Z-axis [11].
value and vice versa. There are two stages in this
algorithm, one is random forest creation, and the
IV. Methodologies other is to make a prediction from the random
The classification of this article is based on forest classifier created in the first stage.
Random Forest Algorithm.
The whole process is explained below.
A. Random forest algorithm
Random Forest creation pseudo code:
Random forests or random decision forests are
1. Randomly select “k” features from total “m”
an ensemble learning method or supervised
features.
classification algorithm for classification,
regression and other tasks, that operate by a. Where k << m
constructing a large number of decision trees
at training time and outputting the class that is 2. Among the “k” features, calculate the node “d”
the mode of the classes (classification) or mean using the best split point.
prediction (regression) of the individual trees. This
3. Split the node into daughter nodes using the best
algorithm creates a forest with a number of trees
split.
and the more trees in the forest the more robust the
forest looks like. 4. Repeat 1 to 3 steps until “l” number of nodes has
reached.
Random decision forests correct the decision trees'
habit of over fitting to their training set. Ensemble 5. Build forest by repeating steps 1 to 4 for “n”
algorithms are those, which combine more than number times to create “n” number of trees.
one algorithms of same or different kind for

- 25 -
SCITECH Nepal, Vol. 14, No. 1

Random forest prediction pseudo-code: is time – consuming. The model is difficult to interpret
 Takes the test features, use the rules of each compared to a decision tree, where you can easily make
randomly created decision tree to predict the a decision by following the path in the tree.
outcome, and stores the predicted outcome (target).

 Calculate the votes for each predicted target.

 Consider the high voted predicted target as the

final prediction from the random forest algorithm.

The over fitting problem will never come when we

use the random forest algorithm in any classification
problem.

 The same random forest algorithm can be used for

both classification and regression task.

 The random forest algorithm could be used for Figure 6: Feature Extraction Diagram
feature engineering.
When a user makes a hand gesture, eight user inputs
 It means identifying the most important features from the glove controller has given to the system in
out of the available features from the training which there is 3- axes accelerometer signal and five flex
dataset. sensor signals. Three axes values and five flex sensor
values that has taken from the user acts as the features
for recognizing a particular gesture. These features for
the gesture has passed through random forest classifier,
then the gestures has classified according to the
features, and the output is recognized.

Figure 5: Working of Random Forest algorithm [15]

Random forest considered as a highly accurate and

robust method. It does not suffer from the over
fitting problem, this algorithm can be used in both
classification and regression problems.

Random forest can also handle missing values, with

two ways: using median values to replace continuous
variables, and computing the proximity- weighted
average of missing values.

Random forest is slow in generating predictions Figure 7: System Block Diagram

because it has multiple decision trees. This algorithm

- 26 -
 Hand Gesture Vocalizer for Dumb and Deaf People

With the machine, has trained with those dataset, for Step 5: Input: Collect data from the serial port at
the real time application of the trained machine, for real time.
a particular gesture the flex sensor and accelerometer
Step 6: Open new file in write mode.
data were collected from the glove movement and
processed by the Arduino converting those raw data to Step 7: Write the collected data the new file in
meaningful data. These values help in feature extraction CSV format.
process. Those data has sent from the Arduino to the Step 8: Flush the Arduino buffer.
serial port, has collected by the python through the
serial interface and saved as temporary dataset (CSV Step 9: Close the Serial port connection.
format) form. Finally, the temporary dataset passed Step 9: Pass the temporary dataset through the model.
through the machine such that the machine extracts the
Step 10: Trained machine predicts the output.
features of those dataset and predicts the appropriate
output for that gesture with reference to the Random Step 11: Display the prediction on the screen and
Forest Classifier model. As the output has recognized play the audio of the predicted word.
by the model, laptop screen and speaker has used to Step 12: Add 1-second delay
display the output to the user.
Step 13: End
V. Algorithms
VI. Results
A. Algorithm for dataset preparation at Arduino
Mega 2560 Finally, we have obtained a system that can read
the values for a particular gesture done by the user,
Step 1: Start predict the output for the gesture, display it on the laptop
Step 2: Input: For a particular gesture, 8 sensor screen, and provide an audio output via a desktop GUI.
data inputs (5 flex sensors data and 3 accelerometer The dataset for all the alphabets and frequently used
data) words has created. Using those datasets, the machine
has trained and the model created. Then dataset for all
Step 3: Convert raw sensor data to meaningful data
the alphabets combined and shuffled in order to train
Step 4: Set the baud rate the machine with reduced variance and make sure that
Step 5: Send data to the serial port model remains general and over fit less. When the input
has taken for sign language of the alphabet or word,
Step 6: Is Arduino buffer flushed? the input passed through the trained machine and the
If yes, go to step 7 closest value predicted display and the audio played
as output. Then the correlation plot of each alphabet
Else, wait for buffer to flush then continue was compare with other alphabet to check the level
Step 7: Add delay of 10 milliseconds. of closeness of alphabets with each other and predict
the accuracy of the model. The accuracy for our model
Step 8: Go to step 2.
founded to be 96.8 %.
B. Algorithm for real time application
Step 1: Start
Step 2: Train the model using the training dataset.
Step 3: Set the baud rate similar to Arduino serial
baud rate and open the serial connection.
Step 4: Is there data at serial port?
If no, wait till data available then continue.
Else, go to step 5.
Figure 6: Glove Controller

- 27 -
SCITECH Nepal, Vol. 14, No. 1

List of Signs our system is train to predict

Sign ASL Sign ASL

A AND

B NO

Figure: Dataset Values for alphabet C

C EAT

D I NEED
TO GO TO
WASHRO
OM
E I
Figure: Dataset Values for Word Hello

OKAY STOP

HELLO THANK
YOU

CAN BYE

Figure: Dataset Values for Word Okay

VII. Conclusion
Some Dataset Values for Some Gestures of our dataset
Smart gloves which take input from the users’
gestures and gives a textual output. It helps the
mute people to communicate with one another and
with the normal people. The mute people use their
standard sign language to communicate which is not
easily understandable by common people without the
knowledge of the sign language. Thus, this system
converts the sign language into audio, pictorial and
into text form, improving their life in a significant
way Image processing can be applied to take in new
gestures from the user themselves and store it in the
dataset. Make proper, well finished and a wireless
wearable technology to bridge the communication gap
with accurate and precise result.
Figure: Dataset Values for alphabet A

- 28 -
 Hand Gesture Vocalizer for Dumb and Deaf People

Refrences Journal of Advanced Research in Electrical,

[1] Hope M Hurlbut, "A Lexicostatistic Servey of Electronics and Instrumentation Engineering,
the Signed Languages in Nepal," SIL Electronic vol. 5, no. 3, p. 7, Mar. 2016.
Survey Reports, p. 23, June 2012. [8] Sparkfun. (2015) learn.sparkfun.com. [Online].
[2] X Chai et al., "Sign Language Recognition And https://learn.sparkfun.com/tutorials/flex-sensor-
Translation with Kinect," Institute of Computing hookup-guide/all.
Technology, CAS, Microsoft Research Asia, [9] Robotpark. robotpark.com. [Online]. http://
Beijing, China, October 2013. www.robotpark.com/GY-61-DXL335-3-Axis-
[3] Y Quan and P Jinye, "Application of improved sign Accelerometer-Module.
languages recognition and synthesis technology," [10] Arthur Samuel, "Some Studies in Machine
Industrial Electronics and Application, 2008, Learning Using the Game of Checkers," IBM
ICIEA 2008, 3rd IEEE Conference, pp. 1629- Journal of Research and Development, vol. 3, no.
1634, June 2008. 3, pp. 210-229, 1959.
[4] D Jun, "Gesture Recognition," Shan Hai Maritime [11] T. K. Ho, "The Random Subspace Method
University, China, May 2004. for Constructing Decision Forests," IEEE
[5] Ata Ur Rehman, Salman Afghani, Muhammad Transactions on Pattern Analysis and Machine
Akmal, and Raheel Yousaf, "Microcontroller Intelligence, vol. 30, no. 8, pp. 832-844, 1998.
and Sensors Based Gesture Vocalizer," Rachna [12] Savan Patel. (2017, May) medium.com.
College of Engg. and tech., Air University [Online]. https://medium.com/machine-
Islamabad, Islamabad, Research Paper 2008. learning-101/chapter-5-random-forest-classifier-
[6] Prerana K.C, Nishu Mahato, and Bishruti Satyal, 56dc7425c3e1
"Hand Gesture Recognition using KINECT," [13] Avinash Navalani. (2018, May) datacamp.
Nepal Engineering College, Baktapur, Bachelor com. [Online]. https://www.datacamp.com/
Thesis P3-I.0-14-4, 2015. community/tutorials/random-forests-classifier-
[7] M. S Kasar, Anvita Deshmukh, Akshada Gavande, python
and Priyanka Ghadage, "Smart Speaking Glove- ***
Virtual tongue for Deaf and Dumb," Internaltional

- 29 -

View publication stats