Deep Learning-Powered Sign Language

Detection for Video Calls

Team Members:
Guide Name: Gopikanth Tirumani (21761A4221)
Ms. Razeena Begum Shaik, Jonnadula Malleswari (21761A4224)
Sr.Assistant Professor Komme Tarun Sai (21761A4232)
 ABSTRACT
Sign Language is the primary means of communication for millions of deaf and hard-of-hearing individuals worldwide. However,
the ability to understand SL is limited among the general population, hindering effective communication and social inclusion for
the deaf and hard-of-hearing community. To address this challenge, we propose a deep learning-based Sign Language recognition
system. Our system aims to bridge the communication gap between the deaf and hearing communities by enabling real-time
translation of SL gestures into text or speech in video call application. Utilizing convolutional neural networks (CNNs), the system
extracts intricate spatial features from video inputs, while recurrent neural networks (RNNs) model the dynamic temporal aspects
of sign language expressions. Unlike strict rules, our system learns and adjusts to how people uniquely express themselves through
signs. This makes it much better at recognizing different sign language patterns compared to traditional methods. With this
technology, we're not just recognizing signs; we're making communication much easier for deaf individuals. This system is useful
in online platforms, education tools, and everyday conversations, making communication more accessible for the deaf community.

Keywords: Computer Vision, Sign Language Recognition, Gesture Recognition, Deaf Communication, Deep Learning,
Convolutional Neural Networks (CNNs), Deep Learning, Recurrent Neural Networks (RNNs), Communication Accessibility, vue
js, Django, Video call.
 INTRODUCTION
A sign language recognition system for video call applictation is a technology that
uses various tools such as Mediapipe, OpenCV, and deep learning algorithms like
CNN to interpret and understand sign language gestures. This system plays a crucial
role in bridging the communication gap between hearing-impaired and hearing
individuals. This makes it much better at recognizing different sign language
patterns compared to traditional methods. With this technology, we're not just
recognizing signs; we're making communication much easier for deaf individuals.
This system is useful in online platforms, education tools, and everyday
conversations, making communication more accessible for the deaf community.
 Motivation

The main reason behind this project is to help people who are deaf or hard of
hearing to communicate better during video calls. Many video call platforms don't
consider how to make communication easier for these individuals. By using
advanced technology like deep learning, we want to make it possible for people to
use sign language during video calls, and have their signs translated into text or
speech for others to understand. Our goal is to make sure everyone can participate
in conversations, regardless of their hearing abilities.
 LITERATURE SURVEY
DeepSign: Sign Language Detection and Recognition Using Deep Learning:
• Aim: Develop a real-time signer-independent SLR system with high accuracy.
• Approach: Combines Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) layers with hand and
face detection in a deep learning architecture.
• Accuracy: Achieves 93.2% accuracy on RWTH-PHOENIX-Weather 2014T dataset and 91.4% on RWTH-
PHOENIX-Weather 2014T 8k dataset.

Sign Language Recognition System using TensorFlow Object Detection API:

• Aim: Create a real-time SLR system using pre-trained models and TensorFlow Object Detection API.
• Approach: Employs TensorFlow's EfficientDet-D0 model for object detection and a custom CNN for gesture
classification.
• Accuracy: Achieves 85% confidence in real-time alphabet recognition and 70% accuracy for specific signs.
Convolutional Neural Networks for Sign Language Recognition:
• Aim: Review and analyze various CNN architectures for SLR applications.
• Approach: Discusses different CNN models, their advantages and limitations, and their performance on various SLR
datasets.
• Accuracy: Presents comparisons of accuracy rates for different models on specific datasets, highlighting trends and
challenges.

Sign Language Recognition for Real-Time Communication Enhancement:

• Aim: Explore integration of SLR systems into real-time communication scenarios to improve accessibility.
• Approach: Discusses advantages and challenges of using SLR in video calls, education, and healthcare settings.
• Accuracy: No specific accuracy metrics presented, focuses on broader impact and potential benefits.
 PROBLEM STATEMENT

Many video call platforms don't support sign language

communication, making it hard for deaf individuals to participate
fully. This project aims to develop a system that can detect sign
language gestures during video calls and translate them into text or
speech, ensuring better communication accessibility for the deaf
community.
 Research Objective
• Communication Accessibility: Develop a sign language detection system integrated into video call platforms to
facilitate easier communication for individuals who are deaf or hard of hearing, ensuring they can participate
fully in conversations.

• Development of a Video Call Application: Build a user-friendly video call application with integrated sign
language detection capabilities, providing a seamless platform for individuals who are deaf or hard of hearing
to communicate effectively.

• Sign Language Detection Integration: Integrate advanced sign language detection algorithms into the video
call application, allowing users to communicate using sign language gestures while ensuring accurate
interpretation and translation.

• Enabling Easy Communication for Deaf Individuals via the Internet: Develop and implement innovative
solutions to make communication more accessible and intuitive for deaf individuals through internet-based
platforms, with a focus on leveraging technology to overcome barriers to communication.
 PROPOSED METHODOLOGY
• Data Collection: Gather diverse video datasets of sign language gestures.
• Data Preprocessing: Standardize and augment video data for model training.
• Model Architecture Design: Design CNN-RNN architecture for sign language detection.
• Model Training: Train model parameters using video dataset.
• Integration with Video Call Platform: Integrate model into video call software.
• Real-Time Inference: Implement real-time sign language detection during video calls.
• User Testing and Evaluation: Evaluate accuracy and user experience with diverse participants.
• Optimization and Fine-Tuning: Refine model for improved accuracy and adaptability.
• Deployment and Monitoring: Deploy system within video call platform and monitor performance.
• Continuous Improvement: Iterate based on feedback and advancements for enhanced effectiveness.
PROJECT PAHSE - 2
SYSTEM REQUIREMENT SPECIFICATIONS

Vue Js
 Requirement Engineering Process
• Process Model

Frames extraction
 • Project deliverables
• Deep Learning Model:Developed model for real-time sign language detection in video calls.
• Integration Module:Module integrated within video call platform for seamless communication.
• User Interface:User-friendly interface for enabling and customizing sign language detection.
• Documentation:Comprehensive documentation including model architecture and user instructions.
• Training Dataset:Curated dataset for training sign language gesture recognition.
• Evaluation Report:Report outlining model performance, accuracy, and usability metrics.
• Deployment Package:Package for easy integration into existing video call applications.
• Support Plan:Support and maintenance plan for ongoing system updates and improvements.
• Training Materials:Materials for platform administrators and support staff.
• Continuous Improvement Roadmap:Roadmap based on user feedback and technological advancements.
 Team Roles & Responsibilities

• Gopikanth Tirumani: Deep learning Coding, Backened

• Jonnadula Malleswari: Documentation, Backend

• Komme Tarun Sai: Dataset collection, Frontend Coding

 Conclusion and Future Enhancements
Future Enhancements:
• Expansion of Languages: Exploration of extending the system to recognize and interpret multiple sign
languages to cater to a wider user base.
• Continuous Learning: Implement a continuous learning mechanism to allow the system to adapt and
improve over time, learning from user feedback and evolving sign language expressions.
• Expand Gesture Vocabulary: Continuously expand the system's gesture Mobile Application
• Integration: Create a user-friendly mobile application that allows individuals to use the system on their
smartphones, promoting widespread adoption and accessibility on-the-go.
• Web Application Integration: Create a user-friendly mobile application that allows individuals to use the
system on their smartphones, promoting widespread adoption and accessibility on-the-go.
 Conclusion and Future Enhancements

Conclusion:
In conclusion, the development of a deep learning-powered sign language detection system
for video calls holds promise for significantly enhancing communication accessibility for the deaf and
hard of hearing population. By seamlessly integrating this technology into existing video call platforms,
we aim to promote inclusivity and equal participation in digital communication channels.In essence, this
groundbreaking system promises to make online communication more inclusive for the deaf community.
It's a step towards a future where everyone can participate fully in digital conversations.
 REFERENCES

• https://www.researchgate.net/publication/354066737_Sign_Language_Recognition

• https://www.mdpi.com/1424-8220/22/24/9864

• https://www.ijrpr.com/uploads/V2ISSUE5/IJRPR462.pdf

• https://github.com/nicknochnack/RealTimeObjectDetection
THANK YOU