Real Time Conversion of Sign Language using Deep Learning for Programming Basics

Abstract

Sign Languages are languages that use the visual- manual modality to convey meaning. It serves as a medium of communication between speech-impaired people and normal people. They are full-fledged natural languages with their own grammar, lexicon and classifications such as American, Indian, Chinese and so on. However, due to the non-universality of sign languages, there is a huge communication gap between the speech or hearing-impaired people and others. Hence, they also suffer from access to basic education. In this work, a low-latency real-time sign language recognition application is developed for detecting and processing gestures performed from the Indian Sign Language (ISL) dictionary using a Convolutional Neural Network model, and identify the words that are being communicated. We focus our application on words from a specific domain, i.e., basic programming keywords, through a custom-made dataset containing 500 different images of the gesture corresponding to each word. Our application strives to detect both static and dynamic gestures performed by the user, and generate Python- like syntax for various constructs such as if and loop, with a minimal response time of less than 0.1s. This can potentially make inroads into educating disabled people in the field of programming.