Marcelo Norberto

Virtual Sign – A Real Time Bidirectional Translator of Portuguese Sign Language

Abstract Promoting equity, equal opportunities to all and social inclusion of people with disabilities is a concern of modern societies at large and a key topic in the agenda of European Higher Education. Despite all the progress, we cannot ignore the fact that the conditions provided by the society for the deaf are still far from being perfect. The communication with deaf by means of written text is not as efficient as it might seem at first. In fact, there is a very deep gap between sign language and spoken/written language. The vocabulary, the sentence construction and the grammatical rules are quite different among these two worlds. These facts bring significant difficulties in reading and understanding the meaning of text for deaf people and, on the other hand, make it quite difficult for people with no hearing disabilities to understand sign language. The deployment of tools to assist the daily communication, in schools, in public services, in museums and other, between deaf people and the rest may be a significant contribution to the social inclusion of the deaf community. The work described in this paper addresses the development of a bidirectional translator between Portuguese Sign Language and Portuguese text. The translator from sign language to text resorts to two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. The hands configurations are classified using Support Vector Machines. The classification of the movement and orientation of the hands are achieved through the use of Dynamic Time Warping algorithm. The translator exhibits a precision higher than 90%. In the other direction, the translation of Portuguese text to Portuguese Sign Language is supported by a 3D avatar which interprets the entered text and performs the corresponding animations.

Digital Assisted Communication.

The communication with the deaf community can prove to be very challenging without the use of sign language. There is a considerable difference between sign and written language as they differ in both syntax and semantics. The work described in this paper addresses the development of a bidirectional translator between several sign languages and their respective text, as well as the evaluation methods and results of those tools. A multiplayer game is using the translator is also described on this paper. The translator from sign language to text employs two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. This translator is being adapted to allow the communication with the blind as well. The Quantitative Evaluation Framework (QEF) and the ten-fold cross-validation methods were used to evaluate the project and show promising results. Also, the product goes through a validation process by sign language experts and deaf users who provide their feedback answering a questionnaire. The translator exhibits a precision higher than 90% and the projects overall quality rates are close to 90% based on the QEF.

Recognition of hand configuration: A critical factor in automatic sign language translation

Identifying hand configuration is a critical feature of sign language translation. In this paper, we describe our approach to recognize hand configurations in real time with the purpose of providing accurate predictions to be used in automatic sign language translation. To capture the hand configuration we rely on data gloves with 14 sensors that measure finger joints bending. These inputs are sampled at a frequency of 100Hz and fed to a classifier that predicts the current hand configuration. The classification model is created from an annotated sample of hand configurations previously acquired. We expect this approach to be accurate and robust in the sense that the performance of the classification model should not vary significantly when the classifier is being used by one or another user. The results from our experimental evaluation show that there is a very high accuracy, meaning that data gloves are a good approach to capture the descriptive features of hand configurations. However, the robustness of such an approach is not as good as desirable since the accuracy of the classifier depends on the user, i.e., the accuracy is high when the classifier is used by a user who trained it but decreases in other cases.

VirtualSign Game Evaluation

This paper presents the evaluation methods and techniques applied to the serious game developed within the VirtualSign project as well as the results achieved through those methods. VirtualSign is a Portuguese sign language bi-directional translator with three main components. The components are a gesture to text translator, a text to gesture translator and a serious game. The serious game aims to make the process of learning sign language easier and enjoyable using the VirtualSign bi-directional translator. The method used to evaluate the VirtualSign was the Quantitative Evaluation Framework (QEF). The translator undertakes a rigid validation process by both sign language experts and the deaf community. The evaluation process was also supported by questionnaires. The precision of the translator has a direct impact on the game performance. To evaluate the precision of the automatic translator we have used QEF and 10-fold cross validation to estimate the accuracy of the translator. The experimental results show a precision above 90%. In the identification of the Portuguese Sign Language terms As for the game the quality rate is 88% based on the QEF evaluation.

Virtualsign translator as a base for a serious game

The goal of this paper is to present the development of a game aimed at making the process of learning sign language enjoyable and interactive, using the VirtualSign Translator. This game aims to make the process of learning sign language easier and enjoyable. In the game the player can control an avatar and interact with several objects and Non-player characters in order to obtain signs. Through the connection with VirtualSign Translator the data gloves and Kinect support this interaction and the gestures can then be represented by the character. This allows for the user to visualize and learn or train the various existing configurations of gestures. To improve the interactivity and to make the game more interesting and motivating, several checkpoints were placed along game levels. The game has as an inventory system where the signs are kept and can be checked allowing for the user to visualize and learn or train the various existing configurations of gestures. A High Scores system was also created, as well as a History option, to ensure that the game is a continuous and motivating learning process.

Real Time Bidirectional Translator of Portuguese Sign Language

The communication with deaf by means of written text is not as efficient as it might seem. In fact, there is a very deep gap between sign language and spoken/written language. The deployment of tools to assist the daily communication between deaf people and the rest may be a significant contribution to the social inclusion of the deaf community. The work described in this paper addresses the development of a bidirectional translator between Portuguese Sign Language and Portuguese text and a serious game to promote the learning of the Portuguese Sign Language. The translator from sign language to text employs two devices, namely the Microsoft Kinect and 5DT Sensor Gloves in order to gather data about the motion and shape of the hands. The hands configurations are classified using Support Vector Machines. The classification of the movement and orientation of the hands is achieved through the use of Dynamic Time Warping algorithm. The translator exhibits a precision higher than 90%. In the other direction, the translation of Portuguese text to Portuguese Sign Language is supported by a 3D avatar which interprets the entered text and performs the corresponding animations. As a complement, we also present a serious game directed to assist in the difficult task of learning the Portuguese Sign Language.

Serious Game on Sign Language

The quantity of educational digital content available for the hearing impaired community is very scarce. However, due to extraordinary progress of the new technologies, remarkable opportunities to bring a better quality of life to the public in general arose. Making these opportunities available to those who endure handicap and disabilities is a core concern in todays society and a must to promote equity and inclusion. The target community addressed by our work, the hearing impaired community, has its own language, known as Sign language. The work presented in this paper consists in the development of a game to make the process of learning sign language enjoyable and interactive. In order to do this, a game was created in which the player controls a character and interacts with various objects and non-player characters with the aim of collecting several gestures from the Portuguese Sign Language. These gestures can then be represented by the character. This allows the user to visualize and learn or train the various existing gestures. To raise the interactivity and to make the game more interesting and motivating, several checkpoints were placed along the level. This will provide the players a chance to test the knowledge they have acquired so far on the checkpoints by using Kinect. A High Scores system was also created as well as a history to ensure that the game is a continuous motivating process as well as a learning process.