Péter Mátételki

An assistive interpreter tool using glove-based hand gesture recognition

An assistive tool (InterpreterGlove) for hearing- and speech impaired people is created, enabling them to easily communicate with the non-disabled using hand gestures and sign language. An integrated hardware and software solution is built to improve their standard of living, consisting of sensor network based motion-capture gloves, a low-level signal processing unit and a mobile application for high-level natural language processing. This paper introduces the overall system architecture and describes our automatic sign language interpreter software solution that processes the gesture descriptor stream of the motion-capture gloves, produces understandable text and reads it out as audible speech. The main logic of our automatic sign language interpreter consists of two algorithms: sign descriptor stream segmentation and text auto-correction. The software architecture of this time-sensitive complex application and the semantics of the developed hand gesture descriptor are described. We also present how the beta testers feedback from the deaf community influenced our work and achievements.