Noureddine Aboutabit

Lip Shape and Hand Position Fusion for Automatic Vowel Recognition in Cued Speech for French

Cued speech is a visual mode of communication that uses handshapes and placements in combination with the mouth movements of speech to make the phonemes of a spoken language look different from each other and clearly understandable to deaf and hearing-impaired people. The aim of cued speech is to overcome the problems of lip reading and thus enable deaf children and adults to wholly understand spoken language. Cued speech recognition requires hand gesture recognition and lip shape recognition, and also integration of the two components. This article presents hidden Markov model (HMM)-based vowel recognition as used in Cued Speech for French. Based on concatenative feature fusion and multistream HMM decision fusion, lip shape and hand position components were integrated into a single component, and automatic vowel recognition was realized. In the case of multistream HMM decision fusion, the obtained vowel classification accuracy using lip shape and hand position information was 87.6%, showing absolute improvement of 19.6% in comparison with a use restricted only to lip parameters.

Image and video for hearing impaired people

We present a global overview of image- and video-processing-based methods to help the communication of hearing impaired people. Two directions of communication have to be considered: from a hearing person to a hearing impaired person and vice versa. In this paper, firstly, we describe sign language (SL) and the cued speech (CS) language which are two different languages used by the deaf community. Secondly, we present existing tools which employ SL and CS video processing and recognition for the automatic communication between deaf people and hearing people. Thirdly, we present the existing tools for reverse communication, from hearing people to deaf people that involve SL and CS video synthesis.

Hand and Lip Desynchronization Analysis in French Cued Speech: Automatic Temporal Segmentation of Hand Flow

In the context of cued speech gesture phonetic translation, the automatic recognition of lip and hand movements is a key factor. The hand and the lip parameters are not synchronized, thus the fusion of the two channels (hand and lips) needs the knowledge of the desynchronized delay. This contribution focuses on the presentation of an automatic algorithm for temporal segmentation of the hand cue information based on Gaussian modeling of the hand position and minimum of velocity. The segmentation delivers the beginning of the hand transition and the instant of attained position. The hand segmentation is used to calculate the delay between hand and lip targets, in relation with the corresponding acoustic realization in the case of French CV syllables extracted from a corpus of phrases uttered and coded by a cued speech speaker. This study confirms in a more complex context the importance of the instant of attained hand position as pointed out by Attina and colleagues, in terms of control and for the fusion process