Gaolin Fang

Large-Vocabulary Continuous Sign Language Recognition Based on Transition-Movement Models

The major challenges that sign language recognition (SLR) now faces are developing methods that solve large-vocabulary continuous sign problems. In this paper, transition-movement models (TMMs) are proposed to handle transition parts between two adjacent signs in large-vocabulary continuous SLR. For tackling mass transition movements arisen from a large vocabulary size, a temporal clustering algorithm improved from k-means by using dynamic time warping as its distance measure is proposed to dynamically cluster them; then, an iterative segmentation algorithm for automatically segmenting transition parts from continuous sentences and training these TMMs through a bootstrap process is presented. The clustered TMMs due to their excellent generalization are very suitable for large-vocabulary continuous SLR. Lastly, TMMs together with sign models are viewed as candidates of the Viterbi search algorithm for recognizing continuous sign language. Experiments demonstrate that continuous SLR based on TMMs has good performance over a large vocabulary of 5113 Chinese signs and obtains an average accuracy of 91.9%

Transition movement models for large vocabulary continuous sign language recognition

The major challenges that sign language recognition (SLR) now faces are developing methods that solve large vocabulary continuous sign problems. In this paper, large vocabulary continuous SLR based on transition movement models is proposed. The proposed method employs the temporal clustering algorithm to cluster a large amount of transition movements, and then the corresponding training algorithm is also presented for automatically segmenting and training these transition movement models. The clustered models can improve the generalization of transition movement models, and are very suitable for large vocabulary continuous SLR. At last, the estimated transition movement models, together with sign models, are viewed as candidate models of the Viterbi search algorithm for recognizing continuous sign language. Experiments show that continuous SLR based on transition movement models has good performance over a large vocabulary of 5113 signs.

A vision-based sign language recognition system using tied-mixture density HMM

In this paper, a vision-based medium vocabulary Chinese sign language recognition (SLR) system is presented. The proposed recognition system consists of two modules. In the first module, techniques of robust hands detection, background subtraction and pupils detection are efficiently combined to precisely extract the feature information with the aid of simple colored gloves in the unconstrained environment. Meanwhile, an effective and efficient hierarchical feature description scheme with different scale features to characterize sign language is proposed, where principal component analysis (PCA) is employed to characterize the finger features more elaborately. In the second part, a Tied-Mixture Density Hidden Markov Models (TMDHMM) framework for SLR is proposed, which can speed up the recognition without the significant loss of recognition accuracy compared with the continuous hidden Markov models (CHMM). Experimental results based on 439 frequently used Chinese sign language (CSL) words show that the proposed methods can work well for the medium vocabulary SLR in the environment without special constraints and the recognition accuracy is up to 92.5%.

Signer-Independent Continuous Sign Language Recognition Based on SRN/HMM

A divide-and-conquer approach is presented for signer-independent continuous Chinese Sign Language(CSL) recognition in this paper. The problem of continuous CSL recognition is divided into the subproblems of isolated CSL recognition. The simple recurrent network (SRN) and the hidden Markov models(HMM) are combined in this approach. The improved SRN is introduced for segmentation of continuous CSL. Outputs of SRN are regarded as the states of HMM, and the Lattice Viterbi algorithm is employed to search the best word sequence in the HMM framework. Experimental results show SRN/HMM approach has better performance than the standard HMM one.

Large vocabulary sign language recognition based on hierarchical decision trees

The major difficulty for large vocabulary sign language or gesture recognition lies in the huge search space due to a variety of recognized classes. How to reduce the recognition time without loss of accuracy is a challenge issue. In this paper, a hierarchical decision tree is first presented for large vocabulary sign language recognition based on the divide-and-conquer principle. As each sign feature has the different importance to gestures, the corresponding classifiers are proposed for the hierarchical decision to gesture attributes. One- or two- handed classifier with little computational cost is first used to eliminate many impossible candidates. The subsequent hand shape classifier is performed on the possible candidate space. SOFM/HMM classifier is employed to get the final results at the last non-leaf nodes that only include few candidates. Experimental results on a large vocabulary of 5113-signs show that the proposed method drastically reduces the recognition time by 11 times and also improves the recognition rate about 0.95% over single SOFM/HMM.

A SRN/HMM system for signer-independent continuous sign language recognition

Sign language recognition is to provide an efficient and accurate mechanism to transcribe sign language into text or speech. State-of-the-art sign language recognition should be able to solve the signer-independent continuous problem for practical applications. A divide-and-conquer approach, which takes the problem of continuous Chinese Sign Language (CSL) recognition as subproblems of isolated CSL recognition, is presented for signer-independent continuous CSL recognition. In the proposed approach, the improved simple recurrent network (SRN) is used to segment the continuous CSL. The outputs of SRN are regarded as the states of hidden Markov models (HMM) in which the Lattice Viterbi algorithm is employed for searching for the best word sequence. Experimental results show that the SRN/HMM approach has a better performance than the standard HMM.

CSLDS: Chinese sign language dialog system

We present a Chinese sign language dialog system (CSLDS) based on the technique of large vocabulary continuous Chinese sign language recognition (CSLR) and Chinese sign language synthesis (CSLS). This system can show the advance technology on gesture recognition and synthesis well and can apply to more powerful system combined with speech recognition and synthesis technology, which then can allow the convenient communication between deaf and hearing society.

HandTalker II: a Chinese sign language recognition and synthesis system

This paper presents a Chinese sign language/spoken language dialog system based on the technique of large vocabulary continuous Chinese sign language recognition (SLR) and Chinese sign language synthesis (SLS), which is new development for HandTalker. In the SLR module, a fuzzy decision tree with heterogeneous classifiers is presented for large vocabulary signer-independent SLR, and then large vocabulary continuous SLR based on transition movement models is proposed. In SLS module, three key techniques: realistic 3D facial animation and gesture retargeting technique and synchronization modal on gesture and lip motion are employed to improve sign language synthesis vividness.

Vision-Based sign language recognition using sign-wise tied mixture HMM

In this paper, a new sign-wise tied mixture HMM (SWTM-HMM) is proposed and applied in vision-based sign language recognition (SLR). In the SWTMHMM, the mixture densities of the same sign model are tied so that the states belonging to the same sign share a common local codebook, which leads to robust model parameters estimation and efficient computation of probability densities. For the sign feature extraction, an effective hierarchical feature description scheme with different scales of features to characterize sign language is presented. Experimental results based on 439 frequently used Chinese sign language (CSL) signs show that the proposed methods can work well for the medium vocabulary SLR in the unconstrained environment.

Incorporating linguistic structure into maximum entropy language models

In statistical language models, how to integrate diverse linguistic knowledge in a general framework for long-distance dependencies is a challenging issue. In this paper, an improved language model incorporating linguistic structure into maximum entropy framework is presented. The proposed model combines trigram with the structure knowledge of base phrase in which trigram is used to capture the local relation between words, while the structure knowledge of base phrase is considered to represent the long-distance relations between syntactical structures. The knowledge of syntax, semantics and vocabulary is integrated into the maximum entropy framework. Experimental results show that the proposed model improves by 24% for language model perplexity and increases about 3% for sign language recognition rate compared with the trigram model.