Hee Deok Yang

Sign Language Spotting with a Threshold Model Based on Conditional Random Fields

Sign language spotting is the task of detecting and recognizing signs in a signed utterance, in a set vocabulary. The difficulty of sign language spotting is that instances of signs vary in both motion and appearance. Moreover, signs appear within a continuous gesture stream, interspersed with transitional movements between signs in a vocabulary and nonsign patterns (which include out-of-vocabulary signs, epentheses, and other movements that do not correspond to signs). In this paper, a novel method for designing threshold models in a conditional random field (CRF) model is proposed which performs an adaptive threshold for distinguishing between signs in a vocabulary and nonsign patterns. A short-sign detector, a hand appearance-based sign verification method, and a subsign reasoning method are included to further improve sign language spotting accuracy. Experiments demonstrate that our system can spot signs from continuous data with an 87.0 percent spotting rate and can recognize signs from isolated data with a 93.5 percent recognition rate versus 73.5 percent and 85.4 percent, respectively, for CRFs without a threshold model, short-sign detection, subsign reasoning, and hand appearance-based sign verification. Our system can also achieve a 15.0 percent sign error rate (SER) from continuous data and a 6.4 percent SER from isolated data versus 76.2 percent and 14.5 percent, respectively, for conventional CRFs.

Reconstruction of 3D human body pose from stereo image sequences based on top-down learning

This paper presents a novel method for reconstructing a 3D human body pose from stereo image sequences based on a top-down learning method. However, it is inefficient to build a statistical model using all training data. Therefore, the training data is hierarchically divided into several clusters to reduce the complexity of the learning problem. In the learning stage, the human body model database is hierarchically constructed by classifying the training data into several sub-clusters with silhouette images. The data of each cluster in the bottom level is represented by a linear combination of examples. In the reconstruction stage, the proposed method hierarchically searches a cluster for the best matching silhouette image using a silhouette history image (SHI). Then, the 3D human body pose is reconstructed from a depth image using a linear combination of examples method. By using depth information to reconstruct 3D human body pose, the similar poses in silhouette images are estimated as different 3D human body poses. The experimental results demonstrate that the proposed method is efficient and effective for reconstructing 3D human body poses.

Simultaneous spotting of signs and fingerspellings based on hierarchical conditional random fields and boostmap embeddings

A sign language consists of two types of action; signs and fingerspellings. Signs are dynamic gestures discriminated by continuous hand motions and hand configurations, while fingerspellings are a combination of continuous hand configurations. Sign language spotting is the task of detection and recognition of signs and fingerspellings in a signed utterance. The internal structures of signs and fingerspellings differ significantly. Therefore, it is difficult to spot signs and fingerspellings simultaneously. In this paper, a novel method for spotting signs and fingerspellings is proposed. It can distinguish signs, fingerspellings and non-sign patterns, and is robust to the various sizes, scales and rotations of the signers hand. This is achieved through a hierarchical framework consisting of three steps: (1) Candidate segments of signs and fingerspellings are discriminated using a two-layer conditional random field (CRF). (2) Hand shapes of segmented signs and fingerspellings are verified using BoostMap embeddings. (3) The motions of fingerspellings are verified in order to distinguish those which have similar hand shapes and different hand motions. Experiments demonstrate that the proposed method can spot signs and fingerspellings from utterance data at rates of 83% and 78%, respectively.

Robust sign language recognition by combining manual and non-manual features based on conditional random field and support vector machine

The sign language is composed of two categories of signals: manual signals such as signs and fingerspellings and non-manual ones such as body gestures and facial expressions. This paper proposes a new method for recognizing manual signals and facial expressions as non-manual signals. The proposed method involves the following three steps: First, a hierarchical conditional random field is used to detect candidate segments of manual signals. Second, the BoostMap embedding method is used to verify hand shapes of segmented signs and to recognize fingerspellings. Finally, the support vector machine is used to recognize facial expressions as non-manual signals. This final step is taken when there is some ambiguity in the previous two steps. The experimental results indicate that the proposed method can accurately recognize the sign language at an 84% rate based on utterance data.

Combination of manual and non-manual features for sign language recognition based on conditional random field and active appearance model

Sign language recognition is the task of detection and recognition of manual signals (MSs) and non-manual signals (NMSs) in a signed utterance. In this paper, a novel method for recognizing MS and facial expressions as a NMS is proposed. This is achieved through a framework consisting of three components: (1) Candidate segments of MSs are discriminated using an hierarchical conditional random field (CRF) and Boost-Map embedding. It can distinguish signs, fingerspellings and non-sign patterns, and is robust to the various sizes, scales and rotations of the signers hand. (2) Facial expressions as a NMS are recognized with support vector machine (SVM) and active appearance model (AAM), AAM is used to extract facial feature points. From these facial feature points, several measurements are computed to distinguish each facial component into defined facial expressions with SVM. (3) Finally, the recognition results of MSs and NMSs are fused in order to recognize signed sentences. Experiments demonstrate that the proposed method can successfully combine MSs and NMSs features for recognizing signed sentences from utterance data.

Robust Sign Language Recognition with Hierarchical Conditional Random Fields

Sign language spotting is the task of detection and recognition of signs (words in the predefined vocabulary) and fingerspellings (a combination of continuous alphabets that are not found in signs) in a signed utterance. The internal structures of signs and fingerspellings differ significantly. Therefore, it is difficult to spot signs and fingerspellings simultaneously. In this paper, a novel method for spotting signs and fingerspellings is proposed, which can distinguish signs, fingerspellings, and nonsign patterns. This is achieved through a hierarchical framework consisting of three steps; (1) Candidate segments of signs and fingerspellings are discriminated with a two-layer conditional random field (CRF). (2) Hand shapes of detected signs and fingerspellings are verified by BoostMap embeddings. (3) The motions of fingerspellings are verified in order to distinguish those which have similar hand shapes and differ only in hand trajectories. Experiments demonstrate that the proposed method can spot signs and fingerspellings from utterance data at rates of 83% and 78%, respectively.

Reconstruction of 3d human body pose for gait recognition

In this paper, we propose a novel method to reconstruct 3D human body pose for gait recognition from monocular image sequences based on top-down learning. Human body pose is represented by a linear combination of prototypes of 2D silhouette images and their corresponding 3D body models in terms of the position of a predetermined set of joints. With a 2D silhouette image, we can estimate optimal coefficients for a linear combination of prototypes of the 2D silhouette images by solving least square minimization. The 3D body model of the input silhouette image is obtained by applying the estimated coefficients to the corresponding 3D body model of prototypes. In the learning stage, the proposed method is hierarchically constructed by classifying the training data into several clusters recursively. Also, in the reconstructing stage, the proposed method hierarchically reconstructs 3D human body pose with a silhouette image. The experimental results show that our method can be efficient and effective to reconstruct 3D human body pose for gait recognition.

MULTIPLE HUMAN DETECTION AND TRACKING BASED ON WEIGHTED TEMPORAL TEXTURE FEATURES

In this paper, we present a method of tracking and identifying persons in video images taken by a fixed camera situated at an entrance. In video sequences a person may be totally or partially occlu...

Reconstruction of 3d human body pose based on top-down learning

This paper presents a novel method for reconstructing 3D human body pose from monocular image sequences based on top-down learning. Human body pose is represented by a linear combination of prototypes of 2D silhouette images and their corresponding 3D body models in terms of the position of a predetermined set of joints. With a 2D silhouette image, we can estimate optimal coefficients for a linear combination of prototypes of the 2D silhouette images by solving least square minimization. The 3D body model of the input silhouette image is obtained by applying the estimated coefficients to the corresponding 3D body model of prototypes. In the learning stage, the proposed method is hierarchically constructed by classifying the training data into several clusters recursively. Also, in the reconstructing stage, the proposed method hierarchically reconstructs 3D human body pose with a silhouette image or a silhouette history image. We use a silhouette history image and a blurring silhouette image as the spatio-temporal features for reducing noise due to extraction of silhouette image and for extending the search area of current body pose to related body pose. The experimental results show that our method can be efficient and effective for reconstructing 3D human body pose.

Sign language spotting based on semi-Markov Conditional Random Field

Sign language spotting is the task of detecting the start and end points of signs from continuous data and recognizing the detected signs in the predefined vocabulary. The difficulty with sign language spotting is that instances of signs vary in terms of both motion and shape. Moreover, signs have variable motion in terms of both trajectory and length. Especially, variable sign lengths result in problems with spotting signs in a video sequence, because short signs involve less information and fewer changes than long signs. In this paper, we propose a method for spotting variable lengths signs based on semi-CRF (semi-Markov Conditional Random Field). We performed experiments with ASL (American Sign Language) and KSL (Korean Sign Language) datasets of continuous sign sentences to demonstrate the efficiency of the proposed method. Experimental results showed that the proposed method outperforms both HMM and CRF.