Neil A. Thacker

Speechreading using Probabilistic Models

We describe a robust method for locating and tracking lips in gray-level image sequences. Our approach learns patterns of shape variability from a training set which constrains the model during image search to only deform in ways similar to the training examples. Image search is guided by a learned gray-level model which is used to describe the large appearance variability of lips. Such variability might be due to different individuals, illumination, mouth opening, specularity, or visibility of teeth and tongue. Visual speech features are recovered from the tracking results and represent both shape and intensity information. We describe a speechreading (lip-reading) system, where the extracted features are modeled by Gaussian distributions and their temporal dependencies by hidden Markov models. Experimental results are presented for locating lips, tracking lips, and speechreading. The database used consists of a broad variety of speakers and was recorded in a natural environment with no special lighting or lip markers used. For a speaker independent digit recognition task using visual information only, the system achieved an accuracy about equivalent to that of untrained humans.

Speaker identification by lipreading

This paper describes a new approach for speaker identification based on lipreading. Visual features are extracted from image sequences of the talking face and consist of shape parameters which describe the lip boundary and intensity parameters which describe the grey-level distribution of the mouth area. Intensity information is based on principal component analysis using eigenspaces which deform with the shape model. The extracted parameters account for both, speech dependent and speaker dependent information. We built spatio-temporal speaker models based on these features, using HMMs with mixtures of Gaussians. Promising results were obtained for text dependent and text independent speaker identification tests performed on a small video database.

Active Shape Models for Visual Speech Feature Extraction

Most approaches for lip modelling are based on heuristic constraints imposed by the user. We describe the use of Active Shape Models for extracting visual speech features for use by automatic speechreading systems, where the deformation of the lip model as well as image search is based on a priori knowledge learned from a training set. We demonstrate the robustness and accuracy of the technique for locating and tracking lips on a database consisting of a broad variety of talkers and lighting conditions.

Visual speech recognition using active shape models and hidden Markov models

This paper describes a novel approach for visual speech recognition. The shape of the mouth is modelled by an active shape model which is derived from the statistics of a training set and used to locate, track and parameterise the speakers lip movements. The extracted parameters representing the lip shape are modelled as continuous probability distributions and their temporal dependencies are modelled by hidden Markov models. We present recognition tests performed on a database of a broad variety of speakers and illumination conditions. The system achieved an accuracy of 85.42% for a speaker independent recognition task of the first four digits using lip shape information only.

Speechreading using shape and intensity information

We describe a speechreading system that uses both shape information from the lip contours and intensity information from the mouth area. Shape information is obtained by tracking and parameterizing the inner and outer lip boundary in an image sequence. Intensity information is extracted from a grey level model, based on principal component analysis. In comparison to other approaches, the intensity area deforms with the shape model to ensure that similar object features are represented after non-rigid deformation of the lips. We describe speaker independent recognition experiments based on these features and hidden Markov models. Preliminary results suggest that similar performance can be achieved by using either shape or intensity information and slightly higher performance by their combined use.

Robust recognition of scaled shapes using pairwise geometric histograms

The recognition of shapes in images using Pairwise Geometric Histograms has previously been confined to fixed scale shape. Although the geometric representation used in this algorithm is not scale invariant, the stable behaviour of the similarity metric as shapes are scaled enables the method to be extended to the recognition of shapes over a range of scale. In this paper the necessary additions to the existing algorithm are described and the technique is demonstrated on real image data. Hypotheses generated by matching scene shape data to models have previously been resolved using the generalised Hough transform. The robustness of this method can be attributed to its approximation of maximum likelihood statistics. To further improve the robustness of the recognition algorithm and to improve the accuracy to which an objects location, orientation and scale can be determined the generalised Hough transform has been replaced by the probabilistic Hough transform.

Locating and tracking facial speech features

This paper describes a robust method for extracting visual speech information from the shape of lips to be used for an automatic speechreading (lipreading) systems. Lip deformation is modelled by a statistically based deformable contour model which learns typical lip deformation from a training set. The main difficulty in locating and tracking lips consists of finding dominant image features for representing the lip contours. We describe the use of a statistical profile model which learns dominant image features from a training set. The model captures global intensity variation due to different illumination and different skin reflectance as well as intensity changes at the inner lip contour due to mouth opening and visibility of teeth and tongue. The method is validated for locating and tracking lip movements on a database of a broad variety of speakers.

Developing a virtual reality environment in petrous bone surgery: a state-of-the-art review.

*Imaging Science and Biomedical Engineering, The Medical School, University of Manchester, Manchester, England, U.K.; †Manchester Visualisation Center, Manchester Computing, University of Manchester, Manchester, England, U.K.; ‡Department of Otolaryngology, Central Manchester Healthcare Trust, Manchester, England, U.K.; §Department of Diagnostic Radiology, Central Manchester Healthcare Trust, Manchester, England, U.K.; Centre for Advanced Studies, Research and Development in Sardinia, Sardinia, Italy; ¶Virtual Presence, Sale, England, U.K., and Department of Virtual Reality Studies, Department of Surgery, and North of England Wolfson Centre for Minimally Invasive Therapy, Manchester Royal Infirmary, Manchester, England, U.K.; #Departments of Medical Physics and Bioengineering and Otolaryngology, University College London, London, England, U.K.; **Department of Otolaryngology, University College London, London, England, U.K.; ††Department of Otolaryngology, University Hospital of Pisa, Pisa, Italy; ‡‡Department of Computer Science, Dresden University of Technology, Dresden, Germany; and §§Genias Benelux, Almere, The Netherlands.

Stretch-correlation as a real-time alternative to feature-based stereo matching algorithms

Abstract We have analysed the requirements for a robust stereo vision algorithm for use in typical industrial applications. For such applications the views obtained in the two cameras have large differences in visual appearance due to the orientation difference between the two cameras and the close proximity of illumination sources. We have concluded that for this category of problem, feature-based methods should be more robust than conventional, area-based approaches, and this conclusion appears to be borne out in the published literature. However, correlation-based approaches are more suited to efficient implementation on available hardware. The technique which we have developed, called Stretch-Correlation, is based on the cross-correlation of warped image blocks which have been preprocessed to maximize the useful information content. Our new method models the severe warping effects encountered in difficult stereo problems and effectively relaxes the front-o-parallel constraint which is normally imposed in area-based disparity calculation. This algorithm imposes effectively most of the local constraints present in feature-based algorithms, and can be efficiently implemented on available hardware.

Algorithmic modelling for performance evaluation

Many of the machine vision algorithms described in the literature are tested on a very small number of images. It is generally agreed that algorithms need to be tested on much larger numbers if any statistically meaningful measure of performance is to be obtained. However, these tests are rarely performed; in our opinion this is normally due to two reasons. Firstly, the scale of the testing problem is daunting when high levels of reliability are sought, since it is the proportion of failure cases that allows the reliability to be assessed and a large number of failure cases are needed to form an accurate estimation of reliability. For reliable and robust algorithms, this requires an inordinate number of test cases. Secondly, there is the difficulty of selecting test images to ensure that they are representative. This is aggravated by the fact that the assumptions made may be valid in one application domain but not in another. Hence, it is very difficult to relate the results of one evaluation to other users’ requirements. While it is true that published papers in the vision area must contain some evidence of the successful application of the suggested technique, a whole host of reasons have been put forward as to why researchers do not attempt to evaluate their algorithms more rigorously. These objections are valid only within a closely defined context and do not stand up to critical examination [13]. The real problem seems to be the time required for the various stages of algorithm development. The ratiotheory: implementation: evaluation seems to scale according to the rule of thumb 1 : 10 : 100 [13]. The effort required to get a new idea published is thus far less than an extensive empirical evaluation, which is a considerable demotivation for researchers to do evaluation work, particularly as evaluation is not much valued as publishable material in either conferences or journals. However, the truth of the matter is that unless algorithms are evaluated – and in a manner that can be used to predict the capabilities of a technique on an unseen data set – it is unlikely to be re-implemented and used. Moreover, the subject cannot advance without a well-founded scientific methodology, which it will not have without an acknowledged system for