Bruce Denby

Robust contour tracking in ultrasound tongue image sequences

ABSTRACT A new contour-tracking algorithm is presented for ultrasound tongue image sequences, which can follow the motion of tongue contours over long durations with good robustness. To cope with missing segments caused by noise, or by the tongue midsagittal surface being parallel to the direction of ultrasound wave propagation, active contours with a contour-similarity constraint are introduced, which can be used to provide ‘prior’ shape information. Also, in order to address accumulation of tracking errors over long sequences, we present an automatic re-initialization technique, based on the complex wavelet image similarity index. Experiments on synthetic data and on real 60 frame per second (fps) data from different subjects demonstrate that the proposed method gives good contour tracking for ultrasound image sequences even over durations of minutes, which can be useful in applications such as speech recognition where very long sequences must be analyzed in their entirety.

Comparison of DCT and autoencoder-based features for DNN-HMM multimodal silent speech recognition

Hidden Markov Model and Deep Neural Network-Hidden Markov Model speech recognition performance for a portable ultrasound + video multimodal silent speech interface is investigated using Discrete Cosine Transform and Deep Auto Encoder-based features with a range of dimensionalities. Experimental results show that the two types of features achieve similar Word Error Rate, but that the autoencoder features maintain good performance even for very low-dimension feature vectors, demonstrating potential as a very compact representation of the information in multimodal silent speech data. It is also shown for the first time that the Deep Network/Markov approach, which has been demonstrated to be beneficial for acoustic speech recognition and for articulatory sensor-based silent speech, improves the silent speech recognition performance for video-based silent speech recognition as well.

Updating the Silent Speech Challenge benchmark with deep learning

The 2010 Silent Speech Challenge benchmark is updated with new results obtained in a Deep Learning strategy, using the same input features and decoding strategy as in the original article. A Word Error Rate of 6.4% is obtained, compared to the published value of 17.4%. Additional results comparing new auto-encoder-based features with the original features at reduced dimensionality, as well as decoding scenarios on two different language models, are also presented. The Silent Speech Challenge archive has been updated to contain both the original and the new auto-encoder features, in addition to the original raw data.

Intangible Cultural Heritage and New Technologies: Challenges and Opportunities for Cultural Preservation and Development

Intangible cultural heritage (ICH) is a relatively recent term coined to represent living cultural expressions and practices, which are recognised by communities as distinct aspects of identity. The safeguarding of ICH has become a topic of international concern primarily through the work of United Nations Educational, Scientific and Cultural Organization (UNESCO). However, little research has been done on the role of new technologies in the preservation and transmission of intangible heritage. This chapter examines resources, projects and technologies providing access to ICH and identifies gaps and constraints. It draws on research conducted within the scope of the collaborative research project, i-Treasures. In doing so, it covers the state of the art in technologies that could be employed for access, capture and analysis of ICH in order to highlight how specific new technologies can contribute to the transmission and safeguarding of ICH.

Contour-based 3D tongue motion visualization using ultrasound image sequences

This article describes a contour-based 3D tongue deformation visualization framework using B-mode ultrasound image sequences. A robust, automatic tracking algorithm characterizes tongue motion via a contour, which is then used to drive a generic 3D Finite Element Model (FEM). A novel contour-based 3D dynamic modeling method is presented. Modal reduction and modal warping techniques are applied to model the deformation of the tongue physically and efficiently. This work can be helpful in a variety of fields, such as speech production, silent speech recognition, articulation training, speech disorder study, etc.

Automatic tongue contour tracking in ultrasound sequences without manual initialization

Tracking the movement of the tongue is important for understanding how tongue shape change contributes to speech production and control. Ultrasound imaging is widely used to record real time information on the tongue surface; however, noise, artefacts, and the presence of spurious edges render automatic detection of tongue contours without manual initialization difficult. In this paper, we propose a method to extract ultrasound tongue surface contour in a totally automatic way using a three-step procedure: 1) noise reduction using a non-local mean filter; 2) use of a quadratic function to roughly fit the surface contour based on points obtained with a Robert cross operator; and 3) an automatic refinement based on gradient shift and relative distance of candidate points to the initial rough contour point. Experiments are conducted on isolated vowels and on a continuous utterance of vowel sequence. The Mean Sum of Distances criterion shows that the proposed method provides results on a par with the popular EdgeTrak algorithm on these two data sets, as compared to hand-scanned contours, but without any manual initialization.

Acoustic Data Analysis from Multi-Sensor Capture in Rare Singing: Cantu in Paghjella Case Study

This paper deals with new capturing technologies to safeguard and transmit endangered intangible cultural heritage including Corsican multipart singing technique. The described work, part of the European FP7 i-Treasures project, aims at increasing our knowledge on rare singing techniques. This paper includes (i) a presentation of our light hyper-helmet with 5 non-invasive sensors (microphone, camera, ultrasound sensor, piezoelectric sensor, electroglottograph), (ii) the data acquisition process and software modules for visualization and data analysis, (iii) a case study on acoustic analysis of voice quality for the UNESCO labelled traditional Cantu in Paghjella. We have identified specific features for this singing style, such as changes in vocal quality, especially concerning the energy in the speaking and singing formant frequency region, a nasal vibration that seems to occur during singing, as well as laryngeal mechanism characteristics. These capturing and analysis technologies will contribute to defi...