Stephen R. Ell

Small-vocabulary speech recognition using a silent speech interface based on magnetic sensing

This paper reports on word recognition experiments using a silent speech interface based on magnetic sensing of articulator movements. A magnetic field was generated by permanent magnet pellets fixed to relevant speech articulators. Magnetic field sensors mounted on a wearable frame measured the fluctuations of the magnetic field during speech articulation. These sensor data were used in place of conventional acoustic features for the training of hidden Markov models. Both small vocabulary isolated word recognition and connected digit recognition experiments are presented. Their results demonstrate the ability of the system to capture phonetic detail at a level that is surprising for a device without any direct access to voicing information.

A User-centric Design of Permanent Magnetic Articulography based Assistive Speech Technology

This paper addresses the design considerations and challenges faced in developing a wearable silent speech interface (SSI) based on Permanent Magnetic Articulography (PMA). To improve its usability, a new prototype was developed with the involvement of end users in the design process. Hence, desirable features such as appearance, portability, ease of use and light weight were incorporated into the prototype. The device showed a comparable performance with its predecessor, but has a much improved appearance, portability and hardware in terms of miniaturisation and cost.

A Wearable Silent Speech Interface based on Magnetic Sensors with Motion-Artefact Removal

For a silent speech interface (SSI) to be truly practical, it has to be able to tolerate motion artefacts generated by the user while engaging in normal activities of everyday living. This paper presents a wearable speech restoration system based on magnetic sensors with an integrated background cancellation technique to eliminate the effect of motion-induced interference. The background cancellation technique is assessed when the user makes no intentional movement, when they performs a set of defined head movements and when they make more natural, conversational head movements. The performance is measured for the TIDigits corpus in terms of whole word recognition rate using a Hidden Markov Model and through Mel Cepstral Distortion for a Direct Synthesis of speech using Deep Neural Networks. The results indicate the robustness of the sensing system with background cancellation against the undesirable motion-induced artefacts.

Direct Speech Reconstruction From Articulatory Sensor Data by Machine Learning

This paper describes a technique that generates speech acoustics from articulator movements. Our motivation is to help people who can no longer speak following laryngectomy, a procedure that is carried out tens of thousands of times per year in the Western world. Our method for sensing articulator movement, permanent magnetic articulography, relies on small, unobtrusive magnets attached to the lips and tongue. Changes in magnetic field caused by magnet movements are sensed and form the input to a process that is trained to estimate speech acoustics. In the experiments reported here this “Direct Synthesis” technique is developed for normal speakers, with glued-on magnets, allowing us to train with parallel sensor and acoustic data. We describe three machine learning techniques for this task, based on Gaussian mixture models, deep neural networks, and recurrent neural networks (RNNs). We evaluate our techniques with objective acoustic distortion measures and subjective listening tests over spoken sentences read from novels (the CMU Arctic corpus). Our results show that the best performing technique is a bidirectional RNN (BiRNN), which employs both past and future contexts to predict the acoustics from the sensor data. BiRNNs are not suitable for synthesis in real time but fixed-lag RNNs give similar results and, because they only look a little way into the future, overcome this problem. Listening tests show that the speech produced by this method has a natural quality that preserves the identity of the speaker. Furthermore, we obtain up to 92% intelligibility on the challenging CMU Arctic material. To our knowledge, these are the best results obtained for a silent-speech system without a restricted vocabulary and with an unobtrusive device that delivers audio in close to real time. This work promises to lead to a technology that truly will give people whose larynx has been removed their voices back.

Towards an intraoral-based silent speech restoration system for post-laryngectomy voice replacement

Silent Speech Interfaces (SSIs) are alternative assistive speech technologies that are capable of restoring speech communication for those individuals who have lost their voice due to laryngectomy or diseases affecting the vocal cords. However, many of these SSIs are still deemed as impractical due to a high degree of intrusiveness and discomfort, hence limiting their transition to outside of the laboratory environment. We aim to address the hardware challenges faced in developing a practical SSI for post-laryngectomy speech rehabilitation. A new Permanent Magnet Articulography (PMA) system is presented which fits within the palatal cavity of the user’s mouth, giving unobtrusive appearance and high portability. The prototype is comprised of a miniaturized circuit constructed using commercial off-the-shelf (COTS) components and is implemented in the form of a dental retainer, which is mounted under roof of the user’s mouth and firmly clasps onto the upper teeth. Preliminary evaluation via speech recognition experiments demonstrates that the intraoral prototype achieves reasonable word recognition accuracy and is comparable to the external PMA version. Moreover, the intraoral design is expected to improve on its stability and robustness, with a much improved appearance since it can be completely hidden inside the user’s mouth.

Voice Restoration After Laryngectomy Based on Magnetic Sensing of Articulator Movement and Statistical Articulation-to-Speech Conversion

In this work, we present a silent speech system that is able to generate audible speech from captured movement of speech articulators. Our goal is to help laryngectomy patients, i.e. patients who have lost the ability to speak following surgical removal of the larynx most frequently due to cancer, to recover their voice. In our system, we use a magnetic sensing technique known as Permanent Magnet Articulography (PMA) to capture the movement of the lips and tongue by attaching small magnets to the articulators and monitoring the magnetic field changes with sensors close to the mouth. The captured sensor data is then transformed into a sequence of speech parameter vectors from which a time-domain speech signal is finally synthesised. The key component of our system is a parametric transformation which represents the PMA-to-speech mapping. Here, this transformation takes the form of a statistical model (a mixture of factor analysers, more specifically) whose parameters are learned from simultaneous recordings of PMA and speech signals acquired before laryngectomy. To evaluate the performance of our system on voice reconstruction, we recorded two PMA-and-speech databases with different phonetic complexity for several non-impaired subjects. Results show that our system is able to synthesise speech that sounds as the original voice of the subject and also is intelligible. However, more work still need to be done to achieve a consistent synthesis for phonetically-rich vocabularies.

Integrating User-Centred Design in the Development of a Silent Speech Interface Based on Permanent Magnetic Articulography

A new wearable silent speech interface (SSI) based on Permanent Magnetic Articulography (PMA) was developed with the involvement of end users in the design process. Hence, desirable features such as appearance, portability, ease of use and light weight were integrated into the prototype. The aim of this paper is to address the challenges faced and the design considerations addressed during the development. Evaluation on both hardware and speech recognition performances are presented here. The new prototype shows a comparable performance with its predecessor in terms of speech recognition accuracy (i.e. ~ 95 % of word accuracy and ~ 75 % of sequence accuracy), but significantly improved appearance, portability and hardware features in terms of miniaturization and cost.