Jeffrey Berry

Deep Belief Networks for Real-Time Extraction of Tongue Contours from Ultrasound During Speech

Ultrasound has become a useful tool for speech scientists studying mechanisms of language sound production. State-of-the-art methods for extracting tongue contours from ultrasound images of the mouth, typically based on active contour snakes, require considerable manual interaction by an expert linguist. In this paper we describe a novel method for fully automatic extraction of tongue contours based on a hierarchy of restricted Boltzmann machines (RBMs), i.e. deep belief networks (DBNs). Usually, DBNs are first trained generatively on sensor data, then discriminatively to predict human-provided labels of the data. In this paper we introduce the translational RBM (tRBM), which allows the DBN to make use of both human labels and raw sensor data at all stages of learning. This method yields performance in contour extraction comparable to human labelers, without any temporal smoothing or human intervention, and runs in real-time.

Listening to speech recruits specific tongue motor synergies as revealed by transcranial magnetic stimulation and tissue-Doppler ultrasound imaging

The activation of listeners motor system during speech processing was first demonstrated by the enhancement of electromyographic tongue potentials as evoked by single-pulse transcranial magnetic stimulation (TMS) over tongue motor cortex. This technique is, however, technically challenging and enables only a rather coarse measurement of this motor mirroring. Here, we applied TMS to listeners’ tongue motor area in association with ultrasound tissue Doppler imaging to describe fine-grained tongue kinematic synergies evoked by passive listening to speech. Subjects listened to syllables requiring different patterns of dorso-ventral and antero-posterior movements (/ki/, /ko/, /ti/, /to/). Results show that passive listening to speech sounds evokes a pattern of motor synergies mirroring those occurring during speech production. Moreover, mirror motor synergies were more evident in those subjects showing good performances in discriminating speech in noise demonstrating a role of the speech-related mirror system in feed-forward processing the speakers ongoing motor plan.

Vision of tongue movements bias auditory speech perception

Audiovisual speech perception is likely based on the association between auditory and visual information into stable audiovisual maps. Conflicting audiovisual inputs generate perceptual illusions such as the McGurk effect. Audiovisual mismatch effects could be either driven by the detection of violations in the standard audiovisual statistics or via the sensorimotor reconstruction of the distal articulatory event that generated the audiovisual ambiguity. In order to disambiguate between the two hypotheses we exploit the fact that the tongue is hidden to vision. For this reason, tongue movement encoding can solely be learned via speech production but not via others׳ speech perception alone. Here we asked participants to identify speech sounds while matching or mismatching visual representations of tongue movements which were shown. Vision of congruent tongue movements facilitated auditory speech identification with respect to incongruent trials. This result suggests that direct visual experience of an articulator movement is not necessary for the generation of audiovisual mismatch effects. Furthermore, we suggest that audiovisual integration in speech may benefit from speech production learning.

Tone space reduction in Mandarin Chinese.

Work on clear versus conversational speech has shown that speakers expand the vowel space in clear speech, which maximizes the perceptual contrast, making it easier for listeners to parse the speech signal. This study investigates how the realization of lexical tones changes for different speech tasks, and whether it is sensitive to the number of repetitions of the same word in a conversation. Native Mandarin speakers participated in a map task in which they were instructed to explain the route on the map to a partner. The first and last tokens of each target word were extracted from the dialogues and compared to the same word read from a list. Pitch and duration were normalized and quadratic curve fitting was performed on the test items. Results of statistical analyses show that contour tones from later occurrences of the target words have a smaller slope, are more linear, and tend to have pitch values closer to the overall average pitch of the speaker. In other words, all tones are realized closer to a ...

Second language pronunciation training using acoustic-to-articulatory inversion

The current work presents articulatory kinematic, acoustic, and perceptual data characterizing the effects of how visual biofeedback derived from acoustic-to-articulatory inversion may influence vowel pronunciation training for native-Mandarin speakers of English. Ten participants were engaged in a six-week pronunciation training program that included a focus on English vowel production. As an addition to traditional pronunciation training techniques often used by speech-language pathologists, half of the participants were also provided with visual biofeedback displays detailing aspects of their current tongue position as well as idealized positions for vowel targets. Visual displays were obtained using an acoustic-to-articulatory inversion model based on the Parallel Reference Speaker Weighting (PRSW) method for model adaptation. Pre- and post-training changes in articulation were compared between participants that used only traditional pronunciation training methods and those who were given visual biofe...

Sensitivity and specificity of auditory feedback driven articulatory learning in virtual speech

The current work presents articulatory kinematic and acoustic data characterizing how the form of synthesized auditory feedback in virtual speech affects the sensitivity and specificity of articulatory learning. The term “virtual speech” refers to talker-manipulated synthesized speech controlled in real time using electromagnetic articulography (EMA). In the current work, 36 participants (4 with dysarthria) participated in a learning experiment requiring them to control an articulatory speech synthesizer using movements of the tongue, lips, and jaw. Participants were divided among two experimental conditions: (1) an “unmatched” condition, during which all participants received auditory feedback based on common articulatory synthesis settings (neither formant working space nor fundamental frequency were distinguishable between talkers); and (2) a “matched” condition, during which the articulatory synthesis parameters were adjusted to mimic the formant working space and average fundamental frequency of the ...

Articulatory kinematics during connected speech across dialects and dysarthria

The current work presents an analysis of articulatory kinematics during connected speech in typical talkers and talkers with dysarthria from two different dialects of American English. Instrumental methods for obtaining articulatory kinematic data during speech (particularly electromagnetic articulography) are becoming increasingly viable within the clinical setting. Yet almost no existing clinical standards for collecting and interpreting articulatory kinematic data have been established. Moreover, there is little basis for differentiating the impact of dialect from dysarthria on articulatory kinematics. We examine articulatory kinematics obtained via electromagnetic articulography during a standard connected speech passage read by typical talkers (n = 30) and talkers with dysarthria (n = 15). Participants are divided among upper Midwestern and Southern American English dialects. Analyses focus on kinematic measures of articulatory movement (range-of-motion, speed, acceleration, and jerk) within and acro...

Vowel production in Mandarin accented English and American English: Kinematic and acoustic data from the Marquette University Mandarin accented English corpus

Few electromagnetic articulography (EMA) datasets are publicly available, and none have focused systematically on non-native accented speech. We introduce a kinematic-acoustic database of speech from 40 (gender and dialect balanced) participants producing upper-Midwestern American English (AE) L1 or Mandarin Accented English (MAE) L2 (Beijing or Shanghai dialect base). The Marquette University EMA-MAE corpus will be released publicly to help advance research in areas such as pronunciation modeling, acoustic-articulatory inversion, L1-L2 comparisons, pronunciation error detection, and accent modification training. EMA data were collected at a 400 Hz sampling rate with synchronous audio using the NDI Wave System. Articulatory sensors were placed on the midsagittal lips, lower incisors, and tongue blade and dorsum, as well as on the lip corner and lateral tongue body. Sensors provide five degree-of-freedom measurements including three-dimensional sensor position and two-dimensional orientation (pitch and rol...

Entropy coding for training deep belief networks with imbalanced and unlabeled data

Training deep belief networks (DBNs) is normally done with large data sets. In this work, the goal is to predict traces of the surface of the tongue in ultrasound images of the mouth during speech. Performance on this task can be dramatically enhanced by pre-training a DBN jointly on human-supplied traces and ultrasound images, then training a modified version of the network to predict traces from ultrasound only. However, hand-tracing the entire dataset of ultrasound images is extremely labor intensive. Moreover, the dataset is highly imbalanced since many images are extremely similar. This work presents a bootstrapping method which takes advantage of this imbalance, iteratively selecting a small subset of images to be hand-traced, then (re)training the DBN, making use of an entropy-based diversity measure for the initial selection. With this approach, a three-fold reduction in human time required to trace an entire dataset with human-level accuracy was achieved.

A Reference Grammar of Modern Hebrew (review)

Vol. 25, No. 3 ♦ 2007 healthy or cooperative. Indeed, he misses a point that economics and community approval may drive the competition among the militant organizations. In fact, the competition might itself explain a deteriorating political situation and an increasing lethality of the attacks as groups attempt to outdo each other in a process I refer to as “outbidding.” In sum, Hafez argues that suicide terror becomes popular among Palestinians partly because of the dynamics of the Israeli occupation and the increased humiliation Palestinians must endure. This is partially a tactical reaction to Israel’s superior military capability, and partially the result of Israel’s rather short-sighted policy of expelling Hamas members and supporters to Marj al Zahour in Lebanon where they spent several months learning the “how to’s” of suicide terrorism from Hizbullah. The tactical and strategic reasons that underlie the support for violence (i.e., a response to bad policies by the State of Israel) is somewhat reversed in chapter five of the book, entitled “societal motives.” Here, Hafez argues that there was a shift in the nature of conflict between the first and second Intifadas because of varying resources the Palestinians had available to them. Thus, the role played by Israeli policies is mooted by tactical concerns of supply. Hafez returns to his original thesis that Israeli responses goad the terrorism and feed the militant propaganda machines. The author ends with a section on policy recommendations which are straightforward and common-sensical. He argues against any kind of negotiation with terrorists and in favor of leaning how to better contain the threat by using better human intelligence and deterrent capabilities. All in all, and despite some minor issues, Manufacturing Human Bombs is a great addition to the growing literature on suicide terrorism and provides the reader with a valuable inside look into the world of martyrs. Mia Bloom University of Cincinnati