Boukje Habets

The role of synchrony and ambiguity in speech-gesture integration during comprehension

During face-to-face communication, one does not only hear speech but also see a speakers communicative hand movements. It has been shown that such hand gestures play an important role in communication where the two modalities influence each others interpretation. A gesture typically temporally overlaps with coexpressive speech, but the gesture is often initiated before (but not after) the coexpressive speech. The present ERP study investigated what degree of asynchrony in the speech and gesture onsets are optimal for semantic integration of the concurrent gesture and speech. Videos of a person gesturing were combined with speech segments that were either semantically congruent or incongruent with the gesture. Although gesture and speech always overlapped in time, gesture and speech were presented with three different degrees of asynchrony. In the SOA 0 condition, the gesture onset and the speech onset were simultaneous. In the SOA 160 and 360 conditions, speech was delayed by 160 and 360 msec, respectively. ERPs time locked to speech onset showed a significant difference between semantically congruent versus incongruent gesture–speech combinations on the N400 for the SOA 0 and 160 conditions. No significant difference was found for the SOA 360 condition. These results imply that speech and gesture are integrated most efficiently when the differences in onsets do not exceed a certain time span because of the fact that iconic gestures need speech to be disambiguated in a way relevant to the speech context.

Neural correlates of conceptualisation difficulty during the preparation of complex utterances

Background: In language production, conceptualisation of the utterance precedes lemma retrieval, phonological encoding, and articulation. Knowledge about the neural correlates of conceptualisation is scarce. The instructions in German were (in the same order): Beschreiben Sie die Richtung!; Beschreiben Sie Richtung und Form!; Beschreiben Sie Richtung, Form und Farbe! Aims: The study aimed at the delineation of neurophysiological correlates of the macro‐planning aspect of conceptualisation by manipulating difficulty of conceptualisation. Methods & Procedures: Utterances were elicited by visual arrays containing a network of eight different shapes (e.g., circle, square) of different colours. Upon the appearance of an arrow in the display, participants had to describe either the direction of the arrow only (simple condition), the direction and the destination shape (medium condition), or the direction, the destination shape, and its colour (complex condition). Event‐related brain potentials (ERPs) were recorded from young healthy native speakers of German and analysed for epochs starting 100 ms prior to the onset of the arrow stimulus until 600 ms thereafter, i.e., prior to the onset of the vocalisation. ERPs were quantified by mean amplitude measures. Outcomes & Results: ERPs uncontaminated by vocalisation artefacts were obtained. Brain potentials in the medium and complex conditions were more positive going than those from the simple condition from 300 ms onwards. This effect had a centro‐parietal distribution akin the P300 component. Conclusions: Reliable electrophysiological effects of conceptualisation difficulty were obtained, opening new possibilities for the neurophysiological investigation of language production in healthy participants and those with non‐aphasic language disorders. The distribution of the conceptualisation effect suggests that it reflects general effects of conceptualisation difficulty (e.g., demand for processing resources) rather than specific steps of the language planning process.

Neural basis of linearization in speech production

An initial stage of speech production is conceptual planning, where a speaker determines which information to convey first (the linearization problem). This fMRI study investigated the linearization process during the production of “before” and “after” sentences. In “after” sentences, a series of events is expressed in the order of event occurrence. In “before” sentences, however, the order of event mention is achieved by reversing the chronological order. We suggested that the linearization process may be supported by a neural network connecting the left middle temporal gyrus (MTG) with the medial superior frontal gyrus, left middle frontal gyrus, and left angular gyrus/inferior parietal gyrus. Within this network, regions were more activated and interregional interactions were strongly enhanced for producing “before” than “after” sentences. The left MTG was also functionally connected with the left orbital inferior frontal gyrus, contributing to the retrieval of necessary world knowledge and linguistic knowledge. Connectivity between these two regions was not different between conditions.

Experience with crossmodal statistics reduces the sensitivity for audio-visual temporal asynchrony

Bayesian models propose that multisensory integration depends on both sensory evidence (the likelihood) and priors indicating whether or not two inputs belong to the same event. The present study manipulated the prior for dynamic auditory and visual stimuli to co-occur and tested the predicted enhancement of multisensory binding as assessed with a simultaneity judgment task. In an initial learning phase participants were exposed to a subset of auditory-visual combinations. In the test phase the previously encountered audio-visual stimuli were presented together with new combinations of the auditory and visual stimuli from the learning phase, audio-visual stimuli containing one learned and one new sensory component, and audio-visual stimuli containing completely new auditory and visual material. Auditory-visual asynchrony was manipulated. A higher proportion of simultaneity judgements was observed for the learned cross-modal combinations than for new combinations of the same auditory and visual elements, as well as for all other conditions. This result suggests that prior exposure to certain auditory-visual combinations changed the expectation (i.e., the prior) that their elements belonged to the same event. As a result, multisensory binding became more likely despite unchanged sensory evidence of the auditory and visual elements.

Infants are superior in implicit crossmodal learning and use other learning mechanisms than adults

During development internal models of the sensory world must be acquired which have to be continuously adapted later. We used event-related potentials (ERP) to test the hypothesis that infants extract crossmodal statistics implicitly while adults learn them when task relevant. Participants were passively exposed to frequent standard audio-visual combinations (A1V1, A2V2, p=0.35 each), rare recombinations of these standard stimuli (A1V2, A2V1, p=0.10 each), and a rare audio-visual deviant with infrequent auditory and visual elements (A3V3, p=0.10). While both six-month-old infants and adults differentiated between rare deviants and standards involving early neural processing stages only infants were sensitive to crossmodal statistics as indicated by a late ERP difference between standard and recombined stimuli. A second experiment revealed that adults differentiated recombined and standard combinations when crossmodal combinations were task relevant. These results demonstrate a heightened sensitivity for crossmodal statistics in infants and a change in learning mode from infancy to adulthood.

Intra- and crossmodal refractory effects in auditory and somatosensory ERPs

Refractory period effects are defined as a temporal decrement in neural response due to a previous activation of the same system. We varied the ISI and modality of a preceding stimulus to investigate overlapping and distinct neural systems processing auditory and tactile stimuli. Auditory stimuli and tactile stimuli were presented in a sequential, random manner with a duration of 50 ms and an ISI of 1000 or 2000 ms. The P1–N1–P2 complex of event-related potentials (ERP) was analyzed separately for auditory and tactile stimuli, as a function of preceding ISI and modality. Main effects of ISI and modality were found within the time-window of the P1 and P2 (auditory) and P1, N1 and P2 (tactile). These results suggest an overlap in underlying neural systems when stimuli from different modalities are being processed.

Recovery periods of event-related potentials indicating crossmodal interactions between the visual, auditory and tactile system

In sequential unimodal stimulus designs the time it takes for an event-related potential (ERP)-amplitude to recover is often interpreted as a transient decrement in responsiveness of the generating cortical circuits. This effect has been called neural refractoriness, which is the larger the more similar the repeated stimuli are and thus indicates the degree of overlap between the neural generator systems activated by two sequential stimuli. We hypothesize that crossmodal refractoriness-effects in a crossmodal sequential design might be a good parameter to assess the ‘modality overlap’ in the involved neural generators and the degree of crossmodal interaction. In order to investigate crossmodal ERP refractory period effects we presented visual and auditory (Experiment 1) and visual and tactile stimuli (Experiment 2) with inter stimulus intervals of 1 and 2 s to adult participants. Participants had to detect rare auditory and visual stimuli. Both, intra- and crossmodal ISI effects for all modalities were found for three investigated ERP-deflections (P1, N1, P2). The topography of the crossmodal refractory period effect of the N1- and P2-deflections in Experiment 1 and of P1 and N1 in Experiment 2 of both modalities was similar to the corresponding intramodal refractory effect, yet more confined and crossmodal effects were generally weaker. The crossmodal refractory effect for the visual P1, however, had a distinct, less circumscribed topography with respect to the intramodal effect. These results suggest that ERP refractory effects might be a promising indicator of the neural correlates of crossmodal interactions.

Intramodal and crossmodal refractory effects: Evidence from oscillatory brain activity

Event-related potentials (ERP) to the second stimulus of a pair are known to be reduced in amplitude. The magnitude of this ‘refractoriness’ is modulated by both the interstimulus interval and the similarity between the two stimuli. Intramodal refractoriness is interpreted as an index of a temporary decrement in neural responsiveness. So, cross-modal refractoriness might be an indicator of shared neural generators between modalities. We analysed oscillatory neuronal activity while participants were engaged in an oddball paradigm with auditory (4000 Hz, 50 ms-long, 90 db, bilateral) and tactile stimuli (50 ms-long, 125 Hz-vibrations, index fingers) presented in a random order with an ISI of either 1000 or 2000 ms. Participants were required to detect rare tactile (middle fingers) and auditory deviants (600 Hz). A time–frequency analysis of the brain response to the second stimulus of each pair (T-T, A-A, T-A and A-T) contrasting Short and Long ISIs revealed a reduced refractory effect after Long ISI with respect to Short ISI, in all pairs (both intramodal and cross-modal). This emerged as a broadly distributed increase of evoked theta activity (3–7 Hz, 100–500 ms). Only in intramodal tactile pairs and cross-modal tactile-auditory pairs we also observed that Long ISI with respect to Short ISI determined a decrease of induced alpha (8–12 Hz, 200–700 ms), a typical sign of enhanced neural excitability and thus decreased refractoriness. These data suggest that somatosensory and auditory cortices display different neural markers of refractoriness and that the auditory cortex might have a stronger low level degree of influence on the tactile cortex than vice-versa.