Mirjam Keetels

Perception of intersensory synchrony: A tutorial review

For most multisensory events, observers perceive synchrony among the various senses (vision, audition, touch), despite the naturally occurring lags in arrival and processing times of the different information streams. A substantial amount of research has examined how the brain accomplishes this. In the present article, we review several key issues about intersensory timing, and we identify four mechanisms of how intersensory lags might be dealt with: by ignoring lags up to some point (a wide window of temporal integration), by compensating for predictable variability, by adjusting the point of perceived synchrony on the longer term, and by shifting one stream directly toward the other.

The role of spatial disparity and hemifields in audio-visual temporal order judgments.

We explored whether sensitivity to audio-visual temporal order judgments (TOJs) was affected by the amount of spatial separation between a sound and light, and by whether the sound and light were presented in the same or in different hemifields. Participants made TOJs about noise bursts and light flashes, and judged whether the stimuli came from the same location or not. Flashes were presented either in the left or right hemifield (at ±10° from central fixation), and sounds either came from the same location as the lights, or at small or large disparities (20 or 40° from the light, respectively), thereby crossing the hemifields or not. TOJs became more accurate (i.e., the just noticeable difference, JND, became smaller) when spatial disparity increased and when hemifields were crossed. Location discrimination of the sound and light was affected similarly. These results demonstrate that audio-visual TOJs are critically dependent on both the relative position from which stimuli are presented and on whether stimuli cross hemifields or not.

The spatial constraint in intersensory pairing: no role in temporal ventriloquism

A sound presented in temporal proximity to a light can alter the perceived temporal occurrence of that light (temporal ventriloquism). The authors explored whether spatial discordance between the sound and light affects this phenomenon. Participants made temporal order judgments about which of 2 lights appeared first, while they heard sounds before the 1st and after the 2nd light. Sensitivity was higher (i.e., a lower just noticeable difference) when the sound-light interval was approximately 100 ms rather than approximately 0 ms. This temporal ventriloquist effect was unaffected by whether sounds came from the same or a different position as the lights, whether the sounds were static or moved, or whether they came from the same or opposite sides of fixation. Yet, discordant sounds interfered with speeded visual discrimination. These results challenge the view that intersensory interactions in general require spatial correspondence between the stimuli.

Temporal recalibration to tactile–visual asynchronous stimuli

Here we demonstrate that the perceptual system adapts to tactile-visual temporal asynchronies (i.e., temporal recalibration). Participants were exposed to a train of tactile and visual stimuli with a constant time lag (either -100ms, 0ms, or 100ms; with negative values indicating that the tactile stimulus came first). Following exposure, they were presented tactile-visual test stimulus pairs and judged whether the tactile or the visual stimulus was presented first (Temporal Order Judgement). Results show that subjective simultaneity (the PSS) was shifted in the direction of the exposure lag. The results fit reports on auditory-visual temporal recalibration and indicate that the brain adapts to temporal incongruencies between modalities in general.

Auditory grouping occurs prior to intersensory pairing: evidence from temporal ventriloquism.

The authors examined how principles of auditory grouping relate to intersensory pairing. Two sounds that normally enhance sensitivity on a visual temporal order judgement task (i.e. temporal ventriloquism) were embedded in a sequence of flanker sounds which either had the same or different frequency (Exp. 1), rhythm (Exp. 2), or location (Exp. 3). In all experiments, we found that temporal ventriloquism only occurred when the two capture sounds differed from the flankers, demonstrating that grouping of the sounds in the auditory stream took priority over intersensory pairing. By combining principles of auditory grouping with intersensory pairing, we demonstrate that capture sounds were, counter-intuitively, more effective when their locations differed from that of the lights rather than when they came from the same position as the lights.

Selective adaptation and recalibration of auditory speech by lipread information: dissipation

Recently, we have shown that lipread speech can recalibrate auditory speech identification when there is a conflict between the auditory and visual information (Bertelson, P., Vroomen, J., De Gelder, B, 2003. Visual recalibration of auditory speech identification: a McGurk aftereffect. Psychol. Sci. 14 (2003) 592–597). When an ambiguous sound intermediate between /aba/ and /ada/ was dubbed onto a face articulating /aba/ (or /ada/), the proportion of responses consistent with the visual stimulus increased in subsequent unimodal auditory sound identification trials, revealing recalibration. In contrast, when an unambiguous /aba/ or /ada/ sound was dubbed onto the face (with no conflict between vision and audition), the proportion of responses decreased, revealing selective adaptation. In the present study we show that recalibration and selective adaptation not only differ in the direction of their aftereffects, but also that they dissipate at different rates, confirming that the effects are caused by different mechanisms.

Tactile—visual temporal ventriloquism: No effect of spatial disparity

Participants made visual temporal order judgments (TOJs) about which of two lights appeared first while taskirrelevant vibrotactile stimuli delivered to the index finger were presented before the first and after the second light. Temporally misaligned tactile stimuli captured the onsets of the lights, thereby improving sensitivity on the visual TOJ task, indicative of tactile-visual (TV) temporal ventriloquism (Experiment 1). The size of this effect was comparable to auditory-visual (AV) temporal ventriloquism (Experiment 2). Spatial discordance between the TV stimuli, as in the AV case, did not harm the effect (Experiments 3 and 4). TV stimuli thus behaved like AV stimuli, demonstrating that spatial co-occurrence is not a necessary constraint for intersensory pairing to occur.

The Build-Up and Transfer of Sensorimotor Temporal Recalibration Measured via a Synchronization Task.

The timing relation between a motor action and the sensory consequences of that action can be adapted by exposing participants to artificially delayed feedback (temporal recalibration). Here, we demonstrate that a sensorimotor synchronization task (i.e., tapping the index finger in synchrony with a pacing signal) can be used as a measure of temporal recalibration. Participants were first exposed to a constant delay (~150 ms) between a voluntary action (a finger tap) and an external feedback stimulus of that action (a visual flash or auditory tone). A subjective “no-delay” condition (~50 ms) served as baseline. After a short exposure phase to delayed feedback participants performed the tapping task in which they tapped their finger in synchrony with a flash or tone. Temporal recalibration manifested itself in that taps were given ~20 ms earlier after exposure to 150 ms delays than in the case of 50 ms delays. This effect quickly built up (within 60 taps) and was bigger for auditory than visual adapters. In Experiment 2, we tested whether temporal recalibration would transfer across modalities by switching the modality of the adapter and pacing signal. Temporal recalibration transferred from visual adapter to auditory test, but not from auditory adapter to visual test. This asymmetric transfer suggests that sensory-specific effects are at play.

No evidence for impaired multisensory integration of low-level audiovisual stimuli in adolescents and young adults with autism spectrum disorders

Abrupt click sounds can improve the visual processing of flashes in several ways. Here, we examined this in high functioning adolescents with Autism Spectrum Disorders (ASD) using three tasks: (1) a task where clicks improve sensitivity for visual temporal order (temporal ventriloquism); (2) a task where a click improves visual search (pip-and-pop), and (3) a task where a click speeds up the visual orienting to a peripheral target (clock reading). Adolescents with ASD were, compared to adolescents with typical development (TD), impaired in judgments of visual temporal order, but they were unimpaired in visual search and orienting. Importantly, in all tasks visual performance of the ASD group improved by the presence of clicks by at least equal amounts as in the TD group. This suggests that adolescents and young adults with ASD show no generalized deficit in the multisensory integration of low-level audiovisual stimuli and/or the phasic alerting by abrupt sounds.

Sounds change four-dot masking.

The temporal occurrence of a flash can be shifted towards a slightly offset sound (temporal ventriloquism). Here we examined whether four-dot masking is affected by this phenomenon. In Experiment 1, we demonstrate that there is release from four-dot masking if two sounds--one before the target and one after the mask--are presented at approximately 100 ms intervals rather than at approximately 0 ms intervals or a silent condition. In Experiment 2, we show that the release from masking originates from an alerting effect of the first sound, and a temporal ventriloquist effect from the first and second sounds that lengthened the perceived interval between target and mask, thereby leaving more time for the target to consolidate. Results thus show that sounds penetrate the visual system at more than one level.