Paul Bertelson

Does awareness of speech as a sequence of phones arise spontaneously

It was found that illiterate adults could neither delete nor add a phone at the beginning of a non-word; but these tasks were rather easily performed by people with similar environment and childhood experiences, who learned to read rudimentarily as adults. Awareness of speech as a sequence of phones is thus not attained spontaneously in the course of general cognitive growth, but demands some specific training, which, for most persons, is probably provided by learning to read in the alphabetic system.

Literacy training and speech segmentation

Abstract New groups of illiterate and ex-illiterate adults, comparable to those of Morais et. al (1979), were given a battery of tasks designed to assess the specificity of the effect of literacy training on speech segmentation. As in the previous study, a strong difference was observed between the two groups on the task of deleting the initial consonant of an utterance. The illeterates displayed the same incapacity to deal with phonetic segments in a detection task and in a progressive free segmentation task. Their performance was better, although still inferior to that of ex-illiterates, on both deletion and detection when the critical unit was a syllable rather than a consonant, as well as in a task of rhyme detection. No significant difference was observed in a task of melody segmentation, on which both groups performed poorly. The high specificity of the differences in performance level implies that they cannot result to an important extent from differences in general ability or motivation between the two groups of subjects. They rather mean that while sensitivity to rhyme and analysis into syllables can develop up to some point in the absence of the experience normally provided by reading instruction, analysis into phonetic segments requires that experience. Finally, in a picture memory task, the illiterates showed a phonological similarity effect, which is consistent with other results suggesting that the use of phonological codes for short-term retention does not require explicit phonetic analysis.

Multisensory integration, perception and ecological validity

Studies of multimodal integration have relied to a large extent on conflict situations, in which two sensory modalities receive incongruent data concerning one aspect of the source. Exposure to such situations produces immediate crossmodal biases as well as longer lasting aftereffects, revealing recalibrations of data-to-percept matches. In the natural environment, such phenomena might be adaptive, by reducing the perturbing effects of factors like noise or growth-induced changes in receptor organs, and by enriching the percept. However, experimental results generalize to real life only when they reflect automatic perceptual processes, and not response strategies adopted to satisfy the particular demands of laboratory tasks. Here, we focus on this issue and review ways of addressing it that have been developed recently.

The time course of preparation

The time course of the adjustments triggered by a warning signal was studied by measuring choice reaction times (RTs) at different predictable foreperiods after such a signal. Before the warning signal, a high time uncertainty situation was created by imposing either a long constant foreperiod of 5 sec. or one varying in the range 1.5 to 5 sec. The warning signal was a click. Foreperiods ranging from 0 to 300 millisec. were used in different blocks of trials. The stimulus was the onset of one of two lamps calling for the pressing of one of two keys. A control condition, without click, was used also. RTs were found to decrease continuously when the forperiod was increased from 0 to 100-150 millisec. The click delivered simultaneously with the stimulus permitted reactions significantly faster than in the control condition. It is concluded (a) that the latency of preparation can be much shorter than the 2 to 4 sec. reported by Woodrow; (b) that the warning signal can be used as a time cue to start preparatory adjustments without starting a refractory period of the order of magnitude found in experiments with pairs of successive reactions, and thus that such refractory periods are not the inevitable cost of paying attention to a signal. There is also some suggestion that in this situation the click not only triggers preparatory adjustments, but also causes an immediate facilitation of the reaction to the visual stimulus.

Sequential redundancy and speed in a serial two-choice responding task

The task used is a self-paced 2-choice serial responding task, involving two neon lamps as signals and two response keys in simple spatial correspondence with the signals. In Experiment 1, three conditions with respectively 1/4, 1/2 and 3/4 alternations between successive signals were compared. It appeared that subjects go faster with 1/4 than with 3/4 alternations. From the point of view of information theory, both conditions present the same amount of information per signal. The asymmetry in the effects of both forms of sequential redundancy could be attributed either to some bias in the expectations of the subjects or to some inertia phenomenon favouring responses to repeated signals. It was reasoned that if the last explanation was true, the asymmetry would be reduced if a time-lag was introduced between end of response and appearance of next signal. This was done for one group of subjects in Experiment 2. These subjects worked on the same three conditions, with a time-lag of 0.5 sec. The asymmetry was significantly reduced when compared with another group of subjects who worked with the same 0.05 sec. time-lag as in Experiment 1. Analysis of separate RTs confirmed the main conclusion that choice processes involve a transitory residual effect favouring repetitions.

The ventriloquist effect does not depend on the direction of deliberate visual attention

It is well known that discrepancies in the location of synchronized auditory and visual events can lead to mislocalizations of the auditory source, so-called ventriloquism. In two experiments, we tested whether such cross-modal influences on auditory localization depend on deliberate visual attention to the biasing visual event. In Experiment 1, subjects pointed to the apparent source of sounds in the presence or absence of a synchronous peripheral flash. They also monitored for target visual events, either at the location of the peripheral flash or in a central location. Auditory localization was attracted toward the synchronous peripheral flash, but this was unaffected by where deliberate visual attention was directed in the monitoring task. In Experiment 2, bilateral flashes were presented in synchrony with each sound, to provide competing visual attractors. When these visual events were equally salient on the two sides, auditory localization was unaffected by which side subjects monitored for visual targets. When one flash was larger than the other, auditory localization was slightly but reliably attracted toward it, but again regardless of where visual monitoring was required. We conclude that ventriloquism largely reflects automatic sensory interactions, with little or no role for deliberate spatial attention.

Serial choice reaction-time as a function of response versus signal-and-response repetition.

IN serial self-paced choice reaction, reaction times (RTs) to repeated signals, that is, to a signal identical to the immediately preceding one, are shorter than RTs to new signals, that is, to a signal different from the preceding one, provided the time-lag separating the onset of each signal from the end of the previous response is short. This was shown in tasks involving either two signals and two responses1,2 or four signals and four responses2, where repetition of the response was, except for the very rare errors, obtained only through repetition of the signal. The present letter reports a first trial to distinguish the effort of signal repetition, not confounded with that of response repetition, by considering a task where more than one signal is associated with each response. In such a situation, the relationship of a cycle to the preceding cycle can be one of ‘identity’ (same signal), of ‘equivalence’ (different signal but same response) or of ‘difference’ (different response). By comparing RTs on cycles belonging to the first two categories, one could then examine if repetition of the signal has an effect per se.

Visual Recalibration of Auditory Speech Identification A McGurk Aftereffect

The kinds of aftereffects, indicative of cross-modal recalibration, that are observed after exposure to spatially incongruent inputs from different sensory modalities have not been demonstrated so far for identity incongruence. We show that exposure to incongruent audiovisual speech (producing the well-known McGurk effect) can recalibrate auditory speech identification. In Experiment 1, exposure to an ambiguous sound intermediate between /aba/ and /ada/ dubbed onto a video of a face articulating either /aba/ or /ada/ increased the proportion of /aba/ or /ada/ responses, respectively, during subsequent sound identification trials. Experiment 2 demonstrated the same recalibration effect or the opposite one, fewer /aba/ or /ada/ responses, revealing selective speech adaptation, depending on whether the ambiguous sound or a congruent nonambiguous one was used during exposure. In separate forced-choice identification trials, bimodal stimulus pairs producing these contrasting effects were identically categorized, which makes a role of postperceptual factors in the generation of the effects unlikely.

The ventriloquist effect does not depend on the direction of automatic visual attention

Previously, we showed that the visual bias of auditory sound location, or ventriloquism, does not depend on the direction of deliberate, orendogenous, attention (Bertelson, Vroomen, de Gelder, & Driver, 2000). In the present study, a similar question concerning automatic, orexogenous, attention was examined. The experimental manipulation was based on the fact that exogenous visual attention can be attracted toward asingleton—that is, an item different on some dimension from all other items presented simultaneously. A display was used that consisted of a row of four bright squares with one square, in either the left- or the rightmost position,smaller than the others, serving as the singleton. In Experiment 1, subjects made dichotomous left-right judgments concerning sound bursts, whose successivelocations were controlled by a psychophysical staircase procedure and which were presented in synchrony with a display with the singleton either left or right. Results showed that the apparent location of the sound was attractednot toward the singleton, but instead toward the big squares at the opposite end of the display. Experiment 2 was run to check that the singleton effectively attracted exogenous attention. The task was to discriminate target letters presented either on the singleton or on the opposite big square. Performance deteriorated when the target was on the big square opposite the singleton, in comparison with control trials with no singleton, thus showing that the singleton attracted attention away from the target location. In Experiment 3, localization and discrimination trials were mixed randomly so as to control for potential differences in subjects’ strategies in the two preceding experiments. Results were as before, showing that the singleton attracted attention, whereas sound localization was shifted away from the singleton. Ventriloquism can thus be dissociated from exogenous visual attention and appears to reflect sensory interactions with little role for the direction of visual spatial attention.

Chapter 14 Ventriloquism: A case of crossmodal perceptual grouping

The term ventriloquism refers to various manifestations of crossmodal spatial interaction between auditory and visual inputs which are observed in sensory conflict situations. These involve on-line manifestations like perceptual fusion in spite of spatial discrepancy and immediate crossmodal bias, by which localization of data in one modality are attracted toward conflicting data in the other modality, and also off-line aftereffects. Existing data concerning the variables on which the occurrence of the phenomena depend are reviewed, with insistence on the distinction between sensory and cognitive factors. A question which has been neglected in the literature consists of separating genuine perceptual contributions to the phenomena from voluntary post-perceptual adjustments. Using a new paradigm based on psychophysical staircases, we have shown that the visual bias of auditory location occurs even when the subject is not aware of the auditory-visual discrepancy, hence cannot reduce to voluntary corrections. Other experiments have demonstrated that ventriloquism cannot be explained by deliberate shifts of spatial attention. It is concluded that ventriloquism reflects a phenomenon of automatic crossmodal pairing, that is, formation of a cross-modal perceptual unit which takes place at a pre-conscious processing stage and thus must be clearly distinguished from conscious perceptual fusion.