Chiara Gambi

A Cognitive Architecture for the Coordination of Utterances

Dialog partners coordinate with each other to reach a common goal. The analogy with other joint activities has sparked interesting observations (e.g., about the norms governing turn-taking) and has informed studies of linguistic alignment in dialog. However, the parallels between language and action have not been fully explored, especially with regard to the mechanisms that support moment-by-moment coordination during language use in conversation. We review the literature on joint actions to show (i) what sorts of mechanisms allow coordination and (ii) which types of experimental paradigms can be informative of the nature of such mechanisms. Regarding (i), there is converging evidence that the actions of others can be represented in the same format as one’s own actions. Furthermore, the predicted actions of others are taken into account in the planning of one’s own actions. Similarly, we propose that interlocutors are able to coordinate their acts of production because they can represent their partner’s utterances. They can then use these representations to build predictions, which they take into account when planning self-generated utterances. Regarding (ii), we propose a new methodology to study interactive language. Psycholinguistic tasks that have traditionally been used to study individual language production are distributed across two participants, who either produce two utterances simultaneously or complete each other’s utterances.

If you stay, it might be easier: Switch costs from comprehension to production in a joint switching task.

Switching language is costly for bilingual speakers and listeners, suggesting that language control is effortful in both modalities. But are the mechanisms underlying language control similar across modalities? In this study, we attempted to answer this question by testing whether bilingual speakers incur a cost when switching to a different language than the one just used by their interlocutor. Pairs of unbalanced Dutch (L1)-English (L2) bilinguals took turns naming pictures in a pure Dutch, a pure English, and a mixed-language block. In the mixed block, one participant (Switching Participant) voluntarily switched between Dutch and English, whereas the other (Non-switching Participant) named all pictures in Dutch. Within the mixed block, the Non-switching participant took longer to name pictures when the Switching participants response on the preceding trial had been in English rather than Dutch, and this local switch cost was larger the more the Non-switching participant was proficient in English. Additionally, there was strong cross-person, item-level interference: The Non-switching participant named pictures more slowly in Dutch if the Switching participant had previously named those same pictures in English rather than Dutch. These findings indicate that comprehension of utterances produced by another speaker in L2 makes subsequent production of L1 utterances more costly. We interpret this as evidence that language control mechanisms are shared between comprehension and production, and specifically that bottom-up factors have a considerable influence on language selection processes in both modalities.

Learning to predict or predicting to learn

ABSTRACT Humans complete complex commonplace tasks, such as understanding sentences, with striking speed and accuracy. This expertise is dependent on anticipation: predicting upcoming words gets us ahead of the game. But how do we master the game in the first place? To make accurate predictions, children must first learn their language. One possibility is that prediction serves double duty, enabling rapid language learning as well as understanding. Children could master the structures of their language by predicting how speakers will behave and, when those guesses are wrong, revising their linguistic representations. A number of prominent computational models assume that children learn in this way. But is that assumption correct? Here, we lay out the requirements for showing that children use “predictive learning”, and review the current evidence for this position. We argue that, despite widespread enthusiasm for the idea, we cannot yet conclude that children “predict to learn”.

Prediction at all levels: forward model predictions can enhance comprehension

We discuss two limitations of Hickoks account. First, we propose that ideas from motor control and planning should be brought wholesale into psycholinguistics so that processing at every level of the linguistic hierarchy (from concepts to sounds) should be recast in terms of forward model predictions and implementation. Second, we argue that motor involvement can sometimes enhance perception. We conclude that our account is consistent with a dual route model of comprehension in which different routes to prediction can interact.

Interference in Joint Picture Naming.

In 4 experiments we showed that picture naming latencies are affected by beliefs about the task concurrently performed by another speaker. Participants took longer to name pictures when they believed that their partner concurrently named pictures than when they believed their partner was silent (Experiments 1 and 4) or concurrently categorized the pictures as being from the same or from different semantic categories (Experiment 2). However, picture naming latencies were not affected by beliefs about what ones partner said, as it did not matter whether participants believed their partner produced the same utterance, or an utterance that differed by ordering (Experiments 1 and 2) or lexical content (Experiments 3 and 4). These findings are consistent with the proposal that speakers represent whether another speaker is preparing to speak but not what they are preparing to say.

Predicting and imagining language

ABSTRACT To what extent is predicting language akin to imagining language? Recently, researchers have argued that covert simulation of the production system underlies both articulation imagery and predicting what somebody is about to say. Moreover, experimental evidence implicates potentially similar production-related mechanisms in prediction during language comprehension and in mental imagery tasks. We discuss evidence in favour of this proposal and argue that imagining others’ utterances can also implicate covert simulation. Finally, we briefly review evidence that speakers in joint language tasks cannot help but mentally represent (i.e., imagine) whether others are engaging in language production, and that they do so using mechanisms that are also implicated in preparing to speak.

Talking to each other and talking together: joint language tasks and degrees of interactivity

A second-person perspective in neuroscience is particularly appropriate for the study of communication. We describe how the investigation of joint language tasks can contribute to our understanding of the mechanisms underlying interaction.

Coordinating Utterances During Turn-Taking: The Role of Prediction, Response Preparation, and Articulation

ABSTRACT During conversation, interlocutors rapidly switch between speaker and listener roles and take turns at talk. How do they achieve such fine coordination? Most research has concentrated on the role of prediction, but listeners must also prepare a response in advance (assuming they wish to respond) and articulate this response at the appropriate moment. Such mechanisms may overlap with the processes of comprehending the speaker’s incoming turn and predicting its end. However, little is known about the stages of response preparation and production. We discuss three questions pertaining to such stages: (1) Do listeners prepare their own response in advance?, (2) Can listeners buffer their prepared response?, and (3) Does buffering lead to interference with concurrent comprehension? We argue that fine coordination requires more than just an accurate prediction of the interlocutor’s incoming turn: Listeners must also simultaneously prepare their own response.

Spontaneous adaptation explains why people act faster when being imitated

The human ability to perform joint actions is often attributed to high-level cognitive processes. For example, the finding that action leaders act faster when imitated by their partners has been interpreted as evidence for anticipation of the other’s actions (Pfister, Dignath, Hommel, & Kunde, 2013). In two experiments, we showed that a low-level mechanism can account for this finding. Action leaders were faster when imitated than when counterimitated, but only if they could observe their partner’s actions (Exp. 1). Crucially, when due to our manipulation the partner’s imitative actions became slower than the counterimitative actions, leaders also became slower when they were imitated, and faster when counterimitated (Exp. 2). Our results suggest that spontaneous temporal adaptation is a key mechanism in joint action tasks. We argue for a reconsideration of other phenomena that have traditionally been attributed solely to high-level processes.

Predicting while comprehending language: a theory and review

Researchers agree that comprehenders regularly predict upcoming language, but they do not always agree on what prediction is (and how to differentiate it from integration) or what constitutes evidence for it. After defining prediction, we show that it occurs at all linguistic levels from semantics to form, and then propose a theory of which mechanisms comprehenders use to predict. We argue that they most effectively predict using their production system (i.e., prediction-by-production): They covertly imitate the linguistic form of the speaker’s utterance and construct a representation of the underlying communicative intention. Comprehenders can then run this intention through their own production system to prepare the predicted utterance. But doing so takes time and resources, and comprehenders vary in the extent of preparation, with many groups of comprehenders (non-native speakers, illiterates, children, and older adults) using it less than typical native young adults. We thus argue that prediction-by-production is an optional mechanism, which is augmented by mechanisms based on association. Support for our proposal comes from many areas of research (electrophysiological, eye-tracking, and behavioral studies of reading, spoken language processing in the context of visual environments, speech processing, and dialogue).