Kathleen M. Eberhard

Eye movements as a window into real-time spoken language comprehension in natural contexts.

When listeners follow spoken instructions to manipulate real objects, their eye movements to the objects are closely time locked to the referring words. We review five experiments showing that this time-locked characteristic of eye movements provides a detailed profile of the processes that underlie real-time spoken language comprehension. Together, the first four experiments showed that listerners immediately integrated lexical, sublexical, and prosodic information in the spoken input with information from the visual context to reduce the set of referents to the intended one. The fifth experiment demonstrated that a visual referential context affected the initial structuring of the linguistic input, eliminating even strong syntactic preferences that result in clear garden paths when the referential context is introduced linguistically. We argue that context affected the earliest moments of language processing because it was highly accessible and relevant to the behavioral goals of the listener.

Making Syntax of Sense: Number Agreement in Sentence Production.

Grammatical agreement flags the parts of sentences that belong together regardless of whether the parts appear together. In English, the major agreement controller is the sentence subject, the major agreement targets are verbs and pronouns, and the major agreement category is number. The authors expand an account of number agreement whose tenets are that pronouns acquire number lexically, whereas verbs acquire it syntactically but with similar contributions from number meaning and from the number morphology of agreement controllers. These tenets were instantiated in a model using existing verb agreement data. The model was then fit to a new, more extensive set of verb data and tested with a parallel set of pronoun data. The theory was supported by the models outcomes. The results have implications for the integration of words and structures, for the workings of agreement categories, and for the nature of the transition from thought to language.

Linguistically Mediated Visual Search

During an individuals normal interaction with the environment and other humans, visual and linguistic signals often coincide and can be integrated very quickly. This has been clearly demonstrated in recent eyetracking studies showing that visual perception constrains on-line comprehension of spoken language. In a modified visual search task, we found the inverse, that real-time language comprehension can also constrain visual perception. In standard visual search tasks, the number of distractors in the display strongly affects search time for a target defined by a conjunction of features, but not for a target defined by a single feature. However, we found that when a conjunction target was identified by a spoken instruction presented concurrently with the visual display, the incremental processing of spoken language allowed the search process to proceed in a manner considerably less affected by the number of distractors. These results suggest that perceptual systems specialized for language and for vision interact more fluidly than previously thought.

A real-time robotic model of human reference resolution using visual constraints

Evidence from recent psycholinguistic experiments suggests that humans resolve reference incrementally in the presence of constraining visual context. In this paper, we present and evaluate a computational model of human reference resolution that directly builds a semantic interpretation of an utterance without the need for a separate syntactic analysis phase, which typically involves the construction of parse trees. The model is implemented on a robot using real audio and video inputs, (thus it operates in real time), and is distributed over several computers, which run in parallel. Results from experiments with the model confirm the viability of the algorithm to process semantic interpretations, in particular reference incrementally, as demonstrated to be employed by humans.

Linguistically mediated visual search: The critical role of speech rate

Recent studies have shown that the presentation of concurrent linguistic context can lead to highly efficient performance in a standard conjunction search task by the induction of an incremental search strategy (Spivey, Tyler, Eberhard, & Tanenhaus, 2001). However, these findings were obtained under anomalously slow speech rate conditions. Accordingly, in the present study, the effects of concurrent linguistic context on visual search performance were compared when speech was recorded at both a normal rate and a slow rate. The findings provided clear evidence that the visual search benefit afforded by concurrent linguistic context was contingent on speech rate, with normal speech producing a smaller benefit. Overall, these findings have important implications for understanding how linguistic and visual processes interact in real time and suggest a disparity in the temporal resolution of speech comprehension and visual search processes.

Team Communication as a Collaborative Process

Communication channels can reveal a great deal of information about the effectiveness of a team. This is particularly relevant for teams operating in performance settings such as medical groups, military squads, and mixed human-robot teams. Currently, it is not known how various factors including coordination strategy, speaker role, and time pressure affect communication in collaborative tasks. The purpose of this paper is to systematically explore how these factors interact with team discourse in order to better understand effective communication patterns. In our analysis of a corpus of remote task-oriented dialogue (CReST corpus), we found that a variety of linguistic- and dialogue-level features were influenced by time pressure, speaker role, and team effectiveness. We also found that effective teams had a higher speech rate, and used specific grounding strategies to improve efficiency and coordination under time pressure. These results inform our understanding of the various factors that influence team communication, and highlight ways in which effective teams overcome constraints on their communication channels.

Effects of verb complexity on speech errors

Two experiments examined whether semantically related verbs that contrast with respect to the absence versus the presence of an additional semantic feature differentially compete for selection during the encoding of a sentence for production. In both experiments, a speech error induction task was used to elicit contextual (misordering) errors involving semantically related verbs that contrasted only in their semantic complexity or in both their semantic and morphophonological complexity. The prediction was that an asymmetry in contextual errors would be observed in which the more complex verbs would replace the simpler verbs more often than the reverse. This prediction was confirmed in both experiments, with more perseverations and anticipations involving the semantically more complex verb of antonym pairs in Experiment 1, and more perseverations and anticipations involving the semantically more-specified verb of a heavy-light pair in Experiment 2. The implications of the results for spreading-activation theories of language production are discussed.

Exploring Coordination in Human-Robot Teams in Space

Human-robot teams in space environments are difficult to evaluate, in large part because performance of these teams is influenced by a variety of factors, including team size, structure, and composition. We introduce and describe a novel experimental framework that is sensitive to these factors, and that serves as a testbed to facilitate the study of human-robot teaming in space. We also report on the results of a preliminary study in this framework that involves a human interacting with a simulated Mars rover. Our findings show that people exhibited great variation in strategy and performance, and point to the role that decision-making and task-switching may have played in this result. This study is the first in a larger effort to develop a rich multimodal corpus and to investigate various dimensions of teaming in this domain.