Carol J. Madden

Grounding Cognition: Embodied Sentence Comprehension

There are two views of cognition in general and of language comprehension in particular. According to the traditional view (Chomsky, 1957; Fodor, 1983; Pylyshyn, 1986), the human mind is like a bricklayer, or maybe a contractor, who puts together bricks to build structures. The malleable clay of perception is converted to the neat mental bricks we call words and propositions, units of meaning, which can be used in a variety of structures. But whereas bricklayers and contractors presumably know how bricks are made, cognitive scientists and neuroscientists have no idea how the brain converts perceptual input to abstract lexical and propositional representations – it is simply taken as a given that this occurs (Barsalou, 1999). According to an alternative and emerging view, there are no clear demarcations between perception, action, and cognition. Interactions with the world leave traces of experience in the brain. These traces are (partially) retrieved and used in the mental simulations that make up cognition. Crucially, these traces bear a resemblance to the perceptual/action processes that generated them (Barsalou, 1999) and are highly malleable. Words and grammar are viewed as a set of cues that activate and combine experiential traces in the mental simulation of the described events (Zwaan, 2004). The main purpose of this chapter is to provide a discussion of this view of language comprehension. To set the stage for this discussion we first analyze a series of linguistic examples that present increasingly larger problems for the traditional view.

How does verb aspect constrain event representations

We investigated the relative contribution of perfective and imperfective aspectual cues on situation models. In Experiment 1, participants were more likely to choose pictures showing completed events than pictures showing ongoing events when they had read perfective sentences, but chose either picture after reading imperfective sentences. In Experiment 2, only one picture was presented and participants were faster to respond to completed pictures than to ongoing pictures when they had read perfective sentences, but showed no latency differences after reading the imperfective sentences. In Experiment 3, participants were faster to read perfective sentences after having seen completed pictures rather than intermediate pictures, but there was no difference for imperfective sentences. The consistent pattern of results demonstrates that readers construct mental representations of completed events when the perfective aspect is used to describe an event. The lack of effect on imperfective sentences and pictures suggests that each reader represents an in-progress event at varying stages of completion.

Experiential simulations of negated text information

We investigated the question of whether comprehenders mentally simulate a described situation even when this situation is explicitly negated in the sentence. In two experiments, participants read negative sentences such as There was no eagle in the sky, and subsequently responded to pictures of mentioned entities in the context of a recognition task. Participants’ responses following negative sentences were faster when the depicted entity matched rather than mismatched the negated situation. These results suggest that comprehenders simulate the negated situation when processing a negated sentence. The results thereby provide further support for the experiential-simulations view of language comprehension.

The presence of an event in the narrated situation affects its availability to the comprehender

Narrative descriptions of events often depart from how these events would have occurred in “real time.” For example, narratives often contain time shifts in which events that are irrelevant to the plot are omitted. Zwaan (1996) has shown that these time shifts may affect on-line comprehension. Specifically, they are associated with increases in processing load and a deactivation of previous information. The experiments in the present article show that the situation is more complex. Specifically, there is only a deactivation of previous events if they are not assumed to be ongoing after a time shift. Furthermore, explicit discontinuations of events, as inhe stopped walking also lead to deactivations when compared with explicit continuations and resumptions.

Verb aspect and perceptual simulations.

Two experiments investigate the influence of verb aspect on situation representations. The results demonstrate that comprehenders use verb aspect as a cue to regulate the activation of ongoing simulations of situations over time. Experiment 1 measured word-by-word reading as well as sensibility judgements on sentences in which a target object word had been replaced by a picture. For the past imperfective sentences, participants were faster to process the picture, the two words following the picture, and the sensibility judgements when objects were pictured in use rather than not in use. However, this in-use facilitation was limited to processing of the picture for the past perfect sentences. Experiment 2 served as a control to ensure that the use effect and its interaction with verb aspect were a result of contextual manipulations rather than surface features of the pictures themselves. The results are interpreted within the framework of perceptual simulations during language comprehension.

Perceptual Representation as a Mechanism of Lexical Ambiguity Resolution: An Investigation of Span and Processing Time.

In 2 experiments, the authors investigated the ability of high- and low-span comprehenders to construe subtle shades of meaning through perceptual representation. High- and low-span comprehenders responded to pictures that either matched or mismatched a target objects shape as implied by the preceding sentence context. At 750 ms after hearing the sentence describing the target object, both high- and low-span comprehenders had activated a contextually appropriate perceptual representation of the target object. However, only high-span comprehenders had perceptually represented the contextually appropriate meaning immediately upon hearing the sentence, whereas low-span comprehenders required more processing time before the perceptual representation was activated. The results are interpreted in a framework of co-occurring lexical representations and perceptual-motor representations.

How persistent is the effect of smoking urges on cognitive performance

Several studies have provided empirical support for S. T. Tiffanys (1990) hypothesis that drug urges interfere with cognitive performance. The authors examined the persistence of this effect. Results from an experiment involving 48 smokers and 46 nonsmokers, using a paradigm developed by R. A. Zwaan and T. P. Truitt (1998), suggest that the effect of smoking urges in cognitive performance dissipates over time. The implications of this finding for cognitive theories of drug urges and for future research on the effects of smoking urges are discussed.

Contextual Constraints in Situation Model Construction: An Investigation of Age and Reading Span

ABSTRACT This study examined the effects of age and reading span on the ability to use contextual constraints during language comprehension. Older and younger participants listened to sentences over headphones and named pictures that appeared subsequently. Older adults named pictures faster when the preceding sentence context matched rather than mismatched the shape of the depicted object, but younger adults showed less of a match advantage. This effect of contextual match was especially pronounced in older high-span participants, consistent with models of cognitive aging in which surface level processing declines in older adulthood whereas processing at the situation model level remains intact. Results suggest that the practiced ability to immediately construe word meanings and activate the appropriate stored representations is preserved, if not strengthened throughout the lifespan.

The Sound of Enemies and Friends in the Neighborhood Phonology Mediates Activation of Neighbor Semantics

Previous studies (e.g., Pecher, Zeelenberg, & Wagenmakers, 2005) found that semantic classification performance is better for target words with orthographic neighbors that are mostly from the same semantic class (e.g., living) compared to target words with orthographic neighbors that are mostly from the opposite semantic class (e.g., nonliving). In the present study we investigated the contribution of phonology to orthographic neighborhood effects by comparing effects of phonologically congruent orthographic neighbors (book-hook) to phonologically incongruent orthographic neighbors (sand-wand). The prior presentation of a semantically congruent word produced larger effects on subsequent animacy decisions when the previously presented word was a phonologically congruent neighbor than when it was a phonologically incongruent neighbor. In a second experiment, performance differences between target words with versus without semantically congruent orthographic neighbors were larger if the orthographic neighbors were also phonologically congruent. These results support models of visual word recognition that assume an important role for phonology in cascaded access to meaning.

A hybrid propositional-embodied cognitive architecture for human-robot cooperation

Robot platforms have now reached a level of technical development wherein they are becoming physically capable of useful interaction with humans, while ensuring safety and reasonable cost. The current challenge is for cognitive systems science to provide these robots with the necessary capabilities so that they can interact and cooperate with humans in a natural manner. We are addressing this problem by exploiting two central ideas derived from the human psychological sciences. The first idea is that the human conceptual system is based on situated simulations that are instantiated in the same systems that are used for perception and action, referred to as embodied cognition. The second idea is that human cooperation relies on the cooperating agents sharing a common representation of their shared plan, which involves the actions of both agents. This representation allows them to cooperate, to trade roles, and to help one another if necessary. We have implemented these concepts on multiple robot platforms including the HRP2 humanoid, and the Cooperator and Cooperator II visually guided robot manipulators. This paper will present the motivation for this system and results, and will then outline what we consider to be the crucial issues for human-like cognitive systems.