Wessel O. van Dam

Flexibility in embodied lexical-semantic representations

According to an embodied view of language comprehension, language concepts are grounded in our perceptual systems. Evidence for the idea that concepts are grounded in areas involved in action and perception comes from both behavioral and neuroimaging studies (Glenberg [1997]: Behav Brain Sci 20:1‐55; Barsalou [1999]: Behav Brain Sci 22:577‐660; Pulvermueller [1999]: Behav Brain Sci 22:253‐336; Barsalou et al. [2003]: Trends Cogn Sci 7:84‐91). However, the results from several studies indicate that the activation of information in perception and action areas is not a purely automatic process (Raposo et al. [2009]: Neuropsychologia 47:388‐396; Rueschemeyer et al. [2007]: J Cogn Neurosci 19:855‐865). These findings suggest that embodied representations are flexible. In these studies, flexibility is characterized by the relative presence or absence of activation in our perceptual systems. However, even if the context in which a word is presented does not undermine a motor interpretation, it is possible that the degree to which a modality‐specific region contributes to a representation depends on the context in which conceptual features are retrieved. In the present study, we investigated this issue by presenting word stimuli for which both motor and visual properties (e.g., Tennis ball, Boxing glove) were important in constituting the concept. Conform with the idea that language representations are flexible and context dependent, we demonstrate that the degree to which a modality‐specific region contributes to a representation considerably changes as a function of context. Hum Brain Mapp 33:2322–2333, 2012.

How specifically are action verbs represented in the neural motor system: An fMRI study

Embodied accounts of language processing suggest that sensorimotor areas, generally dedicated to perception and action, are also involved in the processing and representation of word meaning. Support for such accounts comes from studies showing that language about actions selectively modulates the execution of congruent and incongruent motor responses (e.g., Glenberg & Kaschak, 2002), and from functional neuroimaging studies showing that understanding action-related language recruits sensorimotor brain areas (e.g. Hauk, Johnsrude, & Pulvermueller, 2004). In the current experiment we explored the basis of the neural motor systems involvement in representing words denoting actions. Specifically, we investigated whether the motor systems involvement is modulated by the specificity of the kinematics associated with a word. Previous research in the visual domain indicates that words denoting basic level category members lacking a specific form (e.g., bird) are less richly encoded within visual areas than words denoting subordinate level members (e.g., pelican), for which the visual form is better specified (Gauthier, Anderson, Tarr, Skudlarski, & Gore, 1997). In the present study we extend these findings to the motor domain. Modulation of the BOLD response elicited by verbs denoting a general motor program (e.g., to clean) was compared to modulation elicited by verbs denoting a more specific motor program (e.g., to wipe). Conform with our hypothesis, a region within the bilateral inferior parietal lobule, typically serving the representation of action plans and goals, was sensitive to the specificity of motor programs associated with the action verbs. These findings contribute to the growing body of research on embodied language representations by showing that the concreteness of an action-semantic feature is reflected in the neural response to action verbs.

Effects of intentional motor actions on embodied language processing

Embodied theories of language processing suggest that this motor simulation is an automatic and necessary component of meaning representation. If this is the case, then language and action systems should be mutually dependent (i.e., motor activity should selectively modulate processing of words with an action-semantic component). In this paper, we investigate in two experiments whether evidence for mutual dependence can be found using a motor priming paradigm. Specifically, participants performed either an intentional or a passive motor task while processing words denoting manipulable and nonmanipulable objects. The performance rates (Experiment 1) and response latencies (Experiment 2) in a lexical-decision task reveal that participants performing an intentional action were positively affected in the processing of words denoting manipulable objects as compared to nonmanipulable objects. This was not the case if participants performed a secondary passive motor action (Experiment 1) or did not perform a secondary motor task (Experiment 2). The results go beyond previous research showing that language processes involve motor systems to demonstrate that the execution of motor actions has a selective effect on the semantic processing of words. We suggest that intentional actions activate specific parts of the neural motor system, which are also engaged for lexical-semantic processing of action-related words and discuss the beneficial versus inhibitory nature of this relationship. The results provide new insights into the embodiment of language and the bidirectionality of effects between language and action processing.

Context effects in embodied lexical-semantic processing

The embodied view of language comprehension proposes that the meaning of words is grounded in perception and action rather than represented in abstract amodal symbols. Support for embodied theories of language processing comes from behavioral studies showing that understanding a sentence about an action can modulate congruent and incongruent physical responses, suggesting motor involvement during comprehension of sentences referring to bodily movement. Additionally, several neuroimaging studies have provided evidence that comprehending single words denoting manipulable objects elicits specific responses in the neural motor system. An interesting question that remains is whether action semantic knowledge is directly activated as motor simulations in the brain, or rather modulated by the semantic context in which action words are encountered. In the current paper we investigated the nature of conceptual representations using a go/no-go lexical decision task. Specifically, target words were either presented in a semantic context that emphasized dominant action features (features related to the functional use of an object) or non-dominant action features. The response latencies in a lexical decision task reveal that participants were faster to respond to words denoting objects for which the functional use was congruent with the prepared movement. This facilitation effect, however, was only apparent when the semantic context emphasized corresponding motor properties. These findings suggest that conceptual processing is a context-dependent process that incorporates motor-related knowledge in a flexible manner.

Action-effect binding by observational learning

The acquisition of bidirectional action–effect associations plays a central role in the ability to intentionally control actions. Humans learn about actions not only through active experience, but also through observing the actions of others. In Experiment 1, we examined whether action–effect associations can be acquired by observational learning. To this end, participants observed how a model repeatedly pressed two buttons during an observation phase. Each of the buttonpresses led to a specific tone (action effect). In a subsequent test phase, the tones served as target stimuli to which the participants had to respond with buttonpresses. Reaction times were shorter if the stimulus–response mapping in the test phase was compatible with the action–effect association in the observation phase. Experiment 2 excluded the possibility that the impact of perceived action effects on own actions was driven merely by an association of spatial features with the particular tones. Furthermore, we demonstrated that the presence of an agent is necessary to acquire novel action–effect associations through observation. Altogether, the study provides evidence for the claim that bidirectional action–effect associations can be acquired by observational learning. Our findings are discussed in the context of the idea that the acquisition of action–effect associations through observation is an important cognitive mechanism subserving the human ability for social learning.

Flexibility in embodied language processing: context effects in lexical access.

According to embodied theories of language (ETLs), word meaning relies on sensorimotor brain areas, generally dedicated to acting and perceiving in the real world. More specifically, words denoting actions are postulated to make use of neural motor areas, while words denoting visual properties draw on the resources of visual brain areas. Therefore, there is a direct correspondence between word meaning and the experience a listener has had with a words referent on the brain level. Behavioral and neuroimaging studies have provided evidence in favor of ETLs; however, recent studies have also shown that sensorimotor information is recruited in a flexible manner during language comprehension (e.g., Raposo et al.; Van Dam et al.,), leaving open the question as to what level of language processing sensorimotor activations contribute. In this study, we investigated the time course of modality-specific contributions (i.e., the contribution of action information) as to word processing by manipulating both (a) the linguistic and (b) the action context in which target words were presented. Our results demonstrate that processes reflecting sensorimotor information play a role early in word processing (i.e., within 200 ms of word presentation), but that they are sensitive to the linguistic context in which a word is presented. In other words, when sensorimotor information is activated, it is activated quickly; however, specific words do not reliably activate a consistent sensorimotor pattern.

Context-dependent changes in functional connectivity of auditory cortices during the perception of object words

Embodied theories hold that cognitive concepts are grounded in our sensorimotor systems. Specifically, a number of behavioral and neuroimaging studies have buttressed the idea that language concepts are represented in areas involved in perception and action [Pulvermueller, F. Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6, 576–582, 2005; Barsalou, L. W. Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660, 1999]. Proponents of a strong embodied account argue that activity in perception/action areas is triggered automatically upon encountering a word and reflect static semantic representations. In contrast to what would be expected if lexical semantic representations are automatically triggered upon encountering a word, a number of studies failed to find motor-related activity for words with a putative action-semantic component [Raposo, A., Moss, H. E., Stamatakis, E. A., & Tyler, L. K. Modulation of motor and premotor cortices by actions, action words and action sentences. Neuropsychologia, 47, 388–396, 2009; Rueschemeyer, S.-A., Brass, M., & Friederici, A. D. Comprehending prehending: Neural correlates of processing verbs with motor stems. Journal of Cognitive Neuroscience, 19, 855–865, 2007]. In a recent fMRI study, Van Dam and colleagues [Van Dam, W. O., Van Dijk, M., Bekkering, H., & Rueschemeyer, S.-A. Flexibility in embodied lexical-semantic representations. Human Brain Mapping, in press] showed that the degree to which a modality-specific region contributes to a representation considerably changes as a function of context. In the current study, we presented words for which both motor and visual properties (e.g., tennis ball, boxing glove) were important in constituting the concept. Our aim was to corroborate on earlier findings of flexible and context-dependent language representations by testing whether functional integration between auditory brain regions and perception/action areas is modulated by context. Functional connectivity was investigated by means of a psychophysiological interaction analysis, in which we found that bilateral superior temporal gyrus was more strongly connected with brain regions relevant for coding action information: (1) for Action Color words vs. Abstract words, and (2) for Action Color words presented in a context that emphasized action vs. a context that emphasized color properties.

Familiarity differentially affects right hemisphere contributions to processing metaphors and literals

The role of the two hemispheres in processing metaphoric language is controversial. While some studies have reported a special role of the right hemisphere (RH) in processing metaphors, others indicate no difference in laterality relative to literal language. Some studies have found a role of the RH for novel/unfamiliar metaphors, but not conventional/familiar metaphors. It is not clear, however, whether the role of the RH is specific to metaphor novelty, or whether it reflects processing, reinterpretation or reanalysis of novel/unfamiliar language in general. Here we used functional magnetic resonance imaging (fMRI) to examine the effects of familiarity in both metaphoric and non-metaphoric sentences. A left lateralized network containing the middle and inferior frontal gyri, posterior temporal regions in the left hemisphere (LH), and inferior frontal regions in the RH, was engaged across both metaphoric and non-metaphoric sentences; engagement of this network decreased as familiarity decreased. No region was engaged selectively for greater metaphoric unfamiliarity. An analysis of laterality, however, showed that the contribution of the RH relative to that of LH does increase in a metaphor-specific manner as familiarity decreases. These results show that RH regions, taken by themselves, including commonly reported regions such as the right inferior frontal gyrus (IFG), are responsive to increased cognitive demands of processing unfamiliar stimuli, rather than being metaphor-selective. The division of labor between the two hemispheres, however, does shift towards the right for metaphoric processing. The shift results not because the RH contributes more to metaphoric processing. Rather, relative to its contribution for processing literals, the LH contributes less.

Resting state signatures of domain and demand-specific working memory performance

Working memory (WM) is one of the key constructs in understanding higher-level cognition. We examined whether patterns of activity in the resting state of individual subjects are correlated with their off-line working and short-term memory capabilities. Participants completed a resting-state fMRI scan and off-line working and short-term memory (STM) tests with both verbal and visual materials. We calculated fractional amplitude of low frequency fluctuations (fALFF) from the resting state data, and also computed connectivity between seeds placed in frontal and parietal lobes. Correlating fALFF values with behavioral measures showed that the fALFF values in a widespread fronto-parietal network during rest were positively correlated with a combined memory measure. In addition, STM showed a significant correlation with fALFF within the right angular gyrus and left middle occipital gyrus, whereas WM was correlated with fALFF values within the right IPS and left dorsomedial cerebellar cortex. Furthermore, verbal and visuospatial memory capacities were associated with dissociable patterns of low-frequency fluctuations. Seed-based connectivity showed correlations with the verbal WM measure in the left hemisphere, and with the visual WM measure in the right hemisphere. These findings contribute to our understanding of how differences in spontaneous low-frequency fluctuations at rest are correlated with differences in cognitive performance.

Embodied grounding of memory: Toward the effects of motor execution on memory consolidation

Behavioural and neuroscientific research has provided evidence for a strong functional link between the neural motor system and lexical–semantic processing of action-related language. It remains unclear, however, whether the impact of motor actions is restricted to online language comprehension or whether sensorimotor codes are also important in the formation and consolidation of persisting memory representations of the words referents. The current study now demonstrates that recognition performance for action words is modulated by motor actions performed during the retention interval. Specifically, participants were required to learn words denoting objects that were associated with either a pressing or a twisting action (e.g., piano, screwdriver) and words that were not associated to actions. During a 6–8-minute retention phase, participants performed an intervening task that required the execution of pressing or twisting responses. A subsequent recognition task revealed a better memory for words that denoted objects for which the functional use was congruent with the action performed during the retention interval (e.g., pepper mill–twisting action, doorbell–pressing action) than for words that denoted objects for which the functional use was incongruent. In further experiments, we were able to generalize this effect of selective memory enhancement of words by performing congruent motor actions to an implicit perceptual (Experiment 2) and implicit semantic memory test (Experiment 3). Our findings suggest that a reactivation of motor codes affects the process of memory consolidation and emphasizes therefore the important role of sensorimotor codes in establishing enduring semantic representations.