Moritz F. Wurm

Decoding actions at different levels of abstraction

Brain regions that mediate action understanding must contain representations that are action specific and at the same time tolerate a wide range of perceptual variance. Whereas progress has been made in understanding such generalization mechanisms in the object domain, the neural mechanisms to conceptualize actions remain unknown. In particular, there is ongoing dissent between motor-centric and cognitive accounts whether premotor cortex or brain regions in closer relation to perceptual systems, i.e., lateral occipitotemporal cortex, contain neural populations with such mapping properties. To date, it is unclear to which degree action-specific representations in these brain regions generalize from concrete action instantiations to abstract action concepts. However, such information would be crucial to differentiate between motor and cognitive theories. Using ROI-based and searchlight-based fMRI multivoxel pattern decoding, we sought brain regions in human cortex that manage the balancing act between specificity and generality. We investigated a concrete level that distinguishes actions based on perceptual features (e.g., opening vs closing a specific bottle), an intermediate level that generalizes across movement kinematics and specific objects involved in the action (e.g., opening different bottles with cork or screw cap), and an abstract level that additionally generalizes across object category (e.g., opening bottles or boxes). We demonstrate that the inferior parietal and occipitotemporal cortex code actions at abstract levels whereas the premotor cortex codes actions at the concrete level only. Hence, occipitotemporal, but not premotor, regions fulfill the necessary criteria for action understanding. This result is compatible with cognitive theories but strongly undermines motor theories of action understanding.

Squeezing lemons in the bathroom: Contextual information modulates action recognition

Most every day actions take place in domestic rooms that are specific for certain classes of actions. Contextual information derived from domestic settings may therefore influence the efficiency of action recognition. The present studies investigated whether action recognition is modulated by compatibility of the context an action is embedded in. To this end, subjects watched video clips of actions performed in compatible, incompatible, and neutral contexts. Recognition was significantly slower when actions took place in an incompatible as compared to a compatible or a neutral context (Experiment 1). Functional MRI revealed increased activation for incompatible context in Brodmann Areas (BA) 44, 45, and 47 of the left ventrolateral prefrontal cortex (vlPFC; Experiment 2). Results suggest that contextual information - even when task-irrelevant - informs a high processing level of action analysis. In particular, the functional profiles assigned to these prefrontal regions suggest that contextual information activates associated action representations as a function of (in-)compatibility. Thus, incompatibility effects may reflect the attempt to resolve the conflict between action and context by embedding the presented action step into an overarching action that is again compatible with the provided context.

Objects tell us what action we can expect: Dissociating brain areas for retrieval and exploitation of action knowledge during action observation in fMRI

Objects are reminiscent of actions often performed with them: knife and apple remind us on peeling the apple or cutting it. Mnemonic representations of object-related actions (action codes) evoked by the sight of an object may constrain and hence facilitate recognition of unrolling actions. The present fMRI study investigated if and how action codes influence brain activation during action observation. The average number of action codes (NAC) of 51 sets of objects was rated by a group of n = 24 participants. In an fMRI study, different volunteers were asked to recognize actions performed with the same objects presented in short videos. To disentangle areas reflecting the storage of action codes from those exploiting them, we showed object-compatible and object-incompatible (pantomime) actions. Areas storing action codes were considered to positively co-vary with NAC in both object-compatible and object-incompatible action; due to its role in tool-related tasks, we here hypothesized left anterior inferior parietal cortex (aIPL). In contrast, areas exploiting action codes were expected to show this correlation only in object-compatible but not incompatible action, as only object-compatible actions match one of the active action codes. For this interaction, we hypothesized ventrolateral premotor cortex (PMv) to join aIPL due to its role in biasing competition in IPL. We found left anterior intraparietal sulcus (IPS) and left posterior middle temporal gyrus (pMTG) to co-vary with NAC. In addition to these areas, action codes increased activity in object-compatible action in bilateral PMv, right IPS, and lateral occipital cortex (LO). Findings suggest that during action observation, the brain derives possible actions from perceived objects, and uses this information to shape action recognition. In particular, the number of expectable actions quantifies the activity level at PMv, IPL, and pMTG, but only PMv reflects their biased competition while observed action unfolds.

Decoding Concrete and Abstract Action Representations During Explicit and Implicit Conceptual Processing

Action understanding requires a many-to-one mapping of perceived input onto abstract representations that generalize across concrete features. It is debated whether such abstract action concepts are encoded in ventral premotor cortex (PMv; motor hypothesis) or, alternatively, are represented in lateral occipitotemporal cortex (LOTC; cognitive hypothesis). We used fMRI-based multivoxel pattern analysis to decode observed actions at concrete and abstract, object-independent levels of representation. Participants observed videos of 2 actions involving 2 different objects, using either an explicit or implicit task with respect to conceptual action processing. We decoded concrete action representations by training and testing a classifier to discriminate between actions within each object category. To identify abstract action representations, we trained the classifier to discriminate actions in one object and tested the classifier on actions performed on the other object, and vice versa. Region-of-interest and searchlight analyses revealed decoding in LOTC at both concrete and abstract levels during both tasks, whereas decoding in PMv was restricted to the concrete level during the explicit task. In right inferior parietal cortex, decoding was significant for the abstract level during the explicit task. Our findings are incompatible with the motor hypothesis, but support the cognitive hypothesis of action understanding.

Decoding Internally and Externally Driven Movement Plans.

During movement planning, brain activity within parietofrontal networks encodes information about upcoming actions that can be driven either externally (e.g., by a sensory cue) or internally (i.e., by a choice/decision). Here we used multivariate pattern analysis (MVPA) of fMRI data to distinguish between areas that represent (1) abstract movement plans that generalize across the way in which these were driven, (2) internally driven movement plans, or (3) externally driven movement plans. In a delayed-movement paradigm, human volunteers were asked to plan and execute three types of nonvisually guided right-handed reaching movements toward a central target object: using a precision grip, a power grip, or touching the object without hand preshaping. On separate blocks of trials, movements were either instructed via color cues (Instructed condition), or chosen by the participant (Free-Choice condition). Using ROI-based and whole-brain searchlight-based MVPA, we found abstract representations of planned movements that generalize across the way these movements are selected (internally vs externally driven) in parietal cortex, dorsal premotor cortex, and primary motor cortex contralateral to the acting hand. In addition, we revealed representations specific for internally driven movement plans in contralateral ventral premotor cortex, dorsolateral prefrontal cortex, supramarginal gyrus, and in ipsilateral posterior parietotemporal regions, suggesting that these regions are recruited during movement selection. Finally, we observed representations of externally driven movement plans in bilateral supplementary motor cortex and a similar trend in presupplementary motor cortex, suggesting a role in stimulus–response mapping. SIGNIFICANCE STATEMENT The way the human brain prepares the body for action constitutes an essential part of our ability to interact with our environment. Previous studies demonstrated that patterns of neuronal activity encode upcoming movements. Here we used multivariate pattern analysis of human fMRI data to distinguish between brain regions containing movement plans for instructed (externally driven) movements, areas involved in movement selection (internally driven), and areas containing abstract movement plans that are invariant to the way these were generated (i.e., that generalize across externally and internally driven movement plans). Our findings extend our understanding of the neural basis of movement planning and have the potential to contribute to the development of brain-controlled neural prosthetic devices.

Do we mind other minds when we mind other minds' actions? A functional magnetic resonance imaging study

Action observation engages higher motor areas, possibly reflecting an internal simulation. However, actions considered odd or unusual were found to trigger additional activity in the so‐called theory of mind (ToM) network, pointing to deliberations on the actors mental states. In this functional magnetic resonance imaging study, the hypothesis was tested that an allocentric perspective on a normal action, and even more so the sight of the actors face, suffices to evoke ToM activity. Subjects observed short videos of object manipulation filmed from either the egocentric or the allocentric perspective, the latter including the actors face in half of the trials. On the basis of a regions of interest analysis using ToM coordinates, we found increased neural activity in several regions of the ToM network. First, perceiving actions from an allocentric compared with the egocentric perspective enhanced activity in the left temporoparietal junction (TPJ). Second, the presence of the actors face enhanced activation in the TPJ bilaterally, the medial prefrontal cortex (mPFC) and posterior cingulate cortex (PCC). Finally, the mPFC and PCC showed increased responses when the actor changed with respect to the preceding trial. These findings were further corroborated by zmap findings for the latter two contrasts. Together, findings indicate that observation of normal everyday actions can engage ToM areas and that an allocentric perspective, seeing the actors face and seeing a face switch, are effective triggers. Hum Brain Mapp, 2011.

Predicting goals in action episodes attenuates BOLD response in inferior frontal and occipitotemporal cortex

Actions are usually made of several action steps gearing towards an overarching goal. During observation of such action episodes the overarching action goal becomes more and more clear and upcoming action steps can be predicted with increasing precision. To tap this process, the present fMRI study investigated the dynamic changes of neural activity during the observation of distinct action steps that cohere by an overarching goal. Our hypotheses specifically addressed the role of the inferior frontal gyrus (IFG), a region assumed to be a key hub for integration functions during action processing, as well as the role of regions involved in action perception (often referred to as action observation network or AON) that should benefit from the predictability of forthcoming action steps. Participants watched separate action steps that formed a coherent action goal or not (factor goal coherence) and were performed by a single actor or not (factor actor coherence). Independent of actor coherence, neural activity in IFG and occipitotemporal cortex decreased as a function of goal predictability during the unfolding of goal-coherent episodes. In addition, we identified a network (precuneus, dorsolateral prefrontal and orbitofrontal cortex, angular gyrus, and middle temporal gyrus) that showed increased activity for goal coherence. We conclude that IFG fosters the integration of action steps to build overarching goals. Identifying the unifying goal of an action episode allows anticipation, and thus efficient processing, of forthcoming action steps. To this end, past action steps of the action episode are buffered and recollected with recourse to episodic memory.

The context-object-manipulation triad: Cross talk during action perception revealed by fmri

To recognize an action, an observer exploits information about the applied manipulation, the involved objects, and the context where the action occurs. Context, object, and manipulation information are hence expected to be tightly coupled in a triadic relationship (the COM triad hereafter). The current fMRI study investigated the hemodynamic signatures of reciprocal modulation in the COM triad. Participants watched short video clips of pantomime actions, that is, actions performed with inappropriate objects, taking place at compatible or incompatible contexts. The usage of pantomime actions enabled the disentanglement of the neural substrates of context–manipulation (CM) and context–object (CO) associations. There were trials in which (1) both manipulation and objects, (2) only manipulation, (3) only objects, or (4) neither manipulation nor objects were compatible with the context. CM compatibility effects were found in an action-related network comprising ventral premotor cortex, SMA, left anterior intraparietal sulcus, and bilateral occipito-temporal cortex. Conversely, CO compatibility effects were found bilaterally in lateral occipital complex. These effects interacted in subregions of the lateral occipital complex. An overlap of CM and CO effects was observed in the occipito-temporal cortex and the dorsal attention network, that is, superior frontal sulcus/dorsal premotor cortex and superior parietal lobe. Results indicate that contextual information is integrated into the analysis of actions. Manipulation and object information is linked by contextual associations as a function of co-occurrence in specific contexts. Activation of either CM or CO associations shifts attention to either action- or object-related relevant information.

Action Categories in Lateral Occipitotemporal Cortex Are Organized Along Sociality and Transitivity

How neural specificity for distinct conceptual knowledge categories arises is central for understanding the organization of semantic memory in the human brain. Although there is a large body of research on the neural processing of distinct object categories, the organization of action categories remains largely unknown. In particular, it is unknown whether different action categories follow a specific topographical organization on the cortical surface analogously to the category-specific organization of object knowledge. Here, we tested whether the neural representation of action knowledge is organized in terms of nonsocial versus social and object-unrelated versus object-related actions (sociality and transitivity, respectively, hereafter). We hypothesized a major distinction of sociality and transitivity along dorsal and ventral lateral occipitotemporal cortex (LOTC), respectively. Using fMRI-based multivoxel pattern analysis, we identified neural representations of action information associated with sociality and transitivity in bilateral LOTC. Representational similarity analysis revealed a dissociation between dorsal and ventral LOTC. We found that action representations in dorsal LOTC are segregated along features of sociality, whereas action representations in ventral LOTC are segregated along features of transitivity. In addition, representations of sociality and transitivity features were found more anteriorly in LOTC than representations of specific subtypes of actions, suggesting a posterior–anterior gradient from concrete to abstract action features. These findings elucidate how the neural representations of perceptually and conceptually diverse actions are organized in distinct subsystems in the LOTC. SIGNIFICANCE STATEMENT The lateral occipitotemporal cortex (LOTC) is critically involved in the recognition of objects and actions, but our knowledge about the underlying organizing principles is limited. Here, we discovered a dorsal–ventral distinction of actions in LOTC: dorsal LOTC represents actions based on sociality (how much an action is directed to another person) in proximity to person knowledge. In contrast, ventral LOTC represents actions based on transitivity (how much an action involves the interaction with inanimate objects) in proximity to tools/artifacts in ventral LOTC, suggesting a mutually dependent organization of actions and objects. In addition, we found a posterior-to-anterior organization of the LOTC for concrete and abstract representations, respectively. Our findings provide important insights about the organization of actions in LOTC.

Action Observers Implicitly Expect Actors to Act Goal-Coherently, Even if they Do Not: An fMRI Study

Actions observed in everyday life normally consist of one person performing sequences of goal‐directed actions. The present fMRI study tested the hypotheses that observers are influenced by the actors identity, even when this information is task‐irrelevant, and that this information shapes their expectation on subsequent actions of the same actor. Participants watched short video clips of action steps that either pertained to a common action with an overarching goal or not, and were performed by either one or by varying actors (2 × 2 design). Independent of goal coherence, actor coherence elicited activation in dorsolateral and ventromedial frontal cortex, together pointing to a spontaneous attempt to integrate all actions performed by one actor. Interestingly, watching an actor performing unrelated actions elicited additional activation in left inferior frontal gyrus, suggesting a search in semantic memory in an attempt to construct an overarching goal that can reconcile the disparate action steps with a coherent intention. Post‐experimental surveys indicate that these processes occur mostly unconsciously. Findings strongly suggest a spontaneous expectation bias toward actor‐related episodes in action observers, and hence to the immense impact of actor information on action observation. Hum Brain Mapp 35:2178–2190, 2014.