H.T. van Schie

Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge

Our capacity to use tools and objects is often considered one of the hallmarks of the human species. Many objects greatly extend our bodily capabilities to act in the physical world, such as when using a hammer or a saw. In addition, humans have the remarkable capability to use objects in a flexible fashion and to combine multiple objects in complex actions. We prepare coffee, cook dinner and drive our car. In this review we propose that humans have developed declarative and procedural knowledge, i.e. action semantics that enables us to use objects in a meaningful way. A state-of-the-art review of research on object use is provided, involving behavioral, developmental, neuropsychological and neuroimaging studies. We show that research in each of these domains is characterized by similar discussions regarding (1) the role of object affordances, (2) the relation between goals and means in object use and (3) the functional and neural organization of action semantics. We propose a novel conceptual framework of action semantics to address these issues and to integrate the previous findings. We argue that action semantics entails both multimodal object representations and modality-specific sub-systems, involving manipulation knowledge, functional knowledge and representations of the sensory and proprioceptive consequences of object use. Furthermore, we argue that action semantics are hierarchically organized and selectively activated and used depending on the action intention of the actor and the current task context. Our framework presents an integrative account of multiple findings and perspectives on object use that may guide future studies in this interdisciplinary domain.

The role of immediate and final goals in action planning: An fMRI study

To interact effectively with our environment, we need to specify the intended outcomes (goals) of our actions. In this process, immediate goals and final goals can be regarded as different levels within a hierarchically organized system for action planning: immediate goals and movement details are selected to accomplish more remote goals. Behavioral studies support this notion of different levels of action planning, but the neurophysiological basis remains unclear. Using fMRI, we examined the neural correlates of preparing object manipulations based on either the desired end-state (the final goal) or the initial movement towards a target (the immediate goal). Subjects had to insert an object (consisting of a large and a small cube) into one of two corresponding large and small slots. The subjects were cued on either which slot to fill (Final Goal trials) or which object part to grasp (Immediate Goal trials). These actions required similar movements, but different planning. During Final Goal trials, there was differential preparatory activity along the superior frontal gyrus (bilaterally) and in left inferior parietal cortex. Immediate Goal trials evoked differential activity in occipito-parietal and occipito-temporal cortex. These findings support the notion that actions can be planned at different levels. We show that different fronto-parietal circuits plan the same action, by a relative emphasis on either selecting a sequence of movements to achieve a desired end-state, or selecting movements spatially compatible with given object properties.

Semantics in action: An electrophysiological study on the use of semantic knowledge for action

Recent studies have supported close interactions between language and action-related processes, suggesting comparable neural mechanisms. However, relatively little is known about the semantics involved in action planning. The present study investigated the activation of semantic knowledge in meaningful actions by recording event-related potentials (ERPs). Subjects prepared meaningful or meaningless actions with objects and made a semantic categorization response before executing the action. Words presented could be either congruent or incongruent with respect to the goal of the action. Preparation of meaningful actions elicited a larger anterior N400 for words incongruent to the present action goal as compared to congruent words, while no N400 effect was found when subjects prepared meaningless actions. These findings indicate that the preparation of meaningful actions with objects is accompanied by the activation of semantic information representing the usual action goals associated with those objects.

Conceptual knowledge for understanding other’s actions is organized primarily around action goals

Semantic knowledge about objects entails both knowing how to grasp an object (grip-related knowledge) and what to do with an object (goal-related knowledge). Considerable evidence suggests a hierarchical organization in which specific hand-grips in action execution are most often selected to accomplish a remote action goal. The present study aimed to investigate whether a comparable hierarchical organization of semantic knowledge applies to the recognition of other’s object-directed actions as well. Correctness of either the Grip (hand grip applied to the object) or the Goal (end-location at which an object was directed) were manipulated independently in two experiments. In Experiment 1, subjects were required to attend selectively to either the correctness of the grip or the goal of the observed action. Subjects were faster when attending to the goal of the action and a strong interference of goal-violations was observed when subjects attended to the grip of the action. Importantly, observation of irrelevant goal- or grip-related violations interfered with making decisions about the correctness of the relevant dimension only when the relevant dimension was correct. In contrast, in Experiment 2, when subjects attended to an action-irrelevant stimulus dimension (i.e. orientation of the object), no interference of goal- or grip-related violations was found, ruling out the possibility that interference-effects result from perceptual differences between stimuli. These findings suggest that understanding the correctness of an action selectively recruits specialized, but interacting networks, processing the correctness of goal- and grip-specific information during action observation.

Neural and Temporal Dynamics Underlying Visual Selection for Action

The present study investigated the selection for action hypothesis, according to which a subjects action intention to perform a movement influences the way in which visual information is being processed. Subjects were instructed in separate blocks either to grasp or to point to a three-dimensional target-object and event-related potentials were recorded relative to stimulus onset. It was found that grasping compared with pointing resulted in a stronger N1 component and a subsequent selection negativity, which were localized to the lateral occipital complex. These effects suggest that the intention to grasp influences the processing of action-relevant features in ventral stream areas already at an early stage (e.g., enhanced processing of object orientation for grasping). These findings provide new insight in the neural and temporal dynamics underlying perception-action coupling and provide neural evidence for a selection for action principle in early human visual processing.

The N400-concreteness effect reflects the retrieval of semantic information during the preparation of meaningful actions

The present ERP study investigated when and how action semantics, i.e. the conceptual knowledge that we acquired over the course of our lives, is activated during the preparation of object-directed actions. Subjects were required to grasp one of two objects and were implicitly instructed to either perform a meaningful action (e.g. moving a cup towards the mouth) or a meaningless action with the object (e.g. moving a cup towards the eye). A larger anterior N400 was found for the preparation of meaningful compared to meaningless actions, likely reflecting the retrieval of action semantic information in case a meaningful action was required with the object. The distribution and the latency of the anterior N400-effect were strongly related to standard N400-repetition effects, thereby further corroborating the semantic nature of the effect. In sum, the present study provides new insight in the neural and temporal dynamics underlying semantics for action.

Neural evidence for impaired action selection in right hemiparetic cerebral palsy

Recent studies suggest that in addition to low-level motor impairments, individuals with hemiparetic cerebral palsy (HCP) are characterized by anticipatory action planning deficits as well. In the present EEG study we investigated the neural and temporal dynamics of action planning in participants with right-sided HCP (n=10) and in left-handed control subjects (n=10). An anticipatory planning task was used in which participants were required to grasp and rotate a hexagonal knob over different angles (60 degrees, 120 degrees or 180 degrees). At a behavioral level, participants with HCP were slower in their movements and often selected an inappropriate grip when grasping the object. At a neural level, individuals with HCP showed a strong reduction in the amplitude of the P2 component, likely reflecting an impaired process of action selection. In addition, a strong correlation was observed between the P2 amplitude and grasping and rotation times. The P2 component was localized to sources in the dorsal posterior cingulate cortex (dPCC), an area that is known to be involved in orienting visual body parts in space. Together these findings suggest that anticipatory planning deficits in cerebral palsy arise mainly due to an impaired process of action selection.

Pouring or chilling a bottle of wine: an fMRI study on the prospective planning of object-directed actions

This fMRI study investigates the neural mechanisms supporting the retrieval of action semantics. A novel motor imagery task was used in which participants were required to imagine planning actions with a familiar object (e.g. a toothbrush) or with an unfamiliar object (e.g. a pair of pliers) based on either goal-related information (i.e. where to move the object) or grip-related information (i.e. how to grasp the object). Planning actions with unfamiliar compared to familiar objects was slower and was associated with increased activation in the bilateral superior parietal lobe, the right inferior parietal lobe and the right insula. The stronger activation in parietal areas for unfamiliar objects fits well with the idea that parietal areas are involved in motor imagery and suggests that this process takes more effort in the case of novel or unfamiliar actions. In contrast, the planning of familiar actions resulted in increased activation in the anterior prefrontal cortex, suggesting that subjects maintained a stronger goal-representation when planning actions with familiar compared to unfamiliar objects. These findings provide further insight into the neural structures that support action semantic knowledge for the functional use of real-world objects and suggest that action semantic knowledge is activated most readily when actions are planned in a goal-directed manner.

Imitation of hand and tool actions is effector-independent

Following the theoretical notion that tools often extend one’s body, in the present study, we investigated whether imitation of hand or tool actions is modulated by effector-specific information. Subjects performed grasping actions toward an object with either a handheld tool or their right hand. Actions were initiated in response to pictures representing a grip at an object that could be congruent or incongruent with the required action (grip-type congruency). Importantly, actions could be cued by means of a tool cue, a hand cue, and a symbolic cue (effector-type congruency). For both hand and tool actions, an action congruency effect was observed, reflected in faster reaction times if the observed grip type was congruent with the required movement. However, neither hand actions nor tool actions were differentially affected by the effector represented in the picture (i.e., when performing a tool action, the action congruency effect was similar for tool cues and hand cues). This finding suggests that imitation of hand and tool actions is effector-independent and thereby supports generalist rather than specialist theories of imitation.

No two are the same: Body shape is part of identifying others

Abstract Downing and Peelen argue for a clear distinction between body and identity representation, with the former performed by EBA and FBA, and the latter performed elsewhere in the brain. Under a predictive coding account, we argue that this separation is unnecessary: Representing bodies is part of representing identity. While neurons in EBA and FBA may only code for body shape and posture, we propose that they are a part of a reciprocally connected cortical network that functions to minimize prediction error when making identity inferences. We propose a novel way to test the hypothesis that EBA and FBA are critically involved in person identification.