Claudia Gianelli

Acting in perspective: the role of body and language as social tools

We investigated how the reach-to-grasp movement is influenced by the presence of another person (friend or non-friend), who was either invisible (behind) or located in different positions with respect to an object and to the agent, and by the perspective conveyed by linguistic pronouns (“I”, “You”). The interaction between social relationship and relative position influenced the latency of both maximal fingers aperture and velocity peak, showing shorter latencies in the presence of a non-friend than in the presence of a friend. However, whereas the relative position of a non-friend did not affect the kinematics of the movement, the position of a friend mattered: latencies were significantly shorter with friends only in positions allowing them to easily reach for the object. Finally, the investigation of the overall reaching movement time showed an interaction between the speaker and the pronoun: participants reached the object more quickly when the other spoke, particularly if she used the “I” pronoun. This suggests that speaking, and particularly using the “I” pronoun, evokes a potential action. Implications of the results for embodied cognition are discussed.

The Agent is Right: When Motor Embodied Cognition is Space-Dependent

The role of embodied mechanisms in processing sentences endowed with a first person perspective is now widely accepted. However, whether embodied sentence processing within a third person perspective would also have motor behavioral significance remains unknown. Here, we developed a novel version of the Action-sentence Compatibility Effect (ACE) in which participants were asked to perform a movement compatible or not with the direction embedded in a sentence having a first person (Experiment 1: You gave a pizza to Louis) or third person perspective (Experiment 2: Lea gave a pizza to Louis). Results indicate that shifting perspective from first to third person was sufficient to prevent motor embodied mechanisms, abolishing the ACE. Critically, ACE was restored in Experiment 3 by adding a virtual “body” that allowed participants to know “where” to put themselves in space when taking the third person perspective, thus demonstrating that motor embodied processes are space-dependent. A fourth, control experiment, by dissociating motor response from the transfer verbs direction, supported the conclusion that perspective-taking may induce significant ACE only when coupled with the adequate sentence-response mapping.

Sentence comprehension: effectors and goals, self and others. An overview of experiments and implications for robotics

According to theories referring to embodied and grounded cognition (Barsalou, 2008), language comprehension encompasses an embodied simulation of actions. The neural underpinnings of this simulation could be found in wide neural circuits that involve canonical and mirror neurons (Rizzolatti et al., 1996). In keeping with this view, we review behavioral and kinematic studies conducted in our lab which help characterize the relationship existing between language and the motor system. Overall, our results reveal that the simulation evoked during sentence comprehension is fine-grained, primarily in its sensitivity to the different effectors we employ to perform actions. In addition, they suggest that linguistic comprehension also relies on the representation of actions in terms of goals and of the chains of motor acts necessary to accomplish them. Finally, they indicate that these goals are modulated by both the object features the sentence refers to as well as by social aspects such as the characteristics of the agents implied by sentences. We will discuss the implications of these studies for embodied robotics.

Self, Others, Objects: How This Triadic Interaction Modulates Our Behavior

Two experiments investigated whether the triadic interaction between objects, ourselves and other persons modulates motor system activation during language comprehension. Participants were faced with sentences formed by a descriptive part referring to a positive or negative emotively connoted object and an action part composed of an imperative verb implying a motion toward the self or toward other persons (e.g., “The object is attractive/ugly. Bring it toward you/Give it to another person/Give it to a friend”). Participants judged whether each sentence was sensible or not by moving the mouse toward or away from their body. Findings showed that the simulation of a social context influenced both (1) the motor system and (2) the coding of stimulus valence. Implications of the results for theories of embodied and social cognition are discussed.

Does listening to action-related sentences modulate the activity of the motor system? Replication of a combined TMS and behavioral study.

The neurophysiological and behavioral correlates of action-related language processing have been debated for long time. A precursor in this field was the study by Buccino et al. (2005) combining transcranial magnetic stimulation (TMS) and behavioral measures (reaction times, RTs) to study the effect of listening to hand- and foot-related sentences. In the TMS experiment, the authors showed a decrease of motor evoked potentials (MEPs) recorded from hand muscles when processing hand-related verbs as compared to foot-related verbs. Similarly, MEPs recorded from leg muscles decreased when participants processed foot-related as compared to hand-related verbs. In the behavioral experiment, using the same stimuli and a semantic decision task the authors found slower RTs when the participants used the body effector (hand or foot) involved in the actual execution of the action expressed by the presented verb to give their motor responses. These findings were interpreted as an interference effect due to a simultaneous involvement of the motor system in both a language and a motor task. Our replication aimed to enlarge the sample size and replicate the findings with higher statistical power. The TMS experiment showed a significant modulation of hand MEPs, but in the sense of a motor facilitation when processing hand-related verbs. On the contrary, the behavioral experiment did not show significant results. The results are discussed within the general debate on the time-course of the modulation of motor cortex during implicit and explicit language processing and in relation to the studies on action observation/understanding.

Influence of numerical magnitudes on the free choice of an object position.

The link between numerical magnitude and mechanisms of spatial orienting has been underlined in an increasing number of studies. Similarly, the relationship between numerical magnitude and grasping actions has started to be investigated. The present study focuses on the influence of numerical magnitude processing in the free choice of the position of an object. Participants were presented with a digit (1–9 without 5) and were required to decide whether it was smaller or larger than 5. Then, they had to grasp a small cube and change its position before vocally responding “higher” or “lower”. Results showed that in the initial phase of the grasp movement, the grip aperture was modulated by the numerical magnitude. Moreover, participants shifted the position of the cube more leftward with smaller digits compared with larger ones, and they tended to position the object closer to themselves with smaller digits compared with larger ones. These results extend the previous findings indicating that the processing of magnitude is tightly related to the mechanisms of spatial orienting that subserve action execution.

Learning to Manipulate and Categorize in Human and Artificial Agents

This study investigates the acquisition of integrated object manipulation and categorization abilities through a series of experiments in which human adults and artificial agents were asked to learn to manipulate two-dimensional objects that varied in shape, color, weight, and color intensity. The analysis of the obtained results and the comparison of the behavior displayed by human and artificial agents allowed us to identify the key role played by features affecting the agent/environment interaction, the relation between category and action development, and the role of cognitive biases originating from previous knowledge.

The Impact of Social Context and Language Comprehension on Behaviour: A Kinematic Investigation

We investigated whether and how comprehending sentences that describe a social context influences our motor behaviour. Our stimuli were sentences that referred to objects having different connotations (e.g., attractive/ugly vs. smooth/prickly) and that could be directed towards the self or towards “another person” target (e.g., “The object is ugly/smooth. Bring it to you/Give it to another person”). Participants judged whether each sentence was sensible or non-sensible by moving the mouse towards or away from their body. Mouse movements were analysed according to behavioral and kinematics parameters. In order to enhance the social meaning of the linguistic stimuli, participants performed the task either individually (Individual condition) or in a social setting, in co-presence with the experimenter. The experimenter could either act as a mere observer (Social condition) or as a confederate, interacting with participants in an off-line modality at the end of task execution (Joint condition). Results indicated that the different roles taken by the experimenter affected motor behaviour and are discussed within an embodied approach to language processing and joint actions.

Grasping the Agent’s Perspective: A Kinematics Investigation of Linguistic Perspective in Italian and German

Every day, we primarily experience actions as agents, by having a concrete perspective on our actions, their means and goals. This peculiar perspective is what allows us to successfully plan and execute our actions in a dense social environment. Nevertheless, in this environment actions are also perceived from an observer’s perspective. Adopting such a perspective helps us to understand and respond to other’s people actions and their outcomes. Importantly, similar experiences of being agent and observer occur also when actions are not physically acted/perceived but are merely linguistically shared. In this paper we present two exploratory studies, one in Italian and one in German, in which we applied a direct comparison of three singular perspectives in combination with different verb categories. First, second and third person pronouns were combined with action and interaction verbs, i.e., verbs implying an interaction with an object – e.g., grasp – or an interaction with an object and another person – e.g., give. By means of kinematics recording, we analyzed participants’ reaching-grasping responses to a mouse while they were presented with the different combinations of linguistic stimuli (pronouns and verb type). Results of Experiment 1 on reaching show that, when they are preceded by YOU, interaction verbs reached the velocity peak earlier than action verbs, since a further motor act will follow. Thus pronouns influence perspective taking and while comprehending language we are sensitive to the motor chain organization of verbs. The absence of the same effects in Experiment 2 is likely due to the fact that, being the pronoun in German mandatory, it is perceived as less salient than in Italian. Overall our result supports the idea that language is grounded in the motor system in a flexible way, and highlights the need for cross-linguistic studies in the field of embodied language processing.

Embodied language and perspective taking in light of movement disorders

Increasing neuroimaging and physiological findings have shown how the activity of the listeners motor system is shaped by verbally described actions in the absence of any performed or imagined action (Boulenger, Hauk, & Pulvermüller, 2009; Buccino et al., 2005; Hauk, Davis, Ford, Pulvermüller, & Marslen-Wilson, 2006; Hauk, Johnsrude, & Pulvermüller, 2004; Hauk, Shtyrov, & Pulvermüller, 2008; Pulvermüller, Shtyrov, & Ilmoniemi, 2005; Tettamanti et al., 2005). This rapid recruitment of the motor system during language processing has been referred to as “motor resonance” (for a review see Pulvermüller & Fadiga, 2010).