Anna M. Borghi

Putting words in perspective

In this article, we explore the nature of the conceptual knowledge retrieved when people use words to think about objects. If conceptual knowledge is used to simulate and guide action in the world, then how one can interact with an object should be reflected in the speed of retrieval and the content that is retrieved. This prediction was tested in three experiments in which a part verification procedure was used. Experiments 1 and 2 demonstrated that speed of part verification varied with the perspective imposed on the object by the language used to name the object (e.g., “You are driving a car” or “You are fueling a car”). In Experiment 3, parts were chosen so that actions directed toward them (on the real object) require movement upward (e.g., the roof of a car) or downward (e.g., the wheels of a car). Orthogonally, responding “yes” required an upward or a downward movement to a response button. Responding in a direction incompatible with the part’s location (e.g., responding downward to verify that a car has a roof) was slow relative to responding in a direction compatible with the part’s location. These results provide a strong link between concept knowledge and situated action.

Are visual stimuli sufficient to evoke motor information?: Studies with hand primes

In two experiments we assessed whether seeing objects automatically activates information regarding how to manipulate them. In Experiment 1 participants categorized photographs of objects that could be manipulated either with a power or a precision grip into artefacts or natural kinds. Target-objects were preceded by primes consisting of photographs of hands in grasping postures (precision or power grip). Experiment 2 involved a preliminary motor training phase in which each visual prime was associated with the actual motor action. In both experiments, natural kinds graspable with a power grip produced the fastest responses. In Experiment 2 we also found a congruency effect between the prime and the kind of grip required by the object (precision, power). Results suggest that visual stimuli automatically activate motor information. Specific motor programs are, however, activated only if motor training is performed before the categorization task. Implications of the results for the understanding of the organization of conceptual and motor information in the brain are discussed.

When objects are close to me: Affordances in the peripersonal space

In the present study, we investigated, using language, which motor information is automatically activated by observing 3-D objects (i.e., manipulation vs. function) and whether this information is modulated by the objects’ location in space. Participants were shown 3-D pictures of objects located in peripersonal versus extrapersonal space. Immediately after, they were presented with function, manipulation, or observation verbs (e.g., “to drink,” “to grasp,” “to look at”) and were required to judge whether the verb was compatible with the presented object. We found that participants were slower with observation verbs than with manipulation and function verbs. With both function and manipulation verbs, participants were faster when objects were presented in reachable space. Interestingly, the fastest response times were recorded when participants read function verbs while objects were presented in the accessible space. Results suggest that artifacts are first conceived in terms of affordances linked to manipulation and use, and that affordances are differently activated, depending on context.

Theories and computational models of affordance and mirror systems: An integrative review

Neuroscientific and psychological data suggest a close link between affordance and mirror systems in the brain. However, we still lack a full understanding of both the individual systems and their interactions. Here, we propose that the architecture and functioning of the two systems is best understood in terms of two challenges faced by complex organisms, namely: (a) the need to select among multiple affordances and possible actions dependent on context and high-level goals and (b) the exploitation of the advantages deriving from a hierarchical organisation of behaviour based on actions and action-goals. We first review and analyse the psychological and neuroscientific literature on the mechanisms and processes organisms use to deal with these challenges. We then analyse existing computational models thereof. Finally we present the design of a computational framework that integrates the reviewed knowledge. The framework can be used both as a theoretical guidance to interpret empirical data and design new experiments, and to design computational models addressing specific problems debated in the literature.

TRoPICALS: A Computational Embodied Neuroscience Model of Compatibility Effects

Perceiving objects activates the representation of their affordances. For example, experiments on compatibility effects showed that categorizing objects by producing certain handgrips (power or precision) is faster if the requested responses are compatible with the affordance elicited by the size of objects (e.g., small or large). The article presents a neural-network architecture that provides a general framework to account for compatibility effects. The model was designed with a methodological approach (computational embodied neuroscience) that aims to provide increasingly general accounts of brain and behavior (4 sources of constraints are used: neuroscientific data, behavioral data, embodied systems, reproduction of learning processes). The model is based on 4 principles of brain organization that we claim underlie most compatibility effects. First, visual perception and action are organized in the brain along a dorsal neural pathway encoding affordances and a ventral pathway encoding goals. Second, the prefrontal cortex within the ventral pathway gives a top-down bias to action selection by integrating information on stimuli, context, and goals. Third, reaction times depend on dynamic neural competitions for action selection that integrate bottom-up and top-down information. The congruence or incongruence between affordances and goals explains the different reaction times found in the experiments. Fourth, as words trigger internal simulations of their referents, they can cause compatibility effects as objects do. We validated the model by reproducing and explaining 3 types of compatibility effects and showed its heuristic power by producing 2 testable predictions. We also assessed the explicative power of the model by comparing it with related models and showed how it can be extended to account for other compatibility effects.

Simon-like and functional affordance effects with tools: The effects of object perceptual discrimination and object action state

In the present study two separate stimulus–response compatibility effects (functional affordance and Simon-like effects) were investigated with centrally presented pictures of an object tool (a torch) characterized by a structural separation between the graspable portion and the goal-directed portion. In Experiment 1, participants were required to decide whether the torch was red or blue, while in Experiment 2 they were required to decide whether the torch was upright or inverted. Our results showed that with the same stimulus two types of compatibility effect emerged: one based on the direction signalled by the goal-directed portion of the tool (a Simon-like effect as observed in Experiment 1), and the other based on the actions associated with an object (a functional affordance effect as observed in Experiment 2). Both effects emerged independently of the persons intention to act on the stimulus, but depended on the stimulus properties that were processed in order to perform the task.

Sentence processing: linking language to motor chains

A growing body of evidence in cognitive science and neuroscience points towards the existence of a deep interconnection between cognition, perception and action. According to this embodied perspective language is grounded in the sensorimotor system and language understanding is based on a mental simulation process (Jeannerod, 2007; Gallese, 2008; Barsalou, 2009). This means that during action words and sentence comprehension the same perception, action, and emotion mechanisms implied during interaction with objects are recruited. Among the neural underpinnings of this simulation process an important role is played by a sensorimotor matching system known as the mirror neuron system (Rizzolatti and Craighero, 2004). Despite a growing number of studies, the precise dynamics underlying the relation between language and action are not yet well understood. In fact, experimental studies are not always coherent as some report that language processing interferes with action execution while others find facilitation. In this work we present a detailed neural network model capable of reproducing experimentally observed influences of the processing of action-related sentences on the execution of motor sequences. The proposed model is based on three main points. The first is that the processing of action-related sentences causes the resonance of motor and mirror neurons encoding the corresponding actions. The second is that there exists a varying degree of crosstalk between neuronal populations depending on whether they encode the same motor act, the same effector or the same action-goal. The third is the fact that neuronal populations’ internal dynamics, which results from the combination of multiple processes taking place at different time scales, can facilitate or interfere with successive activations of the same or of partially overlapping pools.

One hand, two objects: emergence of affordance in contexts.

Studies on affordances typically focus on single objects. We investigated whether affordances are modulated by the context, defined by the relation between two objects and a hand. Participants were presented with pictures displaying two manipulable objects linked by a functional (knife-butter), a spatial (knife-coffee mug), or by no relation. They responded by pressing a key whether the objects were related or not. To determine if observing others actions and understanding their goals would facilitate judgments, a hand was: (a) displayed near the objects; (b) grasping an object to use it; (c) grasping an object to manipulate/move it; (d) no hand was displayed. RTs were faster when objects were functionally rather than spatially related. Manipulation postures were the slowest in the functional context and functional postures were inhibited in the spatial context, probably due to mismatch between the inferred goal and the context. The absence of this interaction with foot responses instead of hands in Experiment 2 suggests that effects are due to motor simulation rather than to associations between context and hand-postures.

Manipulating Objects and Telling Words: A Study on Concrete and Abstract Words Acquisition

Four experiments (E1–E2–E3–E4) investigated whether different acquisition modalities lead to the emergence of differences typically found between concrete and abstract words, as argued by the words as tools (WAT) proposal. To mimic the acquisition of concrete and abstract concepts, participants either manipulated novel objects or observed groups of objects interacting in novel ways (Training 1). In TEST 1 participants decided whether two elements belonged to the same category. Later they read the category labels (Training 2); labels could be accompanied by an explanation of their meaning. Then participants observed previously seen exemplars and other elements, and were asked which of them could be named with a given label (TEST 2). Across the experiments, it was more difficult to form abstract than concrete categories (TEST 1); even when adding labels, abstract words remained more difficult than concrete words (TEST 2). TEST 3 differed across the experiments. In E1 participants performed a feature production task. Crucially, the associations produced with the novel words reflected the pattern evoked by existing concrete and abstract words, as the first evoked more perceptual properties. In E2–E3–E4, TEST 3 consisted of a color verification task with manual/verbal (keyboard–microphone) responses. Results showed the microphone use to have an advantage over keyboard use for abstract words, especially in the explanation condition. This supports WAT: due to their acquisition modality, concrete words evoke more manual information; abstract words elicit more verbal information. This advantage was not present when linguistic information contrasted with perceptual one. Implications for theories and computational models of language grounding are discussed.

Words as social tools : an embodied view on abstract concepts

Chapter 1. The problem of definition.- Chapter 2. The WAT proposal and the role of language.- Chapter 3. Embodied and hybrid theories of abstract concepts and words.- Chapter 4 Word learning and word acquisition.- Chapter 5. What can neuroscience tell us about abstract concepts.- Chapter 6. Language, languages, and abstract concepts.- Afterword.