Morana Alač

I See What You Are Saying: Action as Cognition in fMRI Brain Mapping Practice

In cognitive neuroscience, functional magnetic resonance imaging (fMRI) is used to produce images of brain functions. These images play a central role in the practice of neuroscience. In this paper we are interested in how these brain images become understandable and meaningful for scientists. In order to explore this problem we observe how scientists use such semiotic resources as gesture, language, and material structure present in the socially and culturally constituted environment. A micro-analysis of video records of scientists interacting with each other and with fMRI images reveals action as cognition , that is, actions that constitute thinking for the scientists.

Modulation of bold response in motion-sensitive lateral temporal cortex by real and fictive motion sentences

Can linguistic semantics affect neural processing in feature-specific visual regions? Specifically, when we hear a sentence describing a situation that includes motion, do we engage neural processes that are part of the visual perception of motion? How about if a motion verb was used figuratively, not literally? We used fMRI to investigate whether semantic content can “penetrate” and modulate neural populations that are selective to specific visual properties during natural language comprehension. Participants were presented audiovisually with three kinds of sentences: motion sentences (“The wild horse crossed the barren field.”), static sentences, (“The black horse stood in the barren field.”), and fictive motion sentences (“The hiking trail crossed the barren field.”). Motion-sensitive visual areas (MT+) were localized individually in each participant as well as face-selective visual regions (fusiform face area; FFA). MT+ was activated significantly more for motion sentences than the other sentence types. Fictive motion sentences also activated MT+ more than the static sentences. Importantly, no modulation of neural responses was found in FFA. Our findings suggest that the neural substrates of linguistic semantics include early visual areas specifically related to the represented semantics and that figurative uses of motion verbs also engage these neural systems, but to a lesser extent. These data are consistent with a view of language comprehension as an embodied process, with neural substrates as far reaching as early sensory brain areas that are specifically related to the represented semantics.

Working with Brain Scans : Digital Images and Gestural Interaction in fMRI Laboratory

A significant part of functional magnetic resonance imaging (fMRI) practice in neuroscience is spent in front of computer screens. To investigate the brain neuroscientists work with digital images. This paper recovers practical dealings with brain scans in fMRI laboratories to focus on the achievement of seeing in the digital realm. While looking at brain images, neuroscientists gesture and manipulate digital displays to manage and make sense of their experimental data. Their gestural engagements are seen as dynamical phenomenal objects enacted at the junction between the digital world of technology and the world of embodied action.

When a robot is social: Spatial arrangements and multimodal semiotic engagement in the practice of social robotics

Social roboticists design their robots to function as social agents in interaction with humans and other robots. Although we do not deny that the robot’s design features are crucial for attaining this aim, we point to the relevance of spatial organization and coordination between the robot and the humans who interact with it. We recover these interactions through an observational study of a social robotics laboratory and examine them by applying a multimodal interactional analysis to two moments of robotics practice. We describe the vital role of roboticists and of the group of preverbal infants, who are involved in a robot’s design activity, and we argue that the robot’s social character is intrinsically related to the subtleties of human interactional moves in laboratories of social robotics. This human involvement in the robot’s social agency is not simply controlled by individual will. Instead, the human–machine couplings are demanded by the situational dynamics in which the robot is lodged.

Moving Android: On Social Robots and Body-in-Interaction

Social robotics studies embodied technologies designed for social interaction. This paper examines the implied idea of embodiment using as data a sequence in which practitioners of social robotics are involved in designing a robots movement. The moments of learning and work in the laboratory enact the social body as material, dynamic, and multiparty: the body-in-interaction. In describing subject—object reconfigurations, the paper explores how the well-known ideas of extending the body with instruments can be applied to a technology designed to function as our surrogate.

Negotiating pictures of numbers

This paper reports on objectivity and knowledge production in the process of submitting, revising, and publishing an experimental research article in cognitive neuroscience. The review process, as part of scientific practice, is of particular interest, since it puts the research team in direct dialog with a larger scientific community concerned with fMRI evidence. By bringing this often ‘black‐boxed’ dimension of the manuscript’s production into the picture, I illustrate the role that the visual brain representations played in the practice of scientific fact production. This allows me to point out how the knowledge is constructed through seeing, and how the showing of the fact through images bears traits of the scientific culture of which it is part.

From trash to treasure: Learning about brain images through multimodality

Abstract Cognitive Neuroscientists use functional magnetic resonance imaging (fMRI) technology to generate digital images of the human brain. An fMRI image, as a final product of the scientific work, does not document movements and sounds that were present when such an image was recorded. Yet, a focus on actual moments of scientific practice reveals that such forgotten elements of practice can play important roles in understanding and knowledge acquisition. The multimodal interaction among scientists and digital screens shows how movements of the experimental subject and the scanner noise are performed to make images meaningful. Moreover, it suggests that the phenomena whose detection is crucial for a scientific reading of the brain images, such as motion artifacts, become visible as a result of coordination of various semiotic modalities (i.e., images, talk, body movements, gesture, etc.).

Design and early evaluation of the RUBI-5 sociable robots

The RUBI project [2,3,4,5,6,7,8,9] started in 2004 with the goal of designing sociable robots for early childhood education. The project relies on an immersive, iterative approach to robot development and scientific progress. From the beginning of the project scientists and engineers immersed themselves at the Early Childhood Education Center at UC San Diego, designed robot prototypes to interact and teach toddlers in close collaboration with teachers, toddlers, and parents. As part of this process we learned many lessons on how to develop hardware and software components, that are safe, reliable, and that can withstand the rigors of daily interactions with toddlers. Here we present our last robot platforms: RUBI-5. The robot components are off-the-shelf and the 3D-CAD files are being made available for free to the research community to ease replication and adaptation by other groups. One of the goals of the new platform is to greatly accelerate the design, data-gathering, and statistical analysis of early education experiments.

Grounding a Sociable Robot’s Movements in Multimodal, Situational Engagements

To deal with the question of what a sociable robot is, we describe how an educational robot is encountered by children, teachers and designers in a preschool. We consider the importance of the robot’s body by focusing on how its movements are contingently embedded in interactional situations. We point out that the effects of agency that these movements generate are inseparable from their grounding in locally coordinated, multimodal actions and interactions.

We like to talk about smell: A worldly take on language, sensory experience, and the Internet

Abstract Western languages have been marked by their lack of specialized vocabulary to express odor qualities, and thus it is stated that it is difficult – if not impossible – to talk about smell. To engage the issue of olfactory ineffability, this paper turns to actual instances of textual renderings of smell by paying attention to how the olfactory language of scent enthusiasts is rendered on the Internet. The methods that enthusiasts’ texts inscribe do not rely on specialized vocabulary but constitute a language that is turned toward the world. To articulate this character of olfactory language, the paper illustrates four discursive procedures employed on the Internet: embedding knowledges that are contoured and made available through their uses by the community; borrowing professional vocabulary and adapting it through humor; providing “recipes” for practical actions; and emphasizing the subjective character of scent to provide openings for conversation. The paper traces how olfactory talk becomes available as it is practiced, challenging the opposition between sensory experiences and the semiotic realm that is inscribed in the idea of olfactory ineffability.