Lisa Gustavsson

Multimodal Language Acquisition Based on Motor Learning and Interaction

This work presents a developmental and ecological approach to language acquisition in robots, which has its roots in the interaction between infants and their caregivers. We show that the signal directed to infants by their caregivers include several hints that can facilitate the language acquisition and reduce the need for preprogrammed linguistic structure. Moreover, infants also produce sounds, which enables for richer types of interactions such as imitation games, and for the use of motor learning. By using a humanoid robot with embodied models of the infant’s ears, eyes, vocal tract, and memory functions, we can mimic the adult-infant interaction and take advantage of the inherent structure in the signal. Two experiments are shown, where the robot learn a number of word-object associations and the articulatory target positions for a number of vowels.

Multimodal word learning from Infant Directed Speech

When adults talk to infants they do that in a different way compared to how they communicate with other adults. This kind of infant directed speech (IDS) typically highlights target words using focal stress and utterance final position. Also, speech directed to infants often refers to objects, people and events in the world surrounding the infant. Because of this, the sound sequences the infant hears are very likely to co-occur with actual objects or events in the infants visual field. In this work we present a model that is able to learn word-like structures from multimodal information sources without any pre-programmed linguistic knowlege, by taking advantage of the characteristics of IDS. The model is implemented on a humanoid robot platform and is able to extract word-like patterns and associating these to objects in the visual surrounding.

Assessing language acquisition from parent-child interaction: An event-related potential study on perception of audio-visual cues in infancy

This paper promotes a theory-driven model development of parent-child interaction. In our project, we identify, test, and simulate some of the fundamental components of speech, gestures, and social-emotional behaviors and the consequences they might have on child language development. Our theoretical position is part of the connectionist tradition; language acquisition is described to be an emergent consequence of the interplay between the infant and the ambient linguistic environment, including sensory information of all modalities. It is well known that speech comprehension and production are significantly influenced by the presence of co-speech gestures. These gestures may be articulatory in nature or hand/beat co-gestures that keep the rhythm of speech. However, since the extent of this integrated relationship is difficult to determine from behavioral research solely, studies addressing neural mechanisms that underlie cognitive processes and behaviors are of importance. This paper reports an electroen...

Assessing language acquisition from parent-child interaction: An event-related potential study on perception of intonation contours in infancy

The aim of this paper is to present our multidisciplinary project to study parent-child interaction. The goal of the project is to identify, test, and simulate components of child and adult speech and gestures and the consequences they might have on child language acquisition. Since typical parent-child interaction is built upon both interlocutors’ intention-reading, responsiveness to joint-attention, and imitation of speech/gestures, we make video recordings along with recordings of speech data to grasp the integration of semantic and pragmatic aspects of language acquisition. The understanding of parent-child interaction benefits further from information on brain activation involved in speech processing. As a first step to achieve the project goals, an electroencephalography/event-related potential (EEG/ERP) study exploring children’s early perception of intonation contours involved in human interactions was performed. This paper discusses the characteristics of integration of multimodal social-emotiona...

Exploring and learning the consequences of vocalizations in early infancy

The ability to discover and learn about the consequences of own vocalizations is currently being studied in three groups of infants, 3–5‐, 6–8‐, and 9–11‐month olds, exploring a situation that resembles an audiovisual memory game. The infants are tested individually, in a quiet and dimmed studio. They sit in front of a black computer monitor connected to an eye‐tracking system. At randomly selected intervals, a circle appears in the center of the screen and expands and shrinks rapidly to a bright circle to signal that an observation trial is in progress. If the infant vocalizes during the trial, an animated actor emerges after about 200‐ms delay at screen coordinates that are contingent on spectral characteristics of the initial part of the infant’s vocalization. The actors brightness and time on display is controlled by the vocalizations overall intensity. By analyzing the changes in predictive looking behavior that probably occur during the session, we expect to gather valuable information on the infant...