Fey Parrill

Viewpoint in speech–gesture integration: Linguistic structure, discourse structure, and event structure

We examine a corpus of narrative data to determine which types of events evoke character viewpoint gestures, and which evoke observer viewpoint gestures. We consider early claims made by McNeill (1992) that character viewpoint tends to occur with transitive utterances and utterances that are causally central to the narrative. We argue that the structure of the event itself must be taken into account: there are some events that cannot plausibly evoke both types of gesture. We show that linguistic structure (transitivity), event structure (visuo-spatial and motoric properties), and discourse structure all play a role. We apply these findings to a recent model of embodied language production, the Gestures as Simulated Action framework.

Subjects in the hands of speakers: An experimental study of syntactic subject and speech-gesture integration

Abstract Work by Russell Tomlin has shown that there is a close relationship between the syntactic subject of an utterance and the entity the speakers attention is focused on while the utterance is being formulated, for descriptions of a simple event (Tomlin 1985, 1995, 1997). The experiment presented in this paper demonstrates that the same effect can be obtained for a more complex event, and that attention also impacts the spontaneous hand gestures produced along with speech. The paper shows that both syntactic subject and the information contained in gesture can be manipulated by changing which entity a speaker is focused on during utterance formulation. This pattern suggests that changes in conceptualization give rise to changes in both speech and gesture.

Linguistic, gestural, and cinematographic viewpoint : An analysis of ASL and English narrative

Abstract Multimodal narrative can help us understand how conceptualizers schematize information when they create mental representations of films and may shed light on why some cinematic conventions are easier or harder for viewers to integrate. This study compares descriptions of a shot/reverse shot sequence (a sequence of camera shots from the viewpoints of different characters) across users of English and American Sign Language (ASL). We ask which gestural and linguistic resources participants use to narrate this event. Speakers and signers tended to represent the same characters via the same point of view and to show a single perspective rather than combining multiple perspectives simultaneously. Neither group explicitly mentioned the shift in cinematographic perspective. We argue that encoding multiple points of view might be a more accurate visual description, but is avoided because it does not create a better narrative.