Heather Shovelton

Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech.

This study tested McNeill’s theory that the iconic gestures that accompany speech in everyday talk convey critical information in interpersonal communication. Using a structured interview to measure the amount of information respondents receive from clause-length clips depicting aspects of a cartoon story, we discovered that when respondents could see the iconic gestures as well as hear the speech they received significantly more accurate information about those aspects of the original story depicted in the clips than when they just heard the speech. We also discovered that it was only with respect to certain semantic categories, namely, the relative position and the size of objects, that the beneficial effect of gestural communication was significant. We then considered in detail how specific attributes of actions and of objects are communicated via the iconic representation within individual gestures. Lastly, we discuss the implications of these findings for models of human communication in this area.

An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.

It has been hypothesized that the iconic hand gestures that accompany talk communicate important semantic information. This research tests whether some gestures, in the absence of speech, are more communicative than others and considers what properties of gestures might affect their communicative power. Our research found that the communicative power of gestures does vary greatly, and that this is significantly affected by the viewpoint from which a gesture is generated, with character viewpoint gestures being more communicative than observer viewpoint gestures. It has also been suggested that gesture viewpoint is connected with the transitivity of the clause that it accompanies, and it was found in our study that respondents appeared to obtain syntactic information about the associated clause from the gesture. This conclusion was based on the observation that when respondents attempted to report what information was contained in gestures, viewed in the absence of speech, there was a significantly higher proportion of transitive structures in their answers after they had watched character viewpoint gestures compared with observer viewpoint gestures. Communication about the syntax of the accompanying clause might thus be a critical, but thus far neglected, aspect of the role of gestures in everyday talk.

What properties of talk are associated with the generation of spontaneous iconic hand gestures

When people talk, they frequently make movements of their arms and hands, some of which appear connected with the content of the speech and are termed iconic gestures. Critical to our understanding of the relationship between speech and iconic gesture is an analysis of what properties of talk might give rise to these gestures. This paper focuses on two such properties, namely the familiarity and the imageability of the core propositional units that the gestures accompany. The study revealed that imageability had a significant effect overall on the probability of the core propositional unit being accompanied by a gesture, but that familiarity did not. Familiarity did, however, have a significant effect on the probability of a gesture in the case of high imageability units and in the case of units associated with frequent gesture use. Those iconic gestures accompanying core propositional units variously defined by the properties of imageability and familiarity were found to differ in their level of idiosyncrasy, the viewpoint from which they were generated and their overall communicative effect. This research thus uncovered a number of quite distinct relationships between gestures and speech in everyday talk, with important implications for future theories in this area.

Why the spontaneous images created by the hands during talk can help make TV advertisements more effective

The design of effective communications depends upon an adequate model of the communication process. The traditional model is that speech conveys semantic information and bodily movement conveys information about emotion and interpersonal attitudes. But McNeill (2000) argues that this model is fundamentally wrong and that some bodily movements, namely spontaneous hand movements generated during talk (iconic gestures), are integral to semantic communication. But can we increase the effectiveness of communication using this new theory? Focusing on advertising we found that advertisements in which the message was split between speech and iconic gesture (possible on TV) were significantly more effective than advertisements in which meaning resided purely in speech or language (radio/newspaper). We also found that the significant differences in communicative effectiveness were maintained across five consecutive trials. We compared the communicative power of professionally made TV advertisements in which a spoken message was accompanied either by iconic gestures or by pictorial images, and found the iconic gestures to be more effective. We hypothesized that iconic gestures are so effective because they illustrate and isolate just the core semantic properties of a product. This research suggests that TV advertisements can be made more effective by incorporating iconic gestures with exactly the right temporal and semantic properties.

An exploration of the other side of semantic communication: How the spontaneous movements of the human hand add crucial meaning to narrative

Abstract Past research has suggested that those spontaneous movements of the human hand made during talk convey significant semantic information over and above the speech, at least when the unit of speech analyzed is the individual clause. However, no previous research has tested whether this information is represented linguistically elsewhere in the narrative (or is directly inferable from the rest of the narrative). The first study, reported here, uses an experimental procedure to identify which specific imagistic gestures add semantic information to the speech. The second study analyzes whether the specific gestures still do this when respondents hear the whole narrative. It was found that two thirds of the semantic information, thought to be carried by the gestures, is, in fact, represented in the linguistic discourse, or is inferable from it. However, one third of the additional semantic information contained in the gestures is not represented linguistically in the narrative nor is it inferable from it. In other words, a proportion of the imagistic gestures that accompany speech are absolutely critical to semantic communication.

Nonverbal indicators of deception: How iconic gestures reveal thoughts that cannot be suppressed.

Abstract This study explores the morphology of iconic gestures during deception. Participants narrated a static cartoon story twice. In one condition they provided an accurate account of the story, in the other they were instructed to introduce false details. Participants produced significantly fewer iconic gestures when describing plot-line events deceptively than when narrating comparable episode units truthfully. Deceptive gestures had significantly fewer post-stroke holds and shorter stroke phase durations than those produced alongside truthful utterances. Following Beattie (Visible thought: The new psychology of body language, Routledge, 2003) three narrators in the deceptive condition produced gestures that in their morphology contradicted the semantic information encoded in their speech stream, and ultimately signaled possible deceit.

A critical appraisal of the relationship between speech and gesture and its implications for the treatment of aphasia

The paper by Rose is an important and timely attempt to outline the implications of research on gesture and speech for the treatment of aphasia. But this is not an easy task given that major theoretical disagreements exist within the field; fundamental disagreements as to whether iconic gestures function primarily for the speaker or for the listener. It is also not made easy by the apparently contradictory research findings that many of the core empirical studies have produced. The current paper argues that Rose needs to take a much more critical perspective on some of the core studies in this area, including research on the tip-of-the-tongue phenomenon, which is one important area of disagreement with regard to the lexical retrieval hypothesis of iconic gesture and research on the possible communicational effects of iconic gesture. This paper argues that much of the research, when critically reviewed, points towards the communicational theory of gesture and away from the lexical retrieval hypothesis and it proposes that only by taking such a critical stance on the published literature will researchers be able to formulate truly effective principles for the treatment of aphasia.

Tracking the distribution of individual semantic features in gesture across spoken discourse: New perspectives in multi-modal interaction

Abstract Speakers frequently produce elaborate hand movements during talk that have been shown to serve a communicative function. Nevertheless, two-thirds of the semantic content of these hand movements is encoded linguistically elsewhere in the discourse (Beattie and Shovelton, Semiotica 184: 33–52, 2011). The present experiment demonstrated that while 62.9% of semantic information in gesture was elsewhere, most gestures (81.8%) retained at least one semantic feature that was never represented linguistically. Semantic features were more explicit when they occurred in gesture than when represented linguistically. Even in cases where the imagistic gesture appeared somewhat redundant, gesture at the narrative level preserves a discernable communicative function.