Geoffrey Beattie

Gesture and Silence as Indicators of Planning in Speech

A characteristic of human talk is that it is typically accompanied by bodily movements, most noticeably of the arms and hands. It is a matter of common observation that a subclass of these hand and arm movements appear intimately linked with the process of speech production: they are rhythmically timed with the speech, and often seem to reflect the meaning which the speech expresses. We call these movements Speech Focussed Movements (SFMs). These can be distinguished from other movements which accompany speech, scratches, twitches and the like, since the latter are not timed with speech, and do not in any apparent way reflect the meaning of what is said.

Mapping the range of information contained in the iconic hand gestures that accompany spontaneous speech.

This study tested McNeill’s theory that the iconic gestures that accompany speech in everyday talk convey critical information in interpersonal communication. Using a structured interview to measure the amount of information respondents receive from clause-length clips depicting aspects of a cartoon story, we discovered that when respondents could see the iconic gestures as well as hear the speech they received significantly more accurate information about those aspects of the original story depicted in the clips than when they just heard the speech. We also discovered that it was only with respect to certain semantic categories, namely, the relative position and the size of objects, that the beneficial effect of gestural communication was significant. We then considered in detail how specific attributes of actions and of objects are communicated via the iconic representation within individual gestures. Lastly, we discuss the implications of these findings for models of human communication in this area.

An experimental investigation of some properties of individual iconic gestures that mediate their communicative power.

It has been hypothesized that the iconic hand gestures that accompany talk communicate important semantic information. This research tests whether some gestures, in the absence of speech, are more communicative than others and considers what properties of gestures might affect their communicative power. Our research found that the communicative power of gestures does vary greatly, and that this is significantly affected by the viewpoint from which a gesture is generated, with character viewpoint gestures being more communicative than observer viewpoint gestures. It has also been suggested that gesture viewpoint is connected with the transitivity of the clause that it accompanies, and it was found in our study that respondents appeared to obtain syntactic information about the associated clause from the gesture. This conclusion was based on the observation that when respondents attempted to report what information was contained in gestures, viewed in the absence of speech, there was a significantly higher proportion of transitive structures in their answers after they had watched character viewpoint gestures compared with observer viewpoint gestures. Communication about the syntax of the accompanying clause might thus be a critical, but thus far neglected, aspect of the role of gestures in everyday talk.

An experimental investigation of the role of iconic gestures in lexical access using the tip-of-the-tongue phenomenon.

The tip-of-the-tongue (TOT) state was induced in participants to test Butterworth & Hadars (1989) theory that iconic gestures have a functional role in lexical access. Participants were given rare word definitions from which they had to retrieve the appropriate lexical item, all of which had been rated high in imageability. Half were free to gesture and the other half were instructed to fold their arms. Butterworth & Hadars theory (1989) would predict, first, that the TOT state should be associated with iconic gesture and, second, that such gestures should assist in this lexical retrieval function. In other words, those who were free to gesture should have less trouble in accessing the appropriate lexical items. The study found that gestures were associated with lexical search. Furthermore, these gestures were sometimes iconic and sufficiently complex and elaborate that naive judges could discriminate the lexical item the speaker was searching for from a set of five alternatives, at a level far above chance. But often the gestures associated with lexical search were not iconic in nature, and furthermore there was no evidence that the presence of the iconic gesture itself actually helped the speaker find the lexical item they were searching for. This experimental result has important implications for models of linguistic production, which posit an important processing role for iconic gestures in the processes of lexical selection.

Contextual Probability and Word Frequency as Determinants of Pauses and Errors in Spontaneous Speech.

This study investigated the relationship between the contextual probability of lexical items in spontaneous speech, as measured by the Cloze procedure, and word frequency. It also attempted to determine the relative importance of the two variables in causing delay, in the form of hesitation, in the production of spontaneous speech. The analysis revealed that content words of low contextual probability tended to be more infrequent than other words, and that both contextual probability and word frequency were associated with hesitation in speech. Contextual probability had an effect on hesitation even when word frequency was held constant, but word frequency had no effect when contextual probability was controlled. Analysis of certain types of errors, also, revealed that word frequency may play an important role in the lexical selection process.

Planning units in spontaneous speech: some evidence from hesitation in speech and speaker gaze direction in conversation

The aim of the present study was to attempt to elucidate the nature of the units of encoding involved in the generation of spontaneous speech, firstly through analysis of the distribution of hesitations in speech, and secondly through analysis of speaker gaze direction in conversation. These analyses suggested that both suprasentential units and simple clausal units are implicated in the encoding process. Moreover, evidence of encoding on a clausal basis was only obtained for speech produced during the planning phases of the larger, suprasentential units.

The temporal structure of natural telephone conversations (directory enquiry calls)

Investigations of face-to-face conversation have suggested that visuallymediated cues play an important role in the regulation of speaker-turns. By implication, conversational structure should be disrupted in no-vision conditions. This study tested this hypothesis by focussing on natural telephone conversations — directory enquiry calls. The results suggest that the management of speaker-switching, as measured by the temporal characteristics of the transitions, was not impaired in natural telephone conversations. The vast majority of the measures also indicated that operators and subscribers were behaving very similarly in terms of the temporal characteristics of switching, despite considerable differences in their experience.

What properties of talk are associated with the generation of spontaneous iconic hand gestures

When people talk, they frequently make movements of their arms and hands, some of which appear connected with the content of the speech and are termed iconic gestures. Critical to our understanding of the relationship between speech and iconic gesture is an analysis of what properties of talk might give rise to these gestures. This paper focuses on two such properties, namely the familiarity and the imageability of the core propositional units that the gestures accompany. The study revealed that imageability had a significant effect overall on the probability of the core propositional unit being accompanied by a gesture, but that familiarity did not. Familiarity did, however, have a significant effect on the probability of a gesture in the case of high imageability units and in the case of units associated with frequent gesture use. Those iconic gestures accompanying core propositional units variously defined by the properties of imageability and familiarity were found to differ in their level of idiosyncrasy, the viewpoint from which they were generated and their overall communicative effect. This research thus uncovered a number of quite distinct relationships between gestures and speech in everyday talk, with important implications for future theories in this area.

The regulation of speaker turns in face-to-face conversation: Some implications for conversation in sound-only communication channels

Conversation in sound-only communication channels has recently been extensively investigated. The specific aim of some of this research has been to elucidate the regulatory role of visual nonverbal signals in conversation. The methodology has involved observation of interactional behavior and conversational structure in conditions in which interactants either were separated by some physical barrier (Jaffe and Feldstein 1970; Cook and Lalljee 1972; Butterworth et al. 1977; Rutter and Stephenson 1977), or were conversing by telephone (Kasl and Mahl 1965; Butterworth et al. 1977). The aim of this paper is to review observational studies of face-to-face interaction which have analyzed the role of various cues in the regulation of conversation and which have suggested that visual nonverbal cues may play an important role in the regulation of speaker turns. Such a review should enable one to derive specific predictions about the effects of the removal of these visually transmitted cues on conversational structure and interactional behavior in sound-only conditions. It will be argued that studies of conversation in no-vision conditions, thus far, have tested rather imprecise and often inaccurate hypotheses, derived from somewhat cursory reviews of the literature on face-to-face interaction. For example, it is noteworthy that none of the studies of conversation in novision conditions which have already been cited have made any mention of the role of kinesic behavior, except speaker gaze, in the regulation of turn-taking. However, Duncan (1972, 1973, 1974, 1975), Duncan and Fiske (1977), De Long (1974, 1975), Kendon (1972), and Wiemann and Knapp (1975) have carried out observational studies of conversational interaction which have suggested that kinesic phenomena, in addition to speaker gaze, are of considerable importance in regulating the flow of social encounters.

Why the spontaneous images created by the hands during talk can help make TV advertisements more effective

The design of effective communications depends upon an adequate model of the communication process. The traditional model is that speech conveys semantic information and bodily movement conveys information about emotion and interpersonal attitudes. But McNeill (2000) argues that this model is fundamentally wrong and that some bodily movements, namely spontaneous hand movements generated during talk (iconic gestures), are integral to semantic communication. But can we increase the effectiveness of communication using this new theory? Focusing on advertising we found that advertisements in which the message was split between speech and iconic gesture (possible on TV) were significantly more effective than advertisements in which meaning resided purely in speech or language (radio/newspaper). We also found that the significant differences in communicative effectiveness were maintained across five consecutive trials. We compared the communicative power of professionally made TV advertisements in which a spoken message was accompanied either by iconic gestures or by pictorial images, and found the iconic gestures to be more effective. We hypothesized that iconic gestures are so effective because they illustrate and isolate just the core semantic properties of a product. This research suggests that TV advertisements can be made more effective by incorporating iconic gestures with exactly the right temporal and semantic properties.