Anne K. Cordes

The reliability of observational data: I. Theories and methods for speech-language pathology.

Much research and clinical work in speech-language pathology depends on the validity and reliability of data gathered through the direct observation of human behavior. This paper reviews several definitions of reliability, concluding that behavior observation data are reliable if they, and the experimental conclusions drawn from them, are not affected by differences among observers or by other variations in the recording context. The theoretical bases of several methods commonly used to estimate reliability for observational data are reviewed, with examples of the use of these methods drawn from a recent volume of the Journal of Speech and Hearing Research (35, 1992). Although most recent research publications in speech-language pathology have addressed the issue of reliability for their observational data to some extent, most reliability estimates do not clearly establish that the data or the experimental conclusions were replicable or unaffected by differences among observers. Suggestions are provided for improving the usefulness of the reliability estimates published in speech-language pathology research.

The Reliability of Observational Data: II. Issues in the Identification and Measurement of Stuttering Events.

Much attention has been directed recently toward the problem of measuring occurrences of stuttering with satisfactory levels of interjudge agreement. This paper reviews the prominent concepts of the stuttering event, arguing that they may be one cause of the stuttering measurement problem. The evidence that has led to concerns about the reliability of stuttering event measurements is also reviewed. Reliability and measurement issues that were discussed in the first paper of this series (Cordes, 1994) emerge as basic to the interpretation of much stuttering research, and it is argued that the stuttering measurement problem is not confined to research on stuttering judgments but actually permeates other important stuttering research areas. Some recent attempts to resolve the stuttering measurement problem are reviewed, and the implications of developing an improved measurement system for this disorder are discussed.

Time-Interval Measurement of Stuttering: Modifying Interjudge Agreement.

This paper reports the results of two experiments that investigated interval-by-interval inter- and intrajudge agreement for stuttered and nonstuttered speech intervals (4.0 sec). The first experiment demonstrated that interval-by-interval interjudge agreement could be significantly improved, and to satisfactory levels, by training judges to discriminate between experimenter-agreed intervals of stuttered and nonstuttered speech. The findings also showed that, independent of training, judges with relatively high intrajudge agreement also showed relatively higher interjudge agreement. The second experiment showed that interval-by-interval interjudge agreement was not significantly different if judges rated 4-sec speech intervals from different samples under three conditions: in random order, separated by 5-sec recording intervals; in correct order, also separated by 5-sec recording intervals; or after brief judgment signals that occurred every 4 sec during continuous samples. The implications of these findings for stuttering measurement are discussed.

Multicultural identification and treatment of stuttering: A continuing need for research☆

Abstract This article presents one view of the past, present, and future of multicultural studies of stuttering. Historical issues are reviewed, and current issues about the multicultural or cross-cultural identification and treatment of stuttering are then considered in light of that history. It appears from this combined analysis of historical and current trends that many multicultural discussions about stuttering continue to be based on ethnocentric speculation and on ideas related to diagnosogenic theory, rather than on dependable empirical or clinical knowledge. Suggestions are provided in the form of issues to be considered and, more importantly, in the form of areas that will need considerably more research if this disorder is to be approached from a truly multicultural perspective.

Time-Interval Measurement of Stuttering: Systematic Replication of Ingham, Cordes, and Gow (1993)

The study reported in this paper was designed to replicate and extend the results of an earlier study (Ingham, Cordes, & Gow, 1993) that investigated time-interval judgments of stuttering. Results confirmed earlier findings that interjudge agreement is higher for these interval-recording tasks than has been previously reported for event-based analyses of stuttering judgments or for time-interval analyses of event judgments. Results also confirmed an earlier finding that judges with intrajudge agreement levels of 90% or better show higher interjudge agreement than judges with lower intrajudge agreement scores. This study failed to find differences between audiovisual and audio-only judgment conditions; between relatively experienced and relatively inexperienced student judges; and, most importantly, between the judgments made, and the agreement levels achieved, by judges from two different clinical research settings. The implications of these findings for attempts to develop a reliable measurement method for stuttering are discussed.

Time-Interval Measurement of Stuttering: Effects of Training With Highly Agreed or Poorly Agreed Exemplars

This study required six groups of judges, three experimental groups and three control groups (all n = 5), to categorize consecutive 5.0-sec speech intervals as Stuttered or Nonstuttered on four judgment occasions. Between the second and third occasions, each experimental group was trained to categorize correctly one of three sets of speech intervals: agreed intervals, which had been unanimously prejudged to be Stuttered or Nonstuttered; disagreed intervals, which had been prejudged to be Stuttered by approximately half of a large group of judges; or randomly selected intervals, including both agreed and disagreed intervals. Results replicated and extended an earlier finding of improved interjudge agreement for judges trained with highly agreed intervals (Ingham, Cordes, & Gow, 1993): Training with highly agreed intervals was shown to be more effective than equivalent exposure to those intervals without feedback, and training with highly agreed intervals was shown to be more effective than training with, or exposure to, poorly agreed or randomly selected intervals.

Comments on Yaruss, LaSalle, and Conture (1998)

In summary, Yaruss et al. have contributed to the literature a set of “recommendation categories” that were based on retrospective analyses of minimal information from children who might have stuttered, gathered originally by student clinicians in one training clinic who were entirely unaware that their data would be used to develop recommendations for other children in other clinics. These categories and recommendations might have led to inaccurate decisions about a significant minority of the children involved in this study (Yaruss et al.’s Footnote 1), but “suggested treatment recommendations” are nevertheless presented as “useful for clinicians to consider” (p. 73). Given some indications that delaying treatment for children who stutter may reduce the effectiveness of any eventual treatment (R. J. Ingham & Cordes, 1999), the decision not to recommend treatment after an initial stuttering evaluation is an important one. The evidence used to support such a decision should be strong, developed from well-...

On valid and reliable identification of normal disfluencies and stuttering disfluencies: A response to Aram, Meyers, and Ekelman (1990)

Aram, Meyers, and Ekelman (1990, Brain and Language, 38, 105-121) recently reported finding that children with unilateral brain lesions produced more stuttering-type nonfluencies than their neurologically normal peers. However, they did not report inter- or intrajudge agreement for the nonfluency types or for their method of measuring speech rate. The speech rates they reported were also unusually fast. We argue that these problems with Aram et al.s study imperil both their results and their conclusions regarding developmental stuttering.