Michael D. Franzen

Reliability of alternate forms of the trail making test

Abstract Initial research in a limited sample of 15 closed-head-injury patients indicated reasonable reliability for alternate forms of the Trail Making Test, Parts A and B, called Form C and D, respectively. The present study examined the reliability of these alternate forms in a larger and more heterogeneous sample of 192 subjects with neurologic, psychiatric, or substance abuse diagnoses. The two forms correlated significantly, and there were no significant differences in mean scores for the two forms. The results indicate adequate reliability, although separate cut-off values for Form C may be necessary due to a nonsignificant tendency for scores on Form C to be lower than for the Trail Making Test, Part A.

Validity as Applied to Neuropsychological Assessment

The concept of test validity basically corresponds to the question of whether a test or an assessment procedure supplies the kind of information needed for a particular interpretation. This issue is very important because the test scores in and of themselves are meaningless unless they refer to a defined realm of observable phenomena. In clinical psychology, the consequence of interest may be a particular personality style or behavioral disorder. For clinical neuropsychology, the consequence of interest may range from whether brain impairment exists to the implications of the test results for adaptive behavior. Before particular issues that affect the validity of neuropsychological tests are described, general applicable concepts of validity and threats to validity are reviewed.

The Assessment of Child Neuropsychological Function

One of the areas in which there has been significant development and activity is in the realm of child neuropsychological assessment. In the past, the assessment of children depended upon the availability of child norms for adult instruments. This approach was always decried as being inappropriate, but there was a lack of child instruments otherwise available. The exception was, of course, the Wechsler Intelligence Scale for Children which can be viewed as a downward extension of the Wechsler adult scales. The optimal situation would be for an instrument to be developed out of a consideration of the types of referral questions asked in child assessment cases, using age-appropriate stimuli, reflecting developmentally relevant domains and methods, and possessing adequate age-related norms. The ideal is always brighter than the real, but significant progress has been accomplished in recent years.

Tests of Verbal Functions

Object-naming tasks have been a popular form of evaluation of neurologically impaired individuals. Some form of object naming is included in many standardized procedures, including the Luria—Nebraska Neuropsychological Battery. The use of nonstandardized procedures is risky, as several variables can influence performance on this test, including the familiarity of the subject with the object, the use of an object itself as opposed to a picture of the object, and whether the object word can also be used as a verb (Barker & Lawson, 1968). To subvert these potential problems, Oldfield and Wingfield (1965) developed a set of pictures that can be used in a standardized fashion. Newcombe, Oldfield, Ratcliffe, and Wingfield (1971) found this task to be sensitive to brain injury. However, the reliability and the specific validities of this procedure still need to be evaluated. Lawson and Barker (1968) administered an object-naming test to 100 individuals with organic dementia and 40 volunteers from an elderly persons’ social club. They scored a failure to respond as well as the latency of response and found that both variables discriminated between the two groups significantly. They also found that demonstration facilitated naming in the impaired subjects but not in the control subjects. This information points to the possible clinical utility of the procedure.

The Halstead—Reitan and Luria—Nebraska Neuropsychological Batteries

North American neuropsychology is largely based on the psychometric tradition of measurement. Many neuropsychologists practicing in North America were trained in the context of graduate psychology departments. Because they received their training from psychologists, rather than from neurologists who may have provided ancillary training experiences, clinical neuropsychologists largely have chosen assessment instruments that involve standardized assessment. Many clinical neuropsychologists utilized a fixed battery approach in the past. In the early 1980s, the Battle of the Batteries consisted of adherents of either the Halstead—Reitan (HRNB) or of the Luria—Nebraska (LNNB) conducting studies, publishing results, and trading comments. It seemed as if neither battery would achieve hegemony. Surveys such as that described by Guilmette and Faust (1991) indicated that although the HRNB had greater frequency of usage, both batteries were commonly used. Since that time, changes in some of the influences on clinical neuropsychology have resulted in a lesser emphasis on comprehensive evaluations using instruments that have been normed and standardized on a single sample. Whether this is a positive trend is subj ect to review.

Elemental Considerations in Validity

Validity is a term that is often invoked in decisions to use neuropsychological tests. Unfortunately, the context of this use is usually negative, as when a test is cited as invalid. The use of the term implies that a test can be determined to be either valid or invalid. Of course, most clinical neuropsychologists agree that a test that is “valid” for one population may be “invalid” for another. If this is true, can a test ever be evaluated as universally valid or invalid? A second question relates to how a test is evaluated as valid or invalid. This is a question of both method (How do we evaluate a test?) and of epistemology (How do we know what we know?). Although method may be discussed separately from epistemology, the obverse is not necessarily true. That is, how we know something is highly related to how we investigate that something. This chapter discusses general issues in the relationship between epistemology and method, and Chapter 5 discusses the methodological issues more directly.

Postscript—Future Directions

To predict the future is a daunting task. At the time of publication of the previous edition of this book I certainly would not have predicted the developments in or the effects of outside forces such as managed care on the field. Perhaps instead of future directions, we should talk of necessary improvements. In general, I think the field will see a greater degree of psychometric sophistication among clinicians and researchers. The increased use of Item Response Theory and Generalizability Theory to evaluate the precision of measurement would be a positive development. Along those lines, the evaluation and conceptualization of validity as a diffractionable entity rather than a monolith will help us to better understand and use the assessment techniques at our disposal. The general conceptualization of a test as a method-construct unit will also aid our understanding.

Methods for Evaluating the Validity of Test Scores

There is another burgeoning area of assessment methods, namely the development and evaluation of methods to estimate the validity of inferences drawn from the test data. In essence, these are meta-methods to determine the validity of the test data themselves as well as to determine the validity of the inferences drawn from the test data. The estimation of premorbid functioning against which to evaluate change demonstrated by the present scores and the estimation of the level of effort exhibited by the subject are two areas that have attracted much recent attention. Because premorbid scores from a prior test session are rarely available, methods have been developed to estimate premorbid functioning. Because the interpretation of scores requires an assumption of optimal effort, methods have been developed to assess the level of effort and aid decisions regarding the possible presence of response bias.

The Rorschach Inkblots

The use of the Rorschach as a neuropsychological instrument remains an issue of controversy. Despite the extensive clinical and theoretical literature related to the Rorschach, those individuals critical of this instrument maintain that the validity of the Rorschach has not been established according to strict psychometric standards. Proponents of the Rorschach maintain that the functional utility of this instrument relies on the clinical skills and sensitivities of the clinician using it. The past few years have seen a great increase in the number of research studies investigating the reliability and validity of the Rorschach. However, the controversy continues. Regardless of one’s bias toward the use of this instrument, there is a paucity of research on its validity and reliability when it is used with neurologically impaired clients. The utility, or the potential utility, of this procedure, however, continues to be of interest (see the symposium listing for the Eighth European Conference of the International Neuropsychological Society, Costa & Rourke, 1985). The proponents of the Rorschach defend its use and conclude that the Rorschach has an adequate empirical base to allow its use in a forensic context (McCann, 1998). The courts seem to agree, as a review of legal cases in which the Rorschach was involved in legal testimony indicates that the findings of an evaluation using the Rorschach were allowed in testimony the majority of times in which the Rorschach was used (Meloy, Hansen, & Weiner, 1997). However, the true test of the Rorschach needs to be conducted in the context of expert research psychologists published in peer-reviewed journals (Exner, Colligan, Boll, Stischer, & Hillman, 1996; Wood, Nezworski, & Stejskal, 1996).

General and Theoretical Considerations in the Assessment of Reliability

The term reliability is often used to describe the temporal stability of a set of measurement procedures. Other frequent uses of the term are related to internal consistency and agreement among different users of the procedures. These different usages all relate to a single concept, namely, the estimation of the influence of error on the scores resulting from the use of the test or set of procedures. A perfectly reliable test is a test that measures without error. Error-free measurement is a practical impossibility. Instead, it is an ideal to which test authors and developers attempt approximations. As well as being an impossible ideal to attain in a practical sense, reliability is also an impossible concept to evaluate directly. All of the methods and designs discussed in Chapter 3 are able only to estimate the reliability of the test. As will be seen in the more complete discussion in Chapter 3, the methods and designs are attempts to estimate the degree of error that influences test scores by systematically varying the possible sources of error. The success of the endeavor is related to the quality of the methods and designs and to the ability of the researcher to comprehensively describe the possible sources of error.