Alan J. Klockars

The Quest for α: Developments in Multiple Comparison Procedures in the Quarter Century Since Games (1971)

In a highly regarded work, Games (1971) presented state-of-the-art multiple comparison procedures (MCPs) for a variety of research scenarios and sought to bring order to the seemingly chaotic array of MCPs being used at that time. The current article is a sequel of sorts, placing Games’s insights in the context of many of the major developments in simultaneous and sequential inference since his article’s publication. Specifically, we address the common MCP scenarios of orthogonal contrasts, nonorthogonal contrasts, comparisons against a reference group, all possible pairwise comparisons, and exploratory post hoc contrasts, all under the assumed conditions of independent scores, normality, and homogeneity of variance. In addition, discussions of the philosophical issues surrounding the control of Type I error rates are presented.

The effect of scale manipulations on validity: targetting frequency rating scales for anticipated performance levels.

A common problem with standard five-point frequency rating scales is their inability to differentiate between objects within a relatively narrow band of the rating dimension. Two alternatives for increasing a scales ability to reflect existing differences are: increasing the number of positions on the rating scale, or packing the rating scale with quantifiers from a particular portion of the frequency dimension. In this study, three types of rating scale - a standard five-point balanced scale, a longer nine-point balanced scale, and a five-point packed scale - were used to rate two videotaped samples of behaviour, one displaying performance levels from 10% to 100% and the other from 70% to 90%. Each subjects ratings were correlated with true performance levels in each sample of behaviour as a measure of validity. Results showed that for ratings of the wide range of behaviour all three types of scales provided average correlations between the ratings and actual frequencies of the event which exceeded 0.90, with the longer nine-point scale yielding a significantly higher mean correlation than the other two scales. For ratings of the narrow performance range, the nine-point scale provided the highest correlation with the actual frequencies, followed by the packed scale and the standard five-point balanced scale. All differences were significant. Findings suggest that increasing the number of scale positions can significantly increase the validity of ratings obtained. Also, though to a lesser degree, validity of ratings may be enhanced in a shorter scale by using quantifiers from the portion of the frequency continuum where performance are anticipated to lie.

A Factor Analysis of the EPPS and PRF Personality Inventories.

Two multi-scale personality inventories have followed Murray’s need structure theory of personality in the formulation of their scale definitions and item domains. Edwards (1957a), on the basis of the list of manifest needs presented by Murray and others (1938), developed the Edwards Personal Preference Schedule (EPPS) which measures the strength of 15 needs: Achievement (ach), Deference (def ), Order (ord), Exhibition (exh), Autonomy (aut), Affiliation (ajy), Intraception (int), Succorance (suc), Dominance (dom), Abasement (aba), Nurturance (nur), Change (chg), Endurance (end), Heterosexuality (het), and Aggression (agg). The EPPS uses a forced-choice item format in which two state-

The Identification of High-School Dropouts Identified as Learning Disabled: Evaluating the Utility of a Discriminant Analysis Function:

This study examined the utility of a linear discriminant function to distinguish between students identified as learning disabled (LD) who had either been released from high school under codes suggestive of school dropout (n = 213) or graduation (n = 92). The discriminant function was comprised of six variables—student ethnicity, reading ability, family intactness, family socioeconomic status, school transfers, and school-initiated interruptions. The analysis determined that differences between the LD dropout sample and LD graduate sample were sufficient to allow for a discrimination between the groups. On the basis of group differences the discriminant function that was constructed correctly classified 83% of the school dropouts and 46% of the school graduates, for an overall 73% accuracy rate. Those factors contributing most to the function were the number of district-initiated interruptions, school transfers, and family intactness. Based on the findings, implications for school districts and future research are noted.

Scheffé’s More Powerful F-Protected Post Hoc Procedure

In 1970 Henry Scheffé proposed a more powerful version of his well known post hoc multiple comparison procedure, only to fail to recommend it by the papers end. The point of the current paper is to bring this simple modification to a wider audience, complete with an original derivation, in hopes that the method will be embraced by researchers despite its creators hesitations. Specifically, whereas Scheffés original (1953) procedure advocates testing any exploratory post hoc contrast or comparison using a critical value assuming k - 1 between-group degrees of freedom, Scheffés later modification (1970) will be demonstrated here showing that a more liberal critical value assuming k - 2 between-group degrees of freedom may be used if an omnibus null hypothesis across all means has been rejected.

Per-Experiment Error Rates: The Hidden Costs of Several Multiple Comparison Procedures

For a number of common multiple comparison procedures controlling Type I error at .05 refers to what Ryan called the experimentwise error rate. This expression represents the chance of making at least one Type I error within a given experiment. This approach ignores, however, the potential detriments of multiple errors within a single experiment, that is, it fails to acknowledge what Ryan referred to as the error rate per experiment. The current study uses a computer simulation to evaluate the differences between experimentwise error rates and error rates per experiment for a variety of multiple comparison procedures. For pairwise comparisons, Newman-Keulss and Tukeys tests are examined, as is Dunnetts test for comparisons with a control group and Scheffes test for all possible post hoc comparisons. For planned contrasts, a standard Bonferroni and Shaffers sequentially rejective Bonferroni are simulated using both a nonorthogonal and an orthogonal set.

A More Powerful Post Hoc Multiple Comparison Procedure in Analysis of Variance

Scheffé’s test (Scheffé, 1953), which is commonly used to conduct post hoc contrasts among k group means, is unnecessarily conservative because it guards against an infinite number of potential post hoc contrasts when only a small set would ever be of interest to a researcher. This paper identifies a set of post hoc contrasts based on subsets of the treatment groups and simulates critical values from the appropriate multivariate F-distribution to be used in place of those associated with Scheffé’s test. The proposed method and its critical values provide a uniformly more powerful post hoc procedure.

Manipulations of Evaluative Rating Scales to Increase Validity

Standard five-point rating scales often do not allow raters to capture perceived differences between objects or individuals within a relatively narrow band of the evaluative dimension. In the frequency domain, using a longer rating scale or packing the rating scale with labels from a particular portion of the dimension of interest have both been shown by Hancock and Klockars in 1991 to increase rating validity for differentiating among a narrow range of performances. The present study investigates the effect of similar manipulations on the validity of ratings from evaluative rating scales. The correlations of evaluative ratings with experimentally manipulated (and hence known) performance tended to be fairly high regardless of the evaluative scale used.

Dichotomous and Multipoint Scales Using Bipolar Adjectives

Subjects responded to a set of self-descnptive bipo lar adjectives in either a dichotomous format or a 7-point bipolar rating scale format. In addition, each subject answered a measure of social desirability re sponding rate The responses of over 400 subjects in each format were factor analyzed, followed by a vari max rotation The factor structure for the formats were strikingly similar and closely reflected the traits repre sented by the adjectives. Social desirability was found to be of minimal influence for either format. However, the factors derived from the multipoint format were slightly clearer and accounted for more variance. In addition, trait measures produced by the sum of four adjective scales from a common content dimension were more internally consistent using the multipoint format.

Bipolar Adjective Rating Scales for Self-Description: Reliability and Validity Data

Bipolar adjective rating scales were constructed to evaluate selected needs from Murrays (1938) theory of personality. Twenty items measuring each of six needs (succorance, autonomy, dominance, social recognition, achievement, and nurturance) were initially developed to provide a preliminary instrument that was administered to 200 individuals. Because of an unacceptably low estimate of internal consistency, the need for succorance was eliminated. For each of the five remaining needs, sets of 12 adjective pairs were selected based on item-total correlations. The resulting 60-item inventory as well as the Personality Research Form (PRF) (Jackson, 1967) scales measuring the same five needs was administered to 254 college students. Internal-consistency estimates of the bipolar adjective measures compared favorably to those of the PRF scales. Examination of the matrix of inter correlations among bipolar and PRF measures (serving as criterion) and consideration of the problems in developing a reliable measure for succorance suggest that certain personality traits are more amenable to valid measurement by bipolar adjective items than are others.