Joanne C. Rogan

A Comparison of the Modified-Tukey and Scheffé Methods of Multiple Comparisons for Pairwise Contrasts

Abstract The Kramer (1956), Spjotvoll and Stoline (1975), Hochberg (1976), and Games and Howell (1976) modified forms of the Tukey test are compared with the Scheffe (1959) test for (1) rates of Type I error and (2) sensitivity to varying degrees of sample-size imbalance and variance heterogeneity when sampling from normal and skewed distributions. Only the Hochberg (1976) and Games and Howell (1976) procedures control their rates of Type I error at or below the nominal five percent level for all conditions investigated; however, the Games and Howell (1976) procedure is more powerful and always results in Type I error values which are closer to the level of significance.

Effect of Very Unequal Group Sizes on Tukey's Multiple Comparison Test.

The harmonic mean and Kramer procedures have been recommended for use with the Tukey multiple comparison test when group sizes are unequal. The only study that has compared these procedures maximally varied the disparity between the groups for a 3: 1 difference. The present investigation varied the disparity between the groups, the number of groups and the nominal significance levels. It was found that rarely did the empirical Type I error probabilities exceed their nominal significance levels by more than 1 percent even when the groups differed by 40:1. The Kramer estimates were typically less than the nominal significance levels while the harmonic mean estimates were generally larger than the true alphas.

Robust Tests of Repeated Measures Means in Educational and Psychological Research

Numerous authors have maintained that when successive measurements are obtained from educational and psychological research, the traditional repeated measures F tests will not be valid (Danford, Hughes, and McNee, 1960; Humphreys, 1960; Jennings and Wood, 1976; McCall and Appelbaum, 1973; Wilson, 1967, 1974). The tests will be prone to too many false rejections because the data do not satisfy the validity assumption of circularity (Huynh and Feldt, 1970; Huynh and Mandeville, 1979; Rogan, Keselman, and Mendoza, 1979; Rouanet and Lepine, 1970). Multiple comparison tests that use a pooled estimate of error variance to obtain the standard error of the contrast of repeated measures means must also satisfy the circularity assumption to be valid. The purposes of this paper are to demonstrate that multiple comparison tests using a pooled error term are also dependent on the circularity assumption, and to show how to compute tests which are insensitive (robust) to this assumption.

Assumption Violations and Rates of Type I Error for the Tukey Multiple Comparison Test

While numerous investigations have examined the effects of assumption violations on the empirical probability of a Type I error for Tukey’s multiple comparison test, no study to date has numerically quantified and systematically varied the degree of total variation resulting from combining unequal variances with unequal sample sizes. The present investigation employed a coefficient of variance variation to index the degree of heterogeneity and compared the effects of varying degrees of heterogeneity on the harmonic mean, Kramer (11) and Miller (14) unequal group forms of the Tukey test. The discrepancies between the empirical and nominal significance rates of Type I error were related to a bias ratio provided by Box (1) and were found to markedly vary as a function of the magnitude of this ratio. The Kramer unequal group form of the Tukey test is recommended, as it consistently resulted in empirical Type I rates of error deviating less from the nominal significance level than either of the other two unequ...

An Addendum to “A Comparison of the Modified-Tukey and Scheffé Methods of Multiple Comparisons for Pairwise Contrasts”

Abstract Type I error and power rates for pairwise comparisons of means were obtained from a Monte Carlo investigation of the Gabriel (1978) and Hochberg (1974) multiple comparison procedures (MCPs). This addendum to the Keselman and Rogan (1978) study indicates that MCPs that obtain a standard error of the comparison using a pooled within-cell estimate of error variance cannot maintain the overall Type I error rate at α when the largest of a set of unequal variances is paired with the smallest of a set of unequal sample sizes and vice versa. Two procedures for computing pairwise multiple comparisons are recommended.

Type I and Type II errors in simultaneous and two-stage multiple comparison procedures.

Responds to T. A. Ryan (see record 1980-29331-001) by emphasizing and demonstrating the usual inferential trade-off between protecting Type I and protecting Type II errors for multiple comparison procedures. Simultaneous test procedures are preferred, but it is felt that there are types of investiga