Donald W. Zimmerman

Are Simple Gain Scores Obsolete

It is widely believed that measures of gain, growth, or change, expressed as simple differences between pretest and posttest scores, are inherently unreliable. It is also believed that gain scores lack predictive validity with respect to other criteria. However, these conclu sions are based on misleading assumptions about the values of parameters in familiar equations in classical test theory. The present paper examines modified equa tions for the validity and reliability of difference scores that describe applied testing situations more realisti cally and reveal that simple gain scores can be more useful in research than commonly believed.

Coefficient Alpha as an Estimate of Test Reliability Under Violation of Two Assumptions

Through use of computer simulation, the central tendency and variability of coefficient alpha were examined under violation of two assumptions made in the derivation of the formula. When assumptions were satisfied, the mean value of coefficient alpha was extremely close to the population reliability coefficient, but values were highly variable. This result was independent of the shape of the population distribution of test scores. Coefficient alpha underestimated reliability under violation of the assumption of essential tauequivalence of subtest scores and overestimated reliability under violation of the assumption of uncorrelated subtest error scores. In both cases, the bias of the estimates varied systematically with the degree of violation of assumptions, while the variability of the estimates remained constant. All these results were independent of the number of persons and the number of subtests.

A note on preliminary tests of equality of variances.

Preliminary tests of equality of variances used before a test of location are no longer widely recommended by statisticians, although they persist in some textbooks and software packages. The present study extends the findings of previous studies and provides further reasons for discontinuing the use of preliminary tests. The study found Type I error rates of a two-stage procedure, consisting of a preliminary Levene test on samples of different sizes with unequal variances, followed by either a Student pooled-variances t test or a Welch separate-variances t test. Simulations disclosed that the twostage procedure fails to protect the significance level and usually makes the situation worse. Earlier studies have shown that preliminary tests often adversely affect the size of the test, and also that the Welch test is superior to the t test when variances are unequal. The present simulations reveal that changes in Type I error rates are greater when sample sizes are smaller, when the difference in variances is slight rather than extreme, and when the significance level is more stringent. Furthermore, the validity of the Welch test deteriorates if it is used only on those occasions where a preliminary test indicates it is needed. Optimum protection is assured by using a separate-variances test unconditionally whenever sample sizes are unequal.

Invalidation of Parametric and Nonparametric Statistical Tests by Concurrent Violation of Two Assumptions

Abstract To provide counterexamples to some commonly held generalizations about the benefits of nonparametric tests, the author concurrently violated in a simulation study 2 assumptions of parametric statistical significance tests—normality and homogeneity of variance. For various combinations of nonnormal distribution shapes and degrees of variance heterogeneity, the Type I error probability of a non-parametric rank test, the Wilcoxon-Mann-Whitney test, was found to be biased to a far greater extent than that of its parametric counterpart, the Student t test. The Welch-Satterthwaite separate-variances version of the t test, together with a preliminary outlier detection and downweighting procedure, protected the statistical significance level more consistently than the nonparametric test did. Those findings reveal that nonparametric methods are not always acceptable substitutes for parametric methods such as the t test and the F test in research studies when parametric assumptions are not satisfied. They ...

Relative Power of the Wilcoxon Test, the Friedman Test, and Repeated-Measures ANOVA on Ranks

Abstract Many introductory statistics textbooks in education, psychology, and the social sciences consider the Friedman test to be a nonparametric counterpart of repeated-measures ANOVA, just as the Kruskal-Wallis test is a counterpart of oneway ANOVA. However, it is known in theoretical statistics that the Friedman test is a generalization of the sign test and possesses the modest statistical power of the sign test for normal as well as many nonnormal distributions. Although not familiar to researchers, another significance test that can be regarded as a nonparametric counterpart of repeated-measures ANOVA is a rank-transformation procedure, in which the usual parametric statistical analysis is performed on ranks replacing the original scores. In the present computer simulation study we compared the ordinary paired-samples Student t test, the Wilcoxon signed-ranks test, and the sign test for correlated samples from normal, uniform, mixed-normal, exponential, Laplace, and Cauchy distributions, for which t...

Teacher’s Corner: A Note on Interpretation of the Paired-Samples t Test:

Explanations of advantages and disadvantages of paired-samples experimental designs in textbooks in education and psychology frequently overlook the change in the Type I error probability which occurs when an independent-samples t test is performed on correlated observations. This alteration of the significance level can be extreme even if the correlation is small. By comparison, the loss of power of the paired-samples t test on difference scores due to reduction of degrees of freedom, which typically is emphasized, is relatively slight. Although paired-samples designs are appropriate and widely used when there is a natural correspondence or pairing of scores, researchers have not often considered the implications of undetected correlation between supposedly independent samples in the absence of explicit pairing.

A Note on the Influence of Outliers on Parametric and Nonparametric Tests

Abstract Extremely deviant scores, or outliers, reduce the probability of Type I errors of the Student t test and, at the same time, substantially increase the probability of Type II errors, so that power declines. The magnitude of the change depends jointly on the probability of occurrence of an outlier and its extremity, or its distance from the mean. Although outliers do not modify the probability of Type I errors of the Mann-Whitney-Wilcoxon test, they nevertheless increase the probability of Type II errors and reduce power. The effect on this nonparametric test depends largely on the probability of occurrence and not the extremity. Because deviant scores influence the t test to a relatively greater extent, the nonparametric method acquires an advantage for outlier-prone densities despite its loss of power.

PROBABILITY SPACES, HILBERT SPACES, AND THE AXIOMS OF TEST THEORY

A branch of probability theory that has been studied extensively in recent years, the theory of conditional expectation, provides just the concepts needed for mathematical derivation of the main results of the classical test theory with minimal assumptions and greatest economy in the proofs. The collection of all random variables with finite variance defined on a given probability space is a Hilbert space; the function that assigns to each random variable its conditional expectation is a linear operator; and the properties of the conditional expectation needed to derive the usual test-theory formulas are general properties of linear operators in Hilbert space. Accordingly, each of the test-theory formulas has a simple geometric interpretation that holds in all Hilbert spaces.

Comparative Power of Student T Test and Mann-Whitney U Test for Unequal Sample Sizes and Variances

AbstractA computer program generated power functions of the Student t test and Mann-Whitney U test under violation of the parametric assumption of homogeneity of variance for equal and unequal sample sizes. In addition to depression and elevation of nominal significance levels of the t test observed by Hsu and by Scheffe, the entire power functions of both the t test and the U test were depressed or elevated. When the smaller sample was associated with a smaller variance, the U test was more powerful in detecting differences over the entire range of possible differences between population means. When sample sizes were equal, or when the smaller sample had the larger variance, the t test was more powerful over this entire range. These results show that replacement of the t test by a nonparametric alternative under violation of homogeneity of variance does not necessarily maximize correct decisions.

Statistical Significance Levels of Nonparametric Tests Biased by Heterogeneous Variances of Treatment Groups

Abstract The statistical significance levels of the Wilcoxon-Mann-Whitney test and the Kruskal-Wallis test are substantially biased by heterogeneous variances of treatment groups—even when sample sizes are equal. Under these conditions, the Type I error probabilities of the nonparametric tests, performed at the .01, .05, and .10 significance levels, increase by as much as 40%-50% in many cases and sometimes as much as 300%. The bias increases systematically as the ratio of standard deviations of treatment groups increases and remains fairly constant for various sample sizes. There is no indication that Type I error probabilities approach the significance level asymptotically as sample size increases.