Bruno D. Zumbo

Coefficient Alpha as an Estimate of Test Reliability Under Violation of Two Assumptions

Through use of computer simulation, the central tendency and variability of coefficient alpha were examined under violation of two assumptions made in the derivation of the formula. When assumptions were satisfied, the mean value of coefficient alpha was extremely close to the population reliability coefficient, but values were highly variable. This result was independent of the shape of the population distribution of test scores. Coefficient alpha underestimated reliability under violation of the assumption of essential tauequivalence of subtest scores and overestimated reliability under violation of the assumption of uncorrelated subtest error scores. In both cases, the bias of the estimates varied systematically with the degree of violation of assumptions, while the variability of the estimates remained constant. All these results were independent of the number of persons and the number of subtests.

Relative Power of the Wilcoxon Test, the Friedman Test, and Repeated-Measures ANOVA on Ranks

Abstract Many introductory statistics textbooks in education, psychology, and the social sciences consider the Friedman test to be a nonparametric counterpart of repeated-measures ANOVA, just as the Kruskal-Wallis test is a counterpart of oneway ANOVA. However, it is known in theoretical statistics that the Friedman test is a generalization of the sign test and possesses the modest statistical power of the sign test for normal as well as many nonnormal distributions. Although not familiar to researchers, another significance test that can be regarded as a nonparametric counterpart of repeated-measures ANOVA is a rank-transformation procedure, in which the usual parametric statistical analysis is performed on ranks replacing the original scores. In the present computer simulation study we compared the ordinary paired-samples Student t test, the Wilcoxon signed-ranks test, and the sign test for correlated samples from normal, uniform, mixed-normal, exponential, Laplace, and Cauchy distributions, for which t...

Reliability of Measurement and Power of Significance Tests Based on Differences

The power of significance tests based on differ ence scores is indirectly influenced by the reliability of the measures from which differences are obtained. Reliability depends on the relative magnitude of true score and error score variance, but statistical power is a function of the absolute magnitude of these components. Explicit power calculations reaffirm the paradox put forward by Overall & Woodward (1975, 1976)—that significance tests of differences can be powerful even if the reliability of the difference scores is 0. This anomaly arises because power is a function of observed score variance but is not a function of reliability unless either true score variance or error score variance is constant. Provided that sample size, significance level, directionality, and the alternative hypothesis associated with a significance test remain the same, power always increases when population variance decreases, independently of reliability.

Parametric Alternatives to the Student T Test under Violation of Normality and Homogeneity of Variance

Introductory statistics textbooks in psychology, education, and social sciences have contributed to the belief that nonparametric tests, such as the Wilcoxon-Mann-Whitney test, are effective against violations of both normality and homogeneity of variance. The present paper emphasizes that, although rank methods often are useful when samples are obtained from heavy-tailed, nonnormal distributions, they are influenced by unequal variances just like parametric tests. Computer programs are now available to perform modified t tests based on unequal sample variances, in which degrees of freedom and critical values are altered from sample to sample. These procedures, although neglected for many years because they are computationally complex, are far more effective than nonparametric methods in protecting against violation of homogeneity of variance.

Further evidence for Coren and Hakstian's "Methodological implications of interaural correlation: Count heads not ears" and an alternative correction formula

Coren and Hakstian (1990) identified a serious methodological problem that arises in auditory research because of interaural correlation. When measures from both ears of the subjects are pooled together in an experimental design that assumes independence of measures, there can be spuriously high apparent statistical significance. The present paper provides further evidence in support of Coren and Hakstian’s argument and also derives a formula that effectively corrects inflated test statistics resulting from interaural correlation. This formula is a special case of a more general one that applies in many other experimental contexts in which nonindependence of measures is a problem. We found that statistical tests based on our formula have somewhat greater power to detect differences than the kind of correction method advocated by Coren and Hakstian.

Significance testing of correlation using scores, ranks, and modified ranks

A computer simulation study compared significance tests of correlation coefficients calculated from initial scores, from ranks assigned by the Spearman method, and from three kinds of modified ranks in which N sample values were replaced by N12, N/3, and N14 integers. Tests based on the initial scores are more powerful than those based on the various ranks for normal distributions, whereas the reverse is true for mixed-normal, exponential, and Cauchy distributions. Probabilities of Type I and Type II errors are unaffected by reduction in the number of ranks. Implications of these findings for the notion that rank correlation is a nonparametric correlation method are discussed.

Reliability, power, functions, and relations: A reply to Humphreys

a conductor?&dquo; Calculations based on simple equations provide specific values of acceleration and of current, and the same might be true of statistical power. As pointed out in our earlier papers (Williams & Zimmerman, 1989; Zimmerman & Williams, 1986; Zimmerman, Williams, & Zumbo, 1993), however, power is not a mathematical function of reliability unless either true score variance or error score variance is constant. Reliability is determined entirely by the relative magnitudes of true score variance and observed score variance.

Correction of the Student T Statistic for Nonindependence of Sample Observations

A computer-simulation study examined the one-sample Student t test under violation of the assumption of independent sample observations. The probability of Type I errors increased, and the probability of Type II errors decreased, spuriously elevating the entire power function. The magnitude of the change depended on the correlation between pairs of sample values as well as the number of sample values that were pairwise correlated. A modified t statistic, derived from an unbiased estimate of the population variance that assumed only exchangeable random variables instead of independent, identically distributed random variables, effectively corrected for nonindependence for all degrees of correlation and restored the probability of Type I and Type II errors to their usual values.

The Comparative Reliability of Simple and Residualized Difference Scores: A Corrigendum

Abstract This short note contains a correction to an essential result in Williams and Zimmerman (1983). Given this correction, the algebraic inequalities presented by Williams and Zimmerman are translated into a series of questions a researcher can ask him/herself to help decide between the simple or residualized difference scores. This correction and these decision rules should encourage practitioners to make better use of Williams and Zimmermans findings.

Correction for Nonindependence of Sample Observations in ANOVA "F" Tests

Abstract This article derived a modified F test that includes a correction for nonindependence of between-groups and within-groups sample values in ANOVA designs. A computer-simulation study induced correlations between and within treatment groups and examined their influence on the probability of Type I and Type II errors for normal and non-normal distributions. Nonindependence severely altered the usual F tests in one-way ANOVA for all distributions. Between-groups correlations, which can result from violations of random assignment, depressed the probability of Type I errors far below the nominal significance level and markedly increased the probability of Type II errors. Within-groups correlations, which can result from violations of random sampling, had the reverse effect. The modified F test restored these probabilities to their previous values and was more powerful than conventional within-subjects ANOVA. Non-normality and nonindependence were additive. An ordinary F test performed on the ranks of s...