Richard H. Williams

Are Simple Gain Scores Obsolete

It is widely believed that measures of gain, growth, or change, expressed as simple differences between pretest and posttest scores, are inherently unreliable. It is also believed that gain scores lack predictive validity with respect to other criteria. However, these conclu sions are based on misleading assumptions about the values of parameters in familiar equations in classical test theory. The present paper examines modified equa tions for the validity and reliability of difference scores that describe applied testing situations more realisti cally and reveal that simple gain scores can be more useful in research than commonly believed.

The effects of albuterol and isokinetic exercise on the quadriceps muscle group.

Subjects performed 9 wk of isokinetic knee extensions twice weekly. Albuterol (N = 13) or placebo (N = 9) was administered for 6 wk; groups received 16 mg.d-1 of either treatment. Training consisted of three sets of 10 repetitions at 45 degrees.s-1. Data were collected at weeks 0, 6, and 9. Concentric and eccentric variables examined included: peak torque (CPT, EPT), total work (CTW, ETW), average power (CAP, EAP), time to peak torque (CTTPT, ETTPT), peak torque to body weight ratio (CPT/BW), and work to body weight ratio (CW/BW, EW/BW). Other variables included: thigh circumference (CIRC), thigh cross-sectional area (CSA), forced vital capacity (FVC), and forced expiratory volume (FEV1), MANOVA and the Dunn-Bonferroni post-hoc found differences within groups for CPT, CTW, CAP, CPR/BW, EPT, ETTPT, and CSA. Interactions were noted for CW/BW, ETW, EAP, EPT/BW, and ETW/BW; with persons administered albuterol yielding superior values. CW/BW, ETW, and EAP showed interactions at post-testing, while EPT/BW and EW/BW interacted at both midtesting and post-testing. Results indicate therapeutic doses of albuterol administered with resistance exercise may augment strength gains.

Properties of the Spearman Correction for Attenuation for Normal and Realistic Non-Normal Distributions

Results are presented of a computer simulation study of the Spearman correction for attenuation using a design originally suggested by Spearman (1904). Two parallel measures were generated for each of two variables with predetermined distribution shapes, correlations between true scores, and population reliability coefficients. The resulting data consisted of error scores, observed scores, sample reliability coefficients, and sample validity coefficients. The correction for attenuation was performed, and means, variances, and relative frequency distributions of both the uncorrected and corrected validity coefficients were analyzed for normal and non-normal distributions. For varying sample sizes and for all population distributions, the means of the corrected sample correlations were very close to the correlation between true scores, provided that the population reliability coefficients were fairly high. The variability of the corrected sample correlations was substantial, even for larger sample sizes. For lower reliability values, there was pronounced overcorrection, combined with extreme variability, especially for smaller sample sizes. Under these conditions, corrections exceeding 1.00 were frequent. The correction for attenuation appears to be useful only if the reliability coefficients of both measures are relatively high and sample size is relatively large. The properties of the correction for attenuation appear to be independent of the shape of the population distribution of test scores, at least for distributions commonly encountered in psychological and educational research.

The relative error magnitude in three measures of change

Formulas for the standard error of measurement of three measures of change—simple difference scores, residualized difference scores, and the measure introduced by Tucker, Damarin, and Messick—are derived. Equating these formulas by pairs yields additional explicit formulas which provide a practical guide for determining the relative error of the three measures in any pretest-posttest design. The functional relationship between the standard error of measurement and the correlation between pretest and posttest observed scores remains essentially the same for each of the three measures despite variations in other test parameters (reliability coefficients, standard deviations), even when pretest and posttest errors of measurement are correlated.

Reliability of Measurement and Power of Significance Tests Based on Differences

The power of significance tests based on differ ence scores is indirectly influenced by the reliability of the measures from which differences are obtained. Reliability depends on the relative magnitude of true score and error score variance, but statistical power is a function of the absolute magnitude of these components. Explicit power calculations reaffirm the paradox put forward by Overall & Woodward (1975, 1976)—that significance tests of differences can be powerful even if the reliability of the difference scores is 0. This anomaly arises because power is a function of observed score variance but is not a function of reliability unless either true score variance or error score variance is constant. Provided that sample size, significance level, directionality, and the alternative hypothesis associated with a significance test remain the same, power always increases when population variance decreases, independently of reliability.

The theory of test validity and correlated errors of measurement

Abstract In the theory of test validity it is assumed that error scores on two distinct tests, a predictor and a criterion, are uncorrelated. The expected-value concept of true score in the calssical test-theory model as formulated by Lord and Novick, Guttman, and others, implies mathematically, without further assumptions, that true scores and error scores are uncorrelated. This concept does not imply, however, that error scores on two arbitrary tests are uncorrelated, and an additional axiom of “experimental independence” is needed in order to obtain familiar results in the theory of test validity. The formulas derived in the present paper do not depend on this assumption and can be applied to all test scores. These more general formulas reveal some unexpected and anomalous properties of test validty and have implications for the interpretation of validity coefficients in practice. Under some conditions there is no attenuation produced by error of measurement, and the correlation between observed scores sometimes can exceed the correlation between true scores, so that the usual correction for attenuation may be inappropriate and misleading. Observed scores on two tests can be positively correlated even when true scores are negatively correlated, and the validity coefficient can exceed the index of reliability. In some cases of practical interest, the validity coefficient will decrease with increase in test length. These anomalies sometimes occur even when the correlation between error scores is quite small, and their magnitude is inversely related to test reliability. The elimination of correlated errors in practice will not enhance a tests predictive value, but will restore the properties of the validity coefficient that are familiar in the classical theory.

Statistical Power Analysis and Reliability of Measurement

Abstract Reliability of measurement as a determinant of the power of significance tests is investigated using concepts from statistical power analysis and from test and measurement theory. Results of specific power calculations based on measures of effect size together with particular reliability coefficients are presented in the form of tables. The importance of assumptions about true variance, error variance, and observed variance in determining the relation between reliability and power is emphasized, and a formula relating power and test length, analogous to the Spearman-Brown formula in test theory, is derived.

CHANCE SUCCESS DUE TO GUESSING AND NON-INDEPENDENCE OF TRUE SCORES AND ERROR SCORES IN MULTIPLE-CHOICE TESTS: COMPUTER TRIALS WITH PREPARED DISTRIBUTIONS

The effect of chance success due to guessing upon the variance of multiple-choice test scores was estimated from prepared distributions of large numbers of scores. Each score consisted of an assumed “true score” component and an “error score” component generated by a computer. A large negative correlation was found between true scores and error scores and a positive correlation between error scores and error scores. The equation showing reliability in terms of components of variance was derived under the more restrictive assumption that there is a correlation between true scores and error scores, and the result r o 1 o 2 = 1 − [ ( s o 2 / s o 2 ) ( 1 − r o 1 o 2 ) ] was obtained. The fact that reliability can be positive even though error variance and observed variance are equal was discussed.

Restriction of Range and Correlation in Outlier-Prone Distributions.

Statistical theory indicates that restriction of the range of possible values of normally distributed variables, and many non-normal variables, reduces correlations in unrestricted populations. Contrary to this typical outcome, results of a simulation study show that range restriction sometimes increased the correlation between variables having outlier-prone distributions. This result occurred in the case of exponential and ex-Gaussian distributions, which are encountered in experimental studies involving response times. It did not occur in truncated versions of the same densities. Chance occurrence of outliers in contaminated-normal, or mixed-normal, distributions reduced the correlation found between samples from uncontaminated populations. Conversely, detection and downweighting of outliers increased the magnitude of sample correlations, and a similar result occurred for many other outlier-prone distributions. Practical implications of these findings are discussed.

Effect of nonindependence of sample observations on some parametric and nonparametric statistical tests

The probabilities of Type I and Type II errors of the Student t test, as well as the Mann-Whitney-Wilcoxon test, are grossly inflated or deflated by violation of within-group and between-group independence of sample observations. A modified t formula with two correlation