Shlomo S. Sawilowsky

A more realistic look at the robustness and type II error properties of the t test to departures from population normality

The Type I and II error properties of the t test were evaluated by means of a Monte Carlo study that sampled 8 real distribution shapes identified by Micceri (1986, 1989) as being representative of types encountered in psychology and education research. Results showed the independent-samples t tests to be reasonably robust to Type I error when (a) sample sizes are equal, (b) sample sizes are fairly large, and (c) tests are two-tailed rather than one-tailed. Nonrobust results were obtained primarily under distributions with extreme skew. The / test was robust to Type II error under these nonnormal distributions, but researchers should not overlook robust nonparametric competitors that are often more powerful than the t test when its underlying assumptions are violated. Along with Pearsons chi-squared test, the independent-samples t test must be counted among the best-known statistical procedures in current use. Given its familiarity and utility, it is not surprising that over the years, this test has received an inor

Implementation of Self-Determination Activities and Student Participation in IEPs

The Council for Exceptional Children conducted an online Web survey to obtain information on the instructional practices and attitudes of educators as they relate to self-determination and student involvement in the individualized education program (IEP) process. We obtained 523 usable responses from teachers, administrators, and related services professionals. Although respondents highly valued both student involvement in IEPs and self-determination skills, only 8% were satisfied with the approach they were using to teach self-determination. Only 34% were satisfied with the level of student involvement in IEP meetings. Implications include the need for longitudinal research and technical assistance, targeting administrators, general educators, and special educators beginning in the elementary grades, to improve the capacity of schools to deliver self-determination instruction.

Increasing physicians' awareness of the impact of statistics on research outcomes: Comparative power of the t-test and Wilcoxon Rank-Sum test in small samples applied research

To effectively evaluate medical literature, practicing physicians and medical researchers must understand the impact of statistical tests on research outcomes. Applying inefficient statistics not only increases the need for resources, but more importantly increases the probability of committing a Type I or Type II error. The t-test is one of the most prevalent tests used in the medical field and is the uniformally most powerful unbiased test (UMPU) under normal curve theory. But does it maintain its UMPU properties when assumptions of normality are violated? A Monte Carlo investigation evaluates the comparative power of the independent samples t-test and its nonparametric counterpart, the Wilcoxon Rank-Sum (WRS) test, to violations from population normality, using three commonly occurring distributions and small sample sizes. The t-test was more powerful under relatively symmetric distributions, although the magnitude of the differences was moderate. Under distributions with extreme skews, the WRS held large power advantages. When distributions consist of heavier tails or extreme skews, the WRS should be the test of choice. In turn, when population characteristics are unknown, the WRS is recommended, based on the magnitude of these power differences in extreme skews, and the modest variation in symmetric distributions.

SIMULATING CORRELATED MULTIVARIATE NONNORMAL DISTRIBUTIONS: EXTENDING THE FLEISHMAN POWER METHOD

A procedure for generating multivariate nonnormal distributions is proposed. Our procedure generates average values of intercorrelations much closer to population parameters than competing procedures for skewed and/or heavy tailed distributions and for small sample sizes. Also, it eliminates the necessity of conducting a factorization procedure on the population correlation matrix that underlies the random deviates, and it is simpler to code in a programming language (e.g., FORTRAN). Numerical examples demonstrating the procedures are given. Monte Carlo results indicate our procedure yields excellent agreement between population parameters and average values of intercorrelation, skew, and kurtosis.

Limitations of the rank transform statistic in tests for interactions

Conover and Iman (1976), Iman (1974), and Iman and Conover (1976) have found the rank transform test to be robust and powerful when testing for interaction in experimental designs.The current study shows that, insofar as tests for interactions are concerned, the rank transform test is robust and powerful in some circumstances but is dramatically nonrobust and manifests power significantly below that of the usual F test in some cases. Therefore, this procedure should be used only withcaution when employed in designs suchas those examined here.

An Investigation of the Type I Error and Power Properties of the Rank Transform Procedure in Factorial ANOVA

This study examined the Type I error and power properties of the rank transform test when employed in the context of a balanced 2x2x2 fixed effects ANOVA. The results showed the rank transform procedure to be erratic with respect to both Type I error and power. Under some circumstances the test was both robust and powerful, whereas in other circumstances it was decidedly nonrobust and manifested power considerably below that of the usual ANOVA F test. It is recommended that researchers avoid this test except in those specific circumstances where its properties are well understood.

Constructive Criticisms of Methodological and Editorial Practices

Abstract Thompson (1999a and elsewhere) has taken strong positions on a variety of methodological issues. In this article, the authors critique some of those positions and provide alternative views for each.

Measurement practices: methods for developing content-valid student examinations

Measurement experts generally agree that a systematic approach to test construction will probably result in an instrument with sound psychometric properties. One fundamental method is called the blueprint approach to test construction. A test blueprint is a tool used in the process for generating content-valid exams by linking the subject matter delivered during instruction and the items appearing on the test. Unfortunately, this procedure as well as other educational measurement practices is often overlooked. A survey of curriculum administrators at 144 United States and international medical schools was conducted to assess the importance and prevalence of test blueprinting in their school. Although most found test blueprinting to be very important, few require the practice. The purpose of this paper is to review the fundamental principals associated with achieving a high level of content validity when developing tests for students. The short-term efforts necessary to develop and integrate measurement theory into practice will lead to long-term gains for students, faculty and academic institutions.

Properties of the rank transformation in factorial analysis of covariance

Real world data often fail to meet the underlying assumption of population normality. The Rank Transformation (RT) procedure has been recommended as an alternative to the parametric factorial analysis of covariance (ANCOVA). The purpose of this study was to compare the Type I error and power properties of the RT ANCOVA to the parametric procedure in the context of a completely randomized balanced 3 × 4 factorial layout with one covariate. This study was concerned with tests of homogeneity of regression coefficients and interaction under conditional (non)normality. Both procedures displayed erratic Type I error rates for the test of homogeneity of regression coefficients under conditional nonnormality. With all parametric assumptions valid, the simulation results demonstrated that the RT ANCOVA failed as a test for either homogeneity of regression coefficients or interaction due to severe Type I error inflation. The error inflation was most severe when departures from conditional normality were extreme. Also associated with the RT procedure was a loss of power. It is recommended that the RT procedure not be used as an alternative to factorial ANCOVA despite its encouragement from SAS, IMSL, and other respected sources.

Weighted Simplex Procedures for Determining Boundary Points and Constants for the Univariate and Multivariate Power Methods

The power methods are simple and efficient algorithms used to generate either univariate or multivariate nonnormal distributions with specified values of (marginal) mean, standard deviation, skew, and kurtosis. The power methods are bounded as are other transformation techniques. Given an exogenous value of skew, there is an associated lower bound of kurtosis. Previous approximations of the boundary for the power methods are either incorrect or inadequate. Data sets from education and psychology can be found to lie within, near, or outside tile boundary of the power methods. In view of this, we derived necessary and sufficient conditions using the Lagrange multiplier method to determine the boundary of the power methods. The conditions for locating and classifying modes for distributions on the boundary were also derived. Self-contained interactive Fortran programs using a Weighted Simplex Procedure were employed to generate tabled values of minimum kurtosis for a given value of skew and power constants for various (non)normal distributions.