Michael R. Harwell

Summarizing Monte Carlo Results in Methodological Research: The One- and Two-Factor Fixed Effects ANOVA Cases

Meta-analytic methods were used to integrate the findings of a sample of Monte Carlo studies of the robustness of the F test in the one- and two-factor fixed effects ANOVA models. Monte Carlo results for theWelch (1947) and Kruskal-Wallis (Kruskal & Wallis, 1952) tests were also analyzed. The meta-analytic results provided strong support for the robustness of the Type I error rate of the F test when certain assumptions were violated. The F test also showed excellent power properties. However, the Type I error rate of the F test was sensitive to unequal variances, even when sample sizes were equal. The error rate of the Welch test was insensitive to unequal variances when the population distribution was normal, but nonnormal distributions tended to inflate its error rate and to depress its power. Meta-analytic and exact statistical theory results were used to summarize the effects of assumption violations for the tests.

Monte Carlo Studies in Item Response Theory

Monte carlo studies are being used in item response theory (IRT) to provide information about how validly these methods can be applied to realistic datasets (e.g., small numbers of examinees and multidimensional data). This paper describes the conditions under which monte carlo studies are appropriate in IRT-based re search, the kinds of problems these techniques have been applied to, available computer programs for gen erating item responses and estimating item and exam inee parameters, and the importance of conceptualizing these studies as statistical sampling experiments that should be subject to the same principles of experimen tal design and data analysis that pertain to empirical studies. The number of replications that should be used in these studies is also addressed.

Student Eligibility for a Free Lunch as an SES Measure in Education Research

The use of eligibility for a free lunch as a measure of a student’s socioeconomic status continues to be a fixture of quantitative education research. Despite its popularity, it is unclear that education researchers are familiar with what student eligibility for a free lunch does (and does not) represent. The authors examine the National School Lunch Program, which is responsible for certifying students as eligible for a free lunch, and conclude that free lunch eligibility is a poor measure of socioeconomic status, which suffers from important deficiencies that can bias inferences. A table characterizing key strengths and weaknesses of variables used as measures of socioeconomic status is provided to facilitate comparisons.

Rescaling Ordinal Data to Interval Data in Educational Research

Many statistical procedures used in educational research are described as requiring that dependent variables follow a normal distribution, implying an interval scale of measurement. Despite the desirability of interval scales, many dependent variables possess an ordinal scale of measurement in which the differences among values composing the scale are unequal in terms of what is being measured, permitting only a rank ordering of scores. This means that data possessing an ordinal scale will not satisfy the assumption of normality needed in many statistical procedures and may produce biased statistical results that threaten the validity of inferences. This article shows how the measurement technique known as item response theory can be used to rescale ordinal data to an interval scale. The authors provide examples of rescaling using student performance data and argue that educational researchers should routinely consider rescaling ordinal data using item response theory.

An Empirical Study of the Effects of Small Datasets and Varying Prior Variances on Item Parameter Estimation in BILOG

Long-standing difficulties in estimating item parameters in item response theory (IRT) have been addressed recently with the application of Bayesian estimation models. The potential of these methods is enhanced by their availability in the BILOG com puter program. This study investigated the ability of BILOG to recover known item parameters under varying conditions. Data were simulated for a two- parameter logistic IRT model under conditions of small numbers of examinees and items, and different variances for the prior distributions of discrimina tion parameters. The results suggest that for samples of at least 250 examinees and 15 items, BILOG accurately recovers known parameters using the default variance. The quality of the estimation suffers for smaller numbers of examinees under the default variance, and for larger prior variances in general. This raises questions about how practi tioners select a prior variance for small numbers of examinees and items.

Summarizing Monte Carlo Results in Methodological Research

Monte Carlo studies provide information that can assist researchers in selecting a statistical test when underlying assumptions of the test are violated. Effective use of this literature is hampered by the lack of an overarching theory to guide the interpretation of Monte Carlo studies. The problem is exacerbated by the impressionistic nature of the studies, which can lead different readers to different conclusions. These shortcomings can be addressed using meta-analytic methods to integrate the results of Monte Carlo studies. Quantitative summaries of the effects of assumption violations on the Type I error rate and power of a test can assist researchers in selecting the best test for their data. Such summaries can also be used to evaluate the validity of previously published statistical results. This article provides a methodological framework for quantitatively integrating Type I error rates and power values for Monte Carlo studies. An example is provided using Monte Carlo studies of Bartlett’s (1937) test of equality of variances. The importance of relating meta-analytic results to exact statistical theory is emphasized.

Distributive Justice Development: Cross-Cultural, Contextual, and Longitudinal Evaluations.

ENRIGHT, ROBERT D.; BJERSTEDT, AKE; ENRIGHT, WILLIAM F.; LEVY, VICTOR M., JR.; LAPSLEY, DANIEL K.; Buss, RAY R.; HARWELL, MICHAEL; and ZINDLER, MONICA. Distributive Justice Development: Cross-cultural, Contextual, and Longitudinal Evaluations. CHILD DEVELOPMENT, 1984, 55, 1737-1751. The development of distributive justice was examined with the Distributive Justice Scale (DJS) in 3 studies. In Study 1, 176 children, ages 7, 9, and 11, from Sweden and the United States were given the DJS and 2 Piagetian logical reasoning tasks. Significant age trends in DJS scores and the relation with logical reasoning were comparable in the 2 cultures. In Study 2, 75 5and 7-year-old children were given the standard peer DJS and a comparable family DJS to assess reasoning in different contexts. Family stimuli elicited higher levels of reasoning than peer stimuli. In Study 3, 84 6and 9-year-old children were administered the DJS twice at 1-year intervals. Age trends with no cohort biases were found. Implications for distributive justice research are drawn.

A Nonparametric Test Statistic for the General Linear Model

Puri and Sen (1969Puri and Sen (1985) presented a nonparametric test statistic based on a general linear model approach that is appropriate for testing a wide class of hypotheses. The two forms of this statistic, pure- and mixed-rank, differ according to whether the original predictor values or their ranks are used. Both forms permit the use of standard statistical packages to perform the analyses. The applicability of these statistics in testing a number of hypotheses is highlighted, and an example of their use is given. A simulation study for the multivariate-multiple-regression case is used to examine the distributional behavior of the pure- and mixed-rank statistics and an important competitor, the rank transformation of Conover and Iman (1981). The results suggest that the pure- and mixed-rank statistics are superior with respect to minimizing liberal Type I error rates, whereas the Conover and Iman statistic produces larger power values.

Muscle strength as an indicator of the habitual level of physical activity

This study focused on age and physical activity as determinants of muscle strength. The study involved 620 women 25-73 yr of age. The five muscle groups assessed were: grip, plantarflexors, hip abductors, trunk flexors, and trunk extensors. Pearson correlations yielded significant negative correlations of muscle strength with age and positive correlations with height as well as physical activity. The greatest decremental differences in muscle strength were registered in the perimenopausal years between the age decades of 45-54 yr and 55-64 yr. In stepwise regression analyses age was the strongest predictor of the strength of all muscle groups, with smaller contributions to the variance by physical activity and anthropometric variables. When the sample population, divided by decades of age, was further subdivided by tertiles of physical activity, the results of factorial analysis indicated that the main effects due to age and physical activity were significant. It was concluded that 1) moderate levels of physical activity tend to improve muscle strength even in older women, and 2) normative values of muscle strength could serve as an indicator of the adequacy of the habitual levels of physical activity.

The Preparation of Students From National Science Foundation–Funded and Commercially Developed High School Mathematics Curricula for Their First University Mathematics Course

The selection of K–12 mathematics curricula has become a polarizing issue for schools, teachers, parents, and other educators and has raised important questions about the long-term influence of these curricula. This study examined the impact of participation in either a National Science Foundation–funded or commercially developed mathematics curriculum on the difficulty level of the first university mathematics course a student enrolled in and the grade earned in that course. The results provide evidence that National Science Foundation–funded curricula do not prepare students to initially enroll in more difficult university mathematics courses as well as commercially developed curricula, but once enrolled students earn similar grades. These findings have important implications for high school mathematics curriculum selection and for future research in this area.