Neal M. Kingston

Comparability of Computer- and Paper-Administered Multiple-Choice Tests for K–12 Populations: A Synthesis

There have been many studies of the comparability of computer-administered and paper-administered tests. Not surprisingly (given the variety of measurement and statistical sampling issues that can affect any one study) the results of such studies have not always been consistent. Moreover, the quality of computer-based test administration systems has changed considerably over recent years, as has the computer-experience of students. This study synthesizes the results of 81 studies performed between 1997 and 2007. The estimated effect size across all studies was very small (–.01 weighted, .00 unweighted). Meta-analytic methods were used to ascertain whether grade (elementary, middle, or high school) or subject (English Language Arts, Mathematics, Reading, Science, or Social Studies) had an impact on comparability. Grade appeared to have no affect on comparability. Subject did appear to affect comparability, with computer administration appearing to provide a small advantage for English Language Arts and Social Studies test (effect sizes of .11 and .15, respectively), and paper administration appearing to provide a small advantage for Mathematics tests (effect size of −.06).

The Analysis of Item-Ability Regressions: An Exploratory IRT Model Fit Tool

The use of item-ability regressions (the comparison of the regression of the observed proportion of people answering an item correctly on estimated θ with the estimated item response function) to investigate the psychometric properties of particular item types in a given population was explored using data from four administrations of 10 item types (a total of 806 items) from the Graduate Record Examinations General Test. Although the method does not allow an absolute deter mination of fit for a latent trait model (in this case, for the three-parameter logistic model), it does show that certain item types consistently fit the model worse than other item types, and it led to and supported a specific hypothesis as to why the model probably did not fit these item types.

The Relationship Among Self-Determination, Self-Concept, and Academic Achievement for Students With Learning Disabilities

Research suggests that self-determination skills are positively correlated with factors that have been shown to improve academic achievement, but the direct relationship among self-determination, self-concept, and academic achievement is not fully understood. This study offers an empirical explanation of how self-determination and self-concept affect academic achievement for adolescents with learning disabilities after taking into consideration the covariates of gender, income, and urbanicity. In a nationally representative sample (N = 560), the proposed model closely fit the data, with all proposed path coefficients being statistically significant. The results indicated that there were significant correlations among the three latent variables (i.e., self-determination, self-concept, and academic achievement), with self-determination being a potential predictor of academic achievement for students with learning disabilities.

Elementary Teachers’ Knowledge and Use of Interventions and Barriers to Promoting Student Self-Determination

The No Child Left Behind Act (2001) and the Individuals with Disabilities Education Improvement Act (2004) emphasize accountability to improve student academic achievement. Promoting self-determination has been proposed as a means to achieving this outcome. Elementary teachers in 30 states were surveyed to measure (a) their perceived importance of self-determination, (b) to what extent they teach it, and (c) the barriers that inhibit them from teaching it. Both general and special educators assigned considerable importance to self-determination instruction and reported at least occasionally providing such instruction. The authors found a lack of congruence between the value teachers place on promoting self-determination and the time they devote to teaching it. Limitations and implications are discussed, and suggestions for future research are offered.

The Effect of Social and Classroom Ecological Factors on Promoting Self-Determination in Elementary School.

The authors surveyed 233 elementary special educators in 23 states to determine (a) how the teaching of self-regulation strategies and classroom setting affected their perceptions of the importance of teaching self-determination, (b) the frequency with which they did so, and (c) the barriers to promoting self-determination. Results indicated that the frequency of teaching self-determination, more than the ratings of the importance of teaching it, was affected by the teaching of self-regulation strategies. Classroom setting affected only the perception of importance of teaching self-determination, and then only for teachers who taught some of the self-regulation strategies. The authors discuss the limitations and implications as well as offer suggestions for future research.

Why IEP Teams Assign Low Performers With Mild Disabilities to the Alternate Assessment Based on Alternate Achievement Standards

The purpose of this case study was to determine teachers’ rationales for assigning students with mild disabilities to alternate assessment based on alternate achievement standards (AA-AAS). In interviews, special educators stated that their primary considerations in making the assignments were low academic performance, student use of extended standard modifications, and the inflexible 1% cap. None of the teachers provided their students with grade-level content or appropriate modifications. Some students were competent in grade-level reading, but were assigned to the 2010 AA-AAS because read-aloud accommodation is not permitted in the reading passages of the general assessment. Findings raised concerns about the susceptibility of eligibility guidelines to lead to subjective decisions. Future research, implications of these findings, and limitations of the study are discussed.

Examining Teachers’ Decisions on Test-Type Assignment for Statewide Assessments

Recent studies have raised concerns about the vagueness of alternate assessment eligibility guidelines, specifically, that students with mild disabilities (SWMD) have been inappropriately assigned to alternate assessment–alternate achievement standards (AA-AAS). In this study, special education teachers (N = 317) were surveyed about SWMD in vignettes to explore (a) perceptions of the clarity of eligibility guidelines, (b) teachers’ test-type decisions, (c) factors considered when making test-type decisions, and (d) teachers’ priority rankings of these factors. Teachers’ test-type decisions varied by state, subjectivity, and noninstructional factors. Findings raise concerns about the inconsistency and lack of specificity of the current state guidelines, their vulnerability to subjective interpretation of eligibility, and states’ different alternate assessment systems. Directions for future research, implications of the current findings, and limitations are discussed.

Can today's standardized achievement tests yield instructionally useful data? Challenges, promises and the state of the art

Purpose – Against a backdrop of high-stakes assessment policies in the USA, this paper explores the challenges, promises and the “state of the art” with regard to designing standardized achievement tests and educational assessment systems that are instructionally useful. Authors deliberate on the consequences of using inappropriately designed tests, and in particular tests that are insensitive to instruction, for teacher and/or school evaluation purposes. Methodology/approach – The method used is a “moderated policy discussion”. The six invited commentaries represent voices of leading education scholars and measurement experts, juxtaposed against views of a prominent leader and nationally recognized teacher from two American education systems. The discussion is moderated with introductory and concluding remarks from the guest editor, and is excerpted from a recent blog published by Education Week. References and author biographies are presented at the end of the article. Findings – In the education assess...

Fostering Inclusion of Students With Significant Cognitive Disabilities by Using Learning Map Models and Map-Based Assessments

Abstract Although there is widespread agreement among both special education experts and general classroom teachers that students with significant cognitive disabilities should participate in inclu...

Standard Errors and Confidence Intervals from Bootstrapping for Ramsay-Curve Item Response Theory Model Item Parameters.

Ramsay-curve item response theory (RC-IRT) is a nonparametric procedure that estimates the latent trait using splines, and no distributional assumption about the latent trait is required (Woods & Thissen, 2006). Description of this procedure can be found, for example, in the technical manual of RCLOG v.2, software for RC-IRT (Woods, 2006b). For item parameters of the twoparameter logistic (2-PL), three-parameter logistic (3-PL), and polytomous IRT models, RC-IRT can provide more accurate estimates than the commonly used marginal maximum likelihood estimation (MMLE) when the latent trait is not normally distributed (Woods, 2006a, 2007, 2008). However, standard errors (SEs) for the item parameter estimates have not been developed in RC-IRT as no analytical solution is readily available (Woods, 2006a, 2007, 2008; Woods & Lin, 2009). In such cases, bootstrapping provides an alternative way to estimate SEs. Using bootstrapping, the observed sample is treated as the pseudopopulation from which n repeated random samples are drawn with replacement. The same estimation procedure is employed on each random sample and the point estimates are retained. Then, the SE of a particular parameter estimate is the standard deviation of the retained estimates, and the associated confidence interval (CI) can be determined by two percentiles. For example, a 95% CI can be determined by the range between the 2.5th and 97.5th percentiles. In this research, bootstrapping was utilized to estimate SEs and CIs for item parameters in the 2-PL model, and the performance of bootstrapping was compared with that of MMLE.