J. Patrick Meyer

A Mixture Rasch Model with Item Response Time Components

An examinee faced with a test item will engage in solution behavior or rapid-guessing behavior. These qualitatively different test-taking behaviors bias parameter estimates for item response models that do not control for such behavior. A mixture Rasch model with item response time components was proposed and evaluated through application to real test data and a simulation study. The analysis of extant data indicated that a two-class solution fit better than a one-class solution and that 15% of examinees engaged in rapid-guessing behavior. Moreover, solution behavior examinees had substantially higher average ability scores than rapid-guessing examinees. Results of the simulation study indicated that the parameters were recovered well.

Using Evidence-Based Multimedia to Improve Vocabulary Performance of Adolescents With LD: A UDL Approach

Universal Design for Learning (UDL) is a framework that is commonly used for guiding the construction and delivery of instruction intended to support all students. In this study, we used a related model to guide creation of a multimedia-based instructional tool called content acquisition podcasts (CAPs). CAPs delivered vocabulary instruction during two concurrent social studies units to 32 SWD and 109 students without disabilities. We created CAPs using a combination of evidence-based practices for vocabulary instruction, UDL, and Mayer’s instructional design principles. High school students with and without learning disabilities completed weekly curriculum-based measurement (CBM) probes (vocabulary matching) over an 8-week period along with two corresponding posttests. Students were nested within sections of world history and randomly assigned to alternating treatments (CAPs and business as usual) that were administered sequentially to each group. Results revealed that students with and without disabilities made significant growth on CBMs and scored significantly higher on the posttests when taught using CAPs.

The Effect of Observation Length and Presentation Order on the Reliability and Validity of an Observational Measure of Teaching Quality

Observational methods are increasingly being used in classrooms to evaluate the quality of teaching. Operational procedures for observing teachers are somewhat arbitrary in existing measures and vary across different instruments. To study the effect of different observation procedures on score reliability and validity, we conducted an experimental study that manipulated the length of observation and order of presentation of 40-minute videotaped lessons from secondary grade classrooms. Results indicate that two 20-minute observation segments presented in random order produce the most desirable effect on score reliability and validity. This suggests that 20-minute occasions may be sufficient time for a rater to observe true characteristics of teaching quality assessed by the measure used in the study, and randomizing the order in which segments were rated may reduce construct irrelevant variance arising from carry over effects and rater drift.

Occasions and the Reliability of Classroom Observations: Alternative Conceptualizations and Methods of Analysis

Student–teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of student–teacher interaction or simply reflect measurement error. Classroom observation protocols should be designed to minimize measurement error while allowing measureable changes in the construct of interest. Treating occasions as fixed multivariate outcomes allows true changes to be separated from random measurement error. These outcomes may also be summarized through trend score composites to reflect different types of growth over the school year. We demonstrate the use of multivariate generalizability theory to estimate reliability for trend score composites, and we compare the results to traditional methods of analysis. Reliability estimates computed for average, linear, quadratic, and cubic trend scores from 118 classrooms participating in the MyTeachingPartner study indicate that universe scores account for between 57% and 88% of observed score variance.

A Comparison of the Exact Kruskal-Wallis Distribution to Asymptotic Approximations for All Sample Sizes up to 105

The authors generated exact probability distributions for sample sizes up to 35 in each of three groups (n ≤ 105) and up to 10 in each of four groups (n ≤ 40). They compared the exact distributions to the chi-square, gamma, and beta approximations. The beta approximation was best in terms of the root mean square error. At specific significance levels, either the gamma or beta approximation was best. These results suggest that the most common approximation, the chi-square approximation, is not a good choice. The authors demonstrate this point using an applied example. Critical value tables for the exact distribution are available online at http://faculty.virginia.edu/kruskal-wallis. The portion of these tables that provides critical values for equal sample sizes appears in this article. The authors recommend that researchers use critical values from the exact distribution whenever possible. If sample sizes exceed those included in the authors’ exact probability tables, they recommend using the beta approximation instead of the chi-square and gamma approximations.

Consistency and Predictive Nature of Vertically Moderated Standards for South Carolina's 1999 Palmetto Achievement Challenge Tests of Language Arts and Mathematics

This article reports on the consistency of the achievement-level classifications (below basic, basic, proficient, and advanced) established in 1999 for the South Carolina Palmetto Achievement Challenge Tests (PACT; Huynh, Meyer, & Barton, 2000) of English language arts and mathematics. It also utilizes the PACT longitudinal data files of student records from 2000 to 2002 to assess the predictive nature of these classifications. It was found that the proportion of students who obtained a basic or higher level classification, which is considered passing, on the same subject at the next grade level was about 80% for students at the basic category and at 99% for the combined proficient and advanced category. For school accountability purposes, the original below basic category was split into a low category, below basic-1, and a high category, below basic-2. The passing proportion was 17% for below basic-1 students and about 43% for below basic-2 students. It was concluded that the PACT 1999 achievement categories fulfilled the function of identifying and characterizing students prepared for work at the next grade level as formulated in the policy definitions for these categories.

Model Invariance Across Genders of the Broad Autism Phenotype Questionnaire

ASD is one of the most heritable neuropsychiatric disorders, though comprehensive genetic liability remains elusive. To facilitate genetic research, researchers employ the concept of the broad autism phenotype (BAP), a milder presentation of traits in undiagnosed relatives. Research suggests that the BAP Questionnaire (BAPQ) demonstrates psychometric properties superior to other self-report measures. To examine evidence regarding validity of the BAPQ, the current study used confirmatory factor analysis to test the assumption of model invariance across genders. Results of the current study upheld model invariance at each level of parameter constraint; however, model fit indices suggested limited goodness-of-fit between the proposed model and the sample. Exploratory analyses investigated alternate factor structure models but ultimately supported the proposed three-factor structure model.

A Practical Solution to Optimizing the Reliability of Teaching Observation Measures under Budget Constraints.

Researchers often use generalizability theory to estimate relative error variance and reliability in teaching observation measures. They also use it to plan future studies and design the best possible measurement procedures. However, designing the best possible measurement procedure comes at a cost, and researchers must stay within their budget when designing a study. In this study, we applied the LaGrange multiplier method to obtain facet sample size equations that minimize relative error variance (hence maximize reliability) under budget constraints. We did this for a crossed design and three nested designs that are more typical of data collection in teaching observation studies. Using an example budget and variance components similar to those found in practice, we demonstrate the use of these equations. We also show the way variance components from fully crossed designs can be combined to use our equations for a nested design.

Eqboot and Eqwinboot: Java Applications for Estimating Equating Constants and the Standard Error of Equating Using IRT Methods:

Eqboot implements the Mean/Sigma (Marco, 1977), Mean/Mean (Loyd & Hoover, 1980), Haebara (Haebara, 1980), and Stocking-Lord (Stocking & Lord, 1983) equating methods for binary items. Rasch, 2PL, and 3PL models are supported. For the characteristic curve equating methods, the user has control over the type of criterion function and the ability distributions over which the criterion functions are minimized. Criterion functions may be forward, backward, or symmetric (see Kim & Kolen, 2007). Ability distribution may involve all estimated ability parameters, a histogram of ability estimates with the number of bins specified by the user or computed via Sturges’s rule (Sturges, 1926), or a kernel density estimate of the estimated abilities (see Silverman, 1986). Nonlinear optimization of the criterion functions is conducted using methods described in Dennis and Schnabel (1996). Eqboot also computes the standard error of equating for all four equating methods using bootstrap techniques (see Kolen & Brennan, 2004; Tsai, Hanson, Kolen, & Forsyth, 2001). The user must provide three comma delimited files for estimating the standard error of equating. One file contains the item parameter estimates for both forms, where each row in the file lists item parameter estimates from each bootstrap replication. The other two files contain the estimated abilities for each form. Each row of these ability files lists the estimated abilities for each bootstrap replication. The user may generate these plain text files using any available software. Users applying the Rasch model can use the support program eqwinboot. Eqwinboot works with a licensed version of WINSTEPS (Linacre, 2008) to generate the three input files eqboot uses to estimate the standard error of equating. It supports nonparametric and parametric bootstrap procedures. Two types of bootstrap sampling are possible: sampling examinees only (see Tsai et al., 2001) or sampling common items (see Michaelides & Haertel, 2004). Users must obtain their own licensed copy of WINSTEPS. It is not distributed with eqboot.

On the Use of Covariates in Bridging Single- and Multiple-Race Categories.

Federal standards established in 1997 allow respondents to select multiple-race categories. These new standards changed the singlerace subgroup definitions that the government has required since 1977. Meta-analysis, research on long-term assessment trends, and other research involving historical comparisons must account for the definitional changes. Researchers have proposed bridging methods as techniques for linking the old and new definitions. The authors describe covariate and noncovariate bridging methods and evaluate their efficacy. The authors conceptualized covariate procedures as finite mixture models that researchers may use in either single or multiple imputation. A simulation study of single-imputation bridging indicated that the use of covariates notably improves the efficacy of bridging.