Cengiz Zopluoglu

Curriculum-Based Measurement of Oral Reading: Multi-study evaluation of schedule, duration, and dataset quality on progress monitoring outcomes

Curriculum-Based Measurement of Oral Reading (CBM-R) is used to collect time series data, estimate the rate of student achievement, and evaluate program effectiveness. A series of 5 studies were carried out to evaluate the validity, reliability, precision, and diagnostic accuracy of progress monitoring across a variety of progress monitoring durations, schedules, and dataset quality conditions. A sixth study evaluated the relation between the various conditions of progress monitoring (duration, schedule, and dataset quality) and the precision of weekly growth estimates. Model parameters were derived from a large extant progress monitoring dataset of second-grade (n=1517) and third-grade students (n=1561) receiving supplemental reading intervention as part of a Tier II response-to-intervention program. A linear mixed effects regression model was used to simulate true and observed CBM-R progress monitoring data. The validity and reliability of growth estimates were evaluated with squared correlations between true and observed scores along with split-half reliabilities of observed scores. The precision of growth estimates were evaluated with root mean square error between true and observed estimates of growth. Finally, receiver operator curves were used to evaluate the diagnostic accuracy and optimize decision thresholds. Results are interpreted to guide progress monitoring practices and inform future research.

Curriculum-Based Measurement of Oral Reading: Quality of Progress Monitoring Outcomes:

Curriculum-based measurement of oral reading (CBM-R) is frequently used to set student goals and monitor student progress. This study examined the quality of growth estimates derived from CBM-R progress monitoring data. The authors used a linear mixed effects regression (LMER) model to simulate progress monitoring data for multiple levels of progress monitoring duration (i.e., 6, 8, 10 … 20 weeks) and data set quality which was operationalized as residual/error in the model (σε = 5, 10, 15, and 20). The number of data points, quality of data, and method used to estimate growth all influenced the reliability and precision of estimated growth rates. Results indicated that progress monitoring outcomes are sufficient to guide educational decisions if (a) ordinary least-squares regression is used to derive trend lines estimates, (b) a very good progress monitoring data set is used, and (c) the data set comprises a minimum of 14 CBMs-R. The article discusses implications and future directions.

Curriculum-Based Measurement of Oral Reading: Evaluation of Growth Estimates Derived With Pre–Post Assessment Methods

Curriculum-based measurement of oral reading (CBM-R) is used to index the level and rate of student growth across the academic year. The method is frequently used to set student goals and monitor student progress. This study examined the diagnostic accuracy and quality of growth estimates derived from pre–post measurement using CBM-R data. A linear mixed effects regression model was used to simulate progress-monitoring data for multiple levels of progress-monitoring duration (6, 8, 10, . . ., 20 weeks) and data set quality, which was operationalized as residual/error in the model (σε= 5, 10, 15, and 20). Results indicate that the duration of instruction, quality of data, and method used to estimate growth influenced the reliability and precision of estimated growth rates, in addition to the diagnostic accuracy. Pre–post methods to derive CBM-R growth estimates are likely to require 14 or more weeks of instruction between pre–post occasions. Implications and future directions are discussed.

Fitting a linear-linear piecewise growth mixture model with unknown knots: A comparison of two common approaches to inference.

A linear-linear piecewise growth mixture model (PGMM) is appropriate for analyzing segmented (disjointed) change in individual behavior over time, where the data come from a mixture of 2 or more latent classes, and the underlying growth trajectories in the different segments of the developmental process within each latent class are linear. A PGMM allows the knot (change point), the time of transition from 1 phase (segment) to another, to be estimated (when it is not known a priori) along with the other model parameters. To assist researchers in deciding which estimation method is most advantageous for analyzing this kind of mixture data, the current research compares 2 popular approaches to inference for PGMMs: maximum likelihood (ML) via an expectation-maximization (EM) algorithm, and Markov chain Monte Carlo (MCMC) for Bayesian inference. Monte Carlo simulations were carried out to investigate and compare the ability of the 2 approaches to recover the true parameters in linear-linear PGMMs with unknown knots. The results show that MCMC for Bayesian inference outperformed ML via EM in nearly every simulation scenario. Real data examples are also presented, and the corresponding computer codes for model fitting are provided in the Appendix to aid practitioners who wish to apply this class of models.

A Finite Mixture of Nonlinear Random Coefficient Models for Continuous Repeated Measures Data

Nonlinear random coefficient models (NRCMs) for continuous longitudinal data are often used for examining individual behaviors that display nonlinear patterns of development (or growth) over time in measured variables. As an extension of this model, this study considers the finite mixture of NRCMs that combine features of NRCMs with the idea of finite mixture (or latent class) models. The efficacy of this model is that it allows the integration of intrinsically nonlinear functions where the data come from a mixture of two or more unobserved subpopulations, thus allowing the simultaneous investigation of intra-individual (within-person) variability, inter-individual (between-person) variability, and subpopulation heterogeneity. Effectiveness of this model to work under real data analytic conditions was examined by executing a Monte Carlo simulation study. The simulation study was carried out using an R routine specifically developed for the purpose of this study. The R routine used maximum likelihood with the expectation–maximization algorithm. The design of the study mimicked the output obtained from running a two-class mixture model on task completion data.

Rotation to a partially specified target matrix in exploratory factor analysis in practice

The purpose of the present study was to explore the influence of the number of targets specified on the quality of exploratory factor analysis solutions with a complex underlying structure and incomplete substantive measurement theory. We extended previous research in this area by (a) exploring this phenomenon in situations in which both the common factor model and the targeted pattern matrix contained specification errors and (b) comparing the performance of target rotation to an easier-to-use default rotation criterion (i.e., geomin) under conditions commonly observed in practice. A Monte Carlo study manipulated target error, number of targets, model error, overdetermination, communality, and sample size. Outcomes included bias (i.e., accuracy) and variability (i.e., stability) with regard to the rotated pattern matrix. The effects of target error were negligible for both accuracy and stability, whereas small effects were observed for the number of targets for both outcomes. Further, target rotation outperformed geomin rotation with regard to accuracy but generally performed worse than geomin rotation with regard to stability. These findings underscore the potential importance (or caution, in the case of stability) of using extant, even if incomplete and somewhat inaccurate, substantive measurement theory to inform the rotation criterion in a nonmechanical way.

The Empirical Power and Type I Error Rates of the GBT and ω Indices in Detecting Answer Copying on Multiple-Choice Tests

The generalized binomial test (GBT) and ω indices are the most recent methods suggested in the literature to detect answer copying behavior on multiple-choice tests. The ω index is one of the most studied indices, but there has not yet been a systematic simulation study for the GBT index. In addition, the effect of the ability levels of the examinees in answer copying pairs on the statistical properties of the GBT and ω indices have not been systematically addressed as yet. The current study simulated 500 answer copying pairs for each of 1,440 conditions (12 source ability level × 12 cheater ability level × 10 amount of copying) to study the empirical power and 10,000 pairs of independent response vectors for each of 144 conditions (12 source ability level × 12 cheater ability level) to study the empirical Type I error rates of the GBT and ω indices. Results indicate that neither GBT nor ω inflated the Type I error rates, and they are reliable to use in practice. The difference in statistical power of these two methods was very small, and GBT performs slightly better than does ω. The main effect for the amount of copying and the interaction effect between source ability level and the amount of copying are found to be very strong while all other main and interactions effects are negligible.

FitPMM: An R Routine to Fit Finite Mixture of Piecewise Mixed-Effect Models With Unknown Random Knots

Piecewise mixed-effect models are frequently used in education and psychology to model segmented growth over time. For data that exhibit distinct phases of growth, piecewise models are attractive alternatives to familiar quadratic and higher order polynomial models because the parameters in piecewise models provide more relevant information about the mechanism underlying the change process (Fitzmaurice, Laird, & Ware, 2011). Different types of trajectories can be specified in the different phases of a piecewise model; however, a linear–linear piecewise model seems to dominate practical applications. An interesting feature of a linear–linear piecewise model is the knot, the change point on the time axis where two linear splines join (Cudeck & Harring, 2010). In many applications, practitioners locate the knot in piecewise models a priori based on the subject-matter knowledge, and the knot is assumed to be known in subsequent statistical analyses. Moreover, the knot is commonly treated as a fixed parameter, indicating that each individual’s change point is assumed to be the same. An appealing alternative model for practical applications would allow the knot in the piecewise model to be an estimable subject-specific parameter. This model is a type of nonlinear random coefficient model like that described in Cudeck and Klebe (2002) and du Toit and Cudeck (2009). A mixture component can be added to the model to relax the assumption of a single population and to allow latent subpopulations (Everitt & Hand, 1981; Harring, 2012; Kohli, Harring, & Zopluoglu, under review; Titterington, Smith, & Makov, 1985). Fitting finite mixture of nonlinear random coefficient models can be quite challenging. However, du Toit and Cudeck provided a simplified estimation scheme for models with a combination of linear and nonlinear coefficients, and finite mixture of these models can be fitted following the extension provided by Harring (2012). An R routine, fitPMM.R, was designed to fit the finite mixture of piecewise mixed-effect models with unknown random knot parameters using the estimation

The Effects of Baseline Estimation on the Reliability, Validity, and Precision of CBM-R Growth Estimates.

This study examined the effect of baseline estimation on the quality of trend estimates derived from Curriculum Based Measurement of Oral Reading (CBM-R) progress monitoring data. The authors used a linear mixed effects regression (LMER) model to simulate progress monitoring data for schedules ranging from 6-20 weeks for datasets with high and low levels of residual variance (poor and good quality datasets respectively). Three observations per day for the first three days of data collection were generated for baseline estimation. As few as one and as many as nine observations were used to calculate baseline. The number of weeks of progress monitoring and the quality of the dataset were highly influential on the reliability, validity, and precision of simulated growth estimates. Results supported the use of using the median of three observations collected on the first day to estimate baseline, particularly when the first observation of that day systematically underestimated student performance. Collecting a large number of observations to estimate baseline does not appear to improve the quality of CBM-R growth estimates.

A comparison of two estimation algorithms for Samejima’s continuous IRT model

This study compares two algorithms, as implemented in two different computer softwares, that have appeared in the literature for estimating item parameters of Samejima’s continuous response model (CRM) in a simulation environment. In addition to the simulation study, a real-data illustration is provided, and CRM is used as a potential psychometric tool for analyzing measurement outcomes in the context of curriculum-based measurement (CBM) in the field of education. The results indicate that a simplified expectation-maximization (EM) algorithm is as effective and efficient as the traditional EM algorithm for estimating the CRM item parameters. The results also show promise for using this psychometric model to analyze CBM outcomes, although more research is needed in order to recommend CRM as a standard practice in the CBM context.