Okan Bulut

Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review.

Multiple-choice testing is considered one of the most effective and enduring forms of educational assessment that remains in practice today. This study presents a comprehensive review of the literature on multiple-choice testing in education focused, specifically, on the development, analysis, and use of the incorrect options, which are also called the distractors. Despite a vast body of literature on multiple-choice testing, the task of creating distractors has received much less attention. In this study, we provide an overview of what is known about developing distractors for multiple-choice items and evaluating their quality. Next, we synthesize the existing guidelines on how to use distractors and summarize earlier research on the optimal number of distractors and the optimal ordering of distractors. Finally, we use this comprehensive review to provide the most up-to-date recommendations regarding distractor development, analysis, and use, and in the process, we highlight important areas where further research is needed.

Revisiting the Relations Between the WJ-IV Measures of Cattell-Horn-Carroll (CHC) Cognitive Abilities and Reading Achievement During the School-Age Years

This study examined associations between broad cognitive abilities (Fluid Reasoning [Gf], Short-Term Working Memory [Gwm], Long-Term Storage and Retrieval [Glr], Processing Speed [Gs], Comprehension-Knowledge [Gc], Visual Processing [Gv], and Auditory Processing [Ga]) and reading achievement (Basic Reading Skills, Reading Rate, Reading Fluency, and Reading Comprehension) in a nationally representative school-age sample. Findings indicate that some cognitive abilities were stronger predictors of reading achievement than previously found (e.g., Gf, Ga, and Gs). Most notably, the Woodcock-Johnson–IV Gf cluster was found to be the strongest and most consistent predictor of reading achievement. A secondary analysis suggests that this effect was likely due to the new Number Series test. The results of the study suggest revisions to previous conceptualizations of the associations between the broad Cattell-Horn-Carroll abilities and areas of reading achievement.

Multidimensional Extension of Multiple Indicators Multiple Causes Models to Detect DIF

A number of studies have found multiple indicators multiple causes (MIMIC) models to be an effective tool in detecting uniform differential item functioning (DIF) for individual items and item bundles. A recently developed MIMIC-interaction model is capable of detecting both uniform and nonuniform DIF in the unidimensional item response theory (IRT) framework. The goal of the current study is to extend the MIMIC-interaction model for detecting DIF in the context of multidimensional IRT modelling and examine the performance of the multidimensional MIMIC-interaction model under various simulation conditions with respect to Type I error and power rates. Simulation conditions include DIF pattern and magnitude, test length, correlation between latent traits, sample size, and latent mean differences between focal and reference groups. The results of this study indicate that power rates of the multidimensional MIMIC-interaction model under uniform DIF conditions were higher than those of nonuniform DIF conditions. When anchor item length and sample size increased, power for detecting DIF increased. Also, the equal latent mean condition tended to produce higher power rates than the different mean condition. Although the multidimensional MIMIC-interaction model was found to be a reasonably useful tool for identifying uniform DIF, the performance of the model in detecting nonuniform DIF appeared to be questionable.

Test Review: Shipley-2 Manual

ion scales are provided as AutoScore forms, so the responses marked on the outside of the form are transferred to the Scoring Worksheet by the carbon paper. The Block Patterns scale is provided as a form that is scored with an Answer Key. The examinee earns 1 point for each correct answer and 0 points for each incorrect answer or the items left blank. Raw scores for each scale are transferred to the Profile Sheet and then converted to standard scores (M = 100, SD = 15), percentiles, age-equivalents, and confidence intervals as well as a composite score, which reflects overall cognitive ability, and the Impairment Index, which represents the discrepancy between vocabulary and abstract thinking. Test Materials and Stimuli The Shipley-2 kit includes an examiner’s manual, Vocabulary Autoscore form, Abstraction Autoscore form, and Block Pattern form. A two-sided Profile Sheet for Composite A and B is attached to the Vocabulary AutoScore form as its first page. The well-organized manual provides test users with a full understanding of administration, scoring, and interpretation procedures supported by examples and case studies from various settings. The manual also offers detailed interpretive guidance on the issues of response validity, differential cognitive functioning, item analysis, and retesting over time. It also includes multiple strategies to use and interpret the Shipley-2 depending on the referral, setting, and user’s needs. Computerized administration and scoring is also available.

The interactive effects of child maltreatment and the FK506 binding protein 5 gene ( FKBP5 ) on dissociative symptoms in adolescence

The FK506 binding protein 5 gene (FKBP5) has been associated with susceptibility to pathogenic effects of childhood trauma including dissociative symptoms. This study examines the impact of maltreatment on dissociative tendencies in adolescence as moderated by the FKBP5 gene. Dissociative symptoms and variation within FKBP5 were assessed in a high-risk, low socioeconomic status community sample of 279 maltreated and 171 nonmaltreated adolescents. Following the assignment of haplotypes across four single nucleotide polymorphisms (rs3800373, rs9296158, rs1360780, and rs9470080), individuals with one or more copies of the CATT haplotype (N = 230) were grouped together and compared to individuals with zero copies of this haplotype (N = 185). Analyses of covariance were conducted to test hypotheses regarding the effects of developmental timing and the chronicity of maltreatment and the CATT haplotype. We found a significant interactive effect of timing/chronicity of maltreatment and the CATT haplotype on dissociative symptoms. Among adolescents who had no copies of the CATT haplotype, dissociative symptoms were higher for chronically maltreated adolescents who had an infancy onset compared to those who were not maltreated or whose maltreatment experience was either relatively less chronic or not started in infancy. The groups did not differ significantly among subjects who carry one or more copies of the CATT haplotype.

Investigating professional learning communities in Turkish schools: the effects of contextual factors

A great number of studies have focused on professional learning communities in schools, but only a limited number of studies have treated the construct of professional learning communities as a dependent variable. The purpose of this research is to investigate Turkish schools’ capacity for supporting professional learning communities and to examine factors that account for variation in the current level of development. The data for this study were collected from 492 school staff members, including teachers, principals and assistant principals, working at 27 schools across nine provinces of Turkey. Results indicate that school staff had a culture of sharing and collaboration, but suffered from a lack of material and human resources required for supporting effective learning communities. The experience of the staff, as well as the size and socioeconomic status of the school, appeared to be the most important factors in predicting the variation in the available professional learning communities. The results are discussed considering current educational policy and practice in Turkey.

Item and testlet position effects in computer-based alternate assessments for students with disabilities

ABSTRACT Item positions in educational assessments are often randomized across students to prevent cheating. However, if altering item positions results in any significant impact on students’ performance, it may threaten the validity of test scores. Two widely used approaches for detecting position effects – logistic regression and hierarchical generalized linear modelling – are often inconvenient for researchers and practitioners due to some technical and practical limitations. Therefore, this study introduced a structural equation modeling (SEM) approach for examining item and testlet position effects. The SEM approach was demonstrated using data from a computer-based alternate assessment designed for students with cognitive disabilities from three grade bands (3–5, 6–8, and high school). Item and testlet position effects were investigated in the field-test (FT) items that were received by each student at different positions. Results indicated that the difficulty of some FT items in grade bands 3–5 and 6–8 differed depending on the positions of the items on the test. Also, the overall difficulty of the field-test task in grade bands 6–8 increased as students responded to the field-test task in later positions. The SEM approach provides a flexible method for examining different types of position effects.

Retrofitting Diagnostic Classification Models to Responses from IRT-Based Assessment Forms.

Developing a diagnostic tool within the diagnostic measurement framework is the optimal approach to obtain multidimensional and classification-based feedback on examinees. However, end users may seek to obtain diagnostic feedback from existing item responses to assessments that have been designed under either the classical test theory or item response theory frameworks. Retrofitting diagnostic classification models to existing assessments designed under other psychometric frameworks could be a plausible approach to obtain more actionable scores or understand more about the constructs themselves. This study (a) discusses the possibility and problems of retrofitting, (b) proposes a step-by-step retrofitting framework, and (c) explores the information one can gain from retrofitting through an empirical application example. While retrofitting may not always be an ideal approach to diagnostic measurement, this article aims to invite discussions through presenting the possibility, challenges, process, and product of retrofitting.

Investigating Separate and Concurrent Approaches for Item Parameter Drift in 3PL Item Response Theory Equating

This study examines separate and concurrent approaches to combine the detection of item parameter drift (IPD) and the estimation of scale transformation coefficients in the context of the common item nonequivalent groups design with the three-parameter item response theory equating. The study uses real and synthetic data sets to compare the two approaches based on IPD flagging rates, type I error and power rates, and recovery of scale transformation coefficients. Results indicate that the two approaches render similar outcomes with stable anchor sets. However, they can produce dissimilar results with unstable anchor sets because of differences in the performance of their IPD components. Further, the findings of this study caution about working backward from equated cut scores to motivate the selection of an anchor set.

Detecting Multidimensional Differential Item Functioning with the Multiple Indicators Multiple Causes Model, the Item Response Theory Likelihood Ratio Test, and Logistic Regression

Differential item functioning (DIF) is typically evaluated in educational assessments with a simple structure in which items are associated with a single latent trait. This study aims to extend the investigation of DIF for multidimensional assessments with a nonsimple structure in which items can be associated with two or more latent traits. A simulation study was conducted with the multidimensional extensions of the item response theory likelihood ratio (IRT-LR) test, the multiple indicators multiple causes (MIMIC) model, and logistic regression for detecting uniform and nonuniform DIF in multidimensional assessments. The results indicated that the IRT-LR test outperformed the MIMIC and logistic regression approaches in detecting nonuniform DIF. When detecting uniform DIF, the MIMIC and logistic regression approaches appeared to perform better than the IRT-LR test in short tests, while the performances of all three approaches were very similar in longer tests. Type I error rates for logistic regression were severely inflated compared to the other two approaches. The IRT-LR test appears to be a more balanced and powerful method than the MIMIC and logistic regression approaches in detecting DIF in multidimensional assessments with a nonsimple structure.