Claudia Gentile

Q-Matrix Construction: Defining the Link Between Constructs and Test Items in Large-Scale Reading and Listening Comprehension Assessments

In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign LanguageTM Internet-based Test (TOEFL® iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.

Ensuring Data Quality: What Criteria do Monitors Use to Rate Interviewers?

Like most survey research organizations, Mathematica Policy Research monitors telephone interviewers to ensure that they collect high-quality data. Monitors identify problems with survey questions, recommend interviewing techniques, and retrain interviewers whose performance do not meet expectations. Because monitoring is a critical quality assurance tool, we are interested in understanding the behavior of monitors – specifically, their ability to provide effective and consistent feedback on interviewers’ performance. Mathematica implements several best practices to promote monitoring consistency:

Investigating Patterns of Writing Errors for Different L1 Groups through Error-Coded ESL Learners’ Essays

Automated error detection and feedback systems are becoming an important component of online writing practice services for ESL/EFL (English as a second/foreign language) learners. The main purposes of the study are to: (a) collect samples of essays written by ESL learners with different native language (or L1) backgrounds that are error-coded by an early version of an automated error-detection system (CritiqueTM) and trained human coders; and (b) identify some unique patterns of writing errors for different first language (L1) groups. Data analyzed in this study included 18, 439 TOEFL◯R CBT essays error-coded by CritiqueTM and a much smaller, combined sample of 480 TOEFL◯R CBT/TOEFL iBT◯R essays error-coded by trained human coders. A comparison of error rates across five different language groups showed some unique patterns: (a) the Arabic and Spanish groups were the highest on both spelling and punctuation errors; (b) the Korean and Japanese groups had the highest article error frequency; and (c) the Chinese group had the highest number of errors related to verb conjugations or adjective and noun inflections. The implications of these findings are discussed in terms of understanding the nature of L1-related writing errors and enhancing the automated error detection and feedback systems.