Johannes Hartig

Student, School, and Country Differences in Sustained Test-Taking Effort in the 2009 PISA Reading Assessment.

In this article, the change in examinee effort during an assessment, which we will refer to as persistence, is modeled as an effect of item position. A multilevel extension is proposed to analyze hierarchically structured data and decompose the individual differences in persistence. Data from the 2009 Program of International Student Achievement (PISA) reading assessment from N = 467,819 students from 65 countries are analyzed with the proposed model, and the results are compared across countries. A decrease in examinee effort during the PISA reading assessment was found consistently across countries, with individual differences within and between schools. Both the decrease and the individual differences are more pronounced in lower performing countries. Within schools, persistence is slightly negatively correlated with reading ability; but at the school level, this correlation is positive in most countries. The results of our analyses indicate that it is important to model and control examinee effort in low-stakes assessments.

Absolute and Relative Measures of Instructional Sensitivity

Valid inferences on teaching drawn from students’ test scores require that tests are sensitive to the instruction students received in class. Accordingly, measures of the test items’ instructional sensitivity provide empirical support for validity claims about inferences on instruction. In the present study, we first introduce the concepts of absolute and relative measures of instructional sensitivity. Absolute measures summarize a single item’s total capacity of capturing effects of instruction, which is independent of the test’s sensitivity. In contrast, relative measures summarize a single item’s capacity of capturing effects of instruction relative to test sensitivity. Then, we propose a longitudinal multilevel item response theory model that allows estimating both types of measures depending on the identification constraints.

Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes – A Priority Program of the German Research Foundation (DFG)

Social change, social cohesion, and opportunities for societal development are all dependent on the educational level of the members of a society. Current discussion in educational research emphasizes the importance of the products of educational processes, often referred to as educational output or outcomes, for human resources (Klieme & Leutner, 2006).

Analyzing Organizational Growth in Repeated Cross-Sectional Designs Using Multilevel Structural Equation Modeling

In repeated cross-sections of organizations, different individuals are sampled from the same set of organizations at each time point of measurement. As a result, common longitudinal data analysis methods (e.g., latent growth curve models) cannot be applied in the usual way. In this contribution, a multilevel structural equation modeling approach to analyze data from repeated cross-sections is presented. Results from a simulation study are reported which aimed at obtaining guidelines on appropriate sample sizes. We focused on a situation where linear growth occurs at the organizational level, and organizational growth is predicted by a single organizational level variable. The power to identify an effect of this organizational level variable was moderately to strongly positively related to number of measurement occasions, number of groups, group size, intraclass correlation, effect size, and growth curve reliability. The Type I error rate was close to the nominal alpha level under all conditions.

Comparing Attitudes Across Groups: An IRT-Based Item-Fit Statistic for the Analysis of Measurement Invariance:

Questionnaires for the assessment of attitudes and other psychological traits are crucial in educational and psychological research, and item response theory (IRT) has become a viable tool for scaling such data. Many international large-scale assessments aim at comparing these constructs across countries, and the invariance of measures across countries is thus required. In its most recent cycle, the Programme for International Student Assessment (PISA 2015) implemented an innovative approach for testing the invariance of IRT-scaled constructs in the context questionnaires administered to students, parents, school principals, and teachers. On the basis of a concurrent calibration with equal item parameters across all groups (i.e., languages within countries), a group-specific item-fit statistic (root mean square deviance [RMSD]) was used as a measure for the invariance of item parameters for individual groups. The present simulation study examines the statistic’s distribution under different types and extents of (non)invariance in polytomous items. Responses to five 4-point Likert-type items were generated under the generalized partial credit model (GPCM) for 1,000 simulees in 50 groups each. For one of the five items, either location or discrimination parameters were drawn from a normal distribution. In addition to the type of noninvariance, the extent of noninvariance was varied by manipulating the variation of these distributions. The results indicate that the RMSD statistic is better at detecting noninvariance related to between-group differences in item location than in item discrimination. The study’s findings may be used as a starting point to sensitivity analysis aiming to define cutoff values for determining (non)invariance.