Daniel Kasper

On the relevance of assumptions associated with classical factor analytic approaches

A personal trait, for example a person’s cognitive ability, represents a theoretical concept postulated to explain behavior. Interesting constructs are latent, that is, they cannot be observed. Latent variable modeling constitutes a methodology to deal with hypothetical constructs. Constructs are modeled as random variables and become components of a statistical model. As random variables, they possess a probability distribution in the population of reference. In applications, this distribution is typically assumed to be the normal distribution. The normality assumption may be reasonable in many cases, but there are situations where it cannot be justified. For example, this is true for criterion-referenced tests or for background characteristics of students in large scale assessment studies. Nevertheless, the normal procedures in combination with the classical factor analytic methods are frequently pursued, despite the effects of violating this “implicit” assumption are not clear in general. In a simulation study, we investigate whether classical factor analytic approaches can be instrumental in estimating the factorial structure and properties of the population distribution of a latent personal trait from educational test data, when violations of classical assumptions as the aforementioned are present. The results indicate that having a latent non-normal distribution clearly affects the estimation of the distribution of the factor scores and properties thereof. Thus, when the population distribution of a personal trait is assumed to be non-symmetric, we recommend avoiding those factor analytic approaches for estimation of a person’s factor score, even though the number of extracted factors and the estimated loading matrix may not be strongly affected. An application to the Progress in International Reading Literacy Study (PIRLS) is given. Comments on possible implications for the Programme for International Student Assessment (PISA) complete the presentation.

Academic Competencies: Their Interrelatedness and Gender Differences at Their High End.

The present study investigated (a) how a latent profile analysis based on representative data of N = 74,868 4th graders from 17 European countries would cluster the students on the basis of their reading, mathematics, and science achievement test scores; (b) whether there would be gender differences at various competency levels, especially among the top performers; (c) and whether societal gender equity might account for possible cross-national variation in the gender ratios among the top performers. The latent profile analysis revealed an international model with 7 profiles. Across these profiles, the test scores of all achievement domains progressively and consistently increased. Thus, consistent with our expectations, (a) the profiles differed only in their individuals’ overall performance level across all academic competencies and not in their individuals’ performance profile shape. From the national samples, the vast majority of the students could be reliably assigned to 1 of the profiles of the international model. Inspection of the gender ratios revealed (b) that boys were overrepresented at both ends of the competency spectrum. However, there was (c) some cross-national variation in the gender ratios among the top performers, which could be partly explained by women’s access to education and labor market participation. The interrelatedness of academic competencies and its practical implications, the role of gender equity as a possible cause of gender differences among the top performers, and directions for future research are discussed.

Social reproduction and sex in German primary schools

Abstract To understand the relationship between social background and sex in schooling, we use Bourdieu’s theory of social reproduction and a feminist perspective of gender as practice. We pose two questions: (1) What is the relationship between economic and cultural capital and achievement for 4th-grade females versus males studying in Germany? (2) Is the relationship between school composition and student achievement different for 4th-grade females versus males? We report no differences between females and males in the relationships between social background and achievement (p > 0.05). However, the relationship between class-aggregated social background and achievement is halved in female-majority mathematics classrooms (β = -12.6, p < 0.05).

Distributional properties of the PIRLS-home resource for learning scale and observed effects on reading achievement: are measurements of educational inequalities by latent indices without bias?

Abstract In Progress in International Reading Literacy Study (PIRLS) educational inequalities are measured, amongst others, through the relationship between students’ reading achievements and the home resource for learning (HRL) scale. By applying the partial credit model and using the WLE estimates for the person parameters it is accepted that the distribution of this latent variable is asymptotically normal within participating countries. This assumption is challenged from a theoretical perspective and through empirical findings. To find out how far the distributional properties of the HRL index influence the results of educational inequality measurements, the HRL index is rescaled for 21 European countries who participated in PIRLS 2011, assuming three different prior distributions of the latent index and using the EAP estimates for the person parameters. The predictive effects of these latent indices on students reading achievement were estimated with spline regressions. A positively skewed distribution of the latent index in the marginal maximum likelihood for estimating the item parameters results in the best fit of the scaling model in most countries. In addition, the pattern of signs of the estimated spline coefficients across the knots and the non-linear correlation between the latent index and student reading achievement varies considerably across the prior and empirical distributional properties of the latent index. Thus, by interpreting educational inequalities measured through the relationship between students’ reading achievements and the HRL scale in PIRLS, distributional properties of the HRL index should be taken into account.

The OECD’s Programme for International Student Assessment (PISA) Study: A Review of Its Basic Psychometric Concepts

The Programme for International Student Assessment (PISA; e.g., OECD, Sample tasks from the PISA 2000 assessment, 2002a; OECD, Learning for tomorrow’s world: first results from PISA 2003, 2004; OECD, PISA 2006: Science competencies for tomorrow’s world, 2007; OECD, PISA 2009 Technical Report, 2012) is an international large scale assessment study that aims to assess the skills and knowledge of 15-year-old students, and based on the results, to compare education systems across the participating (about 70) countries (with a minimum number of approx. 4,500 tested students per country). Initiator of this Programme is the Organisation for Economic Co-operation and Development (OECD; www.pisa.oecd.org). We review the main methodological techniques of the PISA study. Primarily, we focus on the psychometric procedure applied for scaling items and persons. PISA proficiency scale construction and proficiency levels derived based on discretization of the continua are discussed. For a balanced reflection of the PISA methodology, questions and suggestions on the reproduction of international item parameters, as well as on scoring, classifying and reporting, are raised. We hope that along these lines the PISA analyses can be better understood and evaluated, and if necessary, possibly be improved.

Sensitivity Analyses for the Mixed Coefficients Multinomial Logit Model

For scaling items and persons in large scale assessment studies such as Programme for International Student Assessment (PISA; OECD, PISA 2009 Technical Report. OECD Publishing, Paris, 2012) or Progress in International Reading Literacy Study (PIRLS; Martin et al., PIRLS 2006 Technical Report. TIMSS & PIRLS International Study Center, Chestnut Hill, 2007) variants of the Rasch model (Fischer and Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications. Springer, New York, 1995) are used. However, goodness-of-fit statistics for the overall fit of the models under varying conditions as well as specific statistics for the various testable consequences of the models (Steyer and Eid, Messen und Testen [Measuring and Testing]. Springer, Berlin, 2001) are rarely, if at all, presented in the published reports.In this paper, we apply the mixed coefficients multinomial logit model (Adams et al., The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23, 1997) to PISA data under varying conditions for dealing with missing data. On the basis of various overall and specific fit statistics, we compare how sensitive this model is, across changing conditions. The results of our study will help in quantifying how meaningful the findings from large scale assessment studies can be. In particular, we report that the proportion of missing values and the mechanism behind missingness are relevant factors for estimation accuracy, and that imputing missing values in large scale assessment settings may not lead to more precise results.

Wie viele Punkte auf der TIMSS-Metrik entsprechen einem Lernjahr?

Fur die Interpretation von Ergebnissen der Trends in International Mathematics and Science Study (TIMSS) kann es hilfreich sein, als Kriterium Kompetenzzuwachse heranzuziehen, die Schulerinnen und Schuler fur gewohnlich im Laufe eines Jahres erzielen. In dem Beitrag wird auf Basis von Daten einer fur Deutschland reprasentativen Langsschnittstudie (ADDITION) untersucht, welche durchschnittlichen Leistungszuwachse Grundschulkinder im vierten Schuljahr in Mathematik und Naturwissenschaften auf der TIMSS-Skala erzielen. Hierzu werden unterschiedliche Equatingverfahren angewendet und Resultate verglichen. Je nach genutztem Verfahren liegt der durchschnittliche Leistungszuwachs in Mathematik zwischen 38 und 55 Leistungspunkten, fur die Naturwissenschaften zwischen 25 und 30 Punkten. Analysen wie diese konnen dazu dienen, Disparitaten im Kompetenzerwerb abzuschatzen und deren Bedeutsamkeit zu bewerten.

Using Latent Class Models with Random Effects for Investigating Local Dependence

In psychometric latent variable modeling approaches such as item response theory one of the most central assumptions is local independence (LI), i.e. stochastic independence of test items given a latent ability variable (e.g., Hambleton et al., Fundamentals of item response theory, 1991). This strong assumption, however, is often violated in practice resulting, for instance, in biased parameter estimation. To visualize the local item dependencies, we derive a measure quantifying the degree of such dependence for pairs of items. This measure can be viewed as a dissimilarity function in the sense of psychophysical scaling (Dzhafarov and Colonius, Journal of Mathematical Psychology 51:290–304, 2007), which allows us to represent the local dependencies graphically in the Euclidean 2D space. To avoid problems caused by violation of the local independence assumption, in this paper, we apply a more general concept of “local independence” to psychometric items. Latent class models with random effects (LCMRE; Qu et al., Biometrics 52:797–810, 1996) are used to formulate a generalized local independence (GLI) assumption held more frequently in reality. It includes LI as a special case. We illustrate our approach by investigating the local dependence structures in item types and instances of large scale assessment data from the Programme for International Student Assessment (PISA; OECD, PISA 2009 Technical Report, 2012).