Norman Rose

Dealing with Omitted and Not-Reached Items in Competence Tests: Evaluating Approaches Accounting for Missing Responses in Item Response Theory Models.

Data from competence tests usually show a number of missing responses on test items due to both omitted and not-reached items. Different approaches for dealing with missing responses exist, and there are no clear guidelines on which of those to use. While classical approaches rely on an ignorable missing data mechanism, the most recently developed model-based approaches account for nonignorable missing responses. Model-based approaches include the missing propensity in the measurement model. Although these models are very promising, the assumptions made in these models have not yet been tested for plausibility in empirical data. Furthermore, studies investigating the performance of different approaches have only focused on one kind of missing response at once. In this study, we investigated the performance of classical and model-based approaches in empirical data, accounting for different kinds of missing responses simultaneously. We confirmed the existence of a unidimensional tendency to omit items. Indicating nonignorability of the missing mechanism, missing tendency due to both omitted and not-reached items correlated with ability. However, results on parameter estimation showed that ignoring missing responses was sufficient to account for missing responses, and that the missing propensity was not needed in the model. The results from the empirical study were corroborated in a complete case simulation.

A Cautious Note on Auxiliary Variables That Can Increase Bias in Missing Data Problems

The treatment of missing data in the social sciences has changed tremendously during the last decade. Modern missing data techniques such as multiple imputation and full-information maximum likelihood are used much more frequently. These methods assume that data are missing at random. One very common approach to increase the likelihood that missing at random is achieved consists of including many covariates as so-called auxiliary variables. These variables are either included based on data considerations or in an inclusive fashion; that is, taking all available auxiliary variables. In this article, we point out that there are some instances in which auxiliary variables exhibit the surprising property of increasing bias in missing data problems. In a series of focused simulation studies, we highlight some situations in which this type of biasing behavior can occur. We briefly discuss possible ways how one can avoid selecting bias-inducing covariates as auxiliary variables.

Theory and Analysis of Total, Direct, and Indirect Causal Effects

Mediation analysis, or more generally models with direct and indirect effects, are commonly used in the behavioral sciences. As we show in our illustrative example, traditional methods of mediation analysis that omit confounding variables can lead to systematically biased direct and indirect effects, even in the context of a randomized experiment. Therefore, several definitions of causal effects in mediation models have been presented in the literature (Baron & Kenny, 1986; Imai, Keele, & Tingley, 2010; Pearl, 2012). We illustrate the stochastic theory of causal effects as an alternative foundation of causal mediation analysis based on probability theory. In this theory we define total, direct, and indirect effects and show how they can be identified in the context of our illustrative example. A particular strength of the stochastic theory of causal effects are the causality conditions that imply causal unbiasedness of effect estimates. The causality conditions have empirically testable implications and can be used for covariate selection. In the discussion, we highlight some similarities and differences of the stochastic theory of causal effects with other theories of causal effects.

A Simulation Study on Methods of Correcting for the Effects of Extreme Response Style.

The impact of response styles such as extreme response style (ERS) on trait estimation has long been a matter of concern to researchers and practitioners. This simulation study investigated three methods that have been proposed for the correction of trait estimates for ERS effects: (a) mixed Rasch models, (b) multidimensional item response models, and (c) regression residuals. The methods were compared with respect to their ability of recovering the true latent trait levels. Data were generated according to a unidimensional model with only one trait, a mixed Rasch model with two populations of ERS and non-ERS, and a two-dimensional model incorporating a trait and an ERS dimension. The data were analyzed using the same models as well as linear regression where the trait estimate is regressed on an ERS score and the resulting residual is considered the corrected trait estimate. Over all conditions, the two-dimensional model achieved the best trait recovery, though the difference to the unidimensional model was rather small. Mixed Rasch models were in general inferior to the other correction methods. When the trait and ERS showed no to weak correlations, ERS appeared to have a minor impact on trait estimation.

A Comparison of Unidimensionality and Measurement Precision of the Narcissistic Personality Inventory and the Narcissistic Admiration and Rivalry Questionnaire

The current study compares the closeness to unidimensionality (CU) and measurement precision (MP) of the Narcissistic Personality Inventory (NPI)—with either a pairwise forced-choice or 5-point Likert-type scale response format—to the Narcissistic Admiration and Rivalry Questionnaire (NARQ). Minimum rank factor analysis and item information curves from item response models were utilized. The results mainly confirmed our expectations that NPI subscales are lower in CU and MP compared with NARQ subscales when the NPI was administered with its traditional forced-choice response format. When the NPI was administered with a 5-point Likert-type scale response format, the NPI subscale Leadership/Authority and NPI Grandiose Exhibitionism showed similarly high levels of CU and MP as the two NARQ subscales. While the NPI subscale Entitlement/Exploitativeness had a higher CU than the NARQ subscales it showed considerably lower levels of MP.

Fact or artefact: an item response theory analysis of median split based repressor classification

OBJECTIVE Many studies have investigated the phenomenon of repression. Repressors are defined as individuals who deny or avoid the experience of negative affect. A common method for the identification of repressors is a median split approach using questionnaires that measure anxiety and social desirability. The present study aimed to evaluate this most frequently used procedure using a psychometric model. DESIGN We applied item response theory using model assumptions comparable with those of the median split approach to detect repressors and examine the appropriateness of the median split procedure. METHODS A mixed sample of 655 students and members of the general public completed the two scales usually used to identify repressors, namely the Taylor Manifest Anxiety Scale and the Marlowe-Crowne Social Desirability Scale. RESULTS Employing item response theory mixture-model analyses, we were unable to replicate the median split solution on a latent level. CONCLUSIONS Our results did not support the identification of repressors via dichotomization of the two scales. The median split approach does not appear to detect repressors satisfactorily. The implications of the results for the conceptualization and assessment of repressors are discussed.

Quantitative Bildungsforschung und Assessments

Das Kapitel stellt typische Forschungsdesigns der quantitativen Bildungsforschung vor und beurteilt diese nach ihrer Aussagekraft sowie ihrer externen und internen Validitat. Danach werden drei zentrale Herausforderungen bei der Auswertung von Daten in der quantitativen Bildungsforschung diskutiert (Messfehlerproblem, Mehrebenenstruktur, kausale Inferenz) und kurz die statistischen Methoden beschrieben, mit denen diese Probleme in der Regel adressiert werden.

A multilevel study of position effects in PISA achievement tests: student- and school-level predictors in the German tracked school system

Abstract Position effects (PE) cause decreasing probabilities of correct item responses towards the end of a test. We analysed PEs in science, mathematics and reading tests administered in the German extension to the PISA 2006 study with respect to their variability at the student- and school-level. PEs were strongest in reading and weakest in mathematics. Variability in PEs was found at both levels of analysis. PEs were stronger for male students, for students with a migration background (science and mathematics), and for students with a less favourable socio-economic background (reading). At the school level, PEs were stronger in lower school tracks and in schools with a high proportion of students with a migration background. The relationships of the test scores with the covariates partly reflected the covariates’ relationships with PEs. Our findings suggest that PEs should be taken seriously in large-scale assessments as they have an undesirable impact on the results.

The Development of Narcissistic Admiration and Machiavellianism in Early Adulthood

We investigated the development of narcissistic admiration (i.e., the assertive or extraverted dimension of narcissism; Back et al., 2013) and Machiavellianism (Mach) in early adulthood. Specifically, we examined (a) mean-level changes in narcissistic admiration and Mach during early adulthood and (b) how studying economics and experiencing any of 30 life events were related to individual differences in changes in narcissistic admiration and Mach. We used longitudinal data from 2 cohorts of young adults in Germany (N1 = 4,962 and N2 = 2,572). The mean levels of narcissistic admiration remained stable over time. Life events analyses indicated that narcissistic admiration increased among people who experienced a positively evaluated change in their eating or sleeping habits, a positively evaluated romantic break-up, or a negatively evaluated failure on an important exam. The mean levels of Mach decreased during early adulthood in both cohorts. Life events analyses showed that Mach decreased for only the 91% of young adults who had started a new job and evaluated it positively, suggesting that mastering occupational roles mitigates Mach in early adulthood. The results will be discussed in the light of previous longitudinal studies on narcissism and the Big Five and cross-sectional studies on how age is related to narcissism and Mach.