Dylan Molenaar

The poor availability of psychological research data for reanalysis.

The origin of the present comment lies in a failed attempt to obtain, through e-mailed requests, data reported in 141 empirical articles recently published by the American Psychological Association (APA). Our original aim was to reanalyze these data sets to assess the robustness of the research findings to outliers. We never got that far. In June 2005, we contacted the corresponding author of every article that appeared in the last two 2004 issues of four major APA journals. Because their articles had been published in APA journals, we were certain that all of the authors had signed the APA Certification of Compliance With APA Ethical Principles, which includes the principle on sharing data for reanalysis. Unfortunately, 6 months later, after writing more than 400 e-mails--and sending some corresponding authors detailed descriptions of our study aims, approvals of our ethical committee, signed assurances not to share data with others, and even our full resumes-we ended up with a meager 38 positive reactions and the actual data sets from 64 studies (25.7% of the total number of 249 data sets). This means that 73% of the authors did not share their data.

Cognitive psychology meets psychometric theory: on the relation between process models for decision making and latent variable models for individual differences.

This article analyzes latent variable models from a cognitive psychology perspective. We start by discussing work by Tuerlinckx and De Boeck (2005), who proved that a diffusion model for 2-choice response processes entails a 2-parameter logistic item response theory (IRT) model for individual differences in the response data. Following this line of reasoning, we discuss the appropriateness of IRT for measuring abilities and bipolar traits, such as pro versus contra attitudes. Surprisingly, if a diffusion model underlies the response processes, IRT models are appropriate for bipolar traits but not for ability tests. A reconsideration of the concept of ability that is appropriate for such situations leads to a new item response model for accuracy and speed based on the idea that ability has a natural zero point. The model implies fundamentally new ways to think about guessing, response speed, and person fit in IRT. We discuss the relation between this model and existing models as well as implications for psychology and psychometrics.

The emotional and cognitive effect of immersion in film viewing

This brief report presents an experiment testing the effect of immersion on emotional responses and cognitive genre categorisation of film viewers. Immersion of a film presentation was varied by presenting an animated movie either in a 3D-viewing condition (low immersive condition) or in a CAVE condition (high immersive condition, comparable to virtual reality experience). Viewers rated their emotions and categorised the movies into four basic film genres (action, drama, comedy, and non-fiction). Two distinct types of emotion were measured: Fictional World emotions (e.g., sadness) in response to the presented fictional events and Artefact emotions in response to the film as an artefact (e.g., fascination). Results showed that stronger immersion led to more intense emotions but did not influence genre categorisation. In line with expectations, both types of emotional response were intensified by high immersion. The results are explained by suggesting that highly immersive cinema has its impact on a basic dimension of emotion, namely arousal that underlies both types of emotions.

Testing and modelling non-normality within the one-factor model.

Maximum likelihood estimation in the one-factor model is based on the assumption of multivariate normality for the observed data. This general distributional assumption implies three specific assumptions for the parameters in the one-factor model: the common factor has a normal distribution; the residuals are homoscedastic; and the factor loadings do not vary across the common factor scale. When any of these assumptions is violated, non-normality arises in the observed data. In this paper, a model is presented based on marginal maximum likelihood to enable explicit tests of these assumptions. In addition, the model is suitable to incorporate the detected violations, to enable statistical modelling of these effects. Two simulation studies are reported in which the viability of the model is investigated. Finally, the model is applied to IQ data to demonstrate its practical utility as a means to investigate ability differentiation.

A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times

A generalized linear modeling framework to the analysis of responses and response times is outlined. In this framework, referred to as bivariate generalized linear item response theory (B-GLIRT), separate generalized linear measurement models are specified for the responses and the response times that are subsequently linked by cross-relations. The cross-relations can take various forms. Here, we focus on cross-relations with a linear or interaction term for ability tests, and cross-relations with a curvilinear term for personality tests. In addition, we discuss how popular existing models from the psychometric literature are special cases in the B-GLIRT framework depending on restrictions in the cross-relation. This allows us to compare existing models conceptually and empirically. We discuss various extensions of the traditional models motivated by practical problems. We also illustrate the applicability of our approach using various real data examples, including data on personality and cognitive ability.

Testing Systematic Genotype by Environment Interactions Using Item Level Data

Investigating genotype by environment interactions (GxE) is generally considered challenging due to the scale dependency of the interaction effect. The present paper illustrates the problems associated with testing for GxEs on summed item scores within the well-known ACE model. That is, it is shown how genuine GxEs may be masked and how spurious interactions can arise from scaling issues in the data. A solution is proposed which explicitly distinguishes between a measurement model for the ordinal item responses and a biometric model in which the GxE effects are investigated. The new approach is studied in a simulation study using both a scenario in which the measurement instrument suffers from mild scaling problems and a scenario in which the measurement instrument suffers from severe scaling problems. Results indicate that the severity of the scale problems affects the power to detect GxE, but it rarely results in false positives. We illustrate the new approach on a real dataset concerning affect.

A generalized linear factor model approach to the hierarchical framework for responses and response times

We show how the hierarchical model for responses and response times as developed by van der Linden (2007), Fox, Klein Entink, and van der Linden (2007), Klein Entink, Fox, and van der Linden (2009), and Glas and van der Linden (2010) can be simplified to a generalized linear factor model with only the mild restriction that there is no hierarchical model at the item side. This result is valuable as it enables all well-developed modelling tools and extensions that come with these methods. We show that the restriction we impose on the hierarchical model does not influence parameter recovery under realistic circumstances. In addition, we present two illustrative real data analyses to demonstrate the practical benefits of our approach.

Modeling ability differentiation in the second-order factor model

In this article we present factor models to test for ability differentiation. Ability differentiation predicts that the size of IQ subtest correlations decreases as a function of the general intelligence factor. In the Schmid–Leiman decomposition of the second-order factor model, we model differentiation by introducing heteroscedastic residuals, nonlinear factor loadings, and a skew-normal second-order factor distribution. Using marginal maximum likelihood, we fit this model to Spanish standardization data of the Wechsler Adult Intelligence Scale (3rd ed.) to test the differentiation hypothesis.

Heteroscedastic latent trait models for dichotomous data

Effort has been devoted to account for heteroscedasticity with respect to observed or latent moderator variables in item or test scores. For instance, in the multi-group generalized linear latent trait model, it could be tested whether the observed (polychoric) covariance matrix differs across the levels of an observed moderator variable. In the case that heteroscedasticity arises across the latent trait itself, existing models commonly distinguish between heteroscedastic residuals and a skewed trait distribution. These models have valuable applications in intelligence, personality and psychopathology research. However, existing approaches are only limited to continuous and polytomous data, while dichotomous data are common in intelligence and psychopathology research. Therefore, in present paper, a heteroscedastic latent trait model is presented for dichotomous data. The model is studied in a simulation study, and applied to data pertaining alcohol use and cognitive ability.

The Value of Response Times in Item Response Modeling

A new and very interesting approach to the analysis of responses and response times is proposed by Goldhammer (this issue). In this approach, differences in the speed-ability compromise within respondents are considered to confound the differences in ability between respondents. These confounding effects of speed on the inferences about ability can be controlled for in experimental settings. As a result, the data for psychological or educational inferences consists of the response vectors only. The response time vectors are redundant as these are equal for all respondents (at least in the response signal paradigm as preferred by Goldhammer, this issue, and Goldhammer & Kroehne, 2014). To assess the merit of this promising approach by Goldhammer, a straightforward question that arises is: Why are we interested in differences in response times? Below, I will argue that the natural variability in response times can give valuable information for psychological and educational inferences about response processes and solution strategies but that the approach by Goldhammer is very valuable if a single process or strategy needs to be measured in isolation.