Rogier A. Kievit

An Agenda for Purely Confirmatory Research

The veracity of substantive research claims hinges on the way experimental data are collected and analyzed. In this article, we discuss an uncomfortable fact that threatens the core of psychology’s academic enterprise: almost without exception, psychologists do not commit themselves to a method of data analysis before they see the actual data. It then becomes tempting to fine tune the analysis to the data in order to obtain a desired result—a procedure that invalidates the interpretation of the common statistical tests. The extent of the fine tuning varies widely across experiments and experimenters but is almost impossible for reviewers and readers to gauge. To remedy the situation, we propose that researchers preregister their studies and indicate in advance the analyses they intend to conduct. Only these analyses deserve the label “confirmatory,” and only for these analyses are the common statistical tests valid. Other analyses can be carried out but these should be labeled “exploratory.” We illustrate our proposal with a confirmatory replication attempt of a study on extrasensory perception.

Representational geometry: integrating cognition, computation, and the brain

Highlights • Representational geometry is a framework that enables us to relate brain, computation, and cognition.• Representations in brains and models can be characterized by representational distance matrices.• Distance matrices can be readily compared to test computational models.• We review recent insights into perception, cognition, memory, and action and discuss current challenges.

Lying in the scanner: covert countermeasures disrupt deception detection by functional magnetic resonance imaging.

Functional magnetic resonance imaging (fMRI) studies have documented differences between deceptive and honest responses. Capitalizing on this research, companies marketing fMRI-based lie detection services have been founded, generating methodological and ethical concerns in scientific and legal communities. Critically, no fMRI study has examined directly the effect of countermeasures, methods used by prevaricators to defeat deception detection procedures. An fMRI study was conducted to fill this research gap using a concealed information paradigm in which participants were trained to use countermeasures. Robust group fMRI differences between deceptive and honest responses were found without, but not with countermeasures. Furthermore, in single participants, deception detection accuracy was 100% without countermeasures, using activation in ventrolateral and medial prefrontal cortices, but fell to 33% with countermeasures. These findings show that fMRI-based deception detection measures can be vulnerable to countermeasures, calling for caution before applying these methods to real-world situations.

Simpson's paradox in psychological science: a practical guide

The direction of an association at the population-level may be reversed within the subgroups comprising that population—a striking observation called Simpsons paradox. When facing this pattern, psychologists often view it as anomalous. Here, we argue that Simpsons paradox is more common than conventionally thought, and typically results in incorrect interpretations—potentially with harmful consequences. We support this claim by reviewing results from cognitive neuroscience, behavior genetics, clinical psychology, personality psychology, educational psychology, intelligence research, and simulation studies. We show that Simpsons paradox is most likely to occur when inferences are drawn across different levels of explanation (e.g., from populations to subgroups, or subgroups to individuals). We propose a set of statistical markers indicative of the paradox, and offer psychometric solutions for dealing with the paradox when encountered—including a toolbox in R for detecting Simpsons paradox. We show that explicit modeling of situations in which the paradox might occur not only prevents incorrect interpretations of data, but also results in a deeper understanding of what data tell us about the world.

Cognitive psychology meets psychometric theory: on the relation between process models for decision making and latent variable models for individual differences.

This article analyzes latent variable models from a cognitive psychology perspective. We start by discussing work by Tuerlinckx and De Boeck (2005), who proved that a diffusion model for 2-choice response processes entails a 2-parameter logistic item response theory (IRT) model for individual differences in the response data. Following this line of reasoning, we discuss the appropriateness of IRT for measuring abilities and bipolar traits, such as pro versus contra attitudes. Surprisingly, if a diffusion model underlies the response processes, IRT models are appropriate for bipolar traits but not for ability tests. A reconsideration of the concept of ability that is appropriate for such situations leads to a new item response model for accuracy and speed based on the idea that ability has a natural zero point. The model implies fundamentally new ways to think about guessing, response speed, and person fit in IRT. We discuss the relation between this model and existing models as well as implications for psychology and psychometrics.

Unique semantic space in the brain of each beholder predicts perceived similarity

Significance Everyone is different. Understanding the unique way an individual perceives the world is a fundamental goal of psychology and brain science. Using novel methods for analyzing functional MRI (fMRI) data, we show that each person viewing a set of objects represents the objects uniquely in his or her brain. Moreover, given an individual’s measured brain-activity patterns, idiosyncrasies in his or her perception of the similarities among the objects can be predicted. Prediction accuracy is modest using current technology. However, our results demonstrate that fMRI has the power to reveal individually unique representations of particular objects in the human brain. The novel method might help us understand the biological substrate of individual experience in mental health and disease. The unique way in which each of us perceives the world must arise from our brain representations. If brain imaging could reveal an individual’s unique mental representation, it could help us understand the biological substrate of our individual experiential worlds in mental health and disease. However, imaging studies of object vision have focused on commonalities between individuals rather than individual differences and on category averages rather than representations of particular objects. Here we investigate the individually unique component of brain representations of particular objects with functional MRI (fMRI). Subjects were presented with unfamiliar and personally meaningful object images while we measured their brain activity on two separate days. We characterized the representational geometry by the dissimilarity matrix of activity patterns elicited by particular object images. The representational geometry remained stable across scanning days and was unique in each individual in early visual cortex and human inferior temporal cortex (hIT). The hIT representation predicted perceived similarity as reflected in dissimilarity judgments. Importantly, hIT predicted the individually unique component of the judgments when the objects were personally meaningful. Our results suggest that hIT brain representational idiosyncrasies accessible to fMRI are expressed in an individuals perceptual judgments. The unique way each of us perceives the world thus might reflect the individually unique representation in high-level visual areas.

Distinct aspects of frontal lobe structure mediate age-related differences in fluid intelligence and multitasking

Ageing is characterized by declines on a variety of cognitive measures. These declines are often attributed to a general, unitary underlying cause, such as a reduction in executive function owing to atrophy of the prefrontal cortex. However, age-related changes are likely multifactorial, and the relationship between neural changes and cognitive measures is not well-understood. Here we address this in a large (N=567), population-based sample drawn from the Cambridge Centre for Ageing and Neuroscience (Cam-CAN) data. We relate fluid intelligence and multitasking to multiple brain measures, including grey matter in various prefrontal regions and white matter integrity connecting those regions. We show that multitasking and fluid intelligence are separable cognitive abilities, with differential sensitivities to age, which are mediated by distinct neural subsystems that show different prediction in older versus younger individuals. These results suggest that prefrontal ageing is a manifold process demanding multifaceted models of neurocognitive ageing.

The Two Disciplines of Scientific Psychology, or: The Disunity of Psychology as a Working Hypothesis

Anybody who has some familiarity with the research literature in scientific psychology has probably thought, at one time or another, ‘Well, all these means and correlations are very interesting, but what do they have to do with me, as an individual person?’. The question, innocuous as it may seem, is a deep and complicated one. In contrast to the natural sciences, where researchers can safely assume that, say, all electrons are exchangeable save properties such as location and momentum, people differ from each other. Furthermore, it is not obvious that these differences can be treated as irrelevant to the structure of the organisms in question, i.e., it is not clear that they can be treated as ‘noise’ or ‘error’. The problem permeates virtually every subdiscipline of psychology, and in fact may be one of the reasons that progress in psychology has been limited. As Lykken (1991, pp. 3–4) hypothesizes:

Mind the Gap: a psychometric approach to the reduction problem

Cognitive neuroscience involves the simultaneous analysis of behavioral and neurological data. Common practice in cognitive neuroscience, however, is to limit analyses to the inspection of descriptive measures of association (e.g., correlation coefficients). This practice, often combined with little more than an implicit theoretical stance, fails to address the relationship between neurological and behavioral measures explicitly. This article argues that the reduction problem, in essence, is a measurement problem. As such, it should be solved by using psychometric techniques and models. We show that two influential philosophical theories on this relationship, identity theory and supervenience theory, can be easily translated into psychometric models. Upon such translation, they make explicit hypotheses based on sound theoretical and statistical foundations, which renders them empirically testable. We examine these models, show how they can elucidate our conceptual framework, and examine how they may be used to study foundational questions in cognitive neuroscience. We illustrate these principles by applying them to the relation between personality test scores, intelligence tests, and neurological measures.

Letting the daylight in: Reviewing the reviewers and other ways to maximize transparency in science

With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses.