Moritz Heene

A Vast Graveyard of Undead Theories Publication Bias and Psychological Science’s Aversion to the Null

Publication bias remains a controversial issue in psychological science. The tendency of psychological science to avoid publishing null results produces a situation that limits the replicability assumption of science, as replication cannot be meaningful without the potential acknowledgment of failed replications. We argue that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact.

Anomalous Results in G-Factor Models: Explanations and Alternatives.

G-factor models such as the bifactor model and the hierarchical G-factor model are increasingly applied in psychology. Many applications of these models have produced anomalous and unexpected results that are often not in line with the theoretical assumptions on which these applications are based. Examples of such anomalous results are vanishing specific factors and irregular loading patterns. In this article, the authors show that from the perspective of stochastic measurement theory anomalous results have to be expected when G-factor models are applied to a single-level (rather than a 2-level) sampling process. The authors argue that the application of the bifactor model and related models require a 2-level sampling process that is usually not present in empirical studies. We demonstrate how alternative models with a G-factor and specific factors can be derived that are more well-defined for the actual single-level sampling design that underlies most empirical studies. It is shown in detail how 2 alternative models, the bifactor-(S − 1) model and the bifactor-(S·I − 1) model, can be defined. The properties of these models are described and illustrated with an empirical example. Finally, further alternatives for analyzing multidimensional models are discussed.

Additive conjoint measurement and the resistance toward falsifiability in psychology

The history of the past four decades of the theory and application of additive conjoint measurement (ACM) is characterized by vivid developments of its theoretical foundation (cf. Luce and Tukey, 1964; Krantz et al., 1971, 2006; Narens, 1974), industrious developments of statistical and computational implementations (cf. Karabatsos and Ullrich, 2002; Karabatsos and Sheu, 2004; Karabatsos, 2005; Myung et al., 2005) and heated debates about its applicability and significance in psychology (cf. Michell, 1997, 2009; Borsboom and Mellenbergh, 2004; Barrett, 2008; Borsboom and Scholten, 2008; Kyngdon, 2008a; Trendler, 2009). What started as a promising foundation to solve the everlasting debate about the quantitative nature of psychological attributes (Ferguson et al., 1939) ended in perseverative debates with very little transfer to mainstream psychological science still being dominated by structural equation modeling (SEM) and item response theory (IRT). After reading the aforementioned articles, and comparing their implications with the day-to-day business of mainstream psychological science, even an unbiased reader would certainly agree with Cliff (1992) that ACM was a “… revolution that never happened” (p. 186). It is not the aim of this article, to discredit the efforts of mathematical psychology and proponents of ACM in particular. I just want to address the naive but relevant question why ACM as a stringent way to formalize and to test the requirements of quantitative measurement in psychology has not been embraced by mainstream psychology as a means to an end to test what they always claim: that most of the attributes (e.g., intelligence and personality factors) are quantitative. An attribute possessing a quantitative structure is required to satisfy the three conditions of ordinality (transitivity, antisymmetry, and strong connexity) and the six conditions of additivity (associativity, commutativity, monotonicity, solvability, positivity, and the Archimedean condition; cf. Michell, 1990, p. 52f.). Most of these conditions are testable hypotheses but I have never seen any empirical test in psychological articles before data were analyzed with SEM or IRT models, which already assume the quantitative structure of the attributes under consideration as argued below. Somewhere during my psychology studies at the university I learned that psychology is an empirical science and that there is therefore no room for claims that should just be believed. However, given the assumed but almost never tested quantitative nature of most of the psychological attributes as reflected in factor analysis, SEM and IRT models, I must have missed or misunderstood something.

Assessing drug-specific knowledge as a protective factor in illicit substance use: the Heidelberger Drogenbogen (Heidelberg drug scales – HDB)

Abstract Background: As a modular inventory, the Heidelberger Drogenbogen (HDB) assesses substance-specific knowledge regarding the illicit psychoactive substances cocaine, amphetamines, cannabis, ecstasy, and hallucinogens. This diagnostic tool allows for assessing the level of substance-specific knowledge in most frameworks of prevention. Objectives: This study outlines basic psychometric parameters of the HDB. Different groups of drug users (clinical versus non-clinical, users versus non-users) are compared concerning their knowledge about psychoactive substances. Methods: A community sample (recruited at schools, universities and in subcultural contexts [n = 4212]) was compared to a clinical sample (recruited in in- and out-patient drug abuse treatment departments [n = 582]). Results: Cronbach’s α coefficients (0.61 ≤ α ≤ 0.87) and test–retest correlations (0.81 ≤ rp ≤ 0.87) indicate that the scales are reliable measures of substance-specific knowledge about drugs. Drug users generally have a higher level of knowledge than non-users. Within the group of users, clinical users display a significantly lower level of knowledge than non-clinical users. This confirms the construct validity of the HDB knowledge modules, thus helping to redefine prevention strategies targeting abstinence as well as less harmful consumption patterns.

Phonological Awareness and Rapid Automatized Naming as Longitudinal Predictors of Reading in Five Alphabetic Orthographies with Varying Degrees of Consistency

ABSTRACT Although phonological awareness (PA) and rapid automatized naming (RAN) are confirmed as early predictors of reading in a large number of orthographies, it is as yet unclear whether the predictive patterns are universal or language specific. This was examined in a longitudinal study across Grades 1 and 2 with 1,120 children acquiring one of five alphabetic orthographies with different degrees of orthographic complexity (English, French, German, Dutch, and Greek). Path analyses revealed that a universal model could not be confirmed. When we specified the best-fitting model separately for each language, RAN was a consistent predictor of reading fluency in all orthographies, whereas the association between PA and reading was complex and mostly interactive. We conclude that RAN taps into a language-universal cognitive mechanism that is involved in reading alphabetic orthographies (independent of complexity), whereas the PA–reading relationship depends on many factors like task characteristics, developmental status, and orthographic complexity.

Measuring scientific reasoning – a review of test instruments

ABSTRACT Education systems increasingly emphasize the importance of scientific reasoning skills such as generating hypotheses and evaluating evidence. Despite this importance, we do not know which tests of scientific reasoning exist, which skills they emphasize, how they conceptualize scientific reasoning, and how well they are evaluated. Therefore, this article reviews 38 scientific reasoning tests. They used to primarily consist of multiple-choice questions, but since then have become more diverse, even including tests that automatically analyse virtual experiments. Furthermore, this review revealed that the tests focus on the skills of generating hypotheses, generating evidence, evaluating evidence, and drawing conclusions. Additionally, conceptualizations of scientific reasoning have moved towards a domain-specific set of different but coordinated skills over the years. Finally, concluding from gaps in test evaluation, a future focus should be on testing theoretical assumptions, comparing different scientific reasoning tests, and how relevant test results are in predicting criterion variables like academic performance.

Detecting Violations of Unidimensionality by Order-Restricted Inference Methods

The assumption of unidimensionality and quantitative measurement represents one of the key concepts underlying most of the commonly applied of item response models. The assumption of unidimensionality is frequently tested although most commonly applied methods have been shown having low power against violations of unidimensionality whereas the assumption of quantitative measurement remains in most of the cases only an (implicit) assumption. On the basis of a simulation study it is shown that order restricted inference methods within a Markov Chain Monte Carlo framework can successfully be used to test both assumptions.

A note on the implications of factorial invariance for common factor variable equivalence

ABSTRACT There has come to exist within the psychometric literature a generalized belief to the effect that a determination of the level of factorial invariance that holds over a set of k populations Δj, j = 1..s, is central to ascertaining whether or not the common factor random variables ξj, j = 1..s, are equivalent. In the current manuscript, a technical examination of this belief is undertaken. The chief conclusion of the work is that, as long as technical, statistical senses of random variable equivalence are adhered to, the belief is unfounded.

Assessing Drug Consumption Behavior With the Heidelberger Drogenbogen (Heidelberg Drug Scales): Reliabilities, Validities, and Cut-Off Criteria.

Background: The Heidelberger Drogenbogen (HDB) is a German language assessment of substance-specific knowledge and consumption patterns of the illicit psychoactive substances cannabis, MDMA (ecstasy), amphetamines, cocaine, and hallucinogens. The behavior modules for each of these five drugs/drug groups allow for a diagnostic evaluation of the extent of harmful consumption behavior. Each of the five modules represents a single standardized test. Objectives: This paper outlines several statistical parameters, Cronbachs alpha, retest reliabilities, as well as numerous validity and cut-off-criteria of the behavioral modules. Methods: Participants (N = 4,794) were recruited at schools, universities, in subcultural contexts, and in institutions of substance abuse treatment. Results: Internal consistencies range from = .68 to .79 while test–retest reliabilities between .87 and .94 were found. The behavior modules of the HDB can discriminate between populations with and without clinical levels of substance use. Furthermore, this measure has incremental validity and higher diagnostic accuracy over competing measures. Conclusions: The behavior modules of the HDB are reliable and valid measures of substance use and misuse.