Daniël Lakens

Calculating and reporting effect sizes to facilitate cumulative science : a practical primer for t-tests and ANOVAs

Effect sizes are the most important outcome of empirical studies. Most articles on effect sizes highlight their importance to communicate the practical significance of results. For scientists themselves, effect sizes are most useful because they facilitate cumulative science. Effect sizes can be used to determine the sample size for follow-up studies, or examining effects across studies. This article aims to provide a practical primer on how to calculate and report effect sizes for t-tests and ANOVAs such that effect sizes can be used in a-priori power analyses and meta-analyses. Whereas many articles about effect sizes focus on between-subjects designs and address within-subjects designs only briefly, I provide a detailed overview of the similarities and differences between within- and between-subjects designs. I suggest that some research questions in experimental psychology examine inherently intra-individual effects, which makes effect sizes that incorporate the correlation between measures the best summary of the results. Finally, a supplementary spreadsheet is provided to make it as easy as possible for researchers to incorporate effect size calculations into their workflow.

Weight as an Embodiment of Importance

Four studies show that the abstract concept of importance is grounded in bodily experiences of weight. Participants provided judgments of importance while they held either a heavy or a light clipboard. Holding a heavy clipboard increased judgments of monetary value (Study 1) and made participants consider fair decision-making procedures to be more important (Study 2). It also caused more elaborate thinking, as indicated by higher consistency between related judgments (Study 3) and by greater polarization of agreement ratings for strong versus weak arguments (Study 4). In line with an embodied perspective on cognition, these findings suggest that, much as weight makes people invest more physical effort in dealing with concrete objects, it also makes people invest more cognitive effort in dealing with abstract issues.

An open, large-scale, collaborative effort to estimate the reproducibility of psychological science

Reproducibility is a defining feature of science. However, because of strong incentives for innovation and weak incentives for confirmation, direct replication is rarely practiced or published. The Reproducibility Project is an open, large-scale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science. So far, 72 volunteer researchers from 41 institutions have organized to openly and transparently replicate studies published in three prominent psychological journals in 2008. Multiple methods will be used to evaluate the findings, calculate an empirical rate of replication, and investigate factors that predict reproducibility. Whatever the result, a better understanding of reproducibility will ultimately improve confidence in scientific methodology and findings.

Rewarding Replications: A Sure and Simple Way to Improve Psychological Science

Although replications are vital to scientific progress, psychologists rarely engage in systematic replication efforts. In this article, we consider psychologists’ narrative approach to scientific publications as an underlying reason for this neglect and propose an incentive structure for replications within psychology. First, researchers need accessible outlets for publishing replications. To accomplish this, psychology journals could publish replication reports in files that are electronically linked to reports of the original research. Second, replications should get cited. This can be achieved by cociting replications along with original research reports. Third, replications should become a valued collaborative effort. This can be realized by incorporating replications in teaching programs and by stimulating adversarial collaborations. The proposed incentive structure for replications can be developed in a relatively simple and cost-effective manner. By promoting replications, this incentive structure may greatly enhance the dependability of psychology’s knowledge base.

Sailing From the Seas of Chaos Into the Corridor of Stability Practical Recommendations to Increase the Informational Value of Studies

Recent events have led psychologists to acknowledge that the inherent uncertainty encapsulated in an inductive science is amplified by problematic research practices. In this article, we provide a practical introduction to recently developed statistical tools that can be used to deal with these uncertainties when performing and evaluating research. In Part 1, we discuss the importance of accurate and stable effect size estimates as well as how to design studies to reach a corridor of stability around effect size estimates. In Part 2, we explain how, given uncertain effect size estimates, well-powered studies can be designed with sequential analyses. In Part 3, we (a) explain what p values convey about the likelihood that an effect is true, (b) illustrate how the v statistic can be used to evaluate the accuracy of individual studies, and (c) show how the evidential value of multiple studies can be examined with a p-curve analysis. We end by discussing the consequences of incorporating our recommendations in terms of a reduced quantity, but increased quality, of the research output. We hope that the practical recommendations discussed in this article will provide researchers with the tools to make important steps toward a psychological science that allows researchers to differentiate among all possible truths on the basis of their likelihood.

Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses

Scientists should be able to provide support for the absence of a meaningful effect. Currently, researchers often incorrectly conclude an effect is absent based a nonsignificant result. A widely recommended approach within a frequentist framework is to test for equivalence. In equivalence tests, such as the two one-sided tests (TOST) procedure discussed in this article, an upper and lower equivalence bound is specified based on the smallest effect size of interest. The TOST procedure can be used to statistically reject the presence of effects large enough to be considered worthwhile. This practical primer with accompanying spreadsheet and R package enables psychologists to easily perform equivalence tests (and power analyses) by setting equivalence bounds based on standardized effect sizes and provides recommendations to prespecify equivalence bounds. Extending your statistical tool kit with equivalence tests is an easy way to improve your statistical and theoretical inferences.

Polarity Correspondence in Metaphor Congruency Effects: Structural Overlap Predicts Categorization Times for Bipolar Concepts Presented in Vertical Space.

Previous research has shown that words presented on metaphor congruent locations (e.g., positive words UP on the screen and negative words DOWN on the screen) are categorized faster than words presented on metaphor incongruent locations (e.g., positive words DOWN and negative words UP). These findings have been explained in terms of an interference effect: The meaning associated with UP and DOWN vertical space can automatically interfere with the categorization of words with a metaphorically incongruent meaning. The current studies test an alternative explanation for the interaction between the vertical position of abstract concepts and the speed with which these stimuli are categorized. Research on polarity differences (basic asymmetries in the way dimensions are processed) predicts that +polar endpoints of dimensions (e.g., positive, moral, UP) are categorized faster than -polar endpoints of dimensions (e.g., negative, immoral, DOWN). Furthermore, the polarity correspondence principle predicts that stimuli where polarities correspond (e.g., positive words presented UP) provide an additional processing benefit compared to stimuli where polarities do not correspond (e.g., negative words presented UP). A meta-analysis (Study 1) shows that a polarity account provides a better explanation of reaction time patterns in previous studies than an interference explanation. An experiment (Study 2) reveals that controlling for the polarity benefit of +polar words compared to -polar words did not only remove the main effect of word polarity but also the interaction between word meaning and vertical position due to polarity correspondence. These results reveal that metaphor congruency effects should not be interpreted as automatic associations between vertical locations and word meaning but instead are more parsimoniously explained by their structural overlap in polarities.

Weighty Matters Importance Literally Feels Heavy

Previous work showed that concrete experiences of weight influence people’s judgments of how important certain issues are. In line with an embodied simulation account but contrary to a metaphor-enriched perspective, this work shows that perceived importance of an object influences perceptions of weight. Two studies manipulated information about a book’s importance, after which, participants estimated its weight. Importance information caused participants to perceive the book to be heavier. This was not merely a semantic association, because weight perceptions were affected only when participants physically held the book. Furthermore, importance information influenced weight perceptions but not perceptions of monetary value. These findings extend previous research by showing that the activation direction from weight to importance can be reversed, thus suggesting that the connection between importance and weight goes beyond metaphorical mappings. Implications for the debate on interpretation of findings on the interplay between bodily states and abstract information are discussed.

The Peer Reviewers’ Openness Initiative: Incentivizing open research practices through peer review

Openness is one of the central values of science. Open scientific practices such as sharing data, materials and analysis scripts alongside published articles have many benefits, including easier replication and extension studies, increased availability of data for theory-building and meta-analysis, and increased possibility of review and collaboration even after a paper has been published. Although modern information technology makes sharing easier than ever before, uptake of open practices had been slow. We suggest this might be in part due to a social dilemma arising from misaligned incentives and propose a specific, concrete mechanism—reviewers withholding comprehensive review—to achieve the goal of creating the expectation of open practices as a matter of scientific principle.

But for the Bad, There Would Not Be Good: Grounding Valence in Brightness Through Shared Relational Structures

Light and dark are used pervasively to represent positive and negative concepts. Recent studies suggest that black and white stimuli are automatically associated with negativity and positivity. However, structural factors in experimental designs, such as the shared opposition in the valence (good vs. bad) and brightness (light vs. dark) dimensions might play an important role in the valence-brightness association. In 6 experiments, we show that while black ideographs are consistently judged to represent negative words, white ideographs represent positivity only when the negativity of black is coactivated. The positivity of white emerged only when brightness and valence were manipulated within participants (but not between participants) or when the negativity of black was perceptually activated by presenting positive and white stimuli against a black (vs. gray) background. These findings add to an emerging literature on how structural overlap between dimensions creates associations and highlight the inherently contextualized construction of meaning structures.