Josine Verhagen

Bayesian inference for psychology. Part II: Example applications with JASP

Bayesian hypothesis testing presents an attractive alternative to p value hypothesis testing. Part I of this series outlined several advantages of Bayesian hypothesis testing, including the ability to quantify evidence and the ability to monitor and update this evidence as data come in, without the need to know the intention with which the data were collected. Despite these and other practical advantages, Bayesian hypothesis tests are still reported relatively rarely. An important impediment to the widespread adoption of Bayesian tests is arguably the lack of user-friendly software for the run-of-the-mill statistical problems that confront psychologists for the analysis of almost every experiment: the t-test, ANOVA, correlation, regression, and contingency tables. In Part II of this series we introduce JASP (http://www.jasp-stats.org), an open-source, cross-platform, user-friendly graphical software package that allows users to carry out Bayesian hypothesis tests for standard statistical problems. JASP is based in part on the Bayesian analyses implemented in Morey and Rouder’s BayesFactor package for R. Armed with JASP, the practical advantages of Bayesian hypothesis testing are only a mouse click away.

Bayesian Analysis of Factorial Designs.

This article provides a Bayes factor approach to multiway analysis of variance (ANOVA) that allows researchers to state graded evidence for effects or invariances as determined by the data. ANOVA is conceptualized as a hierarchical model where levels are clustered within factors. The development is comprehensive in that it includes Bayes factors for fixed and random effects and for within-subjects, between-subjects, and mixed designs. Different model construction and comparison strategies are discussed, and an example is provided. We show how Bayes factors may be computed with BayesFactor package in R and with the JASP statistical package.

Bayesian inference for psychology. Part I: Theoretical advantages and practical ramifications

Bayesian parameter estimation and Bayesian hypothesis testing present attractive alternatives to classical inference using confidence intervals and p values. In part I of this series we outline ten prominent advantages of the Bayesian approach. Many of these advantages translate to concrete opportunities for pragmatic researchers. For instance, Bayesian hypothesis testing allows researchers to quantify evidence and monitor its progression as data come in, without needing to know the intention with which the data were collected. We end by countering several objections to Bayesian hypothesis testing. Part II of this series discusses JASP, a free and open source software program that makes it easy to conduct Bayesian estimation and testing for a range of popular statistical scenarios (Wagenmakers et al. this issue).

Why Hypothesis Tests Are Essential for Psychological Science A Comment on Cumming (2014)

In a much-anticipated move, Psychological Science recently announced important changes in its publication practices (Eich, 2014); specifically, the changes promote open science (i.e., open data, open materials, and preregistration) and recommend that researchers report confidence intervals (CIs) instead of p values. As part of these initiatives, Cumming (2014) argued that inference by CI should constitute a “new statistics” in psychology (see also Grant, 1962) and will avoid the pitfalls of null-hypothesis significance testing (NHST). Embracing the new statistics, Cumming recommended a shift to estimation as “usually the most informative approach” (p. 11) and stated that “interpretation [of data] should be based on estimation” (p. 26). We broadly agree with the recommendations for more quantitative thinking, more openness and transparency, and an end to p values. Nonetheless, the benefits of estimation have been overstated, and the mistaken idea that estimation is superior to hypothesis testing is unfortunately becoming the conventional wisdom in psychology. This conventional wisdom is being perpetuated by the American Psychological Association’s (2010) publication manual, the new statistical guidelines for journals of the Psychonomic Society (n.d.), the Society for Personality and Social Psychology Task Force on Publication and Research Practices (Funder et al., 2014), and now also Psychological Science. However, estimation alone is insufficient to move psychology forward as a science; proper hypothesis-testing methods are crucial. Scientific research is often driven by theories that unify diverse observations and make clear predictions. For instance, in physics, one might test for the existence of the Higgs boson (Low, Lykken, & Shaughnessy, 2012). In biology, one might compare various phylogenetic hypotheses about how species are related (Huelsenbeck & Ronquist, 2001). In psychology, one might test whether fluid intelligence can be improved by training (Harrison et al., 2013). Testing a theory requires testing hypotheses that are consequences of the theory, but unfortunately, this is not as simple as looking at the data to see whether they are consistent with the theory. There are three necessary components to testing a theory: First, one must know what one would expect of the data if the theory were true; second, one must know what one would expect if the theory were false; and third, one must have a principled method for using the data to make an inference about the theory. The second and third components are crucial. Inferring support for a theory on the sole basis of agreement between the observations and the theory is a logical fallacy (known as the converse error; Popper, 1935/2005); having no principled method for inferring support leaves one with only ad hoc rules subject to one’s own biases. The difficulties inherent in using estimation to test theory can be illustrated by the example chosen by Cumming (2014, p. 21). Velicer et al. (2008) tested a theoretically motivated model of smokers’ readiness to stop smoking. They predicted the strength of association between 15 inventory scales and the stage of change from the transtheoretical model of behavioral change (Prochaska & Velicer, 1997). To assess the model, they determined whether CIs contained the predicted values. Eleven of the 15 predicted values were included in the 95% CIs, and Velicer et al. concluded that these results provide “overall support for the theoretical model” (p. 602). This assessment raises two issues. First, it is arbitrary. If 10 of the 15 CIs included the predicted values, would the results also support the theory, or instead refute it? If one instead used 99% CIs, would positive results for 12 of the 15 predictions be enough to support the theory? This arbitrariness arises because CIs offer no 525969 PSSXXX10.1177/0956797614525969Morey et al.Hypothesis Testing Versus Estimation research-article2014

An Introduction to Bayesian Hypothesis Testing for Management Research

In management research, empirical data are often analyzed using p-value null hypothesis significance testing (pNHST). Here we outline the conceptual and practical advantages of an alternative analysis method: Bayesian hypothesis testing and model selection using the Bayes factor. In contrast to pNHST, Bayes factors allow researchers to quantify evidence in favor of the null hypothesis. Also, Bayes factors do not require adjustment for the intention with which the data were collected. The use of Bayes factors is demonstrated through an extended example for hierarchical regression based on the design of an experiment recently published in the Journal of Management. This example also highlights the fact that p values overestimate the evidence against the null hypothesis, misleading researchers into believing that their findings are more reliable than is warranted by the data.

A power fallacy

The power fallacy refers to the misconception that what holds on average –across an ensemble of hypothetical experiments– also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a particular outcome from a high-power experiment can be completely uninformative, whereas a particular outcome from a low-power experiment can be highly informative. Although power is useful in planning an experiment, it is less useful—and sometimes even misleading—for making inferences from observed data. To make inferences from data, we recommend the use of likelihood ratios or Bayes factors, which are the extension of likelihood ratios beyond point hypotheses. These methods of inference do not average over hypothetical replications of an experiment, but instead condition on the data that have actually been observed. In this way, likelihood ratios and Bayes factors rationally quantify the evidence that a particular data set provides for or against the null or any other hypothesis.

How to quantify the evidence for the absence of a correlation

We present a suite of Bayes factor hypothesis tests that allow researchers to grade the decisiveness of the evidence that the data provide for the presence versus the absence of a correlation between two variables. For concreteness, we apply our methods to the recent work of Donnellan et al. (in press) who conducted nine replication studies with over 3,000 participants and failed to replicate the phenomenon that lonely people compensate for a lack of social warmth by taking warmer baths or showers. We show how the Bayes factor hypothesis test can quantify evidence in favor of the null hypothesis, and how the prior specification for the correlation coefficient can be used to define a broad range of tests that address complementary questions. Specifically, we show how the prior specification can be adjusted to create a two-sided test, a one-sided test, a sensitivity analysis, and a replication test.

Is there a free lunch in inference

The field of psychology, including cognitive science, is vexed by a crisis of confidence. Although the causes and solutions are varied, we focus here on a common logical problem in inference. The default mode of inference is significance testing, which has a free lunch property where researchers need not make detailed assumptions about the alternative to test the null hypothesis. We present the argument that there is no free lunch; that is, valid testing requires that researchers test the null against a well-specified alternative. We show how this requirement follows from the basic tenets of conventional and Bayesian probability. Moreover, we show in both the conventional and Bayesian framework that not specifying the alternative may lead to rejections of the null hypothesis with scant evidence. We review both frequentist and Bayesian approaches to specifying alternatives, and we show how such specifications improve inference. The field of cognitive science will benefit because consideration of reasonable alternatives will undoubtedly sharpen the intellectual underpinnings of research.

Longitudinal measurement in health‐related surveys. A Bayesian joint growth model for multivariate ordinal responses

Longitudinal surveys measuring physical or mental health status are a common method to evaluate treatments. Multiple items are administered repeatedly to assess changes in the underlying health status of the patient. Traditional models to analyze the resulting data assume that the characteristics of at least some items are identical over measurement occasions. When this assumption is not met, this can result in ambiguous latent health status estimates. Changes in item characteristics over occasions are allowed in the proposed measurement model, which includes truncated and correlated random effects and a growth model for item parameters. In a joint estimation procedure adopting MCMC methods, both item and latent health status parameters are modeled as longitudinal random effects. Simulation study results show accurate parameter recovery. Data from a randomized clinical trial concerning the treatment of depression by increasing psychological acceptance showed significant item parameter shifts. For some items, the probability of responding in the middle category versus the highest or lowest category increased significantly over time. The resulting latent depression scores decreased more over time for the experimental group than for the control group and the amount of decrease was related to the increase in acceptance level.

On the automatic link between affect and tendencies to approach and avoid: Chen and Bargh (1999) revisited

Within the literature on emotion and behavioral action, studies on approach-avoidance take up a prominent place. Several experimental paradigms feature successful conceptual replications but many original studies have not yet been replicated directly. We present such a direct replication attempt of two seminal experiments originally conducted by Chen and Bargh (1999). In their first experiment, participants affectively evaluated attitude objects by pulling or pushing a lever. Participants who had to pull the lever with positively valenced attitude objects and push the lever with negatively valenced attitude objects (i.e., congruent instruction) did so faster than participants who had to follow the reverse (i.e., incongruent) instruction. In Chen and Barghs second experiment, the explicit evaluative instructions were absent and participants merely responded to the attitude objects by either always pushing or always pulling the lever. Similar results were obtained as in Experiment 1. Based on these findings, Chen and Bargh concluded that (1) attitude objects are evaluated automatically; and (2) attitude objects automatically trigger a behavioral tendency to approach or avoid. We attempted to replicate both experiments and failed to find the effects reported by Chen and Bargh as indicated by our pre-registered Bayesian data analyses; nevertheless, the evidence in favor of the null hypotheses was only anecdotal, and definitive conclusions await further study.