Examining replicability in addictions research: How to assess and ways forward.

Abstract

OBJECTIVE\nThe high rate of statistically significant findings in the sciences that do not replicate in a new sample has been described as a replication crisis. Few replication attempts have been conducted in studies of alcohol use disorder (AUD), and the best method for determining whether a finding replicates has not been explored. The goal of the current study was to conduct direct replications within a multisite AUD-randomized controlled trial and to test a range of replication metrics.\n\n\nMETHOD\nWe used data from a large AUD clinical trial (Project Matching Alcoholism Treatments and Client Heterogeneity [Project MATCH], n = 1,726) to simulate direct replication attempts. We examined associations between drinking intensity and negative alcohol-related consequences (Model 1), sex differences in drinking intensity (Model 2), and reductions in drinking following treatment (Model 3). We treated each of the 11 data collection sites as unique studies such that each subsample was treated as an original study, and the remaining 10 subsamples were viewed as replication studies. Replicability metrics included the consistency of statistical significance, overlapping confidence intervals, and consistency of the direction of the effect. We also tested effect replication and heterogeneity using meta-analysis.\n\n\nRESULTS\nWe observed between 0% and 100% replicability across the replicability metrics depending on which subsample was treated as the original study. Meta-analyses indicated results were more similar across subsamples with no significant heterogeneity for Models 1 and 2.\n\n\nCONCLUSIONS\nWe recommend researchers focus on effect sizes and use meta-analysis to evaluate the level of replicability. We also encourage direct replication attempts and sharing of data and code to facilitate direct replication. (PsycInfo Database Record (c) 2021 APA, all rights reserved).