A Train Wreck by Any Other Name

Abstract

This interesting paper by Sherman and Rivers (in press) seems to have a narrower point (which its authors argue for very explicitly) and a broader point (which they argue for much less explicitly.) The narrower point is that the phrase ‘social priming’ (and some other alternative labels that have cropped up in this context recently) are exceptionally poor labels for the body of priming research that has been called into question in a torrent of failed replication attempts appearing within the last 10 years or so (sometimes referred to as “the train wreck”). We agree with this narrower point, with a few important qualifications. Sherman and Rivers’ broader point is a suggestion, which runs through the paper but is never quite explicitly stated, that the onslaught of non-replications has not really brought to light anything terribly worrisome or misguided about the field and its prevailing research practices. The authors’ view seems to be that the field was doing about as well as one should expect of a scientific field, but suffered the misfortune of having a good number of outsiders wander in and stir up trouble in various ways. The biggest problem, they seem to suggest, was basically one of public relations rather than substance (especially, that a Nobel Laureate regrettably used somewhat inflammatory language to describe his frustration with research in the area, an event that got fairly wide coverage). We are one of the groups who have been viewed by some as marauding outsiders, although one of us is a social psychologist. Beginning around 2010, we tried to replicate some of the most (to us) surprising and fascinating priming results coming out of the social cognition field, as did numerous other labs around the same time (e.g., Doyen, Klein, Pichon, & Cleeremans, 2012; Harris, Coburn, Rohrer, & Pashler, 2013, Klein et al., 2014; Pashler, Rohrer, & Harris, 2013; Shanks et al., 2013). Our direct replication attempts, usually with larger n’s than the original studies, resulted in a stream of completely negative results, and left us with a rather different perspective on the situation than Sherman and Rivers. We think that the unreproducibility of such a high fraction of the well-known results on a purported phenomenon (essentially 100%, depending on what is counted) is extremely troubling and should, as it has, engender deep concern about research practices in any field. We see it as reflecting systemic problems that were (and perhaps still are) very widespread in social cognition (and perhaps a far wider swath of research topics), including a lack of recognition of the crucial importance of direct replications, the erroneous belief that conceptual replications offer an adequate substitute for direct replications, and a lack of an ethic of personal responsibility for authors of questioned research to reproduce their own findings and report candidly on their ability or inability to do so whenever possible.