Susann Fiedler

Recommendations for increasing replicability in psychology

Replicability of findings is at the heart of any empirical science. The aim of this article is to move the current replicability debate in psychology towards concrete recommendations for improvement. We focus on research practices but also offer guidelines for reviewers, editors, journal management, teachers, granting institutions, and university promotion committees, highlighting some of the emerging and existing practical solutions that can facilitate implementation of these recommendations. The challenges for improving replicability in psychological science are systemic. Improvement can occur only if changes are made at many levels of practice, evaluation, and reward. Copyright

Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency

Beginning January 2014, Psychological Science gave authors the opportunity to signal open data and materials if they qualified for badges that accompanied published articles. Before badges, less than 3% of Psychological Science articles reported open data. After badges, 23% reported open data, with an accelerating trend; 39% reported open data in the first half of 2015, an increase of more than an order of magnitude from baseline. There was no change over time in the low rates of data sharing among comparison journals. Moreover, reporting openness does not guarantee openness. When badges were earned, reportedly available data were more likely to be actually available, correct, usable, and complete than when badges were not earned. Open materials also increased to a weaker degree, and there was more variability among comparison journals. Badges are simple, effective signals to promote open practices and improve preservation of data and materials by using independent repositories.

The dynamics of decision making in risky choice: an eye-tracking analysis

In the last years, research on risky choice has moved beyond analyzing choices only. Models have been suggested that aim to describe the underlying cognitive processes and some studies have tested process predictions of these models. Prominent approaches are evidence accumulation models such as decision field theory (DFT), simple serial heuristic models such as the adaptive toolbox, and connectionist approaches such as the parallel constraint satisfaction (PCS) model. In two studies involving measures of attention and pupil dilation, we investigate hypotheses derived from these models in choices between two gambles with two outcomes each. We show that attention to an outcome of a gamble increases with its probability and its value and that attention shifts toward the subsequently favored gamble after about two thirds of the decision process, indicating a gaze-cascade effect. Information search occurs mostly within-gambles, and the direction of search does not change over the course of decision making. Pupil dilation, which reflects both cognitive effort and arousal, increases during the decision process and increases with mean expected value. Overall, the results support aspects of automatic integration models for risky choice such as DFT and PCS, but in their current specification none of them can account for the full pattern of results.

Registered Replication Report : Rand, Greene, and Nowak (2012)

In an anonymous 4-person economic game, participants contributed more money to a common project (i.e., cooperated) when required to decide quickly than when forced to delay their decision (Rand, Greene & Nowak, 2012), a pattern consistent with the social heuristics hypothesis proposed by Rand and colleagues. The results of studies using time pressure have been mixed, with some replication attempts observing similar patterns (e.g., Rand et al., 2014) and others observing null effects (e.g., Tinghög et al., 2013; Verkoeijen & Bouwmeester, 2014). This Registered Replication Report (RRR) assessed the size and variability of the effect of time pressure on cooperative decisions by combining 21 separate, preregistered replications of the critical conditions from Study 7 of the original article (Rand et al., 2012). The primary planned analysis used data from all participants who were randomly assigned to conditions and who met the protocol inclusion criteria (an intent-to-treat approach that included the 65.9% of participants in the time-pressure condition and 7.5% in the forced-delay condition who did not adhere to the time constraints), and we observed a difference in contributions of −0.37 percentage points compared with an 8.6 percentage point difference calculated from the original data. Analyzing the data as the original article did, including data only for participants who complied with the time constraints, the RRR observed a 10.37 percentage point difference in contributions compared with a 15.31 percentage point difference in the original study. In combination, the results of the intent-to-treat analysis and the compliant-only analysis are consistent with the presence of selection biases and the absence of a causal effect of time pressure on cooperation.

Processing differences between descriptions and experience: a comparative analysis using eye-tracking and physiological measures

Do decisions from description and from experience trigger different cognitive processes? We investigated this general question using cognitive modeling, eye-tracking, and physiological arousal measures. Three novel findings indeed suggest qualitatively different processes between the two types of decisions. First, comparative modeling indicates that evidence-accumulation models assuming averaging of all fixation-sampled outcomes predict choices best in decisions from experience, whereas Cumulative Prospect Theory predicts choices best in decisions from descriptions. Second, arousal decreased with increasing difference in expected value between gambles in description-based choices but not in experience. Third, the relation between attention and subjective weights given to outcomes was stronger for experience-based than for description-based tasks. Overall, our results indicate that processes in experience-based risky choice can be captured by sampling-and-averaging evidence-accumulation model. This model cannot be generalized to description-based decisions, in which more complex mechanisms are involved.

Psychologists are Open to Change, Yet Wary of Rules

Psychologists must change the way they conduct and report their research—this notion has been the topic of much debate in recent years. One article recently published in Psychological Science proposing six requirements for researchers concerning data collection and reporting practices as well as four guidelines for reviewers aimed at improving the publication process has recently received much attention (Simmons, Nelson, & Simonsohn, 2011). We surveyed 1,292 psychologists to address two questions: Do psychologists support these concrete changes to data collection, reporting, and publication practices, and if not, what are their reasons? Respondents also indicated the percentage of print and online journal space that should be dedicated to novel studies and direct replications as well as the percentage of published psychological research that they believed would be confirmed if direct replications were conducted. We found that psychologists are generally open to change. Five requirements for researchers and three guidelines for reviewers were supported as standards of good practice, whereas one requirement was even supported as a publication condition. Psychologists appear to be less in favor of mandatory conditions of publication than standards of good practice. We conclude that the proposal made by Simmons, Nelson & Simonsohn (2011) is a starting point for such standards.

The reversed description-experience gap: Disentangling sources of presentation format effects in risky choice

Previous literature has suggested that risky choice patterns in general--and probability weighting in particular--are strikingly different in experience-based as compared with description-based formats. In 2 reanalyses and 3 new experiments, we investigate differences between experience-based and description-based decisions using a parametric approach based on cumulative prospect theory (CPT). Once controlling for sampling biases, we consistently find a reversal of the typical description-experience gap, that is, a reduced sensitivity to probabilities and increased overweighting of small probabilities in decisions from experience as compared with decisions from descriptions. This finding supports the hypothesis that regression to the mean effects in probability estimation are a crucial source of differences between both presentation formats. Further analyses identified task specific information asymmetry prevalent in gambles involving certainty as a third source of differences. We present a novel conceptualization of multiple independent sources of bias that contribute to the description-experience gap, namely sampling biases and task specific information asymmetry on the one hand, and regression to the mean effects in probability estimation on the other hand.

The influence of episodic memory decline on value-based choice

ABSTRACT Recent studies suggest the involvement of episodic memory in value-based decisions as a source of information about subjective values of choice options. We therefore tested the link between age-related memory decline and inconsistencies in value-based decisions in 30 cognitively healthy older adults. Within the pre-registered experiment, the inconsistencies were measured in two ways: i) the consistency between stated preferences and revealed choices; ii) the amount of intransitivities in choice triplets, revealed in a forced paired choice task including all possible pairings of 20 food products. Although no significant association of memory functions to number of intransitive triplets was observed, participants with lower memory scores were more likely to choose the item for which they stated a lower preference. The results suggest a higher noise in the underlying preference signal in participants with lower memory. We discuss the results in the context of the unique needs of elderly consumers.