Edward Vul

Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition

Functional magnetic resonance imaging (fMRI) studiesofemotion, personality, and social cognition have drawn much attention in recent years, with high-profile studies frequently reporting extremely high (e.g., >.8) correlations between brain activation and personality measures. We show that these correlations are higher than should be expected given the (evidently limited) reliability of both fMRI and personality measures. The high correlations are all the more puzzling because method sections rarely contain much detail about how the correlations were obtained. We surveyed authors of 55 articles that reported findings of this kind to determine a few details on how these correlations were computed. More than half acknowledged using a strategy that computes separate correlations for individual voxels and reports means of only those voxels exceeding chosen thresholds. We show how this nonindependent analysis inflates correlations while yielding reassuring-looking scattergrams. This analysis technique was used to obtain the vast majority of the implausibly high correlations in our survey sample. In addition, we argue that, in some cases, other analysis problems likely created entirely spurious correlations. We outline how the data from these studies could be reanalyzed with unbiased methods to provide accurate estimates of the correlations in question and urge authors to perform such reanalyses. The underlying problems described here appear to be common in fMRI research of many kinds—not just in studies of emotion, personality, and social cognition.

Distributed practice in verbal recall tasks: A review and quantitative synthesis

The authors performed a meta-analysis of the distributed practice effect to illuminate the effects of temporal variables that have been neglected in previous reviews. This review found 839 assessments of distributed practice in 317 experiments located in 184 articles. Effects of spacing (consecutive massed presentations vs. spaced learning episodes) and lag (less spaced vs. more spaced learning episodes) were examined, as were expanding interstudy interval (ISI) effects. Analyses suggest that ISI and retention interval operate jointly to affect final-test retention; specifically, the ISI producing maximal retention increased as retention interval increased. Areas needing future research and theoretical implications are discussed.

Measuring the Crowd Within Probabilistic Representations Within Individuals

Psychological Science, Short Report, 2008. 19, 645-647. (in press version): This manuscript may differ from the final published version Measuring the crowd within: probabilistic representations within individuals. EDWARD VUL Massachusetts Institute of Technology HAROLD PASHLER University of California, San Diego A crowd often possesses better information than do the individuals it comprises. For example, if people are asked to guess the weight of a prize- winning ox (Galton, 1907), the error of the average response is substantially smaller than the average error of individual estimates. This fact, which Galton interpreted as support for democratic governance, is responsible for the success of polling the audience in the television program “Who Wants to be a Millionaire” (Surowiecki, 2004) and for the superiority of combined over individual financial forecasts (Clemen, 1989). Researchers agree that this wisdom-of-crowds effect depends on a statistical fact: The crowds average will be more accurate as long as some of the error of one individual is statistically independent of the error of other individuals—as seems almost guaranteed to be the case. Whether a similar improvement can be obtained by averaging two estimates from a single individual is not, a priori, obvious. If one estimate represents the best information available to the person, as common intuition suggests, then a second guess will simply add noise, and averaging the two will only decrease accuracy. Researchers have previously assumed this view and focused on improving the best estimate (Hirt & Markman, 1995; Mussweiler, Strack, & Pfeiffer, 2000; Stewart, 2001). Alternatively, single estimates may represent samples drawn from an internal probability Address correspondence to Edward Vul, Department of Brain and Cognitive Science, Massachusetts Institute of Technology, 77 Massachusetts Ave. 46- 4141, Cambridge, MA 02139, e-mail: evul@mit.edu. distribution, rather than deterministic best guesses. According to this account, if the internal probability distribution is unbiased, the average of two estimates from one person will be more accurate than a single estimate. Ariely et al. (2000) predicted that such a benefit would accrue from averaging probability judgments within one individual, but did not find evidence of such an effect. However, probability judgments are known to be biased toward extreme values (0 or 1), and averaging should not reduce the bias of estimates; if guesses are sampled from an unbiased distribution, however, averaging should reduce error (variance; Laplace, 1812/1878; Wallsten, Budescu, Erev, & Diederich, 1997). Probabilistic representations have been postulated in recent models of memory (Steyvers, Griffiths, & Dennis, 2006), perception (Kersten & Yuille, 2003), and neural coding (Ma, Beck, Latham, & Pouget, 2006). It is consistent with such models that responses of many people are distributed probabilistically, as shown by the wisdom-of-crowds effect. However, despite the theoretical appeal of these models, there has been scant evidence that, within a given person, knowledge is represented as a probability distribution. Finding any benefit of averaging two responses from one person would yield support for this hypothesis. METHOD We recruited 428 participants from an Internet- based subject pool and asked them eight questions probing their real-world knowledge (derived from The World Factbook, Central Intelligence Agency, 2007; e.g., “What percentage of the worlds airports are in the

Spacing Effects in Learning A Temporal Ridgeline of Optimal Retention

To achieve enduring retention, people must usually study information on multiple occasions. How does the timing of study events affect retention? Prior research has examined this issue only in a spotty fashion, usually with very short time intervals. In a study aimed at characterizing spacing effects over significant durations, more than 1,350 individuals were taught a set of facts and—after a gap of up to 3.5 months—given a review. A final test was administered at a further delay of up to 1 year. At any given test delay, an increase in the interstudy gap at first increased, and then gradually reduced, final test performance. The optimal gap increased as test delay increased. However, when measured as a proportion of test delay, the optimal gap declined from about 20 to 40% of a 1-week test delay to about 5 to 10% of a 1-year test delay. The interaction of gap and test delay implies that many educational practices are highly inefficient.

One and done? Optimal decisions from very few samples.

In many learning or inference tasks human behavior approximates that of a Bayesian ideal observer, suggesting that, at some level, cognition can be described as Bayesian inference. However, a number of findings have highlighted an intriguing mismatch between human behavior and standard assumptions about optimality: People often appear to make decisions based on just one or a few samples from the appropriate posterior probability distribution, rather than using the full distribution. Although sampling-based approximations are a common way to implement Bayesian inference, the very limited numbers of samples often used by humans seem insufficient to approximate the required probability distributions very accurately. Here, we consider this discrepancy in the broader framework of statistical decision theory, and ask: If people are making decisions based on samples--but as samples are costly--how many samples should people use to optimize their total expected or worst-case reward over a large number of decisions? We find that under reasonable assumptions about the time costs of sampling, making many quick but locally suboptimal decisions based on very few samples may be the globally optimal strategy over long periods. These results help to reconcile a large body of work showing sampling-based or probability matching behavior with the hypothesis that human cognition can be understood in Bayesian terms, and they suggest promising future directions for studies of resource-constrained cognition.

Scanning the horizon: towards transparent and reproducible neuroimaging research

Functional neuroimaging techniques have transformed our ability to probe the neurobiological basis of behaviour and are increasingly being applied by the wider neuroscience community. However, concerns have recently been raised that the conclusions that are drawn from some human neuroimaging studies are either spurious or not generalizable. Problems such as low statistical power, flexibility in data analysis, software errors and a lack of direct replication apply to many fields, but perhaps particularly to functional MRI. Here, we discuss these problems, outline current and suggested best practices, and describe how we think the field should evolve to produce the most meaningful and reliable answers to neuroscientific questions.

Pure Reasoning in 12-Month-Old Infants as Probabilistic Inference

Twelve-month-old infants employ Bayesian statistics. Many organisms can predict future events from the statistics of past experience, but humans also excel at making predictions by pure reasoning: integrating multiple sources of information, guided by abstract knowledge, to form rational expectations about novel situations, never directly experienced. Here, we show that this reasoning is surprisingly rich, powerful, and coherent even in preverbal infants. When 12-month-old infants view complex displays of multiple moving objects, they form time-varying expectations about future events that are a systematic and rational function of several stimulus variables. Infants’ looking times are consistent with a Bayesian ideal observer embodying abstract principles of object motion. The model explains infants’ statistical expectations and classic qualitative findings about object cognition in younger babies, not originally viewed as probabilistic inferences.

Everything You Never Wanted to Know about Circular Analysis, but Were Afraid to Ask

Over the past year, a heated discussion about ‘circular’ or ‘nonindependent’ analysis in brain imaging has emerged in the literature. An analysis is circular (or nonindependent) if it is based on data that were selected for showing the effect of interest or a related effect. The authors of this paper are researchers who have contributed to the discussion and span a range of viewpoints. To clarify points of agreement and disagreement in the community, we collaboratively assembled a series of questions on circularity herein, to which we provide our individual current answers in ≤100 words per question. Although divergent views remain on some of the questions, there is also a substantial convergence of opinion, which we have summarized in a consensus box. The box provides the best current answers that the five authors could agree upon.

Multistability and perceptual inference

Ambiguous images present a challenge to the visual system: How can uncertainty about the causes of visual inputs be represented when there are multiple equally plausible causes? A Bayesian ideal observer should represent uncertainty in the form of a posterior probability distribution over causes. However, in many real-world situations, computing this distribution is intractable and requires some form of approximation. We argue that the visual system approximates the posterior over underlying causes with a set of samples and that this approximation strategy produces perceptual multistability—stochastic alternation between percepts in consciousness. Under our analysis, multistability arises from a dynamic sample-generating process that explores the posterior through stochastic diffusion, implementing a rational form of approximate Bayesian inference known as Markov chain Monte Carlo (MCMC). We examine in detail the most extensively studied form of multistability, binocular rivalry, showing how a variety of experimental phenomena—gamma-like stochastic switching, patchy percepts, fusion, and traveling waves—can be understood in terms of MCMC sampling over simple graphical models of the underlying perceptual tasks. We conjecture that the stochastic nature of spiking neurons may lend itself to implementing sample-based posterior approximations in the brain.

Voodoo and circularity errors

We briefly describe the circularity/non-independence problem, and our perception of the impact the ensuing discussion has had on fMRI research.