Blakeley B. McShane

A statistical analysis of multiple temperature proxies: Are reconstructions of surface temperatures over the last 1000 years reliable?

Predicting historic temperatures based on tree rings, ice cores, and other natural proxies is a difficult endeavor. The relationship between proxies and temperature is weak and the number of proxies is far larger than the number of target data points. Furthermore, the data contain complex spatial and temporal dependence structures which are not easily captured with simple models. In this paper, we assess the reliability of such reconstructions and their statistical significance against various null models. We find that the proxies do not predict temperature significantly better than random series generated independently of temperature. Furthermore, various model specifications that perform similarly at predicting temperature produce extremely different historical backcasts. Finally, the proxies seem unable to forecast the high levels of and sharp run-up in temperature in the 1990s either in-sample or from contiguous holdout blocks, thus casting doubt on their ability to predict such phenomena if in fact they occurred several hundred years ago. We propose our own reconstruction of Northern Hemisphere average annual land temperature over the last millennium, assess its reliability, and compare it to those from the climate science literature. Our model provides a similar reconstruction but has much wider standard errors, reflecting the weak signal and large uncertainty encountered in this setting.

Characterization of the bout durations of sleep and wakefulness

STUDY OBJECTIVES (a) Develop a new statistical approach to describe the microarchitecture of wakefulness and sleep in mice; (b) evaluate differences among inbred strains in this microarchitecture; (c) compare results when data are scored in 4-s versus 10-s epochs. DESIGN Studies in male mice of four inbred strains: AJ, C57BL/6, DBA and PWD. EEG/EMG were recorded for 24h and scored independently in 4-s and 10-s epochs. MEASUREMENTS AND RESULTS Distribution of bout durations of wakefulness, NREM and REM sleep in mice has two distinct components, i.e., short and longer bouts. This is described as a spike (short bouts) and slab (longer bouts) distribution, a particular type of mixture model. The distribution in any state depends on the state the mouse is transitioning from and can be characterized by three parameters: the number of such bouts conditional on the previous state, the size of the spike, and the average length of the slab. While conventional statistics such as time spent in state, average bout duration, and number of bouts show some differences between inbred strains, this new statistical approach reveals more major differences. The major difference between strains is their ability to sustain long bouts of NREM sleep or wakefulness. Scoring mouse sleep/wake in 4-s epochs offered little new information when using conventional metrics but did when evaluating the microarchitecture based on this new approach. CONCLUSIONS Standard statistical approaches do not adequately characterize the microarchitecture of mouse behavioral state. Approaches based on a spike-and-slab provide a quantitative description.

Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes

We review and evaluate selection methods, a prominent class of techniques first proposed by Hedges (1984) that assess and adjust for publication bias in meta-analysis, via an extensive simulation study. Our simulation covers both restrictive settings as well as more realistic settings and proceeds across multiple metrics that assess different aspects of model performance. This evaluation is timely in light of two recently proposed approaches, the so-called p-curve and p-uniform approaches, that can be viewed as alternative implementations of the original Hedges selection method approach. We find that the p-curve and p-uniform approaches perform reasonably well but not as well as the original Hedges approach in the restrictive setting for which all three were designed. We also find they perform poorly in more realistic settings, whereas variants of the Hedges approach perform well. We conclude by urging caution in the application of selection methods: Given the idealistic model assumptions underlying selection methods and the sensitivity of population average effect size estimates to them, we advocate that selection methods should be used less for obtaining a single estimate that purports to adjust for publication bias ex post and more for sensitivity analysis—that is, exploring the range of estimates that result from assuming different forms of and severity of publication bias.

You Cannot Step Into the Same River Twice When Power Analyses Are Optimistic

Statistical power depends on the size of the effect of interest. However, effect sizes are rarely fixed in psychological research: Study design choices, such as the operationalization of the dependent variable or the treatment manipulation, the social context, the subject pool, or the time of day, typically cause systematic variation in the effect size. Ignoring this between-study variation, as standard power formulae do, results in assessments of power that are too optimistic. Consequently, when researchers attempting replication set sample sizes using these formulae, their studies will be underpowered and will thus fail at a greater than expected rate. We illustrate this with both hypothetical examples and data on several well-studied phenomena in psychology. We provide formulae that account for between-study variation and suggest that researchers set sample sizes with respect to our generally more conservative formulae. Our formulae generalize to settings in which there are multiple effects of interest. We also introduce an easy-to-use website that implements our approach to setting sample sizes. Finally, we conclude with recommendations for quantifying between-study variation.

Visual influence and social groups

New car purchases are among the largest and most expensive purchases consumers ever make. While functional and economic concerns are important, the authors examine whether visual influence also plays a role. Using a hierarchical Bayesian probability model and data on 1.6 million new cars sold over nine years, they examine how visual influence affects purchase volume, focusing on three questions: Are people more likely to buy a new car if others around them have recently done so? Are these effects moderated by visibility, the ease of seeing others’ behavior? Do they vary according to the identity (e.g., gender) of prior purchasers and the identity relevance of vehicle type? The authors perform an extensive set of tests to rule out alternatives to visual influence and find that visual effects are (1) present (one additional purchase for approximately every seven prior purchases), (2) larger in areas where others’ behavior should be more visible (i.e., more people commute in car-visible ways), (3) stronger for prior purchases by men than by women in male-oriented vehicle types, (4) extant only for cars of similar price tiers, and (5) subject to saturation effects.

Can small victories help win the war? evidence from consumer debt management

The question of how people should structure goal-directed activity to maximize the likelihood of goal attainment is one of theoretical and practical significance. In particular, should people begin by attempting relatively easy tasks or more difficult ones? How might these differing strategies affect the likelihood of completing the overarching goal? The authors examine this question in the context of an important goal for a large number of consumers—getting out of debt. Using a data set obtained from a debt settlement firm, they find that (1) closing debt accounts is predictive of debt elimination regardless of the dollar balance of the closed accounts, whereas (2) the dollar balance of closed accounts is not predictive of debt elimination when controlling for the fraction of accounts closed. These findings suggest that completing discrete subtasks might motivate consumers to persist in pursuit of a goal. The authors discuss implications for goal pursuit generally and for consumer debt management specifically.

Five ways to fix statistics

As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves. As debate rumbles on about how and how much poor statistics is to blame for poor reproducibility, Nature asked influential statisticians to recommend one change to improve science. The common theme? The problem is not our maths, but ourselves. Illustration by David Parkins

Assessing REM Sleep in Mice Using Video Data

STUDY OBJECTIVES Assessment of sleep and its substages in mice currently requires implantation of chronic electrodes for measurement of electroencephalogram (EEG) and electromyogram (EMG). This is not ideal for high-throughput screening. To address this deficiency, we present a novel method based on digital video analysis. This methodology extends previous approaches that estimate sleep and wakefulness without EEG/EMG in order to now discriminate rapid eye movement (REM) from non-REM (NREM) sleep. DESIGN Studies were conducted in 8 male C57BL/6J mice. EEG/EMG were recorded for 24 hours and manually scored in 10-second epochs. Mouse behavior was continuously recorded by digital video at 10 frames/second. Six variables were extracted from the video for each 10-second epoch (i.e., intraepoch mean of velocity, aspect ratio, and area of the mouse and intraepoch standard deviation of the same variables) and used as inputs for our model. MEASUREMENTS AND RESULTS We focus on estimating features of REM (i.e., time spent in REM, number of bouts, and median bout length) as well as time spent in NREM and WAKE. We also consider the models epoch-by-epoch scoring performance relative to several alternative approaches. Our model provides good estimates of these features across the day both when averaged across mice and in individual mice, but the epoch-by-epoch agreement is not as good. CONCLUSIONS There are subtle changes in the area and shape (i.e., aspect ratio) of the mouse as it transitions from NREM to REM, likely due to the atonia of REM, thus allowing our methodology to discriminate these two states. Although REM is relatively rare, our methodology can detect it and assess the amount of REM sleep.

Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence

Statistical training helps individuals analyze and interpret data. However, the emphasis placed on null hypothesis significance testing in academic training and reporting may lead researchers to interpret evidence dichotomously rather than continuously. Consequently, researchers may either disregard evidence that fails to attain statistical significance or undervalue it relative to evidence that attains statistical significance. Surveys of researchers across a wide variety of fields (including medicine, epidemiology, cognitive science, psychology, business, and economics) show that a substantial majority does indeed do so. This phenomenon is manifest both in researchers’ interpretations of descriptions of evidence and in their likelihood judgments. Dichotomization of evidence is reduced though still present when researchers are asked to make decisions based on the evidence, particularly when the decision outcome is personally consequential. Recommendations are offered. This paper was accepted by Yuval Rot...

Hierarchical Bayesian modeling of hitting performance in baseball

We have developed a sophisticated statistical model for predicting the hitting performance of Major League baseball players. The Bayesian paradigm provides a principled method for balancing past performance with crucial covari- ates, such as player age and position. We share information across time and across players by using mixture distributions to control shrinkage for improved accuracy. We compare the performance of our model to current sabermetric methods on a held-out season (2006), and discuss both successes and limitations.