Katherine S. Button

Power failure: why small sample size undermines the reliability of neuroscience

A study with low statistical power has a reduced chance of detecting a true effect, but it is less well appreciated that low power also reduces the likelihood that a statistically significant result reflects a true effect. Here, we show that the average statistical power of studies in the neurosciences is very low. The consequences of this include overestimates of effect size and low reproducibility of results. There are also ethical dimensions to this problem, as unreliable research is inefficient and wasteful. Improving reproducibility in neuroscience is a key priority and requires attention to well-established but often ignored methodological principles.

A manifesto for reproducible science

Improving the reliability and efficiency of scientific research will increase the credibility of the published scientific literature and accelerate discovery. Here we argue for the adoption of measures to optimize key elements of the scientific process: methods, reporting and dissemination, reproducibility, evaluation and incentives. There is some evidence from both simulations and empirical studies supporting the likely effectiveness of these measures, but their broad adoption by researchers, institutions, funders and journals will require iterative evaluation and improvement. We discuss the goals of these measures, and how they can be implemented, in the hope that this will facilitate action toward improving the transparency, reproducibility and efficiency of scientific research.

Deep impact: unintended consequences of journal rank

Most researchers acknowledge an intrinsic hierarchy in the scholarly journals (“journal rank”) that they submit their work to, and adjust not only their submission but also their reading strategies accordingly. On the other hand, much has been written about the negative effects of institutionalizing journal rank as an impact measure. So far, contributions to the debate concerning the limitations of journal rank as a scientific impact assessment tool have either lacked data, or relied on only a few studies. In this review, we present the most recent and pertinent data on the consequences of our current scholarly communication system with respect to various measures of scientific quality (such as utility/citations, methodological soundness, expert ratings or retractions). These data corroborate previous hypotheses: using journal rank as an assessment tool is bad scientific practice. Moreover, the data lead us to argue that any journal rank (not only the currently-favored Impact Factor) would have this negative impact. Therefore, we suggest that abandoning journals altogether, in favor of a library-based scholarly communication system, will ultimately be necessary. This new system will use modern information technology to vastly improve the filter, sort and discovery functions of the current journal system.

Peer victimization during adolescence and risk for anxiety disorders in adulthood: a prospective cohort study.

Peer victimization is ubiquitous across schools and cultures, and has been suggested as one developmental pathway to anxiety disorders. However, there is a dearth of prospective studies examining this relationship. The purpose of this cohort study was to examine the association between peer victimization during adolescence and subsequent anxiety diagnoses in adulthood. A secondary aim was to investigate whether victimization increases risk for severe anxiety presentations involving diagnostic comorbidity.

Minimal clinically important difference on the Beck Depression Inventory - II according to the patient's perspective

Background The Beck Depression Inventory, 2nd edition (BDI-II) is widely used in research on depression. However, the minimal clinically important difference (MCID) is unknown. MCID can be estimated in several ways. Here we take a patient-centred approach, anchoring the change on the BDI-II to the patients global report of improvement. Method We used data collected (n = 1039) from three randomized controlled trials for the management of depression. Improvement on a ‘global rating of change’ question was compared with changes in BDI-II scores using general linear modelling to explore baseline dependency, assessing whether MCID is best measured in absolute terms (i.e. difference) or as percent reduction in scores from baseline (i.e. ratio), and receiver operator characteristics (ROC) to estimate MCID according to the optimal threshold above which individuals report feeling ‘better’. Results Improvement in BDI-II scores associated with reporting feeling ‘better’ depended on initial depression severity, and statistical modelling indicated that MCID is best measured on a ratio scale as a percentage reduction of score. We estimated a MCID of a 17.5% reduction in scores from baseline from ROC analyses. The corresponding estimate for individuals with longer duration depression who had not responded to antidepressants was higher at 32%. Conclusions MCID on the BDI-II is dependent on baseline severity, is best measured on a ratio scale, and the MCID for treatment-resistant depression is larger than that for more typical depression. This has important implications for clinical trials and practice.

Scientific rigor and the art of motorcycle maintenance

The reliability of scientific research is under scrutiny. A recently convened working group proposes cultural adjustments to incentivize better research practices.

Social anxiety is associated with general but not specific biases in emotion recognition

Misreading facial expressions as signals of social disapproval, such as anger and disgust, may maintain social anxiety. If so, manipulating face processing could be therapeutic. It remains unclear, however, whether socially anxious individuals are in fact more sensitive to disapproving emotions. We assessed decoding of, and cost attributions to, emotional expressions in high and low socially anxious females (n=102) using five emotions (anger, disgust, fear, happiness, and sadness) expressed at 15 intensities (9-65%), providing 75 stimuli (see Supplementary Material). The decoding task briefly presented the stimuli and participants identified the emotion. The cost attribution task asked individuals to rate each stimulus for how costly it would be for them to interact with the person. Random effects regression indicated that social anxiety was not associated with overall decoding accuracy but was associated with a response bias. High socially anxious individuals had a lower threshold for decoding emotions but also more frequently classified low intensity emotions incorrectly. These effects were not emotion-specific. Socially anxious individuals also attributed excessive social cost to expressions of negative valence. Our results provide a novel conceptual framework for understanding emotion decoding in social anxiety, indicating the importance of considering both accuracy and response bias.

Low statistical power in biomedical science:a review of three human research domains

Studies with low statistical power increase the likelihood that a statistically significant finding represents a false positive result. We conducted a review of meta-analyses of studies investigating the association of biological, environmental or cognitive parameters with neurological, psychiatric and somatic diseases, excluding treatment studies, in order to estimate the average statistical power across these domains. Taking the effect size indicated by a meta-analysis as the best estimate of the likely true effect size, and assuming a threshold for declaring statistical significance of 5%, we found that approximately 50% of studies have statistical power in the 0–10% or 11–20% range, well below the minimum of 80% that is often considered conventional. Studies with low statistical power appear to be common in the biomedical sciences, at least in the specific subject areas captured by our search strategy. However, we also observe evidence that this depends in part on research methodology, with candidate gene studies showing very low average power and studies using cognitive/behavioural measures showing high average power. This warrants further investigation.

Psychotic Experiences and Working Memory: A Population-Based Study Using Signal- Detection Analysis

Psychotic Experiences (PEs) during adolescence index increased risk for psychotic disorders and schizophrenia in adult life. Working memory (WM) deficits are a core feature of these disorders. Our objective was to examine the relationship between PEs and WM in a general population sample of young people in a case control study. 4744 individuals of age 17–18 from Bristol and surrounding areas (UK) were analyzed in a cross-sectional study nested within the Avon Longitudinal Study of Parents and Children (ALSPAC) birth cohort study. The dependent variable was PEs, assessed using the semi-structured Psychosis-Like Symptom Interview (PLIKSi). The independent variable was performance on a computerized numerical n-back working memory task. Signal-Detection Theory indices, including standardized hits rate, false alarms rate, discriminability index (d’) and response bias (c) from 2-Back and 3-Back tasks were calculated. 3576 and 3527 individuals had complete data for 2-Back and 3-Back respectively. Suspected/definite PEs prevalence was 7.9% (N = 374). Strongest evidence of association was seen between PEs and false alarms on the 2-Back, (odds ratio (OR) = 1.17 [95% confidence intervals (CI) 1.01, 1.35]) and 3-back (OR = 1.35 [1.18, 1.54]) and with c (OR = 1.59 [1.09, 2.34]), and lower d’ (OR = 0.76 [0.65, 0.89]), on the 3-Back. Adjustment for several potential confounders, including general IQ, drug exposure and different psycho-social factors, and subsequent multiple imputation of missing data did not materially alter the results. WM is impaired in young people with PEs in the general population. False alarms, rather than poor accuracy, are more closely related to PEs. Such impairment is consistent with different neuropsychological models of psychosis focusing on signal-to-noise discrimination, probabilistic reasoning and impaired reality monitoring as a basis of psychotic symptoms.

Cognitive styles and future depressed mood in early adulthood: The importance of global attributions

Background Cognitive theories of depression suggest that beliefs of low self-worth and the tendency to attribute negative events to causes that are global (widespread rather than specific) and stable (will persist rather than change in the future) are associated with the development of depressed mood. Such theories are supported by evidence from prospective studies and have guided the development of successful treatment and prevention strategies such as CBT. However, the relative importance of different psychological constructs within cognitive theories is unknown. This is important to refine cognitive theories and develop more efficient prevention strategies. Method We used prospective data from over 3500 young adults from the Avon Longitudinal Study for Parents and Children (ALSPAC) cohort in the UK to investigate the association between cognitive style, measured by short forms of the Dysfunctional Attitudes Scale (DAS) and Cognitive Styles Questionnaire-Short Form (CSQ-SF) at age 18, and future depressed mood at age 19. Structural equation modelling techniques were used to separate cognitive style constructs. Results Cognitive styles were associated with future depressed mood, independently of baseline mood, both as measured by the DAS-SF and the CSQ-SF. Of the different CSQ-SF constructs, only global attributions were associated with both baseline and future mood independently of other constructs. Limitations The study was subject to attrition and the follow-up was relatively short (10 months). Conclusion The findings suggest that the tendency to attribute negative events specifically to global causes could be particularly important for depression. Reducing global attributions is potentially important in the prevention and treatment of depression.