Thomas E. Nichols

Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate

Finding objective and effective thresholds for voxelwise statistics derived from neuroimaging data has been a long-standing problem. With at least one test performed for every voxel in an image, some correction of the thresholds is needed to control the error rates, but standard procedures for multiple hypothesis testing (e.g., Bonferroni) tend to not be sensitive enough to be useful in this context. This paper introduces to the neuroscience literature statistical procedures for controlling the false discovery rate (FDR). Recent theoretical work in statistics suggests that FDR-controlling procedures will be effective for the analysis of neuroimaging data. These procedures operate simultaneously on all voxelwise test statistics to determine which tests should be considered statistically significant. The innovation of the procedures is that they control the expected proportion of the rejected hypotheses that are falsely rejected. We demonstrate this approach using both simulations and functional magnetic resonance imaging data from two simple experiments.

Nonparametric permutation tests for functional neuroimaging : a primer with examples

Requiring only minimal assumptions for validity, nonparametric permutation testing provides a flexible and intuitive methodology for the statistical analysis of data from functional neuroimaging experiments, at some computational expense. Introduced into the functional neuroimaging literature by Holmes et al. ([ 1996 ]: J Cereb Blood Flow Metab 16:7–22), the permutation approach readily accounts for the multiple comparisons problem implicit in the standard voxel‐by‐voxel hypothesis testing framework. When the appropriate assumptions hold, the nonparametric permutation approach gives results similar to those obtained from a comparable Statistical Parametric Mapping approach using a general linear model with multiple comparisons corrections derived from random field theory. For analyses with low degrees of freedom, such as single subject PET/SPECT experiments or multi‐subject PET/SPECT or fMRI designs assessed for population effects, the nonparametric approach employing a locally pooled (smoothed) variance estimate can outperform the comparable Statistical Parametric Mapping approach. Thus, these nonparametric techniques can be used to verify the validity of less computationally expensive parametric approaches. Although the theory and relative advantages of permutation approaches have been discussed by various authors, there has been no accessible explication of the method, and no freely distributed software implementing it. Consequently, there have been few practical applications of the technique. This article, and the accompanying MATLAB software, attempts to address these issues. The standard nonparametric randomization and permutation testing ideas are developed at an accessible level, using practical examples from functional neuroimaging, and the extensions for multiple comparisons described. Three worked examples from PET and fMRI are presented, with discussion, and comparisons with standard parametric approaches made where appropriate. Practical considerations are given throughout, and relevant statistical concepts are expounded in appendices. Hum. Brain Mapping 15:1–25, 2001.

Valid conjunction inference with the minimum statistic.

In logic a conjunction is defined as an AND between truth statements. In neuroimaging, investigators may look for brain areas activated by task A AND by task B, or a conjunction of tasks (Price, C.J., Friston, K.J., 1997. Cognitive conjunction: a new approach to brain activation experiments. NeuroImage 5, 261-270). Friston et al. (Friston, K., Holmes, A., Price, C., Buchel, C., Worsley, K., 1999. Multisubject fMRI studies and conjunction analyses. NeuroImage 10, 85-396) introduced a minimum statistic test for conjunction. We refer to this method as the minimum statistic compared to the global null (MS/GN). The MS/GN is implemented in SPM2 and SPM99 software, and has been widely used as a test of conjunction. However, we assert that it does not have the correct null hypothesis for a test of logical AND, and further, this has led to confusion in the neuroimaging community. In this paper, we define a conjunction and explain the problem with the MS/GN test as a conjunction method. We present a survey of recent practice in neuroimaging which reveals that the MS/GN test is very often misinterpreted as evidence of a logical AND. We show that a correct test for a logical AND requires that all the comparisons in the conjunction are individually significant. This result holds even if the comparisons are not independent. We suggest that the revised test proposed here is the appropriate means for conjunction inference in neuroimaging.

Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates

The most widely used task fMRI analyses use parametric methods that depend on a variety of assumptions. While individual aspects of these fMRI models have been evaluated, they have not been evaluated in a comprehensive manner with empirical data. In this work, a total of 2 million random task fMRI group analyses have been performed using resting state fMRI data, to compute empirical familywise error rates for the software packages SPM, FSL and AFNI, as well as a standard non-parametric permutation method. While there is some variation, for a nominal familywise error rate of 5% the parametric statistical methods are shown to be conservative for voxel-wise inference and invalid for cluster-wise inference; in particular, cluster size inference with a cluster defining threshold of p = 0.01 generates familywise error rates up to 60%. We conduct a number of follow up analyses and investigations that suggest the cause of the invalid cluster inferences is spatial auto correlation functions that do not follow the assumed Gaussian shape. By comparison, the non-parametric permutation test, which is based on a small number of assumptions, is found to produce valid results for voxel as well as cluster wise inference. Using real task data, we compare the results between one parametric method and the permutation test, and find stark differences in the conclusions drawn between the two using cluster inference. These findings speak to the need of validating the statistical methods being used in the neuroimaging field.Significance Functional MRI (fMRI) is 25 years old, yet surprisingly its most common statistical methods have not been validated using real data. Here, we used resting-state fMRI data from 499 healthy controls to conduct 3 million task group analyses. Using this null data with different experimental designs, we estimate the incidence of significant results. In theory, we should find 5% false positives (for a significance threshold of 5%), but instead we found that the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of a number of fMRI studies and may have a large impact on the interpretation of weakly significant neuroimaging results. The most widely used task functional magnetic resonance imaging (fMRI) analyses use parametric statistical methods that depend on a variety of assumptions. In this work, we use real resting-state data and a total of 3 million random task group analyses to compute empirical familywise error rates for the fMRI software packages SPM, FSL, and AFNI, as well as a nonparametric permutation method. For a nominal familywise error rate of 5%, the parametric statistical methods are shown to be conservative for voxelwise inference and invalid for clusterwise inference. Our results suggest that the principal cause of the invalid cluster inferences is spatial autocorrelation functions that do not follow the assumed Gaussian shape. By comparison, the nonparametric permutation test is found to produce nominal results for voxelwise as well as clusterwise inference. These findings speak to the need of validating the statistical methods being used in the field of neuroimaging.

Controlling the familywise error rate in functional neuroimaging: a comparative review

Functional neuroimaging data embodies a massive multiple testing problem, where 100 000 correlated test statistics must be assessed. The familywise error rate, the chance of any false positives is the standard measure of Type I errors in multiple testing. In this paper we review and evaluate three approaches to thresholding images of test statistics: Bonferroni, random field and the permutation test. Owing to recent developments, improved Bonferroni procedures, such as Hochberg’s methods, are now applicable to dependent data. Continuous random field methods use the smoothness of the image to adapt to the severity of the multiple testing problem. Also, increased computing power has made both permutation and bootstrap methods applicable to functional neuroimaging. We evaluate these approaches on t images using simulations and a collection of real datasets. We find that Bonferroni-related tests offer little improvement over Bonferroni, while the permutation method offers substantial improvement over the random field method for low smoothness and low degrees of freedom. We also show the limitations of trying to find an equivalent number of independent tests for an image of correlated test statistics.

Permutation inference for the general linear model.

Permutation methods can provide exact control of false positives and allow the use of non-standard statistics, making only weak assumptions about the data. With the availability of fast and inexpensive computing, their main limitation would be some lack of flexibility to work with arbitrary experimental designs. In this paper we report on results on approximate permutation methods that are more flexible with respect to the experimental design and nuisance variables, and conduct detailed simulations to identify the best method for settings that are typical for imaging research scenarios. We present a generic framework for permutation inference for complex general linear models (glms) when the errors are exchangeable and/or have a symmetric distribution, and show that, even in the presence of nuisance effects, these permutation inferences are powerful while providing excellent control of false positives in a wide range of common and relevant imaging research scenarios. We also demonstrate how the inference on glm parameters, originally intended for independent data, can be used in certain special but useful cases in which independence is violated. Detailed examples of common neuroimaging applications are provided, as well as a complete algorithm – the “randomise” algorithm – for permutation inference with the glm.

Placebo Effects Mediated by Endogenous Opioid Activity on μ-Opioid Receptors

Reductions in pain ratings when administered a placebo with expected analgesic properties have been described and hypothesized to be mediated by the pain-suppressive endogenous opioid system. Using molecular imaging techniques, we directly examined the activity of the endogenous opioid system on μ-opioid receptors in humans in sustained pain with and without the administration of a placebo. Significant placebo-induced activation of μ-opioid receptor-mediated neurotransmission was observed in both higher-order and sub-cortical brain regions, which included the pregenual and subgenual rostral anterior cingulate, the dorsolateral prefrontal cortex, the insular cortex, and the nucleus accumbens. Regional activations were paralleled by lower ratings of pain intensity, reductions in its sensory and affective qualities, and in the negative emotional state of the volunteers. These data demonstrate that cognitive factors (e.g., expectation of pain relief) are capable of modulating physical and emotional states through the site-specific activation of μ-opioid receptor signaling in the human brain.

Nonstationary cluster-size inference with random field and permutation methods

Because of their increased sensitivity to spatially extended signals, cluster-size tests are widely used to detect changes and activations in brain images. However, when images are nonstationary, the cluster-size distribution varies depending on local smoothness. Clusters tend to be large in smooth regions, resulting in increased false positives, while in rough regions, clusters tend to be small, resulting in decreased sensitivity. Worsley et al. proposed a random field theory (RFT) method that adjusts cluster sizes according to local roughness of images [Worsley, K.J., 2002. Nonstationary FWHM and its effect on statistical inference of fMRI data. Presented at the 8th International Conference on Functional Mapping of the Human Brain, June 2-6, 2002, Sendai, Japan. Available on CD-ROM in NeuroImage 16 (2) 779-780; Hum. Brain Mapp. 8 (1999) 98]. In this paper, we implement this method in a permutation test framework, which requires very few assumptions, is known to be exact [J. Cereb. Blood Flow Metab. 16 (1996) 7] and is robust [NeuroImage 20 (2003) 2343]. We compared our method to stationary permutation, stationary RFT, and nonstationary RFT methods. Using simulated data, we found that our permutation test performs well under any setting examined, whereas the nonstationary RFT test performs well only for smooth images under high df. We also found that the stationary RFT test becomes anticonservative under nonstationarity, while both nonstationary RFT and permutation tests remain valid under nonstationarity. On a real PET data set we found that, though the nonstationary tests have reduced sensitivity due to smoothness estimation variability, these tests have better sensitivity for clusters in rough regions compared to stationary cluster-size tests. We include a detailed and consolidated description of Worsley nonstationary RFT cluster-size test.

Statistical Parametric Mapping

1. INTRODUCTION This chapter is about making regionally specific inferences in neuroimaging. These inferences may be about differences expressed when comparing one group of subjects to another or, within subjects, over a sequence of observations. They may pertain to structural differences (e.g. in voxel-based morphometry-Ashburner and Friston 2000) or neurophysiological indices of brain functions (e.g. fMRI). The principles of data analysis are very similar for all of these applications and constitute the subject of this chapter. We will focus on the analysis of fMRI time-series because this covers most of the issues that are likely to be encountered in other modalities. Generally, 2 Chapter #1 the analysis of structural images and PET scans is simpler because they do not have to deal with correlated errors, from one scan to the next. A general issue, in data analysis, is the relationship between the neurobiological hypothesis one posits and the statistical models adopted to test that hypothesis. This chapter begins by reviewing the distinction between functional specialization and integration and how these principles serve as the motivation for most analyses of neuroimaging data. We will address the design and analysis of neuroimaging studies from both these perspectives but note that both have to be integrated for a full understanding of brain mapping results. Statistical parametric mapping is generally used to identify functionally specialized brain regions and is the most prevalent approach to characterizing functional anatomy and disease-related changes. The alternative perspective, namely that provided by functional integration, requires a different set of [multivariate] approaches that examine the relationship between changes in activity in one brain area and another. Statistical parametric mapping is a voxel-based approach, employing classical inference, to make some comment about regionally specific responses to experimental factors. In order to assign an observed response to a particular brain structure, or cortical area, the data must conform to a known anatomical space. Before considering statistical modeling, this chapter deals briefly with how a time-series of images are realigned and mapped into some standard anatomical space (e.g. a stereotactic space). The general ideas behind statistical parametric mapping are then described and illustrated with attention to the different sorts of inferences that can be #1. Statistical Parametric Mapping 3 made with different experimental designs. fMRI is special, in the sense that the data lend themselves to a signal processing perspective. This can be exploited to ensure that both the design and analysis are …

Validating cluster size inference: random field and permutation methods

Cluster size tests used in analyses of brain images can have more sensitivity compared to intensity based tests. The random field (RF) theory has been widely used in implementation of such tests, however the behavior of such tests is not well understood, especially when the RF assumptions are in doubt. In this paper, we carried out a simulation study of cluster size tests under varying smoothness, thresholds, and degrees of freedom, comparing RF performance to that of the permutation test, which is known to be exact. For Gaussian images, we find that the RF methods are generally conservative, especially for low smoothness and low threshold. For t images, the RF tests are found to be conservative at lower thresholds and do not perform well unless the threshold is high and images are sufficiently smooth. The permutation test performs well for any settings though the discreteness in cluster size must be accounted for. We make specific recommendations on when permutation tests are to be preferred to RF tests.