Nikolaus Kriegeskorte

Representational similarity analysis - connecting the branches of systems neuroscience.

A fundamental challenge for systems neuroscience is to quantitatively relate its three major branches of research: brain-activity measurement, behavioral measurement, and computational modeling. Using measured brain-activity patterns to evaluate computational network models is complicated by the need to define the correspondency between the units of the model and the channels of the brain-activity data, e.g., single-cell recordings or voxels from functional magnetic resonance imaging (fMRI). Similar correspondency problems complicate relating activity patterns between different modalities of brain-activity measurement (e.g., fMRI and invasive or scalp electrophysiology), and between subjects and species. In order to bridge these divides, we suggest abstracting from the activity patterns themselves and computing representational dissimilarity matrices (RDMs), which characterize the information carried by a given representation in a brain or model. Building on a rich psychological and mathematical literature on similarity analysis, we propose a new experimental and data-analytical framework called representational similarity analysis (RSA), in which multi-channel measures of neural activity are quantitatively related to each other and to computational theory and behavior by comparing RDMs. We demonstrate RSA by relating representations of visual objects as measured with fMRI in early visual cortex and the fusiform face area to computational models spanning a wide range of complexities. The RDMs are simultaneously related via second-level application of multidimensional scaling and tested using randomization and bootstrap techniques. We discuss the broad potential of RSA, including novel approaches to experimental design, and argue that these ideas, which have deep roots in psychology and neuroscience, will allow the integrated quantitative analysis of data from all three branches, thus contributing to a more unified systems neuroscience.

Matching categorical object representations in inferior temporal cortex of man and monkey.

Inferior temporal (IT) object representations have been intensively studied in monkeys and humans, but representations of the same particular objects have never been compared between the species. Moreover, ITs role in categorization is not well understood. Here, we presented monkeys and humans with the same images of real-world objects and measured the IT response pattern elicited by each image. In order to relate the representations between the species and to computational models, we compare response-pattern dissimilarity matrices. IT response patterns form category clusters, which match between man and monkey. The clusters correspond to animate and inanimate objects; within the animate objects, faces and bodies form subclusters. Within each category, IT distinguishes individual exemplars, and the within-category exemplar similarities also match between the species. Our findings suggest that primate IT across species may host a common code, which combines a categorical and a continuous representation of objects.

Individual faces elicit distinct response patterns in human anterior temporal cortex

Visual face identification requires distinguishing between thousands of faces we know. This computational feat involves a network of brain regions including the fusiform face area (FFA) and anterior inferotemporal cortex (aIT), whose roles in the process are not well understood. Here, we provide the first demonstration that it is possible to discriminate cortical response patterns elicited by individual face images with high-resolution functional magnetic resonance imaging (fMRI). Response patterns elicited by the face images were distinct in aIT but not in the FFA. Individual-level face information is likely to be present in both regions, but our data suggest that it is more pronounced in aIT. One interpretation is that the FFA detects faces and engages aIT for identification.

Cortical capacity constraints for visual working memory: dissociation of fMRI load effects in a fronto-parietal network

Working memory (WM) capacity limitations and their neurophysiological correlates are of special relevance for the understanding of higher cognitive functions. Evidence from behavioral studies suggests that restricted attentional resources contribute to these capacity limitations. In an event-related functional magnetic resonance imaging (fMRI) study, we probed the capacity of the human visual WM system for up to four complex nonnatural objects using a delayed discrimination task. A number of prefrontal and parietal areas bilaterally showed increased blood oxygen level-dependent activity, relative to baseline, throughout the task when more than one object had to be held in memory. Monotonic increases in response to memory load were observed bilaterally in the dorsolateral prefrontal cortex (DLPFC) and the presupplementary motor area (pre-SMA). Conversely, activity in the frontal eye fields (FEFs) and in areas along the intraparietal sulcus (IPS) peaked when subjects had to maintain only two or three objects and decreased in the highest load condition. This dissociation of memory load effects on cortical activity suggests that the cognitive operations subserved by the IPS and FEF, which are most likely related to attention, fail to support visual WM when the capacity limit is approached. The correlation of brain activity with performance implies that only the operations performed by the DLPFC and pre-SMA, which support an integrated representation of visual information, helped subjects to maintain a reasonable level of performance in the highest load condition. These results indicate that at least two distinct cortical subsystems are recruited for visual WM, and that their interplay changes when the capacity limit is reached.

Revealing representational content with pattern-information fMRI—an introductory guide

Conventional statistical analysis methods for functional magnetic resonance imaging (fMRI) data are very successful at detecting brain regions that are activated as a whole during specific mental activities. The overall activation of a region is usually taken to indicate involvement of the region in the task. However, such activation analysis does not consider the multivoxel patterns of activity within a brain region. These patterns of activity, which are thought to reflect neuronal population codes, can be investigated by pattern-information analysis. In this framework, a regions multivariate pattern information is taken to indicate representational content. This tutorial introduction motivates pattern-information analysis, explains its underlying assumptions, introduces the most widespread methods in an intuitive way, and outlines the basic sequence of analysis steps.

Neural correlates of trust

Trust is a critical social process that helps us to cooperate with others and is present to some degree in all human interaction. However, the underlying brain mechanisms of conditional and unconditional trust in social reciprocal exchange are still obscure. Here, we used hyperfunctional magnetic resonance imaging, in which two strangers interacted online with one another in a sequential reciprocal trust game while their brains were simultaneously scanned. By designing a nonanonymous, alternating multiround game, trust became bidirectional, and we were able to quantify partnership building and maintenance. Using within- and between-brain analyses, an examination of functional brain activity supports the hypothesis that the preferential activation of different neuronal systems implements these two trust strategies. We show that the paracingulate cortex is critically involved in building a trust relationship by inferring another persons intentions to predict subsequent behavior. This more recently evolved brain region can be differently engaged to interact with more primitive neural systems in maintaining conditional and unconditional trust in a partnership. Conditional trust selectively activated the ventral tegmental area, a region linked to the evaluation of expected and realized reward, whereas unconditional trust selectively activated the septal area, a region linked to social attachment behavior. The interplay of these neural systems supports reciprocal exchange that operates beyond the immediate spheres of kinship, one of the distinguishing features of the human species.

Representational geometry: integrating cognition, computation, and the brain

Highlights • Representational geometry is a framework that enables us to relate brain, computation, and cognition.• Representations in brains and models can be characterized by representational distance matrices.• Distance matrices can be readily compared to test computational models.• We review recent insights into perception, cognition, memory, and action and discuss current challenges.

Deep supervised, but not unsupervised, models may explain IT cortical representation.

Inferior temporal (IT) cortex in human and nonhuman primates serves visual object recognition. Computational object-vision models, although continually improving, do not yet reach human performance. It is unclear to what extent the internal representations of computational models can explain the IT representation. Here we investigate a wide range of computational model representations (37 in total), testing their categorization performance and their ability to account for the IT representational geometry. The models include well-known neuroscientific object-recognition models (e.g. HMAX, VisNet) along with several models from computer vision (e.g. SIFT, GIST, self-similarity features, and a deep convolutional neural network). We compared the representational dissimilarity matrices (RDMs) of the model representations with the RDMs obtained from human IT (measured with fMRI) and monkey IT (measured with cell recording) for the same set of stimuli (not used in training the models). Better performing models were more similar to IT in that they showed greater clustering of representational patterns by category. In addition, better performing models also more strongly resembled IT in terms of their within-category representational dissimilarities. Representational geometries were significantly correlated between IT and many of the models. However, the categorical clustering observed in IT was largely unexplained by the unsupervised models. The deep convolutional network, which was trained by supervision with over a million category-labeled images, reached the highest categorization performance and also best explained IT, although it did not fully explain the IT data. Combining the features of this model with appropriate weights and adding linear combinations that maximize the margin between animate and inanimate objects and between faces and other objects yielded a representation that fully explained our IT data. Overall, our results suggest that explaining IT requires computational features trained through supervised learning to emphasize the behaviorally important categorical divisions prominently reflected in IT.

Analyzing for information, not activation, to exploit high-resolution fMRI

High-resolution functional magnetic resonance imaging (hi-res fMRI) promises to help bridge the gap between the macro- and the microview of brain function afforded by conventional neuroimaging and invasive cell recording, respectively. Hi-res fMRI (voxel volume<or=(2 mm)3 is robustly achievable in human studies today using widely available clinical 3-Tesla scanners. However, the neuroscientific exploitation of the greater spatial detail poses four challenges: (1) Hi-res fMRI may give inaccurate (i.e. blurred, displaced and distorted) images of fine-scale neuronal activity patterns. (2) Single small voxels yield very noisy measurements. (3) The greater number of voxels complicates interpretation and poses a more severe multiple-comparisons problem. (4) The functional correspondency mapping between individual brains is unknown at the fine scale of millimeters. Here we argue that these challenges can be met by shifting the focus of brain mapping and visualizing, not the activity patterns themselves, but the amount of information they convey about the experimental conditions.

Primary Visual Cortex Activity along the Apparent-Motion Trace Reflects Illusory Perception

The illusion of apparent motion can be induced when visual stimuli are successively presented at different locations. It has been shown in previous studies that motion-sensitive regions in extrastriate cortex are relevant for the processing of apparent motion, but it is unclear whether primary visual cortex (V1) is also involved in the representation of the illusory motion path. We investigated, in human subjects, apparent-motion-related activity in patches of V1 representing locations along the path of illusory stimulus motion using functional magnetic resonance imaging. Here we show that apparent motion caused a blood-oxygenation-level-dependent response along the V1 representations of the apparent-motion path, including regions that were not directly activated by the apparent-motion-inducing stimuli. This response was unaltered when participants had to perform an attention-demanding task that diverted their attention away from the stimulus. With a bistable motion quartet, we confirmed that the activity was related to the conscious perception of movement. Our data suggest that V1 is part of the network that represents the illusory path of apparent motion. The activation in V1 can be explained either by lateral interactions within V1 or by feedback mechanisms from higher visual areas, especially the motion-sensitive human MT/V5 complex.