Wolfgang M. Pauli

Regional specialization within the human striatum for diverse psychological functions

Significance The subcortical striatum is critical for the planning and execution of motor behavior, and its dysfunction is associated with disorders such as Parkinson’s disease. More recently, the human striatum has also been reported to be involved in heterogeneous nonmotor psychological functions. However, detailed functional mappings of human psychological processes to striatal regions have been bound by theoretical and methodological limitations, including a strong focus on experimental paradigms derived from animal research, and the tendency to infer function from anatomical connectivity, rather than task-related activation. To overcome these limitations, we used a large-scale, unbiased, data-driven approach, and generated a precise, comprehensive functional map, directly associating striatal zones with the broadest range of psychological processes to date. Decades of animal and human neuroimaging research have identified distinct, but overlapping, striatal zones, which are interconnected with separable corticostriatal circuits, and are crucial for the organization of functional systems. Despite continuous efforts to subdivide the human striatum based on anatomical and resting-state functional connectivity, characterizing the different psychological processes related to each zone remains a work in progress. Using an unbiased, data-driven approach, we analyzed large-scale coactivation data from 5,809 human imaging studies. We (i) identified five distinct striatal zones that exhibited discrete patterns of coactivation with cortical brain regions across distinct psychological processes and (ii) identified the different psychological processes associated with each zone. We found that the reported pattern of cortical activation reliably predicted which striatal zone was most strongly activated. Critically, activation in each functional zone could be associated with distinct psychological processes directly, rather than inferred indirectly from psychological functions attributed to associated cortices. Consistent with well-established findings, we found an association of the ventral striatum (VS) with reward processing. Confirming less well-established findings, the VS and adjacent anterior caudate were associated with evaluating the value of rewards and actions, respectively. Furthermore, our results confirmed a sometimes overlooked specialization of the posterior caudate nucleus for executive functions, often considered the exclusive domain of frontoparietal cortical circuits. Our findings provide a precise functional map of regional specialization within the human striatum, both in terms of the differential cortical regions and psychological functions associated with each striatal zone.

Learning, Reward, and Decision Making

&NA; In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward‐related behavior, including a dichotomy between the goal‐directed or model‐based system and the habitual or model‐free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems. We consider the nature of the interactions between these systems and show how these interactions can lead to either adaptive or maladaptive behavioral outcomes. We then review evidence that an additional system guides inference concerning the hidden states of other agents, such as their beliefs, preferences, and intentions, in a social context. We also describe emerging evidence for an arbitration mechanism between model‐based and model‐free reinforcement learning, placing such a mechanism within the broader context of the hierarchical control of behavior.

In vivo delineation of subdivisions of the human amygdaloid complex in a high-resolution group template

The nuclei of the human amygdala remain difficult to distinguish in individual subject structural magnetic resonance images. However, interpretation of the amygdalas role in whole brain networks requires accurate localization of functional activity to a particular nucleus or subgroup of nuclei. To address this, high spatial resolution, three‐dimensional templates, using joint high accuracy diffeomorphic registration of T1‐ and T2‐weighted structural images from 168 typical adults between 22 and 35 years old released by the Human Connectome Project were constructed. Several internuclear boundaries are clearly visible in these templates, which would otherwise be impossible to delineate in individual subject data. A probabilistic atlas of major nuclei and nuclear groups was constructed in this template space and mapped back to individual spaces by inversion of the individual diffeomorphisms. Group level analyses revealed a slight (∼2%) bias toward larger total amygdala and nuclear volumes in the right hemisphere. No substantial sex or age differences were found in amygdala volumes normalized to total intracranial volume, or subdivision volumes normalized to amygdala volume. The current delineation provides a finer parcellation of the amygdala with more accurate external boundary definition than current histology‐based atlases when used in conjunction with high accuracy registration methods, such as diffeomorphic warping. These templates and delineation are intended to be an open and evolving resource for future functional and structural imaging studies of the human amygdala. Hum Brain Mapp 37:3979–3998, 2016.

Neural computations underlying inverse reinforcement learning in the human brain

In inverse reinforcement learning an observer infers the reward distribution available for actions in the environment solely through observing the actions implemented by another agent. To address whether this computational process is implemented in the human brain, participants underwent fMRI while learning about slot machines yielding hidden preferred and non-preferred food outcomes with varying probabilities, through observing the repeated slot choices of agents with similar and dissimilar food preferences. Using formal model comparison, we found that participants implemented inverse RL as opposed to a simple imitation strategy, in which the actions of the other agent are copied instead of inferring the underlying reward structure of the decision problem. Our computational fMRI analysis revealed that anterior dorsomedial prefrontal cortex encoded inferences about action-values within the value space of the agent as opposed to that of the observer, demonstrating that inverse RL is an abstract cognitive process divorceable from the values and concerns of the observer him/herself.

Distinct prediction errors in mesostriatal circuits of the human brain mediate learning about the values of both states and actions: evidence from high-resolution fMRI

Prediction-error signals consistent with formal models of “reinforcement learning” (RL) have repeatedly been found within dopaminergic nuclei of the midbrain and dopaminoceptive areas of the striatum. However, the precise form of the RL algorithms implemented in the human brain is not yet well determined. Here, we created a novel paradigm optimized to dissociate the subtypes of reward-prediction errors that function as the key computational signatures of two distinct classes of RL models—namely, “actor/critic” models and action-value-learning models (e.g., the Q-learning model). The state-value-prediction error (SVPE), which is independent of actions, is a hallmark of the actor/critic architecture, whereas the action-value-prediction error (AVPE) is the distinguishing feature of action-value-learning algorithms. To test for the presence of these prediction-error signals in the brain, we scanned human participants with a high-resolution functional magnetic-resonance imaging (fMRI) protocol optimized to enable measurement of neural activity in the dopaminergic midbrain as well as the striatal areas to which it projects. In keeping with the actor/critic model, the SVPE signal was detected in the substantia nigra. The SVPE was also clearly present in both the ventral striatum and the dorsal striatum. However, alongside these purely state-value-based computations we also found evidence for AVPE signals throughout the striatum. These high-resolution fMRI findings suggest that model-free aspects of reward learning in humans can be explained algorithmically with RL in terms of an actor/critic mechanism operating in parallel with a system for more direct action-value learning.

Computational approaches to habits in a model-free world

Model-free (MF) reinforcement learning (RL) algorithms account for a wealth of neuroscientific and behavioral data pertinent to habits; however, conspicuous disparities between model-predicted response patterns and experimental data have exposed the inadequacy of MF-RL to fully capture the domain of habitual behavior. We review several extensions to generic MF-RL algorithms that could narrow the gap between theory and empirical data. We discuss insights gained from extending RL algorithms to operate in complex environments with multidimensional continuous state spaces. We also review recent advances in hierarchical RL and their potential relevance to habits. Neurobiological evidence suggests that similar mechanisms for habitual learning and control may apply across diverse psychological domains.

Dopamine and the Adolescent Brain: Do Errors in Prediction Make the Difference?

Adolescence is associated with major social and cognitive changes, and it is known to be associated with an increased propensity for risky and impulsive behaviors compared with other developmental stages. These observations have led to a vigorous research agenda addressing the question of how increased risk factors for maladaptive decision making are due to specific differences in the adolescent reward system—for example, a hypersensitivity of the dopaminergic system (1).