Soyoung Q. Park

Prefrontal Cortex Fails to Learn from Reward Prediction Errors in Alcohol Dependence

Patients suffering from addiction persist in consuming substances of abuse, despite negative consequences or absence of positive consequences. One potential explanation is that these patients are impaired at flexibly adapting their behavior to changes in reward contingencies. A key aspect of adaptive decision-making involves updating the value of behavioral options. This is thought to be mediated via a teaching signal expressed as a reward prediction error (PE) in the striatum. However, to exert control over adaptive behavior, value signals need to be broadcast to higher executive regions, such as prefrontal cortex. Here we used functional MRI and a reinforcement learning task to investigate the neural mechanisms underlying maladaptive behavior in human male alcohol-dependent patients. We show that in alcohol-dependent patients the expression of striatal PEs is intact. However, abnormal functional connectivity between striatum and dorsolateral prefrontal cortex (dlPFC) predicted impairments in learning and the magnitude of alcohol craving. These results are in line with reports of dlPFC structural abnormalities in substance dependence and highlight the importance of frontostriatal connectivity in addiction, and its pivotal role in adaptive updating of action values and behavioral regulation. Furthermore, they extend the scope of neurobiological deficits underlying addiction beyond the focus on the striatum.

Connectivity-Based Parcellation of the Human Orbitofrontal Cortex

The primate orbitofrontal cortex (OFC) is involved in reward processing, learning, and decision making. Research in monkeys has shown that this region is densely connected with higher sensory, limbic, and subcortical regions. Moreover, a parcellation of the monkey OFC into two subdivisions has been suggested based on its intrinsic anatomical connections. However, in humans, little is known about any functional subdivisions of the OFC except for a rather coarse medial/lateral distinction. Here, we used resting-state fMRI in combination with unsupervised clustering techniques to investigate whether OFC subdivisions can be revealed based on their functional connectivity profiles with other brain regions. Examination of different cluster solutions provided support for a parcellation into two parts as observed in monkeys, but it also highlighted a much finer hierarchical clustering of the orbital surface. Specifically, we identified (1) a medial, (2) a posterior-central, (3) a central, and (4–6) three lateral clusters spanning the anterior–posterior gradient. Consistent with animal tracing studies, these OFC clusters were connected to other cortical regions such as prefrontal, temporal, and parietal cortices but also subcortical areas in the striatum and the midbrain. These connectivity patterns provide important implications for identifying specific functional roles of OFC subdivisions for reward processing, learning, and decision making. Moreover, this parcellation schema can provide guidance to report results in future studies.

Neurobiology of Value Integration: When Value Impacts Valuation

Everyday choice options have advantages (positive values) and disadvantages (negative values) that need to be integrated into an overall subjective value. For decades, economic models have assumed that when a person evaluates a choice option, different values contribute independently to the overall subjective value of the option. However, human choice behavior often violates this assumption, suggesting interactions between values. To investigate how qualitatively different advantages and disadvantages are integrated into an overall subjective value, we measured the brain activity of human subjects using fMRI while they were accepting or rejecting choice options that were combinations of monetary reward and physical pain. We compared different subjective value models on behavioral and neural data. These models all made similar predictions of choice behavior, suggesting that behavioral data alone are not sufficient to uncover the underlying integration mechanism. Strikingly, a direct model comparison on brain data decisively demonstrated that interactive value integration (where values interact and affect overall valuation) predicts neural activity in value-sensitive brain regions significantly better than the independent mechanism. Furthermore, effective connectivity analyses revealed that value-dependent changes in valuation are associated with modulations in subgenual anterior cingulate cortex–amygdala coupling. These results provide novel insights into the neurobiological underpinnings of human decision making involving the integration of different values.

Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions

It has been suggested that the target areas of dopaminergic midbrain neurons, the dorsal (DS) and ventral striatum (VS), are differently involved in reinforcement learning especially as actor and critic. Whereas the critic learns to predict rewards, the actor maintains action values to guide future decisions. The different midbrain connections to the DS and the VS seem to play a critical role in this functional distinction. Here, subjects performed a dynamic, reward-based decision-making task during fMRI acquisition. A computational model of reinforcement learning was used to estimate the different effects of positive and negative reinforcements on future decisions for each subject individually. We found that activity in both the DS and the VS correlated with reward prediction errors. Using functional connectivity, we show that the DS and the VS are differentially connected to different midbrain regions (possibly corresponding to the substantia nigra [SN] and the ventral tegmental area [VTA], respectively). However, only functional connectivity between the DS and the putative SN predicted the impact of different reinforcement types on future behavior. These results suggest that connections between the putative SN and the DS are critical for modulating action values in the DS according to both positive and negative reinforcements to guide future decision making.

Decoding different roles for vmPFC and dlPFC in multi-attribute decision making

In everyday life, successful decision making requires precise representations of expected values. However, for most behavioral options more than one attribute can be relevant in order to predict the expected reward. Thus, to make good or even optimal choices the reward predictions of multiple attributes need to be integrated into a combined expected value. Importantly, the individual attributes of such multi-attribute objects can agree or disagree in their reward prediction. Here we address where the brain encodes the combined reward prediction (averaged across attributes) and where it encodes the variability of the value predictions of the individual attributes. We acquired fMRI data while subjects performed a task in which they had to integrate reward predictions from multiple attributes into a combined value. Using time-resolved pattern recognition techniques (support vector regression) we find that (1) the combined value is encoded in distributed fMRI patterns in the ventromedial prefrontal cortex (vmPFC) and that (2) the variability of value predictions of the individual attributes is encoded in the dorsolateral prefrontal cortex (dlPFC). The combined value could be used to guide choices, whereas the variability of the value predictions of individual attributes indicates an ambiguity that results in an increased difficulty of the value-integration. These results demonstrate that the different features defining multi-attribute objects are encoded in non-overlapping brain regions and therefore suggest different roles for vmPFC and dlPFC in multi-attribute decision making.

Disentangling neural representations of value and salience in the human brain

Significance The value and salience of predictive cues are important signals for regulating approach–avoidance behavior and attentional processing, respectively. However, the two signals often are confounded in studies of decision-making. Indeed, recent results suggest that neural signals in the primate posterior parietal cortex (PPC) which previously were thought to encode value actually reflect salience. This finding has created considerable uncertainty about previously identified value signals. Here we experimentally dissociate value and salience and use pattern-based functional MRI to demonstrate distinct encoding of both signals in the PPC, thereby reinforcing the earlier reports of value in the PPC. Moreover, we show that the orbitofrontal cortex encodes the predicted value of appetitive and aversive outcomes on a common neural scale. A large body of evidence has implicated the posterior parietal and orbitofrontal cortex in the processing of value. However, value correlates perfectly with salience when appetitive stimuli are investigated in isolation. Accordingly, considerable uncertainty has remained about the precise nature of the previously identified signals. In particular, recent evidence suggests that neurons in the primate parietal cortex signal salience instead of value. To investigate neural signatures of value and salience, here we apply multivariate (pattern-based) analyses to human functional MRI data acquired during a noninstrumental outcome-prediction task involving appetitive and aversive outcomes. Reaction time data indicated additive and independent effects of value and salience. Critically, we show that multivoxel ensemble activity in the posterior parietal cortex encodes predicted value and salience in superior and inferior compartments, respectively. These findings reinforce the earlier reports of parietal value signals and reconcile them with the recent salience report. Moreover, we find that multivoxel patterns in the orbitofrontal cortex correlate with value. Importantly, the patterns coding for the predicted value of appetitive and aversive outcomes are similar, indicating a common neural scale for appetite and aversive values in the orbitofrontal cortex. Thus orbitofrontal activity patterns satisfy a basic requirement for a neural value signal.

Effects of cognitive bias modification training on neural alcohol cue reactivity in alcohol dependence

OBJECTIVE In alcohol-dependent patients, alcohol cues evoke increased activation in mesolimbic brain areas, such as the nucleus accumbens and the amygdala. Moreover, patients show an alcohol approach bias, a tendency to more quickly approach than avoid alcohol cues. Cognitive bias modification training, which aims to retrain approach biases, has been shown to reduce alcohol craving and relapse rates. The authors investigated effects of this training on cue reactivity in alcohol-dependent patients. METHOD In a double-blind randomized design, 32 abstinent alcohol-dependent patients received either bias modification training or sham training. Both trainings consisted of six sessions of the joystick approach-avoidance task; the bias modification training entailed pushing away 90% of alcohol cues and 10% of soft drink cues, whereas this ratio was 50/50 in the sham training. Alcohol cue reactivity was measured with functional MRI before and after training. RESULTS Before training, alcohol cue-evoked activation was observed in the amygdala bilaterally, as well as in the right nucleus accumbens, although here it fell short of significance. Activation in the amygdala correlated with craving and arousal ratings of alcohol stimuli; correlations in the nucleus accumbens again fell short of significance. After training, the bias modification group showed greater reductions in cue-evoked activation in the amygdala bilaterally and in behavioral arousal ratings of alcohol pictures, compared with the sham training group. Decreases in right amygdala activity correlated with decreases in craving in the bias modification but not the sham training group. CONCLUSIONS These findings provide evidence that cognitive bias modification affects alcohol cue-induced mesolimbic brain activity. Reductions in neural reactivity may be a key underlying mechanism of the therapeutic effectiveness of this training.

Neural Correlates of Alcohol-Approach Bias in Alcohol Addiction: the Spirit is Willing but the Flesh is Weak for Spirits

Behavioral studies have shown an alcohol-approach bias in alcohol-dependent patients: the automatic tendency to faster approach than avoid alcohol compared with neutral cues, which has been associated with craving and relapse. Although this is a well-studied psychological phenomenon, little is known about the brain processes underlying automatic action tendencies in addiction. We examined 20 alcohol-dependent patients and 17 healthy controls with functional magnetic resonance imaging (fMRI), while performing an implicit approach-avoidance task. Participants pushed and pulled pictorial cues of alcohol and soft-drink beverages, according to a content-irrelevant feature of the cue (landscape/portrait). The critical fMRI contrast regarding the alcohol-approach bias was defined as (approach alcohol>avoid alcohol)>(approach soft drink>avoid soft drink). This was reversed for the avoid-alcohol contrast: (avoid alcohol>approach alcohol)>(avoid soft drink>approach soft drink). In comparison with healthy controls, alcohol-dependent patients had stronger behavioral approach tendencies for alcohol cues than for soft-drink cues. In the approach, alcohol fMRI contrast patients showed larger blood-oxygen-level-dependent responses in the nucleus accumbens and medial prefrontal cortex, regions involved in reward and motivational processing. In alcohol-dependent patients, alcohol-craving scores were positively correlated with activity in the amygdala for the approach-alcohol contrast. The dorsolateral prefrontal cortex was not activated in the avoid-alcohol contrast in patients vs controls. Our data suggest that brain regions that have a key role in reward and motivation are associated with the automatic alcohol-approach bias in alcohol-dependent patients.

Neural Integration of Risk and Effort Costs by the Frontal Pole: Only upon Request

Rewards in real life are rarely received without incurring costs and successful reward harvesting often involves weighing and minimizing different types of costs. In the natural environment, such costs often include the physical effort required to obtain rewards and potential risks attached to them. Costs may also include potential risks. In this study, we applied fMRI to explore the neural coding of physical effort costs as opposed to costs associated with risky rewards. Using an incentive-compatible valuation mechanism, we separately measured the subjective costs associated with effortful and risky options. As expected, subjective costs of options increased with both increasing effort and increasing risk. Despite the similar nature of behavioral discounting of effort and risk, distinct regions of the brain coded these two cost types separately, with anterior insula primarily processing risk costs and midcingulate and supplementary motor area (SMA) processing effort costs. To investigate integration of the two cost types, we also presented participants with options that combined effortful and risky elements. We found that the frontal pole integrates effort and risk costs through functional coupling with the SMA and insula. The degree to which the latter two regions influenced frontal pole activity correlated with participant-specific behavioral sensitivity to effort and risk costs. These data support the notion that, although physical effort costs may appear to be behaviorally similar to other types of costs, such as risk, they are treated separately at the neural level and are integrated only if there is a need to do so.

Decoding the formation of reward predictions across learning.

The predicted reward of different behavioral options plays an important role in guiding decisions. Previous research has identified reward predictions in prefrontal and striatal brain regions. Moreover, it has been shown that the neural representation of a predicted reward is similar to the neural representation of the actual reward outcome. However, it has remained unknown how these representations emerge over the course of learning and how they relate to decision making. Here, we sought to investigate learning of predicted reward representations using functional magnetic resonance imaging and multivariate pattern classification. Using a pavlovian conditioning procedure, human subjects learned multiple novel cue–outcome associations in each scanning run. We demonstrate that across learning activity patterns in the orbitofrontal cortex, the dorsolateral prefrontal cortex (DLPFC), and the dorsal striatum, coding the value of predicted rewards become similar to the patterns coding the value of actual reward outcomes. Furthermore, we provide evidence that predicted reward representations in the striatum precede those in prefrontal regions and that representations in the DLPFC are linked to subsequent value-based choices. Our results show that different brain regions represent outcome predictions by eliciting the neural representation of the actual outcome. Furthermore, they suggest that reward predictions in the DLPFC are directly related to value-based choices.