Sean B. Ostlund

Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks

Here we challenge the view that reward‐guided learning is solely controlled by the mesoaccumbens pathway arising from dopaminergic neurons in the ventral tegmental area and projecting to the nucleus accumbens. This widely accepted view assumes that reward is a monolithic concept, but recent work has suggested otherwise. It now appears that, in reward‐guided learning, the functions of ventral and dorsal striata, and the cortico‐basal ganglia circuitry associated with them, can be dissociated. Whereas the nucleus accumbens is necessary for the acquisition and expression of certain appetitive Pavlovian responses and contributes to the motivational control of instrumental performance, the dorsal striatum is necessary for the acquisition and expression of instrumental actions. Such findings suggest the existence of multiple independent yet interacting functional systems that are implemented in iterating and hierarchically organized cortico‐basal ganglia networks engaged in appetitive behaviors ranging from Pavlovian approach responses to goal‐directed instrumental actions controlled by action‐outcome contingencies.

The integrative function of the basal ganglia in instrumental conditioning

Recent research in instrumental conditioning has focused on the striatum, particularly the role of the dorsal striatum in the learning processes that contribute to instrumental performance in rats. This research has found evidence of what appear to be parallel, functionally and anatomically distinct circuits involving dorsomedial striatum (DMS) and dorsolateral striatum (DLS) that contribute to two independent instrumental learning processes. Evidence suggests that the formation of the critical action-outcome associations mediating goal-directed action are localized to the dorsomedial striatum, whereas the sensorimotor connections that control the performance of habitual actions are localized to the dorsolateral striatum. In addition to the dorsal striatum, these learning processes appear to engage distinct cortico-striatal networks and to be embedded in a complex of converging and partially segregated loops that constitute the cortico-striatal thalamo-cortical feedback circuit. As the entry point for the basal ganglia, cortical circuits involving the dorsal striatum are clearly in a position to control a variety of motor functions but, as recent studies of various neurodegenerative disorders have made clear, they are also involved in a number of cognitive and executive functions including action selection, planning, and decision-making.

Orbitofrontal Cortex Mediates Outcome Encoding in Pavlovian But Not Instrumental Conditioning

Previous studies have implicated the orbitofrontal cortex (OFC) in outcome encoding. However, it remains unknown whether the OFC is selectively involved in pavlovian stimulus–outcome learning or whether it also contributes to instrumental action–outcome learning. In experiment 1, we investigated this issue by assessing the effects of bilateral lesions of the OFC on the sensitivity of instrumental lever press performance to a reduction in the incentive value of the training outcome (a test of action–outcome encoding) and to outcome-specific pavlovian-instrumental transfer (a test of stimulus–outcome encoding). We found that post-training lesions of the OFC did not affect instrumental outcome devaluation, but abolished the transfer effect. Interestingly, lesions made before training had no effect on either task. In experiment 2, we explored the involvement of the OFC in updating stimulus–outcome associations after the underlying contingency, or predictive relationship, between these two events has been degraded. Shams displayed clear contingency learning, withholding conditioned responding to a stimulus that no longer reliably predicted its outcome while continuing to respond to a control stimulus that remained a good predictor of a different outcome. In contrast, OFC-lesioned rats stopped responding to both stimuli, regardless of their predictive status. Together, these findings suggest that the OFC supports outcome encoding in pavlovian, but not instrumental conditioning.

Lesions of Medial Prefrontal Cortex Disrupt the Acquisition But Not the Expression of Goal-Directed Learning

Several studies have established that pretraining lesions of the medial prefrontal cortex (mPFC) render instrumental actions insensitive to devaluation of the instrumental outcome and degradation of the action-outcome contingency. Nevertheless, it remains to be assessed whether the involvement of the mPFC in goal-directed action is limited to the acquisition or to the expression of the action-outcome association in performance. The current series of experiments investigated this issue by comparing the effects of mPFC lesions made either before or after initial training using sensitivity to outcome devaluation as an assay of goal-directed performance. Whereas pretraining lesions left performance insensitive to outcome devaluation, posttraining lesions spared this effect. To determine whether the effect of mPFC lesions on outcome devaluation was the result of a more fundamental deficit in response selection, experiment 2 assessed the impact of pretraining and posttraining lesions on the ability of the instrumental outcome to selectively reinstate the performance of its associated action after a period of extinction. Although both lesions attenuated the magnitude of instrumental reinstatement generally, they left intact the ability of the instrumental outcome to influence response selection. Experiment 3 investigated the relationship between the outcome-selective devaluation and reinstatement effects and found evidence that these effects are both behaviorally and neurally dissociable at the level of the mPFC. These results indicate that the mPFC is selectively involved in the acquisition, but not the permanent storage or expression, of action-outcome associations in instrumental conditioning.

Still at the choice-point: action selection and initiation in instrumental conditioning.

Abstract:  Contrary to classic stimulus–response (S‐R) theory, recent evidence suggests that, in instrumental conditioning, rats encode the relationship between their actions and the specific consequences that these actions produce. It has remained unclear, however, how encoding this relationship acts to control instrumental performance. Although S‐R theories were able to give a clear account of how learning translates into performance, the argument that instrumental learning constitutes the acquisition of information of the form “response R leads to outcome O” does not directly imply a particular performance rule or policy; this information can be used both to perform R and to avoid performing R. Recognition of this problem has forced the development of accounts that allow the O and stimuli that predict the O (i.e., S‐O) to play a role in the initiation of specific Rs. In recent experiments, we have used a variety of behavioral procedures in an attempt to isolate the processes that contribute to instrumental performance, including outcome devaluation, reinstatement, and Pavlovian–instrumental transfer. Our results, particularly from experiments assessing outcome–selective reinstatement, suggest that both “feed‐forward” (O‐R) and “feed‐back” (R‐O) associations are critical and that although the former appear to be important to response selection, the latter—together with processes that determine outcome value—mediate response initiation. We discuss a conceptual model that integrates these processes and its neural implementation.

Consolidation and Reconsolidation of Incentive Learning in the Amygdala

Incentive learning is the process via which animals update changes in the value of rewards. Current evidence suggests that, for food rewards in rats, this learning process involves the amygdala. However, it remains unclear whether this learning undergoes protein synthesis-dependent consolidation and “reconsolidation” processes in the lateral and basal nuclei of amygdala. Accordingly, we examined this hypothesis by local infusion of protein-synthesis inhibitor after devaluation of a food reward induced by a shift from a food-deprived to a food-sated state in an instrumental conditioning paradigm. Our results show that intra-amygdala infusions of anisomycin, whether given after the initial devaluation or after a second devaluation session, abolished the changes in the value of the food reward produced by incentive learning. This study provides direct evidence that instrumental incentive learning depends on protein synthesis within the amygdala for both consolidation and reconsolidation and extends the demonstrations of protein synthesis-dependent reconsolidation to reward-related memories.

Distinct opioid circuits determine the palatability and the desirability of rewarding events

It generally is assumed that a common neural substrate mediates both the palatability and the reward value of nutritive events. However, recent evidence suggests this assumption may not be true. Whereas opioid circuitry in both the nucleus accumbens and ventral pallidum has been reported to mediate taste-reactivity responses to palatable events, the assignment of reward or inventive value to goal-directed actions has been found to involve the basolateral amygdala. Here we found that, in rats, the neural processes mediating palatability and incentive value are indeed dissociable. Naloxone infused into either the ventral pallidum or nucleus accumbens shell blocked the increase in sucrose palatability induced by an increase in food deprivation without affecting the performance of sucrose-related actions. Conversely, naloxone infused into the basolateral amygdala blocked food deprivation-induced changes in sucrose-related actions without affecting sucrose palatability. This double dissociation of opioid-mediated changes in palatability and incentive value suggests that the role of endogenous opioids in reward processing does not depend on a single neural circuit. Rather, changes in palatability and in the incentive value assigned to rewarding events seem to be mediated by distinct neural processes.

Differential dependence of Pavlovian incentive motivation and instrumental incentive learning processes on dopamine signaling

Here we attempted to clarify the role of dopamine signaling in reward seeking. In Experiment 1, we assessed the effects of the dopamine D(1)/D(2) receptor antagonist flupenthixol (0.5 mg/kg i.p.) on Pavlovian incentive motivation and found that flupenthixol blocked the ability of a conditioned stimulus to enhance both goal approach and instrumental performance (Pavlovian-to-instrumental transfer). In Experiment 2 we assessed the effects of flupenthixol on reward palatability during post-training noncontingent re-exposure to the sucrose reward in either a control 3-h or novel 23-h food-deprived state. Flupenthixol, although effective in blocking the Pavlovian goal approach, was without effect on palatability or the increase in reward palatability induced by the upshift in motivational state. This noncontingent re-exposure provided an opportunity for instrumental incentive learning, the process by which rats encode the value of a reward for use in updating reward-seeking actions. Flupenthixol administered prior to the instrumental incentive learning opportunity did not affect the increase in subsequent off-drug reward-seeking actions induced by that experience. These data suggest that although dopamine signaling is necessary for Pavlovian incentive motivation, it is not necessary for changes in reward experience, or for the instrumental incentive learning process that translates this experience into the incentive value used to drive reward-seeking actions, and provide further evidence that Pavlovian and instrumental incentive learning processes are dissociable.

The Contribution of Orbitofrontal Cortex to Action Selection

Abstract:  A number of recent findings suggest that the orbitofrontal cortex (OFC) influences action selection by providing information about the incentive value of behavioral goals or outcomes. However, much of this evidence has been derived from experiments using Pavlovian conditioning preparations of one form or another, making it difficult to determine whether the OFC is selectively involved in stimulus–outcome learning or whether it plays a more general role in processing reward value. Although many theorists have argued that these are fundamentally similar processes (i.e., that stimulus‐reward learning provides the basis for choosing between actions based on anticipated reward value), several behavioral findings indicate that they are, in fact, dissociable. We have recently investigated the role of the OFC in the control of free operant lever pressing using tests that independently target the effect of stimulus–outcome learning and outcome devaluation on performance. We found that OFC lesions disrupted the tendency of Pavlovian cues to facilitate instrumental performance but left intact the suppressive effects of outcome devaluation. Rather than processing goal value, therefore, we hypothesize that the contribution of the OFC to goal‐directed action is limited to encoding predictive stimulus–outcome relationships that can bias instrumental response selection.

Alcohol-Paired Contextual Cues Produce an Immediate and Selective Loss of Goal-directed Action in Rats

We assessed whether the presence of contextual cues paired with alcohol would disrupt rats’ capacity to express appropriate goal-directed action control. Rats were first given differential context conditioning such that one set of contextual cues was paired with the injection of ethanol and a second, distinctive set of cues was paired with the injection of saline. All rats were then trained in a third, neutral context to press one lever for grain pellets and another lever for sucrose pellets. They were then given two extinction tests to evaluate their ability to choose between the two actions in response to the devaluation of one of the two food outcomes with one test conducted in the alcohol-paired context and the other conducted in the control (saline-paired) context. In the control context, rats exhibited goal-directed action control; i.e., they were able selectively to withhold the action that previously earned the now devalued outcome. However, these same rats were impaired when tested in the alcohol-paired context, performing both actions at the same rate regardless of the current value of their respective outcomes. Subsequent testing revealed that the rats were capable of overcoming this impairment if they were giving response-contingent feedback about the current value of the food outcomes. These results provide a clear demonstration of the disruptive influence that alcohol-paired cues can exert on decision-making in general and goal-directed action selection and choice in particular.