Kathryn A. Burke

The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes

Humans and other animals change their behavior in response to unexpected outcomes. The orbitofrontal cortex (OFC) is implicated in such adaptive responding, based on evidence from reversal tasks. Yet these tasks confound using information about expected outcomes with learning when those expectations are violated. OFC is critical for the former function; here we show it is also critical for the latter. In a Pavlovian overexpectation task, inactivation of OFC prevented learning driven by unexpected outcomes, even when performance was assessed later. We propose this reflects a critical contribution of outcome signaling by OFC to encoding of reward prediction errors elsewhere. In accord with this proposal, we report that signaling of reward predictions by OFC neurons was related to signaling of prediction errors by dopamine neurons in ventral tegmental area (VTA). Furthermore, bilateral inactivation of VTA or contralateral inactivation of VTA and OFC disrupted learning driven by unexpected outcomes.

Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning

In many cases, learning is thought to be driven by differences between the value of rewards we expect and rewards we actually receive. Yet learning can also occur when the identity of the reward we receive is not as expected, even if its value remains unchanged. Learning from changes in reward identity implies access to an internal model of the environment, from which information about the identity of the expected reward can be derived. As a result, such learning is not easily accounted for by model-free reinforcement learning theories such as temporal difference reinforcement learning (TDRL), which predicate learning on changes in reward value, but not identity. Here, we used unblocking procedures to assess learning driven by value- versus identity-based prediction errors. Rats were trained to associate distinct visual cues with different food quantities and identities. These cues were subsequently presented in compound with novel auditory cues and the reward quantity or identity was selectively changed. Unblocking was assessed by presenting the auditory cues alone in a probe test. Consistent with neural implementations of TDRL models, we found that the ventral striatum was necessary for learning in response to changes in reward value. However, this area, along with orbitofrontal cortex, was also required for learning driven by changes in reward identity. This observation requires that existing models of TDRL in the ventral striatum be modified to include information about the specific features of expected outcomes derived from model-based representations, and that the role of orbitofrontal cortex in these models be clearly delineated.

The role of the orbitofrontal cortex in the pursuit of happiness and more specific rewards

Cues that reliably predict rewards trigger the thoughts and emotions normally evoked by those rewards. Humans and other animals will work, often quite hard, for these cues. This is termed conditioned reinforcement. The ability to use conditioned reinforcers to guide our behaviour is normally beneficial; however, it can go awry. For example, corporate icons, such as McDonald’s Golden Arches, influence consumer behaviour in powerful and sometimes surprising ways, and drug-associated cues trigger relapse to drug seeking in addicts and animals exposed to addictive drugs, even after abstinence or extinction. Yet, despite their prevalence, it is not known how conditioned reinforcers control human or other animal behaviour. One possibility is that they act through the use of the specific rewards they predict; alternatively, they could control behaviour directly by activating emotions that are independent of any specific reward. In other words, the Golden Arches may drive business because they evoke thoughts of hamburgers and fries, or instead, may be effective because they also evoke feelings of hunger or happiness. Moreover, different brain circuits could support conditioned reinforcement mediated by thoughts of specific outcomes versus more general affective information. Here we have attempted to address these questions in rats. Rats were trained to learn that different cues predicted different rewards using specialized conditioning procedures that controlled whether the cues evoked thoughts of specific outcomes or general affective representations common to different outcomes. Subsequently, these rats were given the opportunity to press levers to obtain short and otherwise unrewarded presentations of these cues. We found that rats were willing to work for cues that evoked either outcome-specific or general affective representations. Furthermore the orbitofrontal cortex, a prefrontal region important for adaptive decision-making, was critical for the former but not for the latter form of conditioned reinforcement.

Abnormal associative encoding in orbitofrontal neurons in cocaine-experienced rats during decision-making

Recent evidence has linked exposure to addictive drugs to an inability to employ information about adverse consequences, or outcomes, to control behavior. For instance, addicts and drug‐experienced animals fail to adapt their behavior to avoid adverse outcomes in gambling and reversal tasks or after changes in the value of expected rewards. These deficits are similar to those caused by damage to the orbitofrontal cortex, suggesting that addictive drugs may cause long‐lasting changes in the representation of outcome associations in a circuit that includes the orbitofrontal cortex. Here we test this hypothesis by recording from orbitofrontal neurons in a discrimination task in rats previously exposed to cocaine (30 mg/kg i.p. for 14 days). We found that orbitofrontal neurons recorded in cocaine‐experienced rats failed to signal the adverse outcome at the time a decision was made in the task. The loss of this signal was associated with abnormal changes in response latencies on aversive trials. Furthermore, upon reversal of the cue–outcome associations, orbitofrontal neurons in cocaine‐treated rats with enduring reversal impairments failed to reverse their cue‐selectivity, while orbitofrontal neurons in cocaine‐treated rats with normal performance showed an increase in the plasticity of cue‐selective firing after reversal. These results provide direct neurophysiological evidence that exposure to cocaine can cause behaviorally relevant changes in the processing of associative information in a circuit that includes the orbitofrontal cortex.

Should I stay or should I go? : Transformation of time-discounted rewards in orbitofrontal cortex and associated brain circuits

Abstract:  Animals prefer a small, immediate reward over a larger delayed reward (time discounting). Lesions of the orbitofrontal cortex (OFC) can either increase or decrease the breakpoint at which animals abandon the large delayed reward for the more immediate reward as the delay becomes longer. Here we argue that the varied effects of OFC lesions on delayed discounting reflect two different patterns of activity in OFC; one that bridges the gap between a response and an outcome and another that discounts delayed reward. These signals appear to reflect the spatial location of the reward and/or the action taken to obtain it, and are encoded independently from representations of absolute value. We suggest a dual role for output from OFC in both discounting delayed reward, while at the same time supporting new learning for them.

Orbitofrontal inactivation impairs reversal of Pavlovian learning by interfering with ‘disinhibition’ of responding for previously unrewarded cues

Orbitofrontal cortex (OFC) is critical for reversal learning. Reversal deficits are typically demonstrated in complex settings that combine Pavlovian and instrumental learning. Yet recent work has implicated the OFC specifically in behaviors guided by cues and the features of the specific outcomes they predict. To test whether the OFC is important for reversing such Pavlovian associations in the absence of confounding instrumental requirements, we trained rats on a simple Pavlovian task in which two auditory cues were presented, one paired with a food pellet reward and the other presented without reward. After learning, we reversed the cue–outcome associations. For half the rats, OFC was inactivated prior to each reversal session. Inactivation of OFC impaired the ability of the rats to reverse conditioned responding. This deficit reflected the inability of inactivated rats to develop normal responding for the previously unrewarded cue; inactivation of OFC had no impact on the ability of the rats to inhibit responding to the previously rewarded cue. These data show that OFC is critical to reversal of Pavlovian responding, and that the role of OFC in this behavior cannot be explained as a simple deficit in response inhibition. Furthermore, the contrast between the normal inhibition of responding, reported here, and impaired inhibition of responding during Pavlovian over‐expectation, reported previously, suggests the novel hypothesis that OFC may be particularly critical for learning (or behavior) when it requires the subject to generate predictions about outcomes by bringing together or integrating disparate pieces of associative information.

Conditioned Reinforcement can be Mediated by Either Outcome-Specific or General Affective Representations

Conditioned reinforcers are Pavlovian cues that support the acquisition and maintenance of new instrumental responses. Responding on the basis of conditioned rather than primary reinforcers is a pervasive part of modern life, yet we have a remarkably limited understanding of what underlying associative information is triggered by these cues to guide responding. Specifically, it is not certain whether conditioned reinforcers are effective because they evoke representations of specific outcomes or because they trigger general affective states that are independent of any specific outcome. This question has important implications for how different brain circuits might be involved in conditioned reinforcement. Here, we use specialized Pavlovian training procedures, reinforcer devaluation and transreinforcer blocking, to create cues that were biased to preferentially evoke either devaluation-insensitive, general affect representations or, devaluation-sensitive, outcome-specific representations. Subsequently, these cues, along with normally conditioned control cues, were presented contingent on lever pressing. We found that intact rats learned to lever press for either the outcome or the affect cues to the same extent as for a normally conditioned cue. These results demonstrate that conditioned reinforcers can guide responding through either type of associative information. Interestingly, conditioned reinforcement was abolished in rats with basolateral amygdala lesions. Consistent with the extant literature, this result suggests a general role for basolateral amygdala in conditioned reinforcement. The implications of these data, combined with recent reports from our laboratory of a more specialized role of orbitofrontal cortex in conditioned reinforcement, will be discussed.

Cyclin-Dependent Kinase 5 Mediates Adult OPC Maturation and Myelin Repair through Modulation of Akt and GsK-3β Signaling

Failure of remyelination in diseases, such as multiple sclerosis (MS), leads to permanent axonal damage and irreversible functional loss. The mechanisms controlling remyelination are currently poorly understood. Recent studies implicate the cyclin-dependent kinase 5 (Cdk5) in regulating oligodendrocyte (OL) development and myelination in CNS. In this study, we show that Cdk5 is also an important regulator of remyelination. Pharmacological inhibition of Cdk5 inhibits repair of lysolecithin lesions. This inhibition is a consequence of Cdk5 disruption in neural cells because remyelination in slice cultures is blocked by Cdk5 inhibitors, whereas specific deletion of Cdk5 in OLs inhibits myelin repair. In CNP-Cre;Cdk5fl/fl conditional knock-out mouse (Cdk5 cKO), myelin repair was delayed significantly in response to focal demyelinating lesions compared with wild-type animals. The lack of myelin repair was reflected in decreased expression of MBP and proteolipid protein and a reduction in the total number of myelinated axons in the lesion. The number of CC1+ cells in the lesion sites was significantly reduced in Cdk5 cKO compared with wild-type animals although the total number of oligodendrocyte lineage cells (Olig2+ cells) was increased, suggesting that Cdk5 loss perturbs the transition of early OL lineage cell into mature OL and subsequent remyelination. The failure of remyelination in Cdk5 cKO animals was associated with a reduction in signaling through the Akt pathway and an enhancement of Gsk-3β signaling pathways. Together, these data suggest that Cdk5 is critical in regulating the transition of adult oligodendrocyte precursor cells to mature OLs that is essential for myelin repair in adult CNS.

Neural Correlates of Inflexible Behavior in the Orbitofrontal–Amygdalar Circuit after Cocaine Exposure

Abstract: Addiction is characterized by compulsive or inflexible behavior, observed both in the context of drug‐seeking and in contexts unrelated to drugs. One possible contributor to these inflexible behaviors may be drug‐induced dysfunction within circuits that support behavioral flexibility, including the basolateral amygdala (ABL) and the orbitofrontal cortex (OFC). Here we describe data demonstrating that chronic cocaine exposure causes long‐lasting changes in encoding properties in the ABL and the OFC during learning and reversal in an odor‐guided task. In particular, these data suggest that inflexible encoding in ABL neurons may be the proximal cause of cocaine‐induced behavioral inflexibility, and that a loss of outcome‐expectant encoding in OFC neurons could be a more distal contributor to this impairment. A similar mechanism of drug‐induced orbitofrontal–amygdalar dysfunction may cause inflexible behavior when animals and addicts are exposed to drug‐associated cues and contexts.