Michael A. McDannald

Ventral Striatum and Orbitofrontal Cortex Are Both Required for Model-Based, But Not Model-Free, Reinforcement Learning

In many cases, learning is thought to be driven by differences between the value of rewards we expect and rewards we actually receive. Yet learning can also occur when the identity of the reward we receive is not as expected, even if its value remains unchanged. Learning from changes in reward identity implies access to an internal model of the environment, from which information about the identity of the expected reward can be derived. As a result, such learning is not easily accounted for by model-free reinforcement learning theories such as temporal difference reinforcement learning (TDRL), which predicate learning on changes in reward value, but not identity. Here, we used unblocking procedures to assess learning driven by value- versus identity-based prediction errors. Rats were trained to associate distinct visual cues with different food quantities and identities. These cues were subsequently presented in compound with novel auditory cues and the reward quantity or identity was selectively changed. Unblocking was assessed by presenting the auditory cues alone in a probe test. Consistent with neural implementations of TDRL models, we found that the ventral striatum was necessary for learning in response to changes in reward value. However, this area, along with orbitofrontal cortex, was also required for learning driven by changes in reward identity. This observation requires that existing models of TDRL in the ventral striatum be modified to include information about the specific features of expected outcomes derived from model-based representations, and that the role of orbitofrontal cortex in these models be clearly delineated.

Does the orbitofrontal cortex signal value

The orbitofrontal cortex (OFC) has long been implicated in associative learning. Early work by Mishkin and Rolls showed that the OFC was critical for rapid changes in learned behavior, a role that was reflected in the encoding of associative information by orbitofrontal neurons. Over the years, new data—particularly neurophysiological data—have increasingly emphasized the OFC in signaling actual value. These signals have been reported to vary according to internal preferences and judgments and to even be completely independent of the sensory qualities of predictive cues, the actual rewards, and the responses required to obtain them. At the same time, increasingly sophisticated behavioral studies have shown that the OFC is often unnecessary for simple value‐based behavior and instead seems critical when information about specific outcomes must be used to guide behavior and learning. Here, we review these data and suggest a theory that potentially reconciles these two ideas, value versus specific outcomes, and bodies of work on the OFC.

Orbitofrontal Cortex Supports Behavior and Learning Using Inferred But Not Cached Values

Experience Versus Models There is an ongoing debate over what the orbitofrontal cortex contributes to behavior, learning, and decision-making. Jones et al. (p. 953) found that the orbitofrontal cortex was important for value-based computations when value must be inferred from an associative model of the task but not when value estimates based on previous experience are sufficient. This result calls into question the assumption that this region simply signals economic value. However, it would be consistent with a concept of the orbitofrontal cortex as being important for constructing model-based representations of the world that are orthogonal to value. Inferred value can be used to both guide behavior and modulate learning in rats. Computational and learning theory models propose that behavioral control reflects value that is both cached (computed and stored during previous experience) and inferred (estimated on the fly on the basis of knowledge of the causal structure of the environment). The latter is thought to depend on the orbitofrontal cortex. Yet some accounts propose that the orbitofrontal cortex contributes to behavior by signaling “economic” value, regardless of the associative basis of the information. We found that the orbitofrontal cortex is critical for both value-based behavior and learning when value must be inferred but not when a cached value is sufficient. The orbitofrontal cortex is thus fundamental for accessing model-based representations of the environment to compute value rather than for signaling value per se.

Model-based learning and the contribution of the orbitofrontal cortex to the model-free world

Learning is proposed to occur when there is a discrepancy between reward prediction and reward receipt. At least two separate systems are thought to exist: one in which predictions are proposed to be based on model‐free or cached values; and another in which predictions are model‐based. A basic neural circuit for model‐free reinforcement learning has already been described. In the model‐free circuit the ventral striatum (VS) is thought to supply a common‐currency reward prediction to midbrain dopamine neurons that compute prediction errors and drive learning. In a model‐based system, predictions can include more information about an expected reward, such as its sensory attributes or current, unique value. This detailed prediction allows for both behavioral flexibility and learning driven by changes in sensory features of rewards alone. Recent evidence from animal learning and human imaging suggests that, in addition to model‐free information, the VS also signals model‐based information. Further, there is evidence that the orbitofrontal cortex (OFC) signals model‐based information. Here we review these data and suggest that the OFC provides model‐based information to this traditional model‐free circuitry and offer possibilities as to how this interaction might occur.

Learning theory: a driving force in understanding orbitofrontal function.

Since it was demonstrated the orbitofrontal cortex (OFC) is critical to reversal learning, there has been considerable interest in specifying its role in flexible, outcome-guided behavior. Behavioral paradigms from the learning theory tradition, such as outcome devaluation, blocking, Pavlovian to instrumental transfer, and overexpectation have been a driving force in this research. The use of these procedures has revealed OFCs unique role in forming and integrating information about specific features of events and outcomes to drive behavior and learning. These studies highlight the power and importance of learning theory principles in guiding neuroscience research.

Orbitofrontal neurons acquire responses to ‘valueless’ Pavlovian cues during unblocking

The orbitofrontal cortex (OFC) has been described as signaling outcome expectancies or value. Evidence for the latter comes from the studies showing that neural signals in the OFC correlate with value across features. Yet features can co-vary with value, and individual units may participate in multiple ensembles coding different features. Here we used unblocking to test whether OFC neurons would respond to a predictive cue signaling a ‘valueless’ change in outcome flavor. Neurons were recorded as the rats learned about cues that signaled either an increase in reward number or a valueless change in flavor. We found that OFC neurons acquired responses to both predictive cues. This activity exceeded that exhibited to a ‘blocked’ cue and was correlated with activity to the actual outcome. These results show that OFC neurons fire to cues with no value independent of what can be inferred through features of the predicted outcome. DOI: http://dx.doi.org/10.7554/eLife.02653.001

Orbitofrontal neurons infer the value and identity of predicted outcomes

The best way to respond flexibly to changes in the environment is to anticipate them. Such anticipation often benefits us if we can infer that a change has occurred, before we have actually experienced the effects of that change. Here we test for neural correlates of this process by recording single-unit activity in the orbitofrontal cortex in rats performing a choice task in which the available rewards changed across blocks of trials. Consistent with the proposal that orbitofrontal cortex signals inferred information, firing changes at the start of each new block as if predicting the not-yet-experienced reward. This change occurs whether the new reward is different in number of drops, requiring signaling of a new value, or in flavor, requiring signaling of a new sensory feature. These results show that orbitofrontal neurons provide a behaviorally relevant signal that reflects inferences about both value-relevant and value-neutral information about impending outcomes.

Impaired Reality Testing in an Animal Model of Schizophrenia

BACKGROUND Schizophrenia is a chronic and devastating brain disorder characterized by hallucinations and delusions, symptoms reflecting impaired reality testing. Although animal models have captured negative symptoms and cognitive deficits associated with schizophrenia, none have addressed these defining, positive symptoms. METHODS Here we tested the performance of adults given neonatal ventral hippocampal lesions (NVHL), a neurodevelopmental model of schizophrenia, in two taste aversion procedures. RESULTS Normal and NVHL rats formed aversions to a palatable food when the food was directly paired with nausea, but only NVHL rats formed a food aversion when the cue predicting that food was paired with nausea. The failure of NVHL rats to discriminate fully real from imagined food parallels the failure of people with schizophrenia to differentiate internal thoughts and beliefs from reality. CONCLUSIONS These results further validate the NVHL model of schizophrenia and provide a means to assess impaired reality testing in variety of animal models.

Lateral orbitofrontal neurons acquire responses to upshifted, downshifted, or blocked cues during unblocking

The lateral orbitofrontal cortex (lOFC) has been described as signaling either outcome expectancies or value. Previously, we used unblocking to show that lOFC neurons respond to a predictive cue signaling a ‘valueless’ change in outcome features (McDannald, 2014). However, many lOFC neurons also fired to a cue that simply signaled more reward. Here, we recorded lOFC neurons in a variant of this task in which rats learned about cues that signaled either more (upshift), less (downshift) or the same (blocked) amount of reward. We found that neurons acquired responses specifically to one of the three cues and did not fire to the other two. These results show that, at least early in learning, lOFC neurons fire to valued cues in a way that is more consistent with signaling of the predicted outcome’s features than with signaling of a general, abstract or cached value that is independent of the outcome. DOI: http://dx.doi.org/10.7554/eLife.11299.001

The dorsal raphe nucleus is integral to negative prediction errors in Pavlovian fear

Prediction errors are central to modern learning theories. While brain regions contributing to reward prediction errors have been uncovered, the sources of aversive prediction errors remain largely unknown. Here we used probabilistic and deterministic reinforcement procedures, followed by extinction, to examine the contribution of the dorsal raphe nucleus to negative, aversive prediction errors in Pavlovian fear. Rats with dorsal raphe lesions were able to acquire fear and reduce fear to a non‐reinforced deterministic cue. However, dorsal raphe lesions impaired the reduction of fear to a probabilistic cue and fear extinction to a deterministic cue, both of which involve the use of negative prediction errors. The results point to an integral role for the dorsal raphe nucleus in negative prediction error signaling in Pavlovian fear.