Bernard W. Balleine

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Instrumental behaviour is controlled by two systems: a stimulus-response habit mechanism and a goal-directed process that involves two forms of learning. The first is learning about the instrumental contingency between the response and reward, whereas the second consists of the acquisition of incentive value by the reward. Evidence for contingency learning comes from studies of reward devaluation and from demonstrations that instrumental performance is sensitive not only the probability of contiguous reward but also to the probability of unpaired rewards. The process of incentive learning is evident in the acquisition of control over performance by primary motivational states. Preliminary lesion studies of the rat suggest that the prelimbic area of prefrontal cortex plays a role in the contingency learning, whereas the incentive learning for food rewards involves the insular cortex.

Human and Rodent Homologies in Action Control: Corticostriatal Determinants of Goal-Directed and Habitual Action

Recent behavioral studies in both humans and rodents have found evidence that performance in decision-making tasks depends on two different learning processes; one encoding the relationship between actions and their consequences and a second involving the formation of stimulus–response associations. These learning processes are thought to govern goal-directed and habitual actions, respectively, and have been found to depend on homologous corticostriatal networks in these species. Thus, recent research using comparable behavioral tasks in both humans and rats has implicated homologous regions of cortex (medial prefrontal cortex/medial orbital cortex in humans and prelimbic cortex in rats) and of dorsal striatum (anterior caudate in humans and dorsomedial striatum in rats) in goal-directed action and in the control of habitual actions (posterior lateral putamen in humans and dorsolateral striatum in rats). These learning processes have been argued to be antagonistic or competing because their control over performance appears to be all or none. Nevertheless, evidence has started to accumulate suggesting that they may at times compete and at others cooperate in the selection and subsequent evaluation of actions necessary for normal choice performance. It appears likely that cooperation or competition between these sources of action control depends not only on local interactions in dorsal striatum but also on the cortico-basal ganglia network within which the striatum is embedded and that mediates the integration of learning with basic motivational and emotional processes. The neural basis of the integration of learning and motivation in choice and decision-making is still controversial and we review some recent hypotheses relating to this issue.

Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning

Habits are controlled by antecedent stimuli rather than by goal expectancy. Interval schedules of feedback have been shown to generate habits, as revealed by the insensitivity of behaviour acquired under this schedule to outcome devaluation treatments. Two experiments were conducted to assess the role of the dorsolateral striatum in habit learning. In Experiment 1, sham operated controls and rats with dorsolateral striatum lesions were trained to press a lever for sucrose under interval schedules. After training, the sucrose was devalued by inducing taste aversion to it using lithium chloride, whereas saline injections were given to the controls. Only rats given the devaluation treatment reduced their consumption of sucrose and this reduction was similar in both the sham and the lesioned groups. All rats were then returned to the instrumental chamber for an extinction test, in which the lever was extended but no sucrose was delivered. In contrast to sham operated controls, rats with dorsolateral striatum lesions refrained from pressing the lever if the outcome was devalued. To assess the specificity of the role of dorsolateral striatum in this effect a second experiment was conducted in which a group with lesions of dorsomedial striatum was added. In relation now to both the sham and the dorsomedial lesioned groups, only rats with lesions of dorsolateral striatum significantly reduced responding after outcome devaluation. In conclusion, this study provides direct evidence that the dorsolateral striatum is necessary for habit formation. Furthermore, it suggests that, when the habit system is disrupted, control over instrumental performance reverts to the system controlling the performance of goal‐directed instrumental actions.

The Role of the Dorsal Striatum in Reward and Decision-Making

Although the involvement in the striatum in the refinement and control of motor movement has long been recognized, recent description of discrete frontal corticobasal ganglia networks in a range of species has focused attention on the role particularly of the dorsal striatum in executive functions. Current evidence suggests that the dorsal striatum contributes directly to decision-making, especially to action selection and initiation, through the integration of sensorimotor, cognitive, and motivational/emotional information within specific corticostriatal circuits involving discrete regions of striatum. We review key evidence from recent studies in rodent, nonhuman primate, and human subjects.

Motivational control of goal-directed action

The control of goal-directed, instrumental actions by primary motivational states, such as hunger and thirst, is mediated by two processes. The first is engaged by the Pavlovian association between contextual or discriminative stimuli and the outcome or reinforcer presented during instrumental training. Such stimuli exert a motivational influence on instrumental performance that depends upon the relevance of the associated outcome to the current motivational state of the agent. Moreover, the motivational effects of these stimuli operate in the absence of prior experience with the outcome under the relevant motivational state. The second, instrumental, process is mediated by knowledge of the contingency between the action and its outcome and controls the value assigned to this outcome. In contrast to the Pavlovian process, motivational states do not influence the instrumental process directly; rather, the agent has to learn about the value of an outcome in a given motivational state by exposure to it while in that state. This incentive learning is similar in certain respects to the acquisition of “cathexes” envisaged by Tolman (1949a, 1949b).

Reward, Motivation, and Reinforcement Learning

There is substantial evidence that dopamine is involved in reward learning and appetitive conditioning. However, the major reinforcement learning-based theoretical models of classical conditioning (crudely, prediction learning) are actually based on rules designed to explain instrumental conditioning (action learning). Extensive anatomical, pharmacological, and psychological data, particularly concerning the impact of motivational manipulations, show that these models are unreasonable. We review the data and consider the involvement of a rich collection of different neural systems in various aspects of these forms of conditioning. Dopamine plays a pivotal, but complicated, role.

Parallel incentive processing: an integrated view of amygdala function

The amygdala is a heterogeneous structure that has been implicated in a wide variety of functions, most notably in fear conditioning. From this research, an influential serial model of amygdala processes has emerged in which aversive learning is mediated by the amygdala basolateral nucleus whereas performance, in this case of various defensive reflexes, is mediated by the central nucleus. By contrast, recent evidence from appetitive conditioning studies suggests that the basolateral and central nuclei operate in parallel to mediate distinct incentive processes: the basolateral nucleus encodes emotional events with reference to their particular sensory-specific features, whereas the central nucleus encodes their more general motivational or affective significance. Given that there is little if any direct behavioral evidence for the serial model, we suggest that more attention should be given to the claims of the parallel view.

Double Dissociation of Basolateral and Central Amygdala Lesions on the General and Outcome-Specific Forms of Pavlovian-Instrumental Transfer

This series of experiments compared the effects of lesions of the basolateral complex (BLA) and the central nucleus (CN) of the amygdala on a number of tests of instrumental learning and performance and particularly on the contribution of these structures to the specific and general forms of pavlovian-instrumental transfer (PIT). In experiment 1, groups of BLA-, CN-, and sham-lesioned rats were first trained to press two levers, each earning a unique food outcome (pellets or sucrose), after which they were given training in which two auditory stimuli (tone and white noise) were paired with these same outcomes. Tests of specific satiety induced outcome devaluation, and tests of PIT revealed that, although the rats in all of the groups performed similarly during both the instrumental and pavlovian acquisition phases, BLA, but not CN, lesions abolished selective sensitivity to a change in the reward value of the instrumental outcome as well as to the selective excitatory effects of reward-related cues in PIT. In experiment 2, we developed a procedure in which both the general motivational and the specific excitatory effects of pavlovian cues could be assessed in the same animal and found that BLA lesions abolished the outcome-specific but spared the general motivational effects of pavlovian cues. In contrast, lesions of CN abolished the general motivational but spared the specific effects of these cues. Together, these results suggest that the BLA mediates outcome-specific incentive processes, whereas CN is involved in controlling the general motivational influence of reward-related events.

A specific role for posterior dorsolateral striatum in human habit learning

Habits are characterized by an insensitivity to their consequences and, as such, can be distinguished from goal‐directed actions. The neural basis of the development of demonstrably outcome‐insensitive habitual actions in humans has not been previously characterized. In this experiment, we show that extensive training on a free‐operant task reduces the sensitivity of participants’ behavior to a reduction in outcome value. Analysis of functional magnetic resonance imaging data acquired during training revealed a significant increase in task‐related cue sensitivity in a right posterior putamen–globus pallidus region as training progressed. These results provide evidence for a shift from goal‐directed to habit‐based control of instrumental actions in humans, and suggest that cue‐driven activation in a specific region of dorsolateral posterior putamen may contribute to the habitual control of behavior in humans.

Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks

Here we challenge the view that reward‐guided learning is solely controlled by the mesoaccumbens pathway arising from dopaminergic neurons in the ventral tegmental area and projecting to the nucleus accumbens. This widely accepted view assumes that reward is a monolithic concept, but recent work has suggested otherwise. It now appears that, in reward‐guided learning, the functions of ventral and dorsal striata, and the cortico‐basal ganglia circuitry associated with them, can be dissociated. Whereas the nucleus accumbens is necessary for the acquisition and expression of certain appetitive Pavlovian responses and contributes to the motivational control of instrumental performance, the dorsal striatum is necessary for the acquisition and expression of instrumental actions. Such findings suggest the existence of multiple independent yet interacting functional systems that are implemented in iterating and hierarchically organized cortico‐basal ganglia networks engaged in appetitive behaviors ranging from Pavlovian approach responses to goal‐directed instrumental actions controlled by action‐outcome contingencies.