Anthony Dickinson

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Instrumental behaviour is controlled by two systems: a stimulus-response habit mechanism and a goal-directed process that involves two forms of learning. The first is learning about the instrumental contingency between the response and reward, whereas the second consists of the acquisition of incentive value by the reward. Evidence for contingency learning comes from studies of reward devaluation and from demonstrations that instrumental performance is sensitive not only the probability of contiguous reward but also to the probability of unpaired rewards. The process of incentive learning is evident in the acquisition of control over performance by primary motivational states. Preliminary lesion studies of the rat suggest that the prelimbic area of prefrontal cortex plays a role in the contingency learning, whereas the incentive learning for food rewards involves the insular cortex.

Dopamine responses comply with basic assumptions of formal learning theory

According to contemporary learning theories, the discrepancy, or error, between the actual and predicted reward determines whether learning occurs when a stimulus is paired with a reward. The role of prediction errors is directly demonstrated by the observation that learning is blocked when the stimulus is paired with a fully predicted reward. By using this blocking procedure, we show that the responses of dopamine neurons to conditioned stimuli was governed differentially by the occurrence of reward prediction errors rather than stimulus–reward associations alone, as was the learning of behavioural reactions. Both behavioural and neuronal learning occurred predominantly when dopamine neurons registered a reward prediction error at the time of the reward. Our data indicate that the use of analytical tests derived from formal behavioural learning theory provides a powerful approach for studying the role of single neurons in learning.

The neuropsychological basis of addictive behaviour

The argument advanced in this review is that drug addiction can be understood in terms of normal learning and memory systems of the brain which, through the actions of chronically self-administered drugs, are pathologically subverted, thereby leading to the establishment of compulsive drug-seeking habits, strengthened by the motivational impact of drug-associated stimuli and occurring at the expense of other sources of reinforcement. We review data from our studies that have utilized procedures which reveal the various influences of pavlovian stimuli on goal-directed behaviour, namely discriminated approach, pavlovian-to-instrumental transfer and conditioned reinforcement, in order to demonstrate their overlapping and also unique neural bases. These fundamental studies are also reviewed in the context of the neural and psychological mechanisms underlying drug-seeking behaviour that is under the control of drug-associated environmental stimuli. The ways in which such drug-seeking behaviour becomes compulsive and habitual, as well as the propensity for relapse to drug-seeking even after long periods of relapse, are discussed in terms of the aberrant learning set in train by the effects of self-administered drugs on plastic processes in limbic cortical-ventral striatal systems.

Motivational control of goal-directed action

The control of goal-directed, instrumental actions by primary motivational states, such as hunger and thirst, is mediated by two processes. The first is engaged by the Pavlovian association between contextual or discriminative stimuli and the outcome or reinforcer presented during instrumental training. Such stimuli exert a motivational influence on instrumental performance that depends upon the relevance of the associated outcome to the current motivational state of the agent. Moreover, the motivational effects of these stimuli operate in the absence of prior experience with the outcome under the relevant motivational state. The second, instrumental, process is mediated by knowledge of the contingency between the action and its outcome and controls the value assigned to this outcome. In contrast to the Pavlovian process, motivational states do not influence the instrumental process directly; rather, the agent has to learn about the value of an outcome in a given motivational state by exposure to it while in that state. This incentive learning is similar in certain respects to the acquisition of “cathexes” envisaged by Tolman (1949a, 1949b).

Instrumental responding following reinforcer devaluation

In two experiments, hungry rats were given instrumental lever-press training for an appetitive reinforcer and, in addition, were exposed to another type of food which was not contingent on lever pressing. In the first experiment, exposure to each type of food was on separate days, whereas in the second experiment rats were exposed to each type of food in strict alternation within each session. Subsequently, a food aversion was conditioned to the reinforcer for the experimental group and to the non-contingent food for the control group. In both experiments, animals with an aversion to the reinforcer responded less in an extinction test than animals with an aversion to the non-contingent food. Subsequent reacquisition tests confirmed that the aversion to the non-contingent food in the control group was of comparable strength with that to the reinforcer in the experimental group. The results were discussed in terms of whether the reinforcer is encoded in the associative structure set up by exposure to an instrumental contingency.

Planning for the future by western scrub-jays

Knowledge of and planning for the future is a complex skill that is considered by many to be uniquely human. We are not born with it; children develop a sense of the future at around the age of two and some planning ability by only the age of four to five. According to the Bischof-Köhler hypothesis, only humans can dissociate themselves from their current motivation and take action for future needs: other animals are incapable of anticipating future needs, and any future-oriented behaviours they exhibit are either fixed action patterns or cued by their current motivational state. The experiments described here test whether a member of the corvid family, the western scrub-jay (Aphelocoma californica), plans for the future. We show that the jays make provision for a future need, both by preferentially caching food in a place in which they have learned that they will be hungry the following morning and by differentially storing a particular food in a place in which that type of food will not be available the next morning. Previous studies have shown that, in accord with the Bischof-Köhler hypothesis, rats and pigeons may solve tasks by encoding the future but only over very short time scales. Although some primates and corvids take actions now that are based on their future consequences, these have not been shown to be selected with reference to future motivational states, or without extensive reinforcement of the anticipatory act. The results described here suggest that the jays can spontaneously plan for tomorrow without reference to their current motivational state, thereby challenging the idea that this is a uniquely human ability.

Determining the Neural Substrates of Goal-Directed Learning in the Human Brain

Instrumental conditioning is considered to involve at least two distinct learning systems: a goal-directed system that learns associations between responses and the incentive value of outcomes, and a habit system that learns associations between stimuli and responses without any link to the outcome that that response engendered. Lesion studies in rodents suggest that these two distinct components of instrumental conditioning may be mediated by anatomically distinct neural systems. The aim of the present study was to determine the neural substrates of the goal-directed component of instrumental learning in humans. Nineteen human subjects were scanned with functional magnetic resonance imaging while they learned to choose instrumental actions that were associated with the subsequent delivery of different food rewards (tomato juice, chocolate milk, and orange juice). After training, one of these foods was devalued by feeding the subject to satiety on that food. The subjects were then scanned again, while being re-exposed to the instrumental choice procedure (in extinction). We hypothesized that regions of the brain involved in goal-directed learning would show changes in their activity as a function of outcome devaluation. Our results indicate that neural activity in one brain region in particular, the orbitofrontal cortex, showed a strong modulation in its activity during selection of a devalued compared with a nondevalued action. These results suggest an important contribution of orbitofrontal cortex in guiding goal-directed instrumental choices in humans.

Within-compound associations mediate the retrospective revaluation of causality judgements.

The role of within-compound associations in the retrospective revaluation of causality judgements was investigated in a two-stage procedure in which the subjects were asked to learn whether or not different food stimuli caused an allergic reaction in hypothetical patients. In the compound-cue stage a number of compound cues, each consisting of a competing stimulus and a target stimulus, were associated with the reaction across a series of trials, whereas in the single-cue stage the subjects had the opportunity to learn which of the competing cues, when presented alone, caused the reaction. Each target stimulus was presented with the same competing cue across all compound trials in the consistent condition, but with a different competing cue on each trial in the varied condition. In a forward procedure, in which the single-cue stage preceded compound cue training, judgements of the causal effectiveness of the target stimuli were reduced or blocked by training them in compound with a competing cue that had been previously paired with the reaction. Moreover, the magnitude of this reduction was comparable in the consistent and varied conditions. This was not true, however, when the single- and compound-cue stages were reversed in the backward procedure. Judgements for target cues compounded with competing cues that were subsequently paired with the reaction were reduced only in the consistent condition. If it is assumed that stronger associations were formed between the competing and target stimuli during the compound-cue stage in the consistent condition than in the varied condition, this pattern suggests that the retrospective revaluation of causality judgements can be mediated by the formation of within-compound associations.

Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction.

In this review we discuss the evidence that drug addiction, defined as a maladaptive compulsive habit, results from the progressive subversion by addictive drugs of striatum-dependent operant and Pavlovian learning mechanisms that are usually involved in the control over behaviour by stimuli associated with natural reinforcement. Although mainly organized through segregated parallel cortico-striato-pallido-thalamo-cortical loops involved in motor or emotional functions, the basal ganglia, and especially the striatum, are key mediators of the modulation of behavioural responses, under the control of both action-outcome and stimulus-response mechanisms, by incentive motivational processes and Pavlovian associations. Here we suggest that protracted exposure to addictive drugs recruits serial and dopamine-dependent, striato-nigro-striatal ascending spirals from the nucleus accumbens to more dorsal regions of the striatum that underlie a shift from action-outcome to stimulus-response mechanisms in the control over drug seeking. When this progressive ventral to dorsal striatum shift is combined with drug-associated Pavlovian influences from limbic structures such as the amygdala and the orbitofrontal cortex, drug seeking behaviour becomes established as an incentive habit. This instantiation of implicit sub-cortical processing of drug-associated stimuli and instrumental responding might be a key mechanism underlying the development of compulsive drug seeking and the high vulnerability to relapse which are hallmarks of drug addiction.

Judgement of act-outcome contingency: The role of selective attribution

In the first experiment subjects were presented with a number of sets of trials on each of which they could perform a particular action and observe the occurrence of an outcome in the context of a video game. The contingency between the action and outcome was varied across the different sets of trials. When required to judge the effectiveness of the action in controlling the outcome during a set of trials, subjects assigned positive ratings for a positive contingency and negative ratings for a negative contingency. Furthermore, the magnitude of the ratings was related systematically to the strength of the actual contingency. With a fixed probability of an outcome given the action, judgements of positive contingencies decreased as the likelihood that the outcome would occur without the action was raised. Correspondingly, the absolute value of ratings of negative contingencies was increased both by an increment in the probability of the outcome in the absence of the action and by a decrement in the probability of the outcome following the action. A systematic bias was observed, however, in that positive judgements were given under a non-contingent relationship when the outcome frequency was relatively high. However, this bias could be reduced by giving extended exposure to the non-contingent schedule (Experiment 2). This pattern of contingency judgements can be explained if it is assumed that a process of selective attribution operates, whereby people are less likely to attribute an outcome to some potential target cause if another effective cause is present. Experiments 2 and 3 demonstrated the operation of this process by showing that initially establishing another agent as an effective cause of the outcome subsequently reduced or blocked the extent to which the subjects attributed the outcome to the action. Finally, we argue that the pattern and bias in contingency judgements based upon interactions with a causal process can be explained in terms of contemporary conditioning models of associative learning.