Jeffery R. Wickens

A cellular mechanism of reward-related learning.

Positive reinforcement helps to control the acquisition of learned behaviours. Here we report a cellular mechanism in the brain that may underlie the behavioural effects of positive reinforcement. We used intracranial self-stimulation (ICSS) as a model of reinforcement learning, in which each rat learns to press a lever that applies reinforcing electrical stimulation to its own substantia nigra. The outputs from neurons of the substantia nigra terminate on neurons in the striatum in close proximity to inputs from the cerebral cortex on the same striatal neurons. We measured the effect of substantia nigra stimulation on these inputs from the cortex to striatal neurons and also on how quickly the rats learned to press the lever. We found that stimulation of the substantia nigra (with the optimal parameters for lever-pressing behaviour) induced potentiation of synapses between the cortex and the striatum, which required activation of dopamine receptors. The degree of potentiation within ten minutes of the ICSS trains was correlated with the time taken by the rats to learn ICSS behaviour. We propose that stimulation of the substantia nigra when the lever is pressed induces a similar potentiation of cortical inputs to the striatum, positively reinforcing the learning of the behaviour by the rats.

Dopamine-dependent plasticity of corticostriatal synapses

Knowledge of the effect of dopamine on corticostriatal synaptic plasticity has advanced rapidly over the last 5 years. We consider this new knowledge in relation to three factors proposed earlier to describe the rules for synaptic plasticity in the corticostriatal pathway. These factors are a phasic increase in dopamine release, presynaptic activity and postsynaptic depolarisation. A function is proposed which relates the amount of dopamine release in the striatum to the modulation of corticostriatal synaptic efficacy. It is argued that this function, and the experimental data from which it arises, are compatible with existing models which associate the reward-related firing of dopamine neurons with changes in corticostriatal synaptic efficacy.

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Behavioral conditioning of cue-reward pairing results in a shift of midbrain dopamine (DA) cell activity from responding to the reward to responding to the predictive cue. However, the precise time course and mechanism underlying this shift remain unclear. Here, we report a combined single-unit recording and temporal difference (TD) modeling approach to this question. The data from recordings in conscious rats showed that DA cells retain responses to predicted reward after responses to conditioned cues have developed, at least early in training. This contrasts with previous TD models that predict a gradual stepwise shift in latency with responses to rewards lost before responses develop to the conditioned cue. By exploring the TD parameter space, we demonstrate that the persistent reward responses of DA cells during conditioning are only accurately replicated by a TD model with long-lasting eligibility traces (nonzero values for the parameter λ) and low learning rate (α). These physiological constraints for TD parameters suggest that eligibility traces and low per-trial rates of plastic modification may be essential features of neural circuits for reward learning in the brain. Such properties enable rapid but stable initiation of learning when the number of stimulus-reward pairings is limited, conferring significant adaptive advantages in real-world environments.

Dopamine reverses the depression of rat corticostriatal synapses which normally follows high-frequency stimulation of cortex In vitro

Learning deficits resulting from dopamine depletion suggest that striatal dopamine release is crucial for reinforcement. Recently described firing patterns of dopamine neurons in behaving monkeys show that transient increases in dopamine release are brought about by reinforcement. We describe an enduring change in the strength of synaptic transmission following pulsatile application of dopamine intended to mimic the transient increases associated with reinforcement. Intracellular records were made from neurons in slices of the rat corticostriatal system. Neurons having the properties of the medium-sized spiny neurons responded to cortical stimulation with depolarizing potentials (peak amplitude 12.0 +/- 1.3 mV; latency 9.2 +/- 0.1 ms; mean +/- S.D., n = 19), which had the properties of monosynaptic excitatory postsynaptic potentials. After trains of stimuli to the cortex had been applied in conjunction with intracellular depolarizing current, the size of these excitatory postsynaptic potentials was reduced (-27% at 20 min). Application of dopamine (approximately 30 microM) in a solution containing KCl concomitant with depolarization and presynaptic activation increased the subsequent excitatory postsynaptic potentials (+21% at 20 min) without significant lasting change in the membrane properties of the postsynaptic cell. This suggests that dopamine has an enduring, activity-dependent action on the efficacy of corticostriatal transmission, which may be a cellular basis for the learning-related effects of the nigrostriatal system.

Neurobiology of ADHD

Attention-deficit hyperactivity disorder (ADHD) is a prevalent and debilitating disorder diagnosed on the basis of persistent and developmentally-inappropriate levels of overactivity, inattention and impulsivity. The etiology and pathophysiology of ADHD is incompletely understood. There is evidence of a genetic basis for ADHD but it is likely to involve many genes of small individual effect. Differences in the dimensions of the frontal lobes, caudate nucleus, and cerebellar vermis have been demonstrated. Neuropsychological testing has revealed a number of well documented differences between children with and without ADHD. These occur in two main domains: executive function and motivation although neither of these is specific to ADHD. In view of the recent advances in the neurobiology of reinforcement, we concentrate in this review on altered reinforcement mechanisms. Among the motivational differences, many pieces of evidence indicate that an altered response to reinforcement may play a central role in the symptoms of ADHD. In particular, sensitivity to delay of reinforcement appears to be a reliable finding. We review neurobiological mechanisms of reinforcement and discuss how these may be altered in ADHD, with particular focus on the neurotransmitter dopamine and its actions at the cellular and systems level. We describe how dopamine cell firing activity is normally associated with reinforcing events, and transfers to earlier time-points in the behavioural sequence as reinforcement becomes more predictable. We discuss how a failure of this transfer may give rise to many symptoms of ADHD, and propose that methylphenidate might act to compensate for the proposed dopamine transfer deficit.

Neural mechanisms of reward-related motor learning

The analysis of the neural mechanisms responsible for reward-related learning has benefited from recent studies of the effects of dopamine on synaptic plasticity. Dopamine-dependent synaptic plasticity may lead to strengthening of selected inputs on the basis of an activity-dependent conjunction of sensory afferent activity, motor output activity, and temporally related firing of dopamine cells. Such plasticity may provide a link between the reward-related firing of dopamine cells and the acquisition of changes in striatal cell activity during learning. This learning mechanism may play a special role in the translation of reward signals into context-dependent response probability or directional bias in movement responses.

Dopaminergic Mechanisms in Actions and Habits

Recent studies suggest new ways to interpret dopaminergic actions in goal-directed performance and habitual responding. In the early stages of learning dopamine plays an essential role, but with extended training dopamine appears to play a decreasing role in response expression. Experimental manipulation of dopamine levels alters the correlation of cortical and striatal neural activity in behaving animals, and these dopamine-dependent changes in corticostriatal correlations may be reflected in changes in action selection in the basal ganglia. Consistent with this hypothesis, changes in dopamine signaling brought about by sensitization with amphetamine mimic the transition from goal-directed to habit-based instrumental performance. At the cellular level, dopamine-dependent synaptic plasticity may be important initially, and subsequently lead to more persistent changes that no longer require dopamine. The locus of these actions within the cortical and corticostriatal circuitry is a focus on ongoing research.

Dopamine and synaptic plasticity in the neostriatum

After the unilateral destruction of the dopamine input to the neostriatum there are enduring changes in rat behaviour. These have been ascribed to the loss of dopamine and the animals are often referred to as ‘hemiparkinsonian’. In the denervated neostriatum, we have shown that not only are the tyrosine hydroxylase positive boutons missing, but also the medium sized densely spiny output cells have fewer spines. Spines usually have asymmetric synapses on their heads. In a recent stereological study we were able to show that there is a loss of approximately 20% of asymmetric synapses in the lesioned neostriatum by 1 mo after the lesion. Current experiments are trying to establish the specificity of this loss. So far we have evidence suggesting that there is no obvious preferential loss of synapses from either D1 or D2 receptor immunostained dendrites in the neostriatum with damaged dopamine innervation. These experiments suggest that dopamine is somehow necessary for the maintenance of corticostriatal synapses in the neostriatum. In a different series of experiments slices of cortex and neostriatum were maintained in vitro in such a way as to preserve at least some of the corticostriatal connections. In this preparation we have been able to show that cortical stimulation results in robust excitatory postsynaptic potentials (EPSPs) recorded from inside striatal neurons. Using stimulation protocols derived from the experiments on hippocampal synaptic plasticity we have shown that the usual consequence of trains of high frequency stimulation of the cortex is the depression of the size of EPSPs in the striatal cell. In agreement with similar experiments by others, the effect seems to be influenced by NMDA receptors since the unblocking of these receptors with low Mg++ concentrations in the perfusate uncovers a potentiation of the EPSPs after trains of stimulation. Dopamine applied in the perfusion fluid round the slices has no effect but pulsatile application of dopamine, close to the striatal cell being recorded from, and in temporal association with the cortical trains, leads to a similar LTP like effect. The reduction of K+ channel conductance in the bath with TEA also has the effect of making cortical trains induce potentiation of corticostriatal transmission. TEA applied only to the cell being recorded from has no similar effect; the cortical stimulation again depresses the EPSP amplitude, so the site of action of TEA may well be presynaptic to the striatal cell. The morphological and physiological experiments may not necessarily be related but it is tempting to suggest that dopamine protects some corticostriatal synapses by potentiating them but that in the absence of dopamine others simply disconnect and are no longer detectable on electron microscopy.

Timing is not Everything: Neuromodulation Opens the STDP Gate

Spike timing dependent plasticity (STDP) is a temporally specific extension of Hebbian associative plasticity that has tied together the timing of presynaptic inputs relative to the postsynaptic single spike. However, it is difficult to translate this mechanism to in vivo conditions where there is an abundance of presynaptic activity constantly impinging upon the dendritic tree as well as ongoing postsynaptic spiking activity that backpropagates along the dendrite. Theoretical studies have proposed that, in addition to this pre- and postsynaptic activity, a “third factor” would enable the association of specific inputs to specific outputs. Experimentally, the picture that is beginning to emerge, is that in addition to the precise timing of pre- and postsynaptic spikes, this third factor involves neuromodulators that have a distinctive influence on STDP rules. Specifically, neuromodulatory systems can influence STDP rules by acting via dopaminergic, noradrenergic, muscarinic, and nicotinic receptors. Neuromodulator actions can enable STDP induction or – by increasing or decreasing the threshold – can change the conditions for plasticity induction. Because some of the neuromodulators are also involved in reward, a link between STDP and reward-mediated learning is emerging. However, many outstanding questions concerning the relationship between neuromodulatory systems and STDP rules remain, that once solved, will help make the crucial link from timing-based synaptic plasticity rules to behaviorally based learning.

Striatal dopamine in motor activation and reward-mediated learning: steps towards a unifying model

On the basis of behavioural evidence, dopamine is found to be involved in two higher-level functions of the brain: reward-mediated learning and motor activation. In these functions dopamine appears to mediate synaptic enhancement in the corticostriatal pathway. However, in electrophysiological studies, dopamine is often reported to inhibit corticostriatal transmission. These two effects of dopamine seem incompatible. The existence of separate populations of dopamine receptors, differentially modulating cholinergic and glutamatergic synapses, suggests a possible resolution to this paradox. The synaptic enhancement which occurs in reward-mediated learning may also be involved in dopamine-mediated motor activation. The logical form of reward-mediated learning imposes constraints on which mechanisms can be considered possible. Dopamine D1 receptors may mediate enhancement of corticostriatal synapses. On the other hand, dopamine D2 receptors on cholinergic terminals may mediate indirect, inhibitory effects of dopamine on striatal neurons.