Brian I. Hyland

A cellular mechanism of reward-related learning.

Positive reinforcement helps to control the acquisition of learned behaviours. Here we report a cellular mechanism in the brain that may underlie the behavioural effects of positive reinforcement. We used intracranial self-stimulation (ICSS) as a model of reinforcement learning, in which each rat learns to press a lever that applies reinforcing electrical stimulation to its own substantia nigra. The outputs from neurons of the substantia nigra terminate on neurons in the striatum in close proximity to inputs from the cerebral cortex on the same striatal neurons. We measured the effect of substantia nigra stimulation on these inputs from the cortex to striatal neurons and also on how quickly the rats learned to press the lever. We found that stimulation of the substantia nigra (with the optimal parameters for lever-pressing behaviour) induced potentiation of synapses between the cortex and the striatum, which required activation of dopamine receptors. The degree of potentiation within ten minutes of the ICSS trains was correlated with the time taken by the rats to learn ICSS behaviour. We propose that stimulation of the substantia nigra when the lever is pressed induces a similar potentiation of cortical inputs to the striatum, positively reinforcing the learning of the behaviour by the rats.

Dopamine Cells Respond to Predicted Events during Classical Conditioning: Evidence for Eligibility Traces in the Reward-Learning Network

Behavioral conditioning of cue-reward pairing results in a shift of midbrain dopamine (DA) cell activity from responding to the reward to responding to the predictive cue. However, the precise time course and mechanism underlying this shift remain unclear. Here, we report a combined single-unit recording and temporal difference (TD) modeling approach to this question. The data from recordings in conscious rats showed that DA cells retain responses to predicted reward after responses to conditioned cues have developed, at least early in training. This contrasts with previous TD models that predict a gradual stepwise shift in latency with responses to rewards lost before responses develop to the conditioned cue. By exploring the TD parameter space, we demonstrate that the persistent reward responses of DA cells during conditioning are only accurately replicated by a TD model with long-lasting eligibility traces (nonzero values for the parameter λ) and low learning rate (α). These physiological constraints for TD parameters suggest that eligibility traces and low per-trial rates of plastic modification may be essential features of neural circuits for reward learning in the brain. Such properties enable rapid but stable initiation of learning when the number of stimulus-reward pairings is limited, conferring significant adaptive advantages in real-world environments.

Neural mechanisms of reward-related motor learning

The analysis of the neural mechanisms responsible for reward-related learning has benefited from recent studies of the effects of dopamine on synaptic plasticity. Dopamine-dependent synaptic plasticity may lead to strengthening of selected inputs on the basis of an activity-dependent conjunction of sensory afferent activity, motor output activity, and temporally related firing of dopamine cells. Such plasticity may provide a link between the reward-related firing of dopamine cells and the acquisition of changes in striatal cell activity during learning. This learning mechanism may play a special role in the translation of reward signals into context-dependent response probability or directional bias in movement responses.

Pedunculopontine Tegmental Nucleus Controls Conditioned Responses of Midbrain Dopamine Neurons in Behaving Rats

Midbrain dopamine (DA) neurons respond to sensory cues that predict reward. We tested the hypothesis that projections from the pedunculopontine tegmental nucleus (PPTg) are involved in driving this DA cell activity. First, the activity of PPTg and DA neurons was compared in a cued-reward associative learning paradigm. The majority of PPTg neurons showed phasic responses to the onset of sensory cues, at significantly shorter latency than DA cells, consistent with a PPTg-to-DA transmission of information. However, unlike DA cells, PPTg responses were almost entirely independent of whether signals were associated with rewards. Second, DA neuron responses to the cues were recorded in free-moving rats during reversible inactivation of the PPTg by microinfusion of local anesthetic. The results showed clear suppression of conditioned sensory responses of DA neurons after PPTg inactivation that was not seen after saline infusion or in non-DA cells. We propose that the PPTg relays information about the precise timing of attended sensory events, which is integrated with information about reward context by DA neurons.

Striatal Contributions to Reward and Decision Making

Abstract:  The striatum is the major input nucleus of the basal ganglia. It is thought to play a key role in learning on the basis of positive reinforcement and in action selection. One view of the striatum conceives it as comprising a reiterated matrix of processing units that perform common operations in different striatal regions, namely synaptic plasticity according to a three‐factor rule, and lateral inhibition. These operations are required for reinforcement learning and selection of previously reinforced actions. Analysis of the behavioral effects of circumscribed lesions of the striatum, however, suggests regional specialization of learning and decision‐making operations. We consider how a basic processing unit may be modified by regional variations in neurochemical parameters, for example, by the gradient in density of dopamine terminals from dorsal to ventral striatum. These variations suggest subtle differences between dorsolateral and ventromedial striatal regions in the temporal properties of dopamine signaling, which are superimposed on regional differences in connectivity. We propose that these variations make sense in relation to the temporal structure of activity in striatal inputs from different regions, and the requirements of different learning operations. Dorsolateral striatal (DLS) regions may be subject to brief, precisely timed pulses of dopamine, whereas ventromedial striatal regions integrate dopamine signals over a longer time course. These differences may be important for understanding regional variations in the contribution to reinforcement of habits, versus incentive processes that are sensitive to the value of expected rewards.

Cortical cell assemblies: a possible mechanism for motor programs.

The concept of a motor program has been used to interpret a diverse range of empirical findings related to preparation and initiation of voluntary movement. In the absence of an underlying mechanism, its exploratory power has been limited to that of an analogy with running a stored computer program. We argue that the theory of cortical cell assemblies suggests a possible neural mechanism for motor programming. According to this view, a motor program may be conceptualized as a cell assembly, which is stored in the form of strengthened synaptic connections between cortical pyramidal neurons. These connections determine which combinations of corticospinal neurons are activated when the cell assembly is ignited. The dynamics of cell assembly ignition are considered in relation to the problem of serial order. These considerations lead to a plausible neural mechanism for the programming of movements and movement sequences that is compatible with the effects of precue information and sequence length on reaction times. Anatomical and physiological guidelines for future quantitative models of cortical cell assemblies are suggested. By taking into account the parallel re-entrant loops between the cerebral cortex and basal ganglia, the theory of cortical cell assemblies suggests a mechanism for motor plans that involve longer sequences. The suggested model is compared with other existing neural network models for motor programming.

Modulation of an Afterhyperpolarization by the Substantia Nigra Induces Pauses in the Tonic Firing of Striatal Cholinergic Interneurons

Striatal cholinergic interneurons, also known as tonically active neurons (TANs), acquire a pause in firing during learning of stimulus-reward associations. This pause response to a sensory stimulus emerges after repeated pairing with a reward. The conditioned pause is dependent on dopamine from the substantia nigra, but its underlying cellular mechanism is unknown. Using in vivo intracellular recording, we found that both subthreshold and suprathreshold depolarizations in cholinergic interneurons induced a prolonged after-hyperpolarization (AHP) associated with a pause in their tonic firing. The AHP duration was dependent on the level of depolarization, whether elicited by intracellular current injection or by activation of excitatory inputs from the cortex. High-frequency stimulation of the substantia nigra induced potentiation of the cortically evoked excitation and increased the prolonged AHP after the stimulus. These findings from anesthetized animals suggest that a substantia nigra-induced AHP produces stimulus-associated firing pauses in cholinergic interneurons. This mechanism may underlie the acquisition of the pause response in TANs recorded from behaving animals during learning.

Cortical Effects of Subthalamic Stimulation Correlate with Behavioral Recovery from Dopamine Antagonist Induced Akinesia

High-frequency stimulation of around 130 Hz delivered to the subthalamic nucleus (STN-DBS [deep brain stimulation]) is an effective treatment of Parkinsons disease (PD), but the mechanisms of its therapeutic effect remain obscure. Recently, it has been shown in anaesthetized rats that STN-DBS antidromically activates cortical neurons with coincident reduction of the cortical slow wave oscillations that occur in this preparation. Here we extend this work; recording the effect of STN-DBS upon cortical EEG and akinesia, in unanesthetized rats rendered cataleptic by acute dopaminergic blockade. STN-DBS-like stimulation resulted in a short latency, presumed antidromic, evoked potential in the cortex. In cataleptic animals, there was a significant increase in the power of beta oscillations in the electroencephalography which was reversed by stimulation that evoked the cortical response. We also observed a significant rescue of motor function, with the level of akinesia (bar test score) being inversely correlated to the amplitude of the evoked potential (R2 = 0.84). These data confirm that (probably antidromic) short latency cortical responses occur in the awake animal and that these are associated with reductions in abnormal cortical oscillations characteristic of PD and with improvements in akinesia. Our results raise the possibility that STN-DBS reduces PD oscillations and symptoms through antidromic cortical activation.

Tripartite Mechanism of Extinction Suggested by Dopamine Neuron Activity and Temporal Difference Model

Extinction of behavior enables adaptation to a changing world and is crucial for recovery from disorders such as phobias and drug addiction. However, the brain mechanisms underlying behavioral extinction remain poorly understood. Midbrain dopamine (DA) neurons appear to play a central role in most acquisition processes of appetitive conditioning. Here, we show that the responses of putative DA neurons to conditioned reward predicting cues also dynamically encode two classical features of extinction: decrement in amplitude of previously learned excitatory responses and rebound of responding on subsequent retesting (spontaneous recovery). Crucially, this encoding involves development of inhibitory responses in the DA neurons, reflecting new, extinction-specific learning in the brain. We explored the implications of this finding by adding such inhibitory inputs to a standard temporal difference model of DA cell activity. We found that combining extinction-triggered plasticity of these inputs with a time-dependent spontaneous decay of weights, equivalent to a forgetting process as described in classical behavioral extinction literature, enabled the model to simulate several classical features of extinction. A key requirement to achieving spontaneous recovery was differential rates of spontaneous decay for weights representing original conditioning and for subsequent extinction learning. A testable prediction of the model is thus that differential decay properties exist within the wider circuits regulating DA cell activity. These findings are consistent with the hypothesis that extinction processes at both cellular and behavioral levels involve a dynamic interaction between new (inhibitory) learning, forgetting, and unlearning.

Neural activity related to reaching and grasping in rostral and caudal regions of rat motor cortex.

The objective of this study was to assess the relation of motor cortical neural activity in the rat to self-paced reach-to-grasp movements. Overall, around 40% of excitatory and 60% of inhibitory modulations in neuronal activity began prior to reach onset. These data are consistent with a role for rat motor cortex in the initiation and control of the reaching movement. In addition, although the reach only lasted a short time, 30% of excitations and inhibitions began while it was in progress. The existence of such modulations occurring during the reach is consistent with previous data showing activity of cortical neurons late in the reach, and suggests a heavy involvement of cortical neurons in controlling the recently described, complex movements associated with grasping that are seen in the rat. These features were broadly similar in neurones from both the caudal and rostral subdivisions of rat motor cortex.