Shunsuke Kobayashi

Influence of Reward Delays on Responses of Dopamine Neurons

Psychological and microeconomic studies have shown that outcome values are discounted by imposed delays. The effect, called temporal discounting, is demonstrated typically by choice preferences for sooner smaller rewards over later larger rewards. However, it is unclear whether temporal discounting occurs during the decision process when differently delayed reward outcomes are compared or during predictions of reward delays by pavlovian conditioned stimuli without choice. To address this issue, we investigated the temporal discounting behavior in a choice situation and studied the effects of reward delay on the value signals of dopamine neurons. The choice behavior confirmed hyperbolic discounting of reward value by delays on the order of seconds. Reward delay reduced the responses of dopamine neurons to pavlovian conditioned stimuli according to a hyperbolic decay function similar to that observed in choice behavior. Moreover, the stimulus responses increased with larger reward magnitudes, suggesting that both delay and magnitude constituted viable components of dopamine value signals. In contrast, dopamine responses to the reward itself increased with longer delays, possibly reflecting temporal uncertainty and partial learning. These dopamine reward value signals might serve as useful inputs for brain mechanisms involved in economic choices between delayed rewards.

Adaptation of Reward Sensitivity in Orbitofrontal Neurons

Animals depend on a large variety of rewards but their brains have a limited dynamic coding range. When rewards are uncertain, neuronal coding needs to cover a wide range of possible rewards. However, when reward is likely to occur within a specific range, focusing the sensitivity on the predicted range would optimize the discrimination of small reward differences. One way to overcome the trade-off between wide coverage and optimal discrimination is to adapt reward sensitivity dynamically to the available rewards. We investigated how changes in reward distribution influenced the coding of reward in the orbitofrontal cortex. Animals performed an oculomotor task in which a fixation cue predicted the SD of the probability distribution of juice volumes, while the expected mean volume was kept constant. A subsequent cue specified the exact juice volume obtained for a correct saccade response. Population responses of orbitofrontal neurons that reflected the predicted juice volume showed adaptation to the reward distribution. Statistical tests on individual responses revealed that a quarter of value-coding neurons shifted the reward sensitivity slope significantly between two reward distributions, whereas the remaining neurons showed insignificant change or lack of adaptation. Adaptations became more prominent when reward distributions changed less frequently, indicating time constraints for assessing reward distributions and adjusting neuronal sensitivity. The observed neuronal adaptation would optimize discrimination and contribute to the efficient coding of a large variety of potential rewards by neurons with limited dynamic range.

Influences of Rewarding and Aversive Outcomes on Activity in Macaque Lateral Prefrontal Cortex

Both appetitive and aversive outcomes can reinforce animal behavior. It is not clear, however, whether the opposing kinds of reinforcers are processed by specific or common neural mechanisms. To investigate this issue, we studied macaque monkeys that performed a memory-guided saccade task for three different outcomes, namely delivery of liquid reward, avoidance of air puff, and feedback sound only. Animals performed the task best in rewarded trials, intermediately in aversive trials, and worst in sound-only trials. Most task-related activity in lateral prefrontal cortex was differentially influenced by the reinforcers. Aversive avoidance had clear effects on some prefrontal neurons, although the effects of rewards were more common. We also observed neurons modulated by both positive and negative reinforcers, reflecting reinforcement or attentional processes. Our results demonstrate that information about positive and negative reinforcers is processed differentially in prefrontal cortex, which could contribute to the role of this structure in goal-directed behavior.

Functional differences between macaque prefrontal cortex and caudate nucleus during eye movements with and without reward

The prefrontal cortex and the basal ganglia form mutually connected networks and are thought to play essential roles together in guiding goal-directed behaviors. Yet, these structures seem to have independent pathways to motor outputs as well, suggesting differential contributions to goal-directed behaviors. We hypothesized that the prefrontal cortex guides actions to a direction required by external demands and the basal ganglia guide actions to an internally motivated direction. To test this hypothesis, we used a task in which monkeys were required to make a memory-guided saccade to a direction indicated by a visual cue while only one direction was associated with reward. We observed a functional dissociation between the lateral prefrontal cortex (LPFC), which commonly represented the cue direction, and the caudate nucleus (CD), which commonly represented the reward-associated direction. Furthermore, cue-directed and reward-directed signals were integrated differently in the two areas; when the cue direction and the reward direction were opposite, LPFC neurons maintained tuning to the cue direction, whereas CD neurons lost the tuning. Different types of spatial tuning in the two brain areas may contribute to different types of goal-directed behavior.

Reward Contexts Extend Dopamine Signals to Unrewarded Stimuli

Summary Basic tenets of sensory processing emphasize the importance of accurate identification and discrimination of environmental objects [1]. Although this principle holds also for reward, the crucial acquisition of reward for survival would be aided by the capacity to detect objects whose rewarding properties may not be immediately apparent. Animal learning theory conceptualizes how unrewarded stimuli induce behavioral reactions in rewarded contexts due to pseudoconditioning and higher-order context conditioning [2–6]. We hypothesized that the underlying mechanisms may involve context-sensitive reward neurons. We studied short-latency activations of dopamine neurons to unrewarded, physically salient stimuli while systematically changing reward context. Dopamine neurons showed substantial activations to unrewarded stimuli and their conditioned stimuli in highly rewarded contexts. The activations decreased and often disappeared entirely with stepwise separation from rewarded contexts. The influence of reward context suggests that dopamine neurons respond to real and potential reward. The influence of reward context is compatible with the reward nature of phasic dopamine responses. The responses may facilitate rapid, default initiation of behavioral reactions in environments usually containing reward. Agents would encounter more and miss less reward, resulting in survival advantage and enhanced evolutionary fitness.

Neuronal signals for reward risk in frontal cortex

Rewards can be viewed as probability distributions of reward values. Besides expected (mean) value, a key parameter of such distributions is variance (or standard deviation), which constitutes a measure of risk. Single neurons in orbitofrontal cortex signal risk mostly separately from value. Comparable risk signals in human frontal cortex reflect risk attitudes of individual participants. Subjective outcome value constitutes the primary economic decision variable. The terms risk avoidance and risk taking suggest that risk affects subjective outcome value, a basic tenet of economic decision theories. Correspondingly, risk reduces neuronal value signals in frontal cortex of human risk avoiders and enhances value signals in risk takers. Behavioral contrast effects and reference‐dependent valuation demonstrate flexible reward valuation. As a potential correlate, value signals in orbitofrontal neurons adjust reward discrimination to variance (risk). These neurophysiological mechanisms of reward risk on economic decisions inform and validate theories of economic decision making under uncertainty.

Organization of Neural Systems for Aversive Information Processing: Pain, Error, and Punishment

The avoidance of aversive events is critically important for the survival of organisms. It has been proposed that the medial pain system, including the amygdala, periaqueductal gray (PAG), and anterior cingulate cortex (ACC), contains the neural circuitry that signals pain affect and negative value. This system appears to have multiple defense mechanisms, such as rapid stereotyped escape, aversive association learning, and cognitive adaptation. These defense mechanisms vary in speed and flexibility, reflecting different strategies of self-protection. Over the course of evolution, the medial pain system appears to have developed primitive, associative, and cognitive solutions for aversive avoidance. There may be a functional grading along the caudal-rostral axis, such that the amygdala-PAG system underlies automatic and autonomic responses, the amygdala-orbitofrontal system contributes to associative learning, and the ACC controls cognitive processes in cooperation with the lateral prefrontal cortex. A review of behavioral and physiological studies on the aversive system is presented, and a conceptual framework for understanding the neural organization of the aversive avoidance system is proposed.

Bidirectional modulation of sensory cortical excitability by quadripulse transcranial magnetic stimulation (QPS) in humans.

OBJECTIVEnQuadripulse transcranial magnetic stimulation (QPS) is a newly designed patterned repetitive transcranial magnetic stimulation (TMS). Previous studies of QPS showed bidirectional effects on the primary motor cortex (M1), which depended on its inter-stimulus interval (ISI): motor evoked potentials (MEPs) were potentiated at short ISIs and depressed at long ISIs (homotopic effects). These physiological characters were compatible with synaptic plasticity. In this research, we studied effects of QPS on the primary sensory cortex (S1).nnnMETHODSnOne burst consisted of four monophasic TMS pulses at an intensity of 90% active motor threshold. The ISI of four pulses was set at 5 ms (QPS-5) or at 50 ms (QPS-50). Same bursts were given every 5s for 30 min. QPS-5 and QPS-50 were performed over three areas (M1, S1 and dorsal premotor cortex (dPMC)). One sham stimulation session was also performed. Excitability changes of S1 were evaluated by timeline of somatosensory evoked potentials (SEPs).nnnRESULTSnQPS-5 over M1 or dPMC enhanced the P25-N33 component of SEP, and QPS-50 over M1 depressed it. By contrast, QPSs over S1 had no effects on SEPs.nnnCONCLUSIONSnQPSs over motor cortices modulated the S1 cortical excitability (heterotopic effects). Mutual connections between dPMC or M1 and S1 might be responsible for these modulations.nnnSIGNIFICANCEnQPSs induced heterotopic LTP or LTD-like cortical excitability changes.

Operant conditioning of primate prefrontal neurons.

An operant is a behavioral act that has an impact on the environment to produce an outcome, constituting an important component of voluntary behavior. Because the environment can be volatile, the same action may cause different consequences. Thus to obtain an optimal outcome, it is crucial to detect action–outcome relationships and adapt the behavior accordingly. Although prefrontal neurons are known to change activity depending on expected reward, it remains unknown whether prefrontal activity contributes to obtaining reward. We investigated this issue by setting variable relationships between levels of single-neuron activity and rewarding outcomes. Lateral prefrontal neurons changed their spiking activity according to the specific requirements for gaining reward, without the animals making a motor response. Thus spiking activity constituted an operant response. Data from a control task suggested that these changes were unlikely to reflect simple reward predictions. These data demonstrate a remarkable capacity of prefrontal neurons to adapt to specific operant requirements at the single-neuron level.

Components and characteristics of the dopamine reward utility signal.

Rewards are defined by their behavioral functions in learning (positive reinforcement), approach behavior, economic choices, and emotions. Dopamine neurons respond to rewards with two components, similar to higher order sensory and cognitive neurons. The initial, rapid, unselective dopamine detection component reports all salient environmental events irrespective of their reward association. It is highly sensitive to factors related to reward and thus detects a maximal number of potential rewards. It also senses aversive stimuli but reports their physical impact rather than their aversiveness. The second response component processes reward value accurately and starts early enough to prevent confusion with unrewarded stimuli and objects. It codes reward value as a numeric, quantitative utility prediction error, consistent with formal concepts of economic decision theory. Thus, the dopamine reward signal is fast, highly sensitive and appropriate for driving and updating economic decisions. J. Comp. Neurol. 524:1699–1711, 2016.