Saori C. Tanaka

Humans Can Adopt Optimal Discounting Strategy under Real-Time Constraints

Critical to our many daily choices between larger delayed rewards, and smaller more immediate rewards, are the shape and the steepness of the function that discounts rewards with time. Although research in artificial intelligence favors exponential discounting in uncertain environments, studies with humans and animals have consistently shown hyperbolic discounting. We investigated how humans perform in a reward decision task with temporal constraints, in which each choice affects the time remaining for later trials, and in which the delays vary at each trial. We demonstrated that most of our subjects adopted exponential discounting in this experiment. Further, we confirmed analytically that exponential discounting, with a decay rate comparable to that used by our subjects, maximized the total reward gain in our task. Our results suggest that the particular shape and steepness of temporal discounting is determined by the task that the subject is facing, and question the notion of hyperbolic reward discounting as a universal principle.

Serotonin and the Evaluation of Future Rewards Theory, Experiments, and Possible Neural Mechanisms

Abstract:  The ability to select an action by considering both delays and amount of reward outcome is critical for survival and well‐being of animals and humans. Previous animal experiments suggest a role of serotonin in action choice by modulating the evaluation of delayed rewards. It remains unclear, however, through which neural circuits, and through what receptors and intracellular mechanisms, serotonin affects the evaluation of delayed rewards. Here, we review experimental studies and computational theory of decisions under delayed rewards, and propose that serotonin controls the timescale of reward prediction by regulating neural activity in the basal ganglia.

Inter-individual discount factor differences in reward prediction are topographically associated with caudate activation

In general, humans tend to devalue a delayed reward. Such delay discounting is a theoretical and computational concept in which the discount factor influences the time scale of the trade-off between delay of reward and amount of reward. The discount factor relies on the individual’s ability to evaluate the future reward. Using functional magnetic resonance imaging, we investigated brain mechanisms for reward valuation at different individual discount factors in a delayed reward choice task. In the task, participants were required to select small/immediate or large/delayed rewards to maximize the total reward over time. The discount factor for each participant individually was calculated from the behavioral data based on an exponential discounting model. The estimated value of a future reward increases as the expected delivery approaches, so the time course of these estimated values was computed based on each individual’s discount factor; each was entered into the regression analysis as an explanatory (independent) variable. After the region of interest was narrowed anatomically to the caudate, a peak coordinate was detected in each individual. A correlation analysis revealed that the location of the peak along the dorsal–ventral axis in the right caudate was positively correlated with the discount factor. This implies that individuals who showed a larger discount factor had peak activations in a more dorsal part of the right caudate associated with future reward prediction. This evidence also suggests that a higher ability to delay reward prediction might be related to activation of the more dorsal caudate.

Neural Mechanisms of Gain–Loss Asymmetry in Temporal Discounting

Humans typically discount future gains more than losses. This phenomenon is referred to as the “sign effect” in experimental and behavioral economics. Although recent studies have reported associations between the sign effect and important social problems, such as obesity and incurring multiple debts, the biological basis for this phenomenon remains poorly understood. Here, we hypothesized that enhanced loss-related neural processing in magnitude and/or delay representation are causes of the sign effect. We examined participants performing intertemporal choice tasks involving future gains or losses and compared the brain activity of those who exhibited the sign effect and those who did not. When predicting future losses, significant differences were apparent between the two participant groups in terms of striatal activity representing delay length and in insular activity representing sensitivity to magnitude. Furthermore, participants with the sign effect exhibited a greater insular response to the magnitude of loss than to that of gain, and also a greater striatal response to the delay of loss than to that of gain. These findings may provide a new biological perspective for the development of novel treatments and preventive measures for social problems associated with the sign effect.

Model-based action planning involves cortico-cerebellar and basal ganglia networks

Humans can select actions by learning, planning, or retrieving motor memories. Reinforcement Learning (RL) associates these processes with three major classes of strategies for action selection: exploratory RL learns state-action values by exploration, model-based RL uses internal models to simulate future states reached by hypothetical actions, and motor-memory RL selects past successful state-action mapping. In order to investigate the neural substrates that implement these strategies, we conducted a functional magnetic resonance imaging (fMRI) experiment while humans performed a sequential action selection task under conditions that promoted the use of a specific RL strategy. The ventromedial prefrontal cortex and ventral striatum increased activity in the exploratory condition; the dorsolateral prefrontal cortex, dorsomedial striatum, and lateral cerebellum in the model-based condition; and the supplementary motor area, putamen, and anterior cerebellum in the motor-memory condition. These findings suggest that a distinct prefrontal-basal ganglia and cerebellar network implements the model-based RL action selection strategy.

Neural and Personality Correlates of Individual Differences Related to the Effects of Acute Tryptophan Depletion on Future Reward Evaluation

Background/Aims: In general, humans tend to discount the value of delayed reward. An increase in the rate of discounting leads to an inability to select a delayed reward over a smaller immediate reward (reward-delay impulsivity). Although deficits in the serotonergic system are implicated in this reward-delay impulsivity, there is individual variation in response to serotonin depletion. The aim of the present study was to investigate whether the effects of serotonin depletion on the ability to evaluate future reward are affected by individual personality traits or brain activation. Methods: Personality traits were assessed using the NEO-Five Factor Inventory and Temperament and Character Inventory. The central serotonergic levels of 16 healthy volunteers were manipulated by dietary tryptophan depletion. Subjects performed a delayed reward choice task that required the continuous estimation of reward value during functional magnetic resonance imaging scanning. Results: Discounting rates were increased in 9 participants, but were unchanged or decreased in 7 participants in response to tryptophan depletion. Participants whose discounting rate was increased by tryptophan depletion had significantly higher neuroticism and lower self-directedness. Furthermore, tryptophan depletion differentially affected the groups in terms of hemodynamic responses to the value of predicted future reward in the right insula. Conclusions: These results suggest that individuals who have high neuroticism and low self-directedness as personality traits are particularly vulnerable to the effect of low serotonin on future reward evaluation accompanied by altered brain activation patterns.

A Neural Marker of Obsessive-Compulsive Disorder from Whole-Brain Functional Connectivity

Obsessive-compulsive disorder (OCD) is a common psychiatric disorder with a lifetime prevalence of 2–3%. Recently, brain activity in the resting state is gathering attention for exploring altered functional connectivity in psychiatric disorders. Although previous resting-state functional magnetic resonance imaging studies investigated the neurobiological abnormalities of patients with OCD, there are concerns that should be addressed. One concern is the validity of the hypothesis employed. Most studies used seed-based analysis of the fronto-striatal circuit, despite the potential for abnormalities in other regions. A hypothesis-free study is a promising approach in such a case, while it requires researchers to handle a dataset with large dimensions. Another concern is the reliability of biomarkers derived from a single dataset, which may be influenced by cohort-specific features. Here, our machine learning algorithm identified an OCD biomarker that achieves high accuracy for an internal dataset (AUC = 0.81; N = 108) and demonstrates generalizability to an external dataset (AUC = 0.70; N = 28). Our biomarker was unaffected by medication status, and the functional networks contributing to the biomarker were distributed widely, including the frontoparietal and default mode networks. Our biomarker has the potential to deepen our understanding of OCD and to be applied clinically.

Harmonization of resting-state functional MRI data across multiple imaging sites via the separation of site differences into sampling bias and measurement bias

When collecting large neuroimaging data associated with psychiatric disorders, images must be acquired from multiple sites because of the limited capacity of a single site. However, site differences represent the greatest barrier when acquiring multi-site neuroimaging data. We utilized a traveling-subject dataset in conjunction with a multi-site, multi-disorder dataset to demonstrate that site differences are composed of biological sampling bias and engineering measurement bias. Effects on resting-state functional MRI connectivity because of both bias types were greater than or equal to those because of psychiatric disorders. Furthermore, our findings indicated that each site can sample only from among a subpopulation of participants. This result suggests that it is essential to collect large neuroimaging data from as many sites as possible to appropriately estimate the distribution of the grand population. Finally, we developed a novel harmonization method that removed only the measurement bias by using traveling-subject dataset and achieved the reduction of the measurement bias by 29% and the improvement of the signal to noise ratios by 40%.

State-Unspecific Modes of Whole-Brain Functional Connectivity Predict Intelligence and Life Outcomes

Recent functional magnetic resonance imaging (fMRI) studies have increasingly revealed potential neural substrates of individual differences in diverse types of brain function and dysfunction. Although most previous studies have been inherently limited to state-specific characterizations of related brain networks and their functions, several recent studies have examined the potential state-unspecific nature of functional brain networks, such as their global similarities across different experimental conditions (i.e., states) including both task and rest. However, no previous studies have carried out direct, systematic characterizations of state-unspecific brain networks, or their functional implications. Here, we quantitatively identified several modes of state-unspecific individual variation in whole-brain functional connectivity patterns, called “Common Neural Modes (CNMs)”, from a large fMRI dataset including eight task/rest states, obtained from the Human Connectome Project. Furthermore, we tested how CNMs account for variability in individual behavioral measures. The results revealed that three CNMs were robustly extracted under various different preprocessing conditions. Each of these CNMs was significantly correlated with different aspects of behavioral measures of both fluid and crystalized intelligence. The three CNMs were also able to predict several life outcomes, such as income and life satisfaction, achieving the highest performance when combined with behavioral intelligence measures as inputs. Our findings highlight the importance of state-unspecific brain networks to characterize fundamental individual variation.

Preliminary evidence of altered neural response during intertemporal choice of losses in adult attention-deficit hyperactivity disorder

Impulsive behaviours are common symptoms of attention-deficit hyperactivity disorder (ADHD). Although previous studies have suggested functional models of impulsive behaviour, a full explanation of impulsivity in ADHD remains elusive. To investigate the detailed mechanisms behind impulsive behaviour in ADHD, we applied an economic intertemporal choice task involving gains and losses to adults with ADHD and healthy controls and measured brain activity by functional magnetic resonance imaging. In the intertemporal choice of future gains, we observed no behavioural or neural difference between the two groups. In the intertemporal choice of future losses, adults with ADHD exhibited higher discount rates than the control participants. Furthermore, a comparison of brain activity representing the sensitivity of future loss in the two groups revealed significantly lower activity in the striatum and higher activity in the amygdala in adults with ADHD than in controls. Our preliminary findings suggest that an altered size sensitivity to future loss is involved in apparent impulsive choice behaviour in adults with ADHD and shed light on the multifaceted impulsivity underlying ADHD.