Poornima Kumar

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments—prediction error—is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Dissociable cortico-striatal connectivity abnormalities in major depression in response to monetary gains and penalties

BACKGROUND Individuals with major depressive disorder (MDD) are characterized by maladaptive responses to both positive and negative outcomes, which have been linked to localized abnormal activations in cortical and striatal brain regions. However, the exact neural circuitry implicated in such abnormalities remains largely unexplored. METHOD In this study 26 unmedicated adults with MDD and 29 matched healthy controls (HCs) completed a monetary incentive delay task during functional magnetic resonance imaging (fMRI). Psychophysiological interaction (PPI) analyses probed group differences in connectivity separately in response to positive and negative outcomes (i.e. monetary gains and penalties). RESULTS Relative to HCs, MDD subjects displayed decreased connectivity between the caudate and dorsal anterior cingulate cortex (dACC) in response to monetary gains, yet increased connectivity between the caudate and a different, more rostral, dACC subregion in response to monetary penalties. Moreover, exploratory analyses of 14 MDD patients who completed a 12-week, double-blind, placebo-controlled clinical trial after the baseline fMRI scans indicated that a more normative pattern of cortico-striatal connectivity pre-treatment was associated with greater improvement in symptoms 12 weeks later. CONCLUSIONS These results identify the caudate as a region with dissociable incentive-dependent dACC connectivity abnormalities in MDD, and provide initial evidence that cortico-striatal circuitry may play a role in MDD treatment response. Given the role of cortico-striatal circuitry in encoding action-outcome contingencies, such dysregulated connectivity may relate to the prominent disruptions in goal-directed behavior that characterize MDD.

Microstructural brain abnormalities in medication-free patients with major depressive disorder: a systematic review and meta-analysis of diffusion tensor imaging.

Background Multiple meta-analyses of diffusion tensor imaging (DTI) studies have reported impaired white matter integrity in patients with major depressive disorder (MDD). However, owing to inclusion of medicated patients in these studies, it is difficult to conclude whether these reported alterations are associated with MDD or confounded by medication effects. A meta-analysis of DTI studies on medication-free (medication-naive and medication washout) patients with MDD would therefore be necessary to disentangle MDD-specific effects. Methods We analyzed white matter alterations between medication-free patients with MDD and healthy controls using anisotropic effect size–signed differential mapping (AES-SDM). We used DTI query software for fibre tracking. Results Both pooled and subgroup meta-analyses in medication washout patients showed robust fractional anisotropy (FA) reductions in white matter of the right cerebellum hemispheric lobule, body of the corpus callosum (CC) and bilateral superior longitudinal fasciculus III (SLF III), whereas FA reductions in the genu of the CC and right anterior thalamic projections were seen in only medication-naive patients. Fibre tracking showed that the main tracts with observed FA reductions included the right cerebellar tracts, body of the CC, bilateral SLF III and arcuate fascicle. Limitations The analytic techniques, patient characteristics and clinical variables of the included studies were heterogeneous; we could not exclude the effects of nondrug therapies owing to a lack of data. Conclusion By excluding the confounding influences of current medication status, findings from the present study may provide a better understanding of the underlying neuropathology of MDD.

Increased neural response to social rejection in major depression

Being a part of community is critical for survival and individuals with major depressive disorder (MDD) have a greater sensitivity to interpersonal stress that makes them vulnerable to future episodes. Social rejection is a critical risk factor for depression and it is said to increase interpersonal stress and thereby impairing social functioning. It is therefore critical to understand the neural correlates of social rejection in MDD.

Frontostriatal and Dopamine Markers of Individual Differences in Reinforcement Learning: A Multi-modal Investigation

Prior studies have shown that dopamine (DA) functioning in frontostriatal circuits supports reinforcement learning (RL), as phasic DA activity in ventral striatum signals unexpected reward and may drive coordinated activity of striatal and orbitofrontal regions that support updating of action plans. However, the nature of DA functioning in RL is complex, in particular regarding the role of DA clearance in RL behavior. Here, in a multi-modal neuroimaging study with healthy adults, we took an individual differences approach to the examination of RL behavior and DA clearance mechanisms in frontostriatal learning networks. We predicted that better RL would be associated with decreased striatal DA transporter (DAT) availability and increased intrinsic functional connectivity among DA-rich frontostriatal regions. In support of these predictions, individual differences in RL behavior were related to DAT binding potential in ventral striatum and resting-state functional connectivity between ventral striatum and orbitofrontal cortex. Critically, DAT binding potential had an indirect effect on reinforcement learning behavior through frontostriatal connectivity, suggesting potential causal relationships across levels of neurocognitive functioning. These data suggest that individual differences in DA clearance and frontostriatal coordination may serve as markers for RL, and suggest directions for research on psychopathologies characterized by altered RL.

Perceived life stress exposure modulates reward-related medial prefrontal cortex responses to acute stress in depression.

INTRODUCTION Major depressive disorder (MDD) is often precipitated by life stress and growing evidence suggests that stress-induced alterations in reward processing may contribute to such risk. However, no human imaging studies have examined how recent life stress exposure modulates the neural systems that underlie reward processing in depressed and healthy individuals. METHODS In this proof-of-concept study, 12 MDD and 10 psychiatrically healthy individuals were interviewed using the Life Events and Difficulties Schedule (LEDS) to assess their perceived levels of recent acute and chronic life stress exposure. Additionally, each participant performed a monetary incentive delay task under baseline (no-stress) and stress (social-evaluative) conditions during functional MRI. RESULTS Across groups, medial prefrontal cortex (mPFC) activation to reward feedback was greater during acute stress versus no-stress conditions in individuals with greater perceived stressor severity. Under acute stress, depressed individuals showed a positive correlation between perceived stressor severity levels and reward-related mPFC activation (r=0.79, p=0.004), whereas no effect was found in healthy controls. Moreover, for depressed (but not healthy) individuals, the correlations between the stress (r=0.79) and no-stress (r=-0.48) conditions were significantly different. Finally, relative to controls, depressed participants showed significantly reduced mPFC gray matter, but functional findings remained robust while accounting for structural differences. LIMITATION Small sample size, which warrants replication. CONCLUSION Depressed individuals experiencing greater recent life stress recruited the mPFC more under stress when processing rewards. Our results represent an initial step toward elucidating mechanisms underlying stress sensitization and recurrence in depression.

SSRI enhances sensitivity to background outcomes and modulates response rates: A randomized double blind study of instrumental action and depression

Serotonin reuptake inhibitors (SSRIs) have immediate effects on synaptic levels of serotonin but their therapeutic effects are often delayed. This delay has been suggested to reflect time required for new learning and therefore that SSRIs might be having effects on the learning process. We examined the effects of elevating serotonin levels, through short-term SSRI administration (escitalopram), on learning about perceptions of instrumental control. A randomised double blind procedure was used to allocate healthy people, categorised as mildly depressed (high BDI⩾10: n=76) or not depressed (low BDI⩽5: n=78) to either a drug (escitalopram, 10mg/7days) or placebo control group. Following treatment, participants were trained with a simple task that involved learning the effectiveness of an instrumental action (key press) and the background context at eliciting an outcome (auditory cue) where there was no programmed contingency. The effects of the drug were (i) to moderate response rates and (ii) to enhance sensitivity to the background or context rate of occurrence of the outcome. These findings suggest that serotonin modulates learning about the long-term rate of outcomes, which supports perception of instrumental control, and that this may provide a clue to the mechanism for supporting the development of the therapeutic effects of the drug.

Stress and reward processing in bipolar disorder: a functional magnetic resonance imaging study

A link between negative life stress and the onset of mood episodes in bipolar disorder (BD) has been established, but processes underlying such a link remain unclear. Growing evidence suggests that stress can negatively affect reward processing and related neurobiological substrates, indicating that a dysregulated reward system may provide a partial explanation. The aim of this study was to test the impact of stress on reward‐related neural functioning in BD.

Impaired reward prediction error encoding and striatal-midbrain connectivity in depression

Anhedonia (hyposensitivity to rewards) and negative bias (hypersensitivity to punishments) are core features of major depressive disorder (MDD), which could stem from abnormal reinforcement learning. Emerging evidence highlights blunted reward learning and reward prediction error (RPE) signaling in the striatum in MDD, although inconsistencies exist. Preclinical studies have clarified that ventral tegmental area (VTA) neurons encode RPE and habenular neurons encode punishment prediction error (PPE), which are then transmitted to the striatum and cortex to guide goal-directed behavior. However, few studies have probed striatal activation, and functional connectivity between VTA-striatum and VTA-habenula during reward and punishment learning respectively, in unmedicated MDD. To fill this gap, we acquired fMRI data from 25 unmedicated MDD and 26 healthy individuals during a monetary instrumental learning task and utilized a computational modeling approach to characterize underlying neural correlates of RPE and PPE. Relative to controls, MDD individuals showed impaired reward learning, blunted RPE signal in the striatum and overall reduced VTA-striatal connectivity to feedback. Critically, striatal RPE signal was increasingly blunted with more major depressive episodes (MDEs). No group differences emerged in PPE signals in the habenula and VTA or in connectivity between these regions. However, PPE signals in the habenula correlated positively with number of MDEs. These results highlight impaired reward learning, disrupted RPE signaling in the striatum (particularly among individuals with more lifetime MDEs) as well as reduced VTA-striatal connectivity in MDD. Collectively, these findings highlight reward-related learning deficits in MDD and their underlying pathophysiology.

Neuroanatomical Prediction of Anhedonia in Adolescents

Anhedonia is a transdiagnostic risk factor implicated in mental illness onset, treatment non-response, and suicidal behaviors. Prior cross-sectional research in adults has shown that anhedonia is associated with reduced dorsal striatal volume, but it is unknown whether this relationship extends to adolescents and whether reduced striatal volume prospectively predicts anhedonia. To address these gaps, the current study investigated whether striatal volume predicted anhedonia severity in adolescents. At baseline, healthy female adolescents aged 12–14 years (n=50) completed a clinical assessment, and structural MRI data were acquired on a 3 Tesla MR scanner. While in the scanner, participants also completed a peer feedback task where subjective ratings following peer ‘acceptance’ or ‘rejection’ were obtained. At the three-month follow-up, participants provided self-report assessments of anhedonia, depression, and anxiety symptoms. Three main findings emerged. First, in cross-sectional analyses, right nucleus accumbens volume was inversely related to anhedonia severity. Second, reduced bilateral putamen volume prospectively predicted anhedonia severity while controlling for baseline anhedonia, depression, and anxiety symptoms. Third, a blunted subjective response to peer acceptance (ie, neutral response to positive feedback), but not a more negative subjective response to peer rejection, contributed to anhedonia severity, but only among youth with smaller putamen volume. Collectively, these results suggest that smaller volume in striatal regions critically implicated in reward processing is associated with current and future anhedonic symptoms among healthy female youth. These anatomical features may confer vulnerability to anhedonia and thus, may inform early identification of individuals at high risk for mental illness.