Junichiro Yoshimoto

A Kinetic Model of Dopamine- and Calcium-Dependent Striatal Synaptic Plasticity

Corticostriatal synapse plasticity of medium spiny neurons is regulated by glutamate input from the cortex and dopamine input from the substantia nigra. While cortical stimulation alone results in long-term depression (LTD), the combination with dopamine switches LTD to long-term potentiation (LTP), which is known as dopamine-dependent plasticity. LTP is also induced by cortical stimulation in magnesium-free solution, which leads to massive calcium influx through NMDA-type receptors and is regarded as calcium-dependent plasticity. Signaling cascades in the corticostriatal spines are currently under investigation. However, because of the existence of multiple excitatory and inhibitory pathways with loops, the mechanisms regulating the two types of plasticity remain poorly understood. A signaling pathway model of spines that express D1-type dopamine receptors was constructed to analyze the dynamic mechanisms of dopamine- and calcium-dependent plasticity. The model incorporated all major signaling molecules, including dopamine- and cyclic AMP-regulated phosphoprotein with a molecular weight of 32 kDa (DARPP32), as well as AMPA receptor trafficking in the post-synaptic membrane. Simulations with dopamine and calcium inputs reproduced dopamine- and calcium-dependent plasticity. Further in silico experiments revealed that the positive feedback loop consisted of protein kinase A (PKA), protein phosphatase 2A (PP2A), and the phosphorylation site at threonine 75 of DARPP-32 (Thr75) served as the major switch for inducing LTD and LTP. Calcium input modulated this loop through the PP2B (phosphatase 2B)-CK1 (casein kinase 1)-Cdk5 (cyclin-dependent kinase 5)-Thr75 pathway and PP2A, whereas calcium and dopamine input activated the loop via PKA activation by cyclic AMP (cAMP). The positive feedback loop displayed robust bi-stable responses following changes in the reaction parameters. Increased basal dopamine levels disrupted this dopamine-dependent plasticity. The present model elucidated the mechanisms involved in bidirectional regulation of corticostriatal synapses and will allow for further exploration into causes and therapies for dysfunctions such as drug addiction.

Acrobot control by learning the switching of multiple controllers

Reinforcement learning (RL) has been applied to constructing controllers for nonlinear systems in recent years. Since RL methods do not require an exact dynamics model of the controlled object, they have a higher flexibility and potential for adaptation to uncertain or nonstationary environments than methods based on traditional control theory. If the target system has a continuous state space whose dynamic characteristics are nonlinear, however, RL methods often suffer from unstable learning processes. For this reason, it is difficult to apply RL methods to control tasks in the real world. In order to overcome the disadvantage of RL methods, we propose an RL scheme combining multiple controllers, each of which is constructed based on traditional control theory. We then apply it to a swinging-up and stabilizing task of an acrobot with a limited torque, which is a typical but difficult task in the field of nonlinear control theory. Our simulation result showed that our method was able to realize stable learning and to achieve fairly good control.

Toward Probabilistic Diagnosis and Understanding of Depression Based on Functional MRI Data Analysis with Logistic Group LASSO

Diagnosis of psychiatric disorders based on brain imaging data is highly desirable in clinical applications. However, a common problem in applying machine learning algorithms is that the number of imaging data dimensions often greatly exceeds the number of available training samples. Furthermore, interpretability of the learned classifier with respect to brain function and anatomy is an important, but non-trivial issue. We propose the use of logistic regression with a least absolute shrinkage and selection operator (LASSO) to capture the most critical input features. In particular, we consider application of group LASSO to select brain areas relevant to diagnosis. An additional advantage of LASSO is its probabilistic output, which allows evaluation of diagnosis certainty. To verify our approach, we obtained semantic and phonological verbal fluency fMRI data from 31 depression patients and 31 control subjects, and compared the performances of group LASSO (gLASSO), and sparse group LASSO (sgLASSO) to those of standard LASSO (sLASSO), Support Vector Machine (SVM), and Random Forest. Over 90% classification accuracy was achieved with gLASSO, sgLASSO, as well as SVM; however, in contrast to SVM, LASSO approaches allow for identification of the most discriminative weights and estimation of prediction reliability. Semantic task data revealed contributions to the classification from left precuneus, left precentral gyrus, left inferior frontal cortex (pars triangularis), and left cerebellum (c rus1). Weights for the phonological task indicated contributions from left inferior frontal operculum, left post central gyrus, left insula, left middle frontal cortex, bilateral middle temporal cortices, bilateral precuneus, left inferior frontal cortex (pars triangularis), and left precentral gyrus. The distribution of normalized odds ratios further showed, that predictions with absolute odds ratios higher than 0.2 could be regarded as certain.

Evidence for Model-Based Action Planning in a Sequential Finger Movement Task

ABSTRACT In this article, the authors examine whether and how humans use model-free, reflexive strategies and model-based, deliberative strategies in motor sequence learning. They asked subjects to perform the grid-sailing task, which required moving a cursor to different goal positions in a 5 × 5 grid using different key-mapping (KM) rules between 3 finger keys and 3 cursor movement directions. The task was performed under 3 conditions: Condition 1, new KM; Condition 2, new goal position with learned KM; and Condition 3, learned goal position with learned KM; with or without prestart delay time. The performance improvement with prestart delay was significantly larger under Condition 2. This result provides evidence that humans implement a model-based strategy for sequential action selection and learning by using previously learned internal model of state transition by actions.

Phosphoproteomics of the Dopamine Pathway Enables Discovery of Rap1 Activation as a Reward Signal In Vivo.

Dopamine (DA) type 1 receptor (D1R) signaling in the striatum presumably regulates neuronal excitability and reward-related behaviors through PKA. However, whether and how D1Rs and PKA regulate neuronal excitability and behavior remain largely unknown. Here, we developed a phosphoproteomic analysis method to identify known and novel PKA substrates downstream of the D1R and obtained more than 100 candidate substrates, including Rap1 GEF (Rasgrp2). We found that PKA phosphorylation of Rasgrp2 activated its guanine nucleotide-exchange activity on Rap1. Cocaine exposure activated Rap1 in the nucleus accumbens in mice. The expression of constitutively active PKA or Rap1 in accumbal D1R-expressing medium spiny neurons (D1R-MSNs) enhanced neuronal firing rates and behavioral responses to cocaine exposure through MAPK. Knockout of Rap1 in the accumbal D1R-MSNs was sufficient to decrease these phenotypes. These findings demonstrate a novel DA-PKA-Rap1-MAPK intracellular signaling mechanism in D1R-MSNs that increases neuronal excitability to enhance reward-related behaviors.

Application of reinforcement learning to balancing of Acrobot

The Acrobot is a two-link robot, actuated only at the joint between the two links. It is one of difficult tasks in reinforcement learning (RL) to control the Acrobot because it has nonlinear dynamics and continuous state and action spaces. In this article, we discuss applying the RL to the task of balancing control of the Acrobot. Our RL method has an architecture similar to the actor-critic. The actor and the critic are approximated by normalized Gaussian networks, which are trained by an online EM algorithm. We also introduce eligibility traces for our actor-critic architecture. Our computer simulation shows that our method is able to achieve fairly good control with a small number of trials.

Phosphorylation Signals in Striatal Medium Spiny Neurons

Dopamine signaling in the brain is a complex phenomenon that strongly contributes to emotional behaviors. Medium spiny neurons (MSNs) play a major role in dopamine signaling through dopamine D1 receptors (D1Rs) or dopamine D2 receptors (D2Rs) in the striatum. cAMP/protein kinase A (PKA) regulates phosphorylation signals downstream of D1Rs, which affects the excitability of MSNs, leading to reward-associated emotional expression and memory formation. A combination of phosphoproteomic approaches and the curated KANPHOS database can be used to elucidate the physiological and pathophysiological functions of dopamine signaling and other monoamines. Emerging evidence from these techniques suggests that the Rap1 pathway plays a crucial role in the excitability of MSNs, leading to the expression of emotional behaviors.

Model-based action planning involves cortico-cerebellar and basal ganglia networks

Humans can select actions by learning, planning, or retrieving motor memories. Reinforcement Learning (RL) associates these processes with three major classes of strategies for action selection: exploratory RL learns state-action values by exploration, model-based RL uses internal models to simulate future states reached by hypothetical actions, and motor-memory RL selects past successful state-action mapping. In order to investigate the neural substrates that implement these strategies, we conducted a functional magnetic resonance imaging (fMRI) experiment while humans performed a sequential action selection task under conditions that promoted the use of a specific RL strategy. The ventromedial prefrontal cortex and ventral striatum increased activity in the exploratory condition; the dorsolateral prefrontal cortex, dorsomedial striatum, and lateral cerebellum in the model-based condition; and the supplementary motor area, putamen, and anterior cerebellum in the motor-memory condition. These findings suggest that a distinct prefrontal-basal ganglia and cerebellar network implements the model-based RL action selection strategy.

System identification based on online variational bayes method and its application to reinforcement learning

In this article, we present an on-line variational Bayes (VB) method for the identification of linear state space models. The learning algorithm is implemented as alternate maximization of an on-line free energy, which can be used for determining the dimension of the internal state. We also propose a reinforcement learning (RL) method using this system identification method. Our RL method is applied to a simple automatic control problem. The result shows that our method is able to determine correctly the dimension of the internal state and to acquire a good control, even in a partially observable environment.

A model-based prediction of the calcium responses in the striatal synaptic spines depending on the timing of cortical and dopaminergic inputs and post-synaptic spikes

The dopamine-dependent plasticity of the cortico-striatal synapses is considered as the cellular mechanism crucial for reinforcement learning. The dopaminergic inputs and the calcium responses affect the synaptic plasticity by way of the signaling cascades within the synaptic spines. The calcium concentration within synaptic spines, however, is dependent on multiple factors including the calcium influx through ionotropic glutamate receptors, the intracellular calcium release by activation of metabotropic glutamate receptors, and the opening of calcium channels by EPSPs and back-propagating action potentials. Furthermore, dopamine is known to modulate the efficacies of NMDA receptors, some of the calcium channels, and sodium and potassium channels that affect the back propagation of action potentials. Here we construct an electric compartment model of the striatal medium spiny neuron with a realistic morphology and predict the calcium responses in the synaptic spines with variable timings of the glutamatergic and dopaminergic inputs and the postsynaptic action potentials. The model was validated by reproducing the responses to current inputs and could predict the electric and calcium responses to glutamatergic inputs and back-propagating action potential in the proximal and distal synaptic spines during up- and down-states. We investigated the calcium responses by systematically varying the timings of the glutamatergic and dopaminergic inputs relative to the action potential and found that the calcium response and the subsequent synaptic potentiation is maximal when the dopamine input precedes glutamate input and action potential. The prediction is not consistent with the hypothesis that the dopamine input provides the reward prediction error for reinforcement learning. The finding suggests that there is an unknown learning mechanisms at the network level or an unknown cellular mechanism for calcium dynamics and signaling cascades.