Daphna Joel

Tonic dopamine: opportunity costs and the control of response vigor

RationaleDopamine neurotransmission has long been known to exert a powerful influence over the vigor, strength, or rate of responding. However, there exists no clear understanding of the computational foundation for this effect; predominant accounts of dopamine’s computational function focus on a role for phasic dopamine in controlling the discrete selection between different actions and have nothing to say about response vigor or indeed the free-operant tasks in which it is typically measured.ObjectivesWe seek to accommodate free-operant behavioral tasks within the realm of models of optimal control and thereby capture how dopaminergic and motivational manipulations affect response vigor.MethodsWe construct an average reward reinforcement learning model in which subjects choose both which action to perform and also the latency with which to perform it. Optimal control balances the costs of acting quickly against the benefits of getting reward earlier and thereby chooses a best response latency.ResultsIn this framework, the long-run average rate of reward plays a key role as an opportunity cost and mediates motivational influences on rates and vigor of responding. We review evidence suggesting that the average reward rate is reported by tonic levels of dopamine putatively in the nucleus accumbens.ConclusionsOur extension of reinforcement learning models to free-operant tasks unites psychologically and computationally inspired ideas about the role of tonic dopamine in striatum, explaining from a normative point of view why higher levels of dopamine might be associated with more vigorous responding.

The connections of the dopaminergic system with the striatum in rats and primates: an analysis with respect to the functional and compartmental organization of the striatum.

This Commentary compares the connections of the dopaminergic system with the striatum in rats and primates with respect to two levels of striatal organization: a tripartite functional (motor, associative and limbic) subdivision and a compartmental (patch/striosome-matrix) subdivision. The topography of other basal ganglia projections to the dopaminergic system with respect to their tripartite functional subdivision is also reviewed. This examination indicates that, in rats and primates, the following observations can be made. (1) The limbic striatum reciprocates its dopaminergic input and in addition innervates most of the dopaminergic neurons projecting to the associative and motor striatum, whereas the motor and associative striatum reciprocate only part of their dopaminergic input. Therefore, the connections of the three striatal subregions with the dopaminergic system are asymmetrical, but the direction of asymmetry differs between the limbic versus the motor and associative striatum. (2) The limbic striatum provides the main striatal input to dopamine cell bodies and proximal dendrites, with some contribution from a subset of neurons in the associative and motor striatum (patch neurons in rats; an unspecified group of neurons in primates), while striatal input to the ventrally extending dopamine dendrites arises mainly from a subset of neurons in the associative and motor striatum (matrix neurons in rats; an unspecified group of neurons in primates). (3) Projections from functionally corresponding subdivisions of the striatum, pallidum and subthalamic nucleus to the dopaminergic system overlap, but the specific targets (dopamine cells, dopamine dendrites, GABA cells) of these projections differ. Major differences include the following. (1) In rats, neurons projecting to the motor and associative striatum reside in distinct regions, while in primates they are arranged in interdigitating clusters. (2) In rats, the terminal fields of projections arising from the motor and associative striatum are largely segregated, while in primates they are not. (3) In rats, patch- and matrix-projecting dopamine cells are organized in spatially, morphologically, histochemically and hodologically distinct ventral and dorsal tiers, while in primates there is no (bi)division of the dopaminergic system that results in two areas which have all the characteristics of the two tiers in rats. Based on the anatomical data and known dopamine cell physiology, we forward an hypothesis regarding the influence of the basal ganglia on dopamine cell activity which captures at least part of the complex interplay taking place within the substantia nigra between projections arising from the different basal ganglia nuclei. Finally, we incorporate the striatal connections with the dopaminergic system into an open-interconnected scheme of basal ganglia-thalamocortical circuitry.

Reinforcement Learning Signals in the Human Striatum Distinguish Learners from Nonlearners during Reward-Based Decision Making

The computational framework of reinforcement learning has been used to forward our understanding of the neural mechanisms underlying reward learning and decision-making behavior. It is known that humans vary widely in their performance in decision-making tasks. Here, we used a simple four-armed bandit task in which subjects are almost evenly split into two groups on the basis of their performance: those who do learn to favor choice of the optimal action and those who do not. Using models of reinforcement learning we sought to determine the neural basis of these intrinsic differences in performance by scanning both groups with functional magnetic resonance imaging. We scanned 29 subjects while they performed the reward-based decision-making task. Our results suggest that these two groups differ markedly in the degree to which reinforcement learning signals in the striatum are engaged during task performance. While the learners showed robust prediction error signals in both the ventral and dorsal striatum during learning, the nonlearner group showed a marked absence of such signals. Moreover, the magnitude of prediction error signals in a region of dorsal striatum correlated significantly with a measure of behavioral performance across all subjects. These findings support a crucial role of prediction error signals, likely originating from dopaminergic midbrain neurons, in enabling learning of action selection preferences on the basis of obtained rewards. Thus, spontaneously observed individual differences in decision making performance demonstrate the suggested dependence of this type of learning on the functional integrity of the dopaminergic striatal system in humans.

The role of mesolimbic dopaminergic and retrohippocampal afferents to the nucleus accumbens in latent inhibition: implications for schizophrenia

Latent inhibition (LI) consists in a retardation of conditioning seen when the to-be-conditioned stimulus is first presented a number of times without other consequence. Disruption of LI has been proposed as a possible model of the cognitive abnormality that underlies the positive psychotic symptoms of acute schizophrenia. We review here evidence in support of the model, including experiments tending to show that: (1) disruption of LI is characteristic of acute, positively-symptomatic schizophrenia; (2) LI depends upon dopaminergic activity; (3) LI depends specifically upon dopamine release in n. accumbens; (4) LI depends upon the integrity of the hippocampal formation and the retrohippocampal region reciprocally connected to the hippocampal formation; (5) the roles of n. accumbens and the hippocampal system in LI are interconnected.

Open interconnected model of basal ganglia-thalamocortical circuitry and its relevance to the clinical syndrome of Huntington's disease

The early stages of Huntingtons disease (HD) present with motor, cognitive, and emotional symptoms. Correspondingly, current models implicate dysfunction of the motor, associative, and limbic basal ganglia‐thalamocortical circuits. Available data, however, indicate that in the early stages of the disease, striatal damage is mainly restricted to the associative striatum. Based on an open interconnected model of basal ganglia‐thalamocortical organization, we provide a detailed account of the mechanisms by which associative striatal pathology may lead to the complex pattern of motor, cognitive, and emotional symptoms of early HD. According to this account, the degeneration of a direct and several indirect pathways arising from the associative striatum leads to impaired functioning of: (1) the motor circuit, resulting in chorea and bradykinesia, (2) the associative circuit, resulting in abnormal eye movements, “frontal‐like” cognitive deficits and “cognitive disinhibition,” and (3) the limbic circuit, resulting in affective and psychiatric symptoms. When relevant, this analysis is aided by comparing the symptomatology of HD patients to that of patients with mild to moderate Parkinsons disease, since in the latter there is similar dysfunction of direct pathways but opposite dysfunction of indirect pathways. Finally, we suggest a potential novel treatment of HD and provide supportive evidence from a rat model of the disease.

The orbital cortex in rats topographically projects to central parts of the caudate–putamen complex

Disturbances of the orbitofrontal-striatal pathways in humans have been associated with several psychopathologies including obsessive-compulsive disorder and drug addiction. In nonhuman primates, different subareas of the orbitofrontal cortex project topographically to central and ventromedial parts of the striatum. Relatively little is known about the anatomical organization of the rat orbital cortex while there is a growing interest in this cortical area from a functional and behavioral point of view. The aim of the present neuroanatomical tracing study was to determine in rats the striatal target area of the projections of the orbital cortex as well as the topographical organization within these projections. To this end, anterograde tracers were injected in the different cytoarchitectonically distinct subareas of the orbital cortex. The results show that the individual orbital areas, i.e. medial orbital area, ventral orbital area, ventrolateral orbital area and lateral orbital area, project to central parts of the caudate-putamen, exhibiting a mediolateral and, to a lesser degree, rostrocaudal topographical arrangement. Orbital projections avoid the most dorsal, as well as rostral and caudal parts of the caudate-putamen. Terminal fields from cytoarchitectonically different areas show a considerable overlap. Superficial cortical layers project preferentially to the striatal matrix, deep layers to the patch compartment. The projections from the ventrolateral orbital area are strongest and occupy the most extensive striatal area. In addition to projections to the caudate-putamen, the ventrolateral, lateral and dorsolateral orbital areas have a scarce projection to the most lateral part of the nucleus accumbens shell in the ventral striatum. In contrast to nonhuman primates, the remainder of the rat nucleus accumbens is virtually free of orbital projections.

Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviors

Reinforcement learning is a fundamental process by which organisms learn to achieve goals from their interactions with the environment. Using evolutionary computation techniques we evolve (near-)optimal neuronal learning rules in a simple neural network model of reinforcement learning in bumblebees foraging for nectar. The resulting neural networks exhibit efficient reinforcement learning, allowing the bees to respond rapidly to changes in reward contingencies. The evolved synaptic plasticity dynamics give rise to varying exploration/exploitation levels and to the well-documented choice strategies of risk aversion and probability matching. Additionally, risk aversion is shown to emerge even when bees are evolved in a completely risk-less environment. In contrast to existing theories in economics and game theory, risk-averse behavior is shown to be a direct consequence of (near-)optimal reinforcement learning, without requiring additional assumptions such as the existence of a nonlinear subjective utility function for rewards. Our results are corroborated by a rigorous mathematical analysis, and their robustness in real-world situations is supported by experiments in a mobile robot. Thus we provide a biologically founded, parsimonious, and novel explanation for risk aversion and probability matching.

Sex beyond the genitalia: The human brain mosaic

Significance Sex/gender differences in the brain are of high social interest because their presence is typically assumed to prove that humans belong to two distinct categories not only in terms of their genitalia, and thus justify differential treatment of males and females. Here we show that, although there are sex/gender differences in brain and behavior, humans and human brains are comprised of unique “mosaics” of features, some more common in females compared with males, some more common in males compared with females, and some common in both females and males. Our results demonstrate that regardless of the cause of observed sex/gender differences in brain and behavior (nature or nurture), human brains cannot be categorized into two distinct classes: male brain/female brain. Whereas a categorical difference in the genitals has always been acknowledged, the question of how far these categories extend into human biology is still not resolved. Documented sex/gender differences in the brain are often taken as support of a sexually dimorphic view of human brains (“female brain” or “male brain”). However, such a distinction would be possible only if sex/gender differences in brain features were highly dimorphic (i.e., little overlap between the forms of these features in males and females) and internally consistent (i.e., a brain has only “male” or only “female” features). Here, analysis of MRIs of more than 1,400 human brains from four datasets reveals extensive overlap between the distributions of females and males for all gray matter, white matter, and connections assessed. Moreover, analyses of internal consistency reveal that brains with features that are consistently at one end of the “maleness-femaleness” continuum are rare. Rather, most brains are comprised of unique “mosaics” of features, some more common in females compared with males, some more common in males compared with females, and some common in both females and males. Our findings are robust across sample, age, type of MRI, and method of analysis. These findings are corroborated by a similar analysis of personality traits, attitudes, interests, and behaviors of more than 5,500 individuals, which reveals that internal consistency is extremely rare. Our study demonstrates that, although there are sex/gender differences in the brain, human brains do not belong to one of two distinct categories: male brain/female brain.

Behavioral, Pharmacological, and Immunological Abnormalities after Streptococcal Exposure: A Novel Rat Model of Sydenham Chorea and Related Neuropsychiatric Disorders

Group A streptococcal (GAS) infections and autoimmunity are associated with the onset of a spectrum of neuropsychiatric disorders in children, with the prototypical disorder being Sydenham chorea (SC). Our aim was to develop an animal model that resembled the behavioral, pharmacological, and immunological abnormalities of SC and other streptococcal-related neuropsychiatric disorders. Male Lewis rats exposed to GAS antigen exhibited motor symptoms (impaired food manipulation and beam walking) and compulsive behavior (increased induced-grooming). These symptoms were alleviated by the D2 blocker haloperidol and the selective serotonin reuptake inhibitor paroxetine, respectively, drugs that are used to treat motor symptoms and compulsions in streptococcal-related neuropsychiatric disorders. Streptococcal exposure resulted in antibody deposition in the striatum, thalamus, and frontal cortex, and concomitant alterations in dopamine and glutamate levels in cortex and basal ganglia, consistent with the known pathophysiology of SC and related neuropsychiatric disorders. Autoantibodies (IgG) of GAS rats reacted with tubulin and caused elevated calcium/calmodulin-dependent protein kinase II signaling in SK-N-SH neuronal cells, as previously found with sera from SC and related neuropsychiatric disorders. Our new animal model translates directly to human disease and led us to discover autoantibodies targeted against dopamine D1 and D2 receptors in the rat model as well as in SC and other streptococcal-related neuropsychiatric disorders.

Animal models of obsessive-compulsive disorder: exploring pharmacology and neural substrates.

During the last 30 years there have been many attempts to develop animal models of obsessive-compulsive disorder (OCD). Most models have not been studied further following the original publication, and in the past few years, most papers present studies employing a few established animal models, exploring the neural basis of compulsive behavior and developing new treatment strategies. Here we summarize findings from the five most studied animal models of OCD: 8-OHDPAT (8-hydroxy-2-(di-n-propylamino)-tetralin hydrobromide) induced decreased alternation, quinpirole-induced compulsive checking, marble burying, signal attenuation and spontaneous stereotypy in deer mice. We evaluate each models face validity, derived from similarity between the behavior in the model and the specific symptoms of the human condition, predictive validity, derived from similarity in response to treatment (pharmacological or other), and construct validity, derived from similarity in the mechanism (physiological or psychological) that induces behavioral symptoms and in the neural systems involved. We present ideas regarding future clinical research based on each models findings, and on this basis, also emphasize possible new approaches for the treatment of OCD.