Francesco Mannella

The nucleus accumbens as a nexus between values and goals in goal-directed behavior: a review and a new hypothesis

Goal-directed behavior is a fundamental means by which animals can flexibly solve the challenges posed by variable external and internal conditions. Recently, the processes and brain mechanisms underlying such behavior have been extensively studied from behavioral, neuroscientific and computational perspectives. This research has highlighted the processes underlying goal-directed behavior and associated brain systems including prefrontal cortex, basal ganglia and, in particular therein, the nucleus accumbens (NAcc). This paper focusses on one particular process at the core of goal-directed behavior: how motivational value is assigned to goals on the basis of internal states and environmental stimuli, and how this supports goal selection processes. Various biological and computational accounts have been given of this problem and of related multiple neural and behavior phenomena, but we still lack an integrated hypothesis on the generation and use of value for goal selection. This paper proposes an hypothesis that aims to solve this problem and is based on this key elements: (a) amygdala and hippocampus establish the motivational value of stimuli and goals; (b) prefrontal cortex encodes various types of action outcomes; (c) NAcc integrates different sources of value, representing them in terms of a common currency with the aid of dopamine, and thereby plays a major role in selecting action outcomes within prefrontal cortex. The “goals” pursued by the organism are the outcomes selected by these processes. The hypothesis is developed in the context of a critical review of relevant biological and computational literature which offer it support. The paper shows how the hypothesis has the potential to integrate existing interpretations of motivational value and goal selection.

Keep focussing: striatal dopamine multiple functions resolved in a single mechanism tested in a simulated humanoid robot.

The effects of striatal dopamine (DA) on behavior have been widely investigated over the past decades, with “phasic” burst firings considered as the key expression of a reward prediction error responsible for reinforcement learning. Less well studied is “tonic” DA, where putative functions include the idea that it is a regulator of vigor, incentive salience, disposition to exert an effort and a modulator of approach strategies. We present a model combining tonic and phasic DA to show how different outflows triggered by either intrinsically or extrinsically motivating stimuli dynamically affect the basal ganglia by impacting on a selection process this system performs on its cortical input. The model, which has been tested on the simulated humanoid robot iCub interacting with a mechatronic board, shows the putative functions ascribed to DA emerging from the combination of a standard computational mechanism coupled to a differential sensitivity to the presence of DA across the striatum.

The roles of the amygdala in the affective regulation of body, brain, and behaviour

Despite the great amount of knowledge produced by the neuroscientific literature on affective phenomena, current models tackling non-cognitive aspects of behaviour are often bio-inspired but rarely bio-constrained. This paper presents a theoretical account of affective systems centred on the amygdala (Amg). This account aims to furnish a general framework and specific pathways to implement models that are more closely related to biological evidence. The Amg, which receives input from brain areas encoding internal states, innately relevant stimuli, and innately neutral stimuli, plays a fundamental role in the motivational and emotional processes of organisms. This role is based on the fact that Amg implements the two associative processes at the core of Pavlovian learning (conditioned stimulus (CS)–unconditioned stimulus (US) and CS–unconditioned response (UR) associations), and that it has the capacity of modulating these associations on the basis of internal states. These functionalities allow the Amg to play an important role in the regulation of the three fundamental classes of affective responses (namely, the regulation of body states, the regulation of brain states via neuromodulators, and the triggering of a number of basic behaviours fundamental for adaptation) and in the regulation of three high-level cognitive processes (namely, the affective labelling of memories, the production of goal-directed behaviours, and the performance of planning and complex decision-making). Our analysis is conducted within a methodological approach that stresses the importance of understanding the brain within an evolutionary/adaptive framework and with the aim of isolating general principles that can potentially account for the wider possible empirical evidence in a coherent fashion.

Modelling Perception with Artificial Neural Networks: The interplay of Pavlovian and instrumental processes in devaluation experiments: a computational embodied neuroscience model tested with a simulated rat

This paper presents an embodied biologically-plausible model investigating the relationships existing between classical and instrumental conditioning. The architecture and functioning of the model is constrained with some some important anatomical and physiological assumptions drawn from the relevant neuroscientific literature. The model is validated by successfully reproducing the primary outcomes of some instrumentalconditioning devaluation tests conducted with normal and amygdala-lesioned rats. These experiments are particularly important as they show how the sensitivity to internal states (as satiety) exhibited by classical conditioning mechanisms can transfer to behaviors acquired on the basis of instrumental conditioning mechanisms. The results presented are relevant for both neuroscience and behavioural sciences as they are based on a model, constrained and validated at both neural and behavioural level, which indicates how internal states might modulate learning and performance of rigid habits so as to render to action some of the flexibility typical of goal-directed behaviour. The results are also relevant for autonomous robotics as they start to investigate, with an embodied system, how the use of sophisticated motivational systems might allow building robots capable of exhibiting some of the flexibility typical of organisms.

A Computational Model of the Amygdala Nuclei's Role in Second Order Conditioning

The mechanisms underlying learning in classical conditioning experiments play a key role in many learning processes of real organisms. This paper presents a novel computational model that incorporates a biologically plausible hypothesis on the functions that the main nuclei of the amygdala might play in first and second order classical conditioning tasks. The model proposes that in these experiments the first and second order conditioned stimuli (CS) are associated both (a) with the unconditioned stimuli (US) within the basolateral amygdala (BLA), and (b) directly with the unconditioned responses (UR) through the connections linking the lateral amygdala (LA) to the central nucleus of amygdala (CeA). The model, embodied in a simulated robotic rat, is validated by reproducing the results of first and second order conditioning experiments of both sham-lesioned and BLA-lesioned real rats.

Selection of cortical dynamics for motor behaviour by the basal ganglia

The basal ganglia and cortex are strongly implicated in the control of motor preparation and execution. Re-entrant loops between these two brain areas are thought to determine the selection of motor repertoires for instrumental action. The nature of neural encoding and processing in the motor cortex as well as the way in which selection by the basal ganglia acts on them is currently debated. The classic view of the motor cortex implementing a direct mapping of information from perception to muscular responses is challenged by proposals viewing it as a set of dynamical systems controlling muscles. Consequently, the common idea that a competition between relatively segregated cortico-striato-nigro-thalamo-cortical channels selects patterns of activity in the motor cortex is no more sufficient to explain how action selection works. Here, we contribute to develop the dynamical view of the basal ganglia–cortical system by proposing a computational model in which a thalamo-cortical dynamical neural reservoir is modulated by disinhibitory selection of the basal ganglia guided by top-down information, so that it responds with different dynamics to the same bottom-up input. The model shows how different motor trajectories can so be produced by controlling the same set of joint actuators. Furthermore, the model shows how the basal ganglia might modulate cortical dynamics by preserving coarse-grained spatiotemporal information throughout cortico-cortical pathways.

Dysfunctions of the basal ganglia-cerebellar-thalamo-cortical system produce motor tics in Tourette syndrome

Motor tics are a cardinal feature of Tourette syndrome and are traditionally associated with an excess of striatal dopamine in the basal ganglia. Recent evidence increasingly supports a more articulated view where cerebellum and cortex, working closely in concert with basal ganglia, are also involved in tic production. Building on such evidence, this article proposes a computational model of the basal ganglia-cerebellar-thalamo-cortical system to study how motor tics are generated in Tourette syndrome. In particular, the model: (i) reproduces the main results of recent experiments about the involvement of the basal ganglia-cerebellar-thalamo-cortical system in tic generation; (ii) suggests an explanation of the system-level mechanisms underlying motor tic production: in this respect, the model predicts that the interplay between dopaminergic signal and cortical activity contributes to triggering the tic event and that the recently discovered basal ganglia-cerebellar anatomical pathway may support the involvement of the cerebellum in tic production; (iii) furnishes predictions on the amount of tics generated when striatal dopamine increases and when the cortex is externally stimulated. These predictions could be important in identifying new brain target areas for future therapies. Finally, the model represents the first computational attempt to study the role of the recently discovered basal ganglia-cerebellar anatomical links. Studying this non-cortex-mediated basal ganglia-cerebellar interaction could radically change our perspective about how these areas interact with each other and with the cortex. Overall, the model also shows the utility of casting Tourette syndrome within a system-level perspective rather than viewing it as related to the dysfunction of a single brain area.

The Hierarchical Organisation of Cortical and Basal-Ganglia Systems: A Computationally-Informed Review and Integrated Hypothesis

To suitably adapt to the challenges posed by reproduction and survival, animals need to learn to select when to perform different behaviours, to have internal criteria for guiding these learning processes, and to perform behaviours efficiently once selected. To implement these processes, their brains must be organised in a suitable hierarchical fashion. Here we briefly review two types of neural/behavioural/computational literatures, focussed, respectively, on cortex and on sub-cortical areas, and highlight their important limitations. Then we review two computational modelling works of the authors that exemplify the problems, brain areas, experiments, main concepts, and limitations of the two research threads. Finally we propose a theoretical integration of the two views, showing how this allows us to solve most of the problems found by the two accounts if taken in isolation. The overall picture that emerges is that the cortical and the basal ganglia systems form two highly-organised hierarchical systems working in close synergy and jointly solving all the challenges of choice, selection, and implementation needed to acquire and express adaptive behaviour.

Modelling coordination of learning systems: a reservoir systems approach to dopamine modulated pavlovian conditioning

This paper presents a biologically constrained reward prediction model capable of learning cue-outcome associations involving temporally distant stimuli without using the commonly used temporal difference model. The model incorporates a novel use of an adapted echo state network to substitute the biologically implausible delay chains usually used, in relation to dopamine phenomena, for tackling temporally structured stimuli. Moreover, the model is based on a novel algorithm which successfully coordinates two sub systems: one providing Pavlovian conditioning, one providing timely inhibition of dopamine responses to salient anticipated stimuli. The model is validated against the typical profile of phasic dopamine in first and second order Pavlovian conditioning. The model is relevant not only to explaining the mechanisms underlying the biological regulation of dopamine signals, but also for applications in autonomous robotics involving reinforcement-based learning.

Corticolimbic catecholamines in stress: a computational model of the appraisal of controllability

Appraisal of a stressful situation and the possibility to control or avoid it is thought to involve frontal-cortical mechanisms. The precise mechanism underlying this appraisal and its translation into effective stress coping (the regulation of physiological and behavioural responses) are poorly understood. Here, we propose a computational model which involves tuning motivational arousal to the appraised stressing condition. The model provides a causal explanation of the shift from active to passive coping strategies, i.e. from a condition characterised by high motivational arousal, required to deal with a situation appraised as stressful, to a condition characterised by emotional and motivational withdrawal, required when the stressful situation is appraised as uncontrollable/unavoidable. The model is motivated by results acquired via microdialysis recordings in rats and highlights the presence of two competing circuits dominated by different areas of the ventromedial prefrontal cortex: these are shown having opposite effects on several subcortical areas, affecting dopamine outflow in the striatum, and therefore controlling motivation. We start by reviewing published data supporting structure and functioning of the neural model and present the computational model itself with its essential neural mechanisms. Finally, we show the results of a new experiment, involving the condition of repeated inescapable stress, which validate most of the model’s predictions.