Emmanuel Procyk

Behavioral Shifts and Action Valuation in the Anterior Cingulate Cortex

Rapid optimization of behavior requires decisions about when to explore and when to exploit discovered resources. The mechanisms that lead to fast adaptations and their interaction with action valuation are a central issue. We show here that the anterior cingulate cortex (ACC) encodes multiple feedbacks devoted to exploration and its immediate termination. In a task that alternates exploration and exploitation periods, the ACC monitored negative and positive outcomes relevant for different adaptations. In particular, it produced signals specific of the first reward, i.e., the end of exploration. Those signals disappeared in exploitation periods but immediately transferred to the initiation of trials-a transfer comparable to learning phenomena observed for dopaminergic neurons. Importantly, these were also observed for high gamma oscillations of local field potentials shown to correlate with brain imaging signal. Thus, mechanisms of action valuation and monitoring of events/actions are combined for rapid behavioral regulation.

Anterior cingulate error-related activity is modulated by predicted reward.

Learning abilities depend on detection and exploitation of errors. In primates, this function involves the anterior cingulate cortex. However, whether anterior cingulate error‐related activity indicates occurrence of inappropriate responses or results from other computations is debated. Here we have tested whether reward‐related parameters modulate error‐related activity of anterior cingulate neurons. Recordings in monkeys performing stimulus–reward associations and preliminary data obtained with a problem‐solving task revealed major properties of error‐related unit activity: (i) their amplitude varies with the amount of predicted reward or the proximity to reward delivery; (ii) they appear both after execution and performance errors; (iii) they do not indicate which error occurred or which correction to make; and (iv), importantly, the activity of these neurons also increases following an external signal indicating the necessity to shift response. Hence, we conclude that anterior cingulate ‘error’ activity might represent a negative deviation from a predicted goal, and does not only reflect error detection but signals events interrupting potentially rewarded actions.

The primate working memory networks

Working memory has long been associated with the prefrontal cortex, since damage to this brain area can critically impair the ability to maintain and update mnemonic information. Anatomical and physiological evidence suggests, however, that the prefrontal cortex is part of a broader network of interconnected brain areas involved in working memory. These include the parietal and temporal association areas of the cerebral cortex, cingulate and limbic areas, and subcortical structures such as the mediodorsal thalamus and the basal ganglia. Neurophysiological studies in primates confirm the involvement of areas beyond the frontal lobe and illustrate that working memory involves parallel, distributed neuronal networks. In this article, we review the current understanding of the anatomical organization of networks mediating working memory and the neural correlates of memory manifested in each of their nodes. The neural mechanisms of memory maintenance and the integrative role of the prefrontal cortex are also discussed.

Expectations, gains, and losses in the anterior cingulate cortex.

The anterior cingulate cortex (ACC) participates in evaluating actions and outcomes. Little is known on how action-reward values are processed in ACC and if the context in which actions are performed influences this processing. In the present article, we report ACC unit activity of monkeys performing two tasks. The first task tested whether the encoding of reward values is context dependent—that is, dependent on the size of the other rewards that are available in the current block of trials. The second task tested whether unexpected events signaling a change in reward are represented. We show that the context created by a block design (i.e., the context of possible alternative rewards) influences the encoding of reward values, even if no decision or choice is required. ACC activity encodes the relative and not absolute expected reward values. Moreover, cingulate activity signals and evaluates when reward expectations are violated by unexpected stimuli, indicating reward gains or losses.

Modulation of Dorsolateral Prefrontal Delay Activity during Self-Organized Behavior

The regulation of cognitive activity relies on the flexibility of prefrontal cortex functions. To study this mechanism we compared monkey dorsolateral prefrontal activity in two different spatial cognitive tasks: a delayed response task and a self-organized problem-solving task. The latter included two periods, a search by trial and error for a correct response, and a repetition of the response once discovered. We show that (1) delay activity involved in the delayed task also participates in self-generated responses during the problem-solving task and keeps the same location preference, and (2) the amplitude of firing and the strength of spatial selectivity vary with task requirement, even within search periods while approaching the correct response. This variation is dissociated from pure reward probability, but may have a link with uncertainty because the selectivity dropped when reward predictability was maximal. Overall, we show that spatially tuned delay activity of prefrontal neurons reflects the varying level of engagement in control between different spatial cognitive tasks and during self-organized behavior.

Characterization of serial order encoding in the monkey anterior cingulate sulcus

In a previous report we showed that neurons in the anterior cingulate cortex might encode the serial order of the three components (first, second and third) of motor sequences, irrespective of which component is performed, and irrespective of the component that precedes or follows. Here we further explore these data by comparing the magnitude of cell activity at the different ranks. We also compare the activity recorded in the motor sequences and in tasks with only one motor component. We finally discuss functional hypotheses, which may account for the serial order encoding.

Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model

A major challenge in modern robotics is to liberate robots from controlled industrial settings, and allow them to interact with humans and changing environments in the real-world. The current research attempts to determine if a neurophysiologically motivated model of cortical function in the primate can help to address this challenge. Primates are endowed with cognitive systems that allow them to maximize the feedback from their environment by learning the values of actions in diverse situations and by adjusting their behavioral parameters (i.e., cognitive control) to accommodate unexpected events. In such contexts uncertainty can arise from at least two distinct sources – expected uncertainty resulting from noise during sensory-motor interaction in a known context, and unexpected uncertainty resulting from the changing probabilistic structure of the environment. However, it is not clear how neurophysiological mechanisms of reinforcement learning and cognitive control integrate in the brain to produce efficient behavior. Based on primate neuroanatomy and neurophysiology, we propose a novel computational model for the interaction between lateral prefrontal and anterior cingulate cortex reconciling previous models dedicated to these two functions. We deployed the model in two robots and demonstrate that, based on adaptive regulation of a meta-parameter β that controls the exploration rate, the model can robustly deal with the two kinds of uncertainties in the real-world. In addition the model could reproduce monkey behavioral performance and neurophysiological data in two problem-solving tasks. A last experiment extends this to human–robot interaction with the iCub humanoid, and novel sources of uncertainty corresponding to “cheating” by the human. The combined results provide concrete evidence for the ability of neurophysiologically inspired cognitive systems to control advanced robots in the real-world.

Midcingulate Motor Map and Feedback Detection: Converging Data from Humans and Monkeys

The functional and anatomical organization of the cingulate cortex across primate species is the subject of considerable and often confusing debate. The functions attributed to the midcingulate cortex (MCC) embrace, among others, feedback processing, pain, salience, action-reward association, premotor functions, and conflict monitoring. This multiplicity of functional concepts suggests either unresolved separation of functional contributions or integration and convergence. We here provide evidence from recent experiments in humans and from a meta-analysis of monkey data that MCC feedback-related activity is generated in the rostral cingulate premotor area by specific body maps directly related to the modality of feedback. As such, we argue for an embodied mechanism for adaptation and exploration in MCC. We propose arguments and precise tools to resolve the origins of performance monitoring signals in the medial frontal cortex, and to progress on issues regarding homology between human and nonhuman primate cingulate cortex.

The Location of Feedback-Related Activity in the Midcingulate Cortex Is Predicted by Local Morphology

Information processing in the medial frontal cortex is often said to be modulated in pathological conditions or by individual traits. This has been observed in neuroimaging and event-related potential studies centered in particular on midcingulate cortex (MCC) functions. This region of the brain is characterized by considerable intersubject morphological variability. Whereas in a subset of hemispheres only a single cingulate sulcus (cgs) is present, a majority of hemispheres exhibit an additional sulcus referred to as the paracingulate sulcus (pcgs). The present functional magnetic resonance imaging study defined the relationship between the local morphology of the cingulate/paracingulate sulcal complex and feedback-related activity. Human subjects performed a trial-and-error learning task in which they had to discover which one of a set of abstract stimuli was the best option. Feedback was provided by means of fruit juice, as in studies with monkeys. A subject-by-subject analysis revealed that the feedback-related activity during exploration was systematically located in the cgs when no pcgs was observed, but in the pcgs when the latter sulcus was present. The activations had the same functional signature when located in either the cgs or in the pcgs, confirming that both regions were homologues. Together, the results show that the location of feedback-related MCC activity can be predicted from morphological features of the cingulate/paracingulate complex.

Modulation of feedback related activity in the rostral anterior cingulate cortex during trial and error exploration.

The present functional magnetic resonance imaging (fMRI) experiment aims at clarifying the role of the rostral anterior cingulate cortex (rACC) in the evaluation of feedback in a deterministic environment. We tested, in particular, the response of the rACC to the detection of different types of feedback, and to varying levels of outcome expectancy. We used a problem-solving task in which subjects had to discover, in successive trials, which one of the four presented stimuli was associated with a positive feedback, the other ones being associated with error feedback. In this task, two periods alternated: 1. an exploratory period in which error feedback indicated to adapt the following response appropriately (and continue to explore), and first positive feedback indicated to change strategy (i.e. to shift from explorative to exploitative behavior), and 2. an exploitative period in which subjects had to repeat the correct choice. The rACC is recruited in the exploratory period during the analysis of both error and first correct positive feedback. In addition, the rACC activity was modulated by positive reward prediction error values (i.e. the difference between obtained and expected feedback). Altogether, these results reveal the critical role of the rACC in the evaluation of salient feedback for learning optimal strategies.