William H. Alexander

Medial prefrontal cortex as an action-outcome predictor

The medial prefrontal cortex (mPFC) and especially anterior cingulate cortex is central to higher cognitive function and many clinical disorders, yet its basic function remains in dispute. Various competing theories of mPFC have treated effects of errors, conflict, error likelihood, volatility and reward, using findings from neuroimaging and neurophysiology in humans and monkeys. No single theory has been able to reconcile and account for the variety of findings. Here we show that a simple model based on standard learning rules can simulate and unify an unprecedented range of known effects in mPFC. The model reinterprets many known effects and suggests a new view of mPFC, as a region concerned with learning and predicting the likely outcomes of actions, whether good or bad. Cognitive control at the neural level is then seen as a result of evaluating the probable and actual outcomes of ones actions.

Neuromodulation and Plasticity in an autonomous robot

In this paper we implement a computational model of a neuromodulatory system in an autonomous robot. The output of the neuromodulatory system acts as a value signal, modulating widely distributed synaptic changes. The model is based on anatomical and physiological properties of midbrain diffuse ascending systems, in particular parts of the dopamine and noradrenaline systems. During reward conditioning, the model learns to generate tonic and phasic signals that represent predictions and prediction errors, including precisely timed negative signals if expected rewards are omitted or delayed. We test the robots learning and behavior in different environmental contexts and observe changes in the development of the neuromodulatory system that depend upon environmental factors. Simulation of a computational model incorporating both reward-related and aversive stimuli leads to the emergence of conditioned reward and aversive behaviors. These studies represent a step towards investigating computational aspects of neuromodulatory systems in autonomous robots.

From conflict management to reward-based decision making: Actors and critics in primate medial frontal cortex

The role of the medial prefrontal cortex (mPFC) and especially the anterior cingulate cortex has been the subject of intense debate for the last decade. A number of theories have been proposed to account for its function. Broadly speaking, some emphasize cognitive control, whereas others emphasize value processing; specific theories concern reward processing, conflict detection, error monitoring, and volatility detection, among others. Here we survey and evaluate them relative to experimental results from neurophysiological, anatomical, and cognitive studies. We argue for a new conceptualization of mPFC, arising from recent computational modeling work. Based on reinforcement learning theory, these new models propose that mPFC is an Actor-Critic system. This system is aimed to predict future events including rewards, to evaluate errors in those predictions, and finally, to implement optimal skeletal-motor and visceromotor commands to obtain reward. This framework provides a comprehensive account of mPFC function, accounting for and predicting empirical results across different levels of analysis, including monkey neurophysiology, human ERP, human neuroimaging, and human behavior.

Competition between learned reward and error outcome predictions in anterior cingulate cortex

The anterior cingulate cortex (ACC) is implicated in performance monitoring and cognitive control. Non-human primate studies of ACC show prominent reward signals, but these are elusive in human studies, which instead show mainly conflict and error effects. Here we demonstrate distinct appetitive and aversive activity in human ACC. The error likelihood hypothesis suggests that ACC activity increases in proportion to the likelihood of an error, and ACC is also sensitive to the consequence magnitude of the predicted error. Previous work further showed that error likelihood effects reach a ceiling as the potential consequences of an error increase, possibly due to reductions in the average reward. We explored this issue by independently manipulating reward magnitude of task responses and error likelihood while controlling for potential error consequences in an Incentive Change Signal Task. The fMRI results ruled out a modulatory effect of expected reward on error likelihood effects in favor of a competition effect between expected reward and error likelihood. Dynamic causal modeling showed that error likelihood and expected reward signals are intrinsic to the ACC rather than received from elsewhere. These findings agree with interpretations of ACC activity as signaling both perceptions of risk and predicted reward.

An Embodied Model of Learning, Plasticity, and Reward

We describe and discuss a neural network model of the dopaminergic system based on observed anatomical and physiological properties of the primate midbrain. The model relies on value-dependent synaptic modification to acquire temporal information regarding reward-related events and the stimuli with which such events are paired. Experience-dependent changes in synaptic plasticity allow the model to generate neuromodulatory responses corresponding to prediction errors. These phasic neural responses act as a value signal with positive and negative components, representing the unpredicted occurrence of rewarding stimuli and the omission of an expected reward, respectively. The value signal modulates widespread synaptic changes, including afferent connections of the value system itself. The model is embedded in an autonomous robot, and its behavior is tested as changes are applied to the robots motor characteristics and as the stimulus content of the environment is varied. We observe the development of the system as a consequence of environmental stimuli and autonomous movement, leading to the conditioning of reward-related behaviors through the interaction between the robot and its surroundings.

Hierarchical error representation: A computational model of anterior cingulate and dorsolateral prefrontal cortex

Anterior cingulate and dorsolateral prefrontal cortex (ACC and dlPFC, respectively) are core components of the cognitive control network. Activation of these regions is routinely observed in tasks that involve monitoring the external environment and maintaining information in order to generate appropriate responses. Despite the ubiquity of studies reporting coactivation of these two regions, a consensus on how they interact to support cognitive control has yet to emerge. In this letter, we present a new hypothesis and computational model of ACC and dlPFC. The error representation hypothesis states that multidimensional error signals generated by ACC in response to surprising outcomes are used to train representations of expected error in dlPFC, which are then associated with relevant task stimuli. Error representations maintained in dlPFC are in turn used to modulate predictive activity in ACC in order to generate better estimates of the likely outcomes of actions. We formalize the error representation hypothesis in a new computational model based on our previous model of ACC. The hierarchical error representation (HER) model of ACC/dlPFC suggests a mechanism by which hierarchically organized layers within ACC and dlPFC interact in order to solve sophisticated cognitive tasks. In a series of simulations, we demonstrate the ability of the HER model to autonomously learn to perform structured tasks in a manner comparable to human performance, and we show that the HER model outperforms current deep learning networks by an order of magnitude.

Distinct regions of anterior cingulate cortex signal prediction and outcome evaluation

A number of theories have been proposed to account for the role of anterior cingulate cortex (ACC) and the broader medial prefrontal cortex (mPFC) in cognition. The recent Prediction of Response Outcome (PRO) computational model casts the mPFC in part as performing two theoretically distinct functions: learning to predict the various possible outcomes of actions, and then evaluating those predictions against the actual outcomes. Simulations have shown that this new model can account for an unprecedented range of known mPFC effects, but the central theory of distinct prediction and evaluation mechanisms within ACC remains untested. Using combined computational neural modeling and fMRI, we show here that prediction and evaluation signals are indeed each represented in the ACC, and furthermore, they are represented in distinct regions within ACC. Our task independently manipulated both the number of predicted outcomes and the degree to which outcomes violated expectancies, the former providing assessment of regions sensitive to prediction and the latter providing assessment of regions sensitive to evaluation. Using quantitative regressors derived from the PRO computational model, we show that prediction-based model signals load on a network including the posterior and perigenual ACC, but outcome evaluation model signals load on the mid-dorsal ACC. These findings are consistent with distinct prediction and evaluation signals as posited by the PRO model and provide new perspective on a large set of known effects within ACC.

A general role for medial prefrontal cortex in event prediction

A recent computational neural model of medial prefrontal cortex (mPFC), namely the predicted response-outcome (PRO) model (Alexander and Brown, 2011), suggests that mPFC learns to predict the outcomes of actions. The model accounted for a wide range of data on the mPFC. Nevertheless, numerous recent findings suggest that mPFC may signal predictions and prediction errors even when the predicted outcomes are not contingent on prior actions. Here we show that the existing PRO model can learn to predict outcomes in a general sense, and not only when the outcomes are contingent on actions. A series of simulations show how this generalized PRO model can account for an even broader range of findings in the mPFC, including human ERP, fMRI, and macaque single-unit data. The results suggest that the mPFC learns to predict salient events in general and provides a theoretical framework that links mPFC function to model-based reinforcement learning, Bayesian learning, and theories of cognitive control.

Distinct Regions within Medial Prefrontal Cortex Process Pain and Cognition

Neuroimaging studies of the medial prefrontal cortex (mPFC) suggest that the dorsal anterior cingulate cortex (dACC) region is responsive to a wide variety of stimuli and psychological states, such as pain, cognitive control, and prediction error (PE). In contrast, a recent meta-analysis argues that the dACC is selective for pain, whereas the supplementary motor area (SMA) and pre-SMA are specifically associated with higher-level cognitive processes (Lieberman and Eisenberger, 2015). To empirically test this claim, we manipulated effects of pain, conflict, and PE in a single experiment using human subjects. We observed a robust dorsal-ventral dissociation within the mPFC with cognitive effects of PE and conflict overlapping dorsally and pain localized more ventrally. Classification of subjects based on the presence or absence of a paracingulate sulcus showed that PE effects extended across the dorsal area of the dACC and into the pre-SMA. These results begin to resolve recent controversies by showing the following: (1) the mPFC includes dissociable regions for pain and cognitive processing; and (2) meta-analyses are correct in localizing cognitive effects to the dACC, although these effects extend to the pre-SMA as well. These results both provide evidence distinguishing between different theories of mPFC function and highlight the importance of taking individual anatomical variability into account when conducting empirical studies of the mPFC. SIGNIFICANCE STATEMENT Decades of neuroimaging research have shown the mPFC to represent a wide variety of stimulus processing and cognitive states. However, recently it has been argued whether distinct regions of the mPFC separately process pain and cognitive phenomena. To address this controversy, this study directly compared pain and cognitive processes within subjects. We found a double dissociation within the mPFC with pain localized ventral to the cingulate sulcus and cognitive effects localized more dorsally within the dACC and spreading into the pre-supplementary motor area. This provides empirical evidence to help resolve the current debate about the functional architecture of the mPFC.

Computational Models of Anterior Cingulate Cortex: At the Crossroads between Prediction and Effort

In the last two decades the anterior cingulate cortex (ACC) has become one of the most investigated areas of the brain. Extensive neuroimaging evidence suggests countless functions for this region, ranging from conflict and error coding, to social cognition, pain and effortful control. In response to this burgeoning amount of data, a proliferation of computational models has tried to characterize the neurocognitive architecture of ACC. Early seminal models provided a computational explanation for a relatively circumscribed set of empirical findings, mainly accounting for EEG and fMRI evidence. More recent models have focused on ACCs contribution to effortful control. In parallel to these developments, several proposals attempted to explain within a single computational framework a wider variety of empirical findings that span different cognitive processes and experimental modalities. Here we critically evaluate these modeling attempts, highlighting the continued need to reconcile the array of disparate ACC observations within a coherent, unifying framework.