Angela J. Yu

Uncertainty, Neuromodulation, and Attention

Uncertainty in various forms plagues our interactions with the environment. In a Bayesian statistical framework, optimal inference and prediction, based on unreliable observations in changing contexts, require the representation and manipulation of different forms of uncertainty. We propose that the neuromodulators acetylcholine and norepinephrine play a major role in the brains implementation of these uncertainty computations. Acetylcholine signals expected uncertainty, coming from known unreliability of predictive cues within a context. Norepinephrine signals unexpected uncertainty, as when unsignaled context switches produce strongly unexpected observations. These uncertainty signals interact to enable optimal inference and learning in noisy and changeable environments. This formulation is consistent with a wealth of physiological, pharmacological, and behavioral data implicating acetylcholine and norepinephrine in specific aspects of a range of cognitive processes. Moreover, the model suggests a class of attentional cueing tasks that involve both neuromodulators and shows how their interactions may be part-antagonistic, part-synergistic.

Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

Many large and small decisions we make in our daily lives—which ice cream to choose, what research projects to pursue, which partner to marry—require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.

Phasic norepinephrine: A neural interrupt signal for unexpected events

Extensive animal studies indicate that the neuromodulator norepinephrine plays an important role in specific aspects of vigilance, attention and learning, putatively serving as a neural interrupt or reset function. The activity of norepinephrine-releasing neurons in the locus coeruleus during attentional tasks is modulated not only by the animals level of engagement and the sensory inputs, but also by temporally rich aspects of internal decision-making processes. Here, we propose that it is unexpected changes in the world within the context of a task that activate the noradrenergic interrupt signal. We quantify this idea in a Bayesian model of a well-studied visual discrimination task, demonstrating that the model captures a rich repertoire of noradrenergic responses at the sub-second temporal resolution.

Acetylcholine in cortical inference

Acetylcholine (ACh) plays an important role in a wide variety of cognitive tasks, such as perception, selective attention, associative learning, and memory. Extensive experimental and theoretical work in tasks involving learning and memory has suggested that ACh reports on unfamiliarity and controls plasticity and effective network connectivity. Based on these computational and implementational insights, we develop a theory of cholinergic modulation in perceptual inference. We propose that ACh levels reflect the uncertainty associated with top-down information, and have the effect of modulating the interaction between top-down and bottom-up processing in determining the appropriate neural representations for inputs. We illustrate our proposal by means of an hierarchical hidden Markov model, showing that cholinergic modulation of contextual information leads to appropriate perceptual inference.

Bayesian Prediction and Evaluation in the Anterior Cingulate Cortex

The dorsal anterior cingulate cortex (dACC) has been implicated in a variety of cognitive control functions, among them the monitoring of conflict, error, and volatility, error anticipation, reward learning, and reward prediction errors. In this work, we used a Bayesian ideal observer model, which predicts trial-by-trial probabilistic expectation of stop trials and response errors in the stop-signal task, to differentiate these proposed functions quantitatively. We found that dACC hemodynamic response, as measured by functional magnetic resonance imaging, encodes both the absolute prediction error between stimulus expectation and outcome, and the signed prediction error related to response outcome. After accounting for these factors, dACC has no residual correlation with conflict or error likelihood in the stop-signal task. Consistent with recent monkey neural recording studies, and in contrast with other neuroimaging studies, our work demonstrates that dACC reports at least two different types of prediction errors, and beyond contexts that are limited to reward processing.

Uncertainty and Learning

It is a commonplace in statistics that uncertainty about parameters drives learning. Indeed one of the most influential models of behavioural learning has uncertainty at its heart. However, many popular theoretical models of learning focus exclusively on error, and ignore uncertainty. Here we review the links between learning and uncertainty from three perspectives: statistical theories such as the Kalman filter, psychological models in which differential attention is paid to stimuli with an effect on the speed of learning associated with those stimuli, and neurobiological data on the influence of the neuromodulators acetylcholine and norepinephrine on learning and inference.

Emotion and decision-making: affect-driven belief systems in anxiety and depression

Emotion processing and decision-making are integral aspects of daily life. However, our understanding of the interaction between these constructs is limited. In this review, we summarize theoretical approaches that link emotion and decision-making, and focus on research with anxious or depressed individuals to show how emotions can interfere with decision-making. We integrate the emotional framework based on valence and arousal with a Bayesian approach to decision-making in terms of probability and value processing. We discuss how studies of individuals with emotional dysfunctions provide evidence that alterations of decision-making can be viewed in terms of altered probability and value computation. We argue that the probabilistic representation of belief states in the context of partially observable Markov decision processes provides a useful approach to examine alterations in probability and value representation in individuals with anxiety and depression, and outline the broader implications of this approach.

Biophysiologically plausible implementations of the maximum operation

Visual processing in the cortex can be characterized by a predominantly hierarchical architecture, in which specialized brain regions along the processing pathways extract visual features of increasing complexity, accompanied by greater invariance in stimulus properties such as size and position. Various studies have postulated that a nonlinear pooling function such as the maximum (MAX) operation could be fundamental in achieving such selectivity and invariance. In this article, we are concerned with neurally plausible mechanisms that may be involved in realizing the MAX operation. Different canonical models are proposed, each based on neural mechanisms that have been previously discussed in the context of cortical processing. Through simulations and mathematical analysis, we compare the performance and robustness of these mechanisms. We derive experimentally verifiable predictions for each model and discuss the relevant physiological considerations.

Rational Decision-Making in Inhibitory Control

An important aspect of cognitive flexibility is inhibitory control, the ability to dynamically modify or cancel planned actions in response to changes in the sensory environment or task demands. We formulate a probabilistic, rational decision-making framework for inhibitory control in the stop signal paradigm. Our model posits that subjects maintain a Bayes-optimal, continually updated representation of sensory inputs, and repeatedly assess the relative value of stopping and going on a fine temporal scale, in order to make an optimal decision on when and whether to go on each trial. We further posit that they implement this continual evaluation with respect to a global objective function capturing the various reward and penalties associated with different behavioral outcomes, such as speed and accuracy, or the relative costs of stop errors and go errors. We demonstrate that our rational decision-making model naturally gives rise to basic behavioral characteristics consistently observed for this paradigm, as well as more subtle effects due to contextual factors such as reward contingencies or motivational factors. Furthermore, we show that the classical race model can be seen as a computationally simpler, perhaps neurally plausible, approximation to optimal decision-making. This conceptual link allows us to predict how the parameters of the race model, such as the stopping latency, should change with task parameters and individual experiences/ability.

Altered Neural Processing of the Need to Stop in Young Adults at Risk for Stimulant Dependence

Identification of neurocognitive predictors of substance dependence is an important step in developing approaches to prevent addiction. Given evidence of inhibitory control deficits in substance abusers (Monterosso et al., 2005; Fu et al., 2008; Lawrence et al., 2009; Tabibnia et al., 2011), we examined neural processing characteristics in human occasional stimulant users (OSU), a population at risk for dependence. A total of 158 nondependent OSU and 47 stimulant-naive control subjects (CS) were recruited and completed a stop signal task while undergoing functional magnetic resonance imaging (fMRI). A Bayesian ideal observer model was used to predict probabilistic expectations of inhibitory demand, P(stop), on a trial-to-trial basis, based on experienced trial history. Compared with CS, OSU showed attenuated neural activation related to P(stop) magnitude in several areas, including left prefrontal cortex and left caudate. OSU also showed reduced neural activation in the dorsal anterior cingulate cortex (dACC) and right insula in response to an unsigned Bayesian prediction error representing the discrepancy between stimulus outcome and the predicted probability of a stop trial. These results indicate that, despite minimal overt behavioral manifestations, OSU use fewer brain processing resources to predict and update the need for response inhibition, processes that are critical for adjusting and optimizing behavioral performance, which may provide a biomarker for the development of substance dependence.