Mark E. Walton

Learning the value of information in an uncertain world

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

Optimal decision making and the anterior cingulate cortex

Learning the value of options in an uncertain environment is central to optimal decision making. The anterior cingulate cortex (ACC) has been implicated in using reinforcement information to control behavior. Here we demonstrate that the ACCs critical role in reinforcement-guided behavior is neither in detecting nor in correcting errors, but in guiding voluntary choices based on the history of actions and outcomes. ACC lesions did not impair the performance of monkeys (Macaca mulatta) immediately after errors, but made them unable to sustain rewarded responses in a reinforcement-guided choice task and to integrate risk and payoff in a dynamic foraging task. These data suggest that the ACC is essential for learning the value of actions.

Frontal Cortex and Reward-Guided Learning and Decision-Making

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.

Separate neural pathways process different decision costs

Behavioral ecologists and economists emphasize that potential costs, as well as rewards, influence decision making. Although neuroscientists assume that frontal areas are central to decision making, the evidence is contradictory and the critical region remains unclear. Here it is shown that frontal lobe contributions to cost-benefit decision making can be understood by positing the existence of two independent systems that make decisions about delay and effort costs. Anterior cingulate cortex lesions affected how much effort rats decided to invest for rewards. Orbitofrontal cortical lesions affected how long rats decided to wait for rewards. The pattern of disruption suggested the deficit could be related to impaired associative learning. Impairments of the two systems may underlie apathetic and impulsive choice patterns in neurological and psychiatric illnesses. Although the existence of two systems is not predicted by economic accounts of decision making, our results suggest that delay and effort may exert distinct influences on decision making.

Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour.

There is general acknowledgement that both the anterior cingulate and orbitofrontal cortex are implicated in reinforcement-guided decision making, and emotion and social behaviour. Despite the interest that these areas generate in both the cognitive neuroscience laboratory and the psychiatric clinic, ideas about the distinctive contributions made by each have only recently begun to emerge. This reflects an increasing understanding of the component processes that underlie reinforcement-guided decision making, such as the representation of reinforcement expectations, the exploration, updating and representation of action values, and the appreciation that choices are guided not just by the prospect of reward but also by the costs that action entails. Evidence is emerging to suggest that the anterior cingulate and orbitofrontal cortex make distinct contributions to each of these aspects of decision making.

Interactions between decision making and performance monitoring within prefrontal cortex.

Our ability to judge the consequences of our actions is central to rational decision making. A large body of evidence implicates primate prefrontal regions in the regulation of this ability. It has proven extremely difficult, however, to separate functional areas in the frontal lobes. Using functional magnetic resonance imaging, we demonstrate complementary and reciprocal roles for the human orbitofrontal (OFC) and dorsal anterior cingulate cortices (ACd) in monitoring the outcome of behavior. Activation levels in these regions were negatively correlated, with activation increasing in the ACd and decreasing in the OFC when the selected response was the result of the participants own decision. The pattern was reversed when the selected response was guided by the experimenter rather than the participant. These results indicate that the neural mechanisms underlying the way we assess the consequences of choices differ depending on whether we are told what to do or are able to exercise our volition.

Functional organization of the medial frontal cortex.

The anterior cingulate cortex (ACC) and adjacent areas of the medial frontal cortex (MFC) have been implicated in monitoring behaviour and in detecting errors. Recent evidence, however, suggests that the ACC not only registers the occurrence of errors but also represents other aspects of the reinforcement history that are crucial for guiding behaviour. Other studies raise the possibility that dorsal MFC areas not only monitor behaviour but also actually control response selection, particularly when the task in hand is changing. Many decisions are made in social contexts and their chances of success depend on what other individuals are doing. Evaluation of other individuals is therefore crucial for effective action selection, and some ACC regions are implicated in this process.

Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort

RationaleAlthough tasks assessing the role of dopamine in effort-reward decisions are similar to those concerned with the role of serotonin in impulsive choice in that both require analysis of the costs and benefits of possible actions, they have never been directly compared.ObjectivesThis study investigated the involvement of serotonin and dopamine in two cost-benefit paradigms, one in which the cost was delay and the other in which it was physical effort.MethodsSixteen rats were trained on a T-maze task in which they chose between high and low reward arms. In one version, the high reward arm was obstructed by a barrier, in the other, delivery of the high reward was delayed by 15 s. Serotonin and dopamine function were manipulated using systemic pCPA and haloperidol injections, respectively.ResultsHaloperidol-treated rats were less inclined either to exert more effort or to countenance a delay for a higher reward. pCPA had no effect on the performance of the rats on the effortful task, but significantly increased the rats’ preference for an immediate but smaller reward. All animals (drug treated and controls) chose the high reward arm on the majority of trials when the delay or effort costs were matched in both high and low reward arms.ConclusionA dissociation was found between the neurotransmitter systems involved in different types of cost-benefit decision making. While dopaminergic systems were required for decisions about both effort and delay, serotonergic systems were only needed for the latter.

Frontal Cortex Subregions Play Distinct Roles in Choices between Actions and Stimuli

The orbitofrontal cortex (OFC) has been implicated in reinforcement-guided decision making, error monitoring, and the reversal of behavior in response to changing circumstances. The anterior cingulate cortex sulcus (ACCS), however, has also been implicated in similar aspects of behavior. Dissociating the unique functions of these areas would improve our understanding of the decision-making process. The effect of selective OFC lesions on how monkeys used the history of reinforcement to guide choices of either particular actions or particular stimuli was studied and compared with the effects of ACCS lesions. Both lesions disrupted decision making, but their effects were differentially modulated by the dependence on action– or stimulus–value contingencies. OFC lesions caused a deficit in stimulus but not action selection, whereas ACCS lesions had the opposite effect, disrupting action but not stimulus selection. Furthermore, OFC lesions that have previously been found to impair decision making when deterministic stimulus–reward contingencies are switched were found to cause a more general learning impairment in more naturalistic situations in which reward was stochastic. Both OFC and ACCS are essential for reinforcement-guided decision making rather than just error monitoring or behavioral reversal. The OFC and ACCS are both, however, more concerned with learning and making decisions, but their roles in selecting between stimulus and action values are distinct.

A Role for the Macaque Anterior Cingulate Gyrus in Social Valuation

Complex human social interaction is disrupted when the frontal lobe is damaged in disease, and in extreme cases patients are described as having acquired sociopathy. We compared, in macaques, the effects of lesions in subdivisions of the anterior cingulate and the orbitofrontal cortices believed to be anatomically homologous to those damaged in such patients. We show that the anterior cingulate gyrus in male macaques is critical for normal patterns of social interest in other individual male or female macaques. Conversely, the orbitofrontal cortex lesion had a marked effect only on responses to mildly fear-inducing stimuli. These results suggest that damage to the anterior cingulate gyrus may be the cause of changes in social interaction seen after frontal lobe damage.