Matthew F. S. Rushworth

Learning the value of information in an uncertain world

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

Optimal decision making and the anterior cingulate cortex

Learning the value of options in an uncertain environment is central to optimal decision making. The anterior cingulate cortex (ACC) has been implicated in using reinforcement information to control behavior. Here we demonstrate that the ACCs critical role in reinforcement-guided behavior is neither in detecting nor in correcting errors, but in guiding voluntary choices based on the history of actions and outcomes. ACC lesions did not impair the performance of monkeys (Macaca mulatta) immediately after errors, but made them unable to sustain rewarded responses in a reinforcement-guided choice task and to integrate risk and payoff in a dynamic foraging task. These data suggest that the ACC is essential for learning the value of actions.

The role of ipsilateral premotor cortex in hand movement after stroke

Movement of an affected hand after stroke is associated with increased activation of ipsilateral motor cortical areas, suggesting that these motor areas in the undamaged hemisphere may adaptively compensate for damaged or disconnected regions. However, this adaptive compensation has not yet been demonstrated directly. Here we used transcranial magnetic stimulation (TMS) to interfere transiently with processing in the ipsilateral primary motor or dorsal premotor cortex (PMd) during finger movements. TMS had a greater effect on patients than controls in a manner that depended on the site, hemisphere, and time of stimulation. In patients with right hemiparesis (but not in healthy controls), TMS applied to PMd early (100 ms) after the cue to move slowed simple reaction-time finger movements by 12% compared with controls. The relative slowing of movements with ipsilateral PMd stimulation in patients correlated with the degree of motor impairment, suggesting that functional recruitment of ipsilateral motor areas was greatest in the more impaired patients. We also used functional magnetic resonance imaging to monitor brain activity in these subjects as they performed the same movements. Slowing of reaction time after premotor cortex TMS in the patients correlated inversely with the relative hemispheric lateralization of functional magnetic resonance imaging activation in PMd. This inverse correlation suggests that the increased activation in ipsilateral cortical motor areas during movements of a paretic hand, shown in this and previous functional imaging studies, represents a functionally relevant, adaptive response to the associated brain injury.

Choice, uncertainty and value in prefrontal and cingulate cortex.

Reinforcement learning models that focus on the striatum and dopamine can predict the choices of animals and people. Representations of reward expectation and of reward prediction errors that are pertinent to decision making, however, are not confined to these regions but are also found in prefrontal and cingulate cortex. Moreover, decisions are not guided solely by the magnitude of the reward that is expected. Uncertainty in the estimate of the reward expectation, the value of information that might be gained by taking a course of action and the cost of an action all influence the manner in which decisions are made through prefrontal and cingulate cortex.

Frontal Cortex and Reward-Guided Learning and Decision-Making

Reward-guided decision-making and learning depends on distributed neural circuits with many components. Here we focus on recent evidence that suggests four frontal lobe regions make distinct contributions to reward-guided learning and decision-making: the lateral orbitofrontal cortex, the ventromedial prefrontal cortex and adjacent medial orbitofrontal cortex, anterior cingulate cortex, and the anterior lateral prefrontal cortex. We attempt to identify common themes in experiments with human participants and with animal models, which suggest roles that the areas play in learning about reward associations, selecting reward goals, choosing actions to obtain reward, and monitoring the potential value of switching to alternative courses of action.

Connectivity-Based Parcellation of Human Cingulate Cortex and Its Relation to Functional Specialization

Whole-brain neuroimaging studies have demonstrated regional variations in function within human cingulate cortex. At the same time, regional variations in cingulate anatomical connections have been found in animal models. It has, however, been difficult to estimate the relationship between connectivity and function throughout the whole cingulate cortex within the human brain. In this study, magnetic resonance diffusion tractography was used to investigate cingulate probabilistic connectivity in the human brain with two approaches. First, an algorithm was used to search for regional variations in the probabilistic connectivity profiles of all cingulate cortex voxels with the whole of the rest of the brain. Nine subregions with distinctive connectivity profiles were identified. It was possible to characterize several distinct areas in the dorsal cingulate sulcal region. Several distinct regions were also found in subgenual and perigenual cortex. Second, the probabilities of connection between cingulate cortex and 11 predefined target regions of interest were calculated. Cingulate voxels with a high probability of connection with the different targets formed separate clusters within cingulate cortex. Distinct connectivity fingerprints characterized the likelihood of connections between the extracingulate target regions and the nine cingulate subregions. Last, a meta-analysis of 171 functional studies reporting cingulate activation was performed. Seven different cognitive conditions were selected and peak activation coordinates were plotted to create maps of functional localization within the cingulate cortex. Regional functional specialization was found to be related to regional differences in probabilistic anatomical connectivity.

Associative learning of social value.

Our decisions are guided by information learnt from our environment. This information may come via personal experiences of reward, but also from the behaviour of social partners. Social learning is widely held to be distinct from other forms of learning in its mechanism and neural implementation; it is often assumed to compete with simpler mechanisms, such as reward-based associative learning, to drive behaviour. Recently, neural signals have been observed during social exchange reminiscent of signals seen in studies of associative learning. Here we demonstrate that social information may be acquired using the same associative processes assumed to underlie reward-based learning. We find that key computational variables for learning in the social and reward domains are processed in a similar fashion, but in parallel neural processing streams. Two neighbouring divisions of the anterior cingulate cortex were central to learning about social and reward-based information, and for determining the extent to which each source of information guides behaviour. When making a decision, however, the information learnt using these parallel streams was combined within ventromedial prefrontal cortex. These findings suggest that human social valuation can be realized by means of the same associative processes previously established for learning other, simpler, features of the environment.

Separate neural pathways process different decision costs

Behavioral ecologists and economists emphasize that potential costs, as well as rewards, influence decision making. Although neuroscientists assume that frontal areas are central to decision making, the evidence is contradictory and the critical region remains unclear. Here it is shown that frontal lobe contributions to cost-benefit decision making can be understood by positing the existence of two independent systems that make decisions about delay and effort costs. Anterior cingulate cortex lesions affected how much effort rats decided to invest for rewards. Orbitofrontal cortical lesions affected how long rats decided to wait for rewards. The pattern of disruption suggested the deficit could be related to impaired associative learning. Impairments of the two systems may underlie apathetic and impulsive choice patterns in neurological and psychiatric illnesses. Although the existence of two systems is not predicted by economic accounts of decision making, our results suggest that delay and effort may exert distinct influences on decision making.

Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour.

There is general acknowledgement that both the anterior cingulate and orbitofrontal cortex are implicated in reinforcement-guided decision making, and emotion and social behaviour. Despite the interest that these areas generate in both the cognitive neuroscience laboratory and the psychiatric clinic, ideas about the distinctive contributions made by each have only recently begun to emerge. This reflects an increasing understanding of the component processes that underlie reinforcement-guided decision making, such as the representation of reinforcement expectations, the exploration, updating and representation of action values, and the appreciation that choices are guided not just by the prospect of reward but also by the costs that action entails. Evidence is emerging to suggest that the anterior cingulate and orbitofrontal cortex make distinct contributions to each of these aspects of decision making.

How Green Is the Grass on the Other Side? Frontopolar Cortex and the Evidence in Favor of Alternative Courses of Action

Behavioral flexibility is the hallmark of goal-directed behavior. Whereas a great deal is known about the neural substrates of behavioral adjustment when it is explicitly cued by features of the external environment, little is known about how we adapt our behavior when such changes are made on the basis of uncertain evidence. Using a Bayesian reinforcement-learning model and fMRI, we show that frontopolar cortex (FPC) tracks the relative advantage in favor of switching to a foregone alternative when choices are made voluntarily. Changes in FPC functional connectivity occur when subjects finally decide to switch to the alternative behavior. Moreover, interindividual variation in the FPC signal predicts interindividual differences in effectively adapting behavior. By contrast, ventromedial prefrontal cortex (vmPFC) encodes the relative value of the current decision. Collectively, these findings reveal complementary prefrontal computations essential for promoting short- and long-term behavioral flexibility.