Wako Yoshida

Resolution of Uncertainty in Prefrontal Cortex

Making optimal decisions in the face of uncertain or incomplete information arises as a common problem in everyday behavior, but the neural processes underlying this ability remain poorly understood. A typical case is navigation, in which a subject has to search for a known goal from an unknown location. Navigating under uncertain conditions requires making decisions on the basis of the current belief about location and updating that belief based on incoming information. Here, we use functional magnetic resonance imaging during a maze navigation task to study neural activity relating to the resolution of uncertainty as subjects make sequential decisions to reach a goal. We show that distinct regions of prefrontal cortex are engaged in specific computational functions that are well described by a Bayesian model of decision making. This permits efficient goal-oriented navigation and provides new insights into decision making by humans.

Choosing to make an effort: the role of striatum in signaling physical effort of a chosen action.

The possibility that we will have to invest effort influences our future choice behavior. Indeed deciding whether an action is actually worth taking is a key element in the expression of human apathy or inertia. There is a well developed literature on brain activity related to the anticipation of effort, but how effort affects actual choice is less well understood. Furthermore, prior work is largely restricted to mental as opposed to physical effort or has confounded temporal with effortful costs. Here we investigated choice behavior and brain activity, using functional magnetic resonance imaging, in a study where healthy participants are required to make decisions between effortful gripping, where the factors of force (high and low) and reward (high and low) were varied, and a choice of merely holding a grip device for minimal monetary reward. Behaviorally, we show that force level influences the likelihood of choosing an effortful grip. We observed greater activity in the putamen when participants opt to grip an option with low effort compared with when they opt to grip an option with high effort. The results suggest that, over and above a nonspecific role in movement anticipation and salience, the putamen plays a crucial role in computations for choice that involves effort costs.

Game theory of mind

This paper introduces a model of ‘theory of mind’, namely, how we represent the intentions and goals of others to optimise our mutual interactions. We draw on ideas from optimum control and game theory to provide a ‘game theory of mind’. First, we consider the representations of goals in terms of value functions that are prescribed by utility or rewards. Critically, the joint value functions and ensuing behaviour are optimised recursively, under the assumption that I represent your value function, your representation of mine, your representation of my representation of yours, and so on ad infinitum. However, if we assume that the degree of recursion is bounded, then players need to estimate the opponents degree of recursion (i.e., sophistication) to respond optimally. This induces a problem of inferring the opponents sophistication, given behavioural exchanges. We show it is possible to deduce whether players make inferences about each other and quantify their sophistication on the basis of choices in sequential games. This rests on comparing generative models of choices with, and without, inference. Model comparison is demonstrated using simulated and real data from a ‘stag-hunt’. Finally, we note that exactly the same sophisticated behaviour can be achieved by optimising the utility function itself (through prosocial utility), producing unsophisticated but apparently altruistic agents. This may be relevant ethologically in hierarchal game theory and coevolution.

Uncertainty Increases Pain: Evidence for a Novel Mechanism of Pain Modulation Involving the Periaqueductal Gray

Predictions about sensory input exert a dominant effect on what we perceive, and this is particularly true for the experience of pain. However, it remains unclear what component of prediction, from an information-theoretic perspective, controls this effect. We used a vicarious pain observation paradigm to study how the underlying statistics of predictive information modulate experience. Subjects observed judgments that a group of people made to a painful thermal stimulus, before receiving the same stimulus themselves. We show that the mean observed rating exerted a strong assimilative effect on subjective pain. In addition, we show that observed uncertainty had a specific and potent hyperalgesic effect. Using computational functional magnetic resonance imaging, we found that this effect correlated with activity in the periaqueductal gray. Our results provide evidence for a novel form of cognitive hyperalgesia relating to perceptual uncertainty, induced here by vicarious observation, with control mediated by the brainstem pain modulatory system.

Fear reduction without fear through reinforcement of neural activity that bypasses conscious exposure

Fear conditioning is a fundamentally important and preserved process across species1,2. In humans it is linked to fear-related disorders such as phobias and post-traumatic stress disorder (PTSD)3,4. Fear memories can be reduced by counter-conditioning, in which fear conditioned stimuli (CS+s) are repeatedly reinforced with reward5 or with novel non-threatening stimuli6. However, this procedure involves explicit presentations of CS+s, which is itself aversive before fear is successfully reduced. This aversiveness may be a problem when trying to translate such experimental paradigms into clinical settings7. It also raises the fundamental question as to whether explicit presentations of feared objects is necessary for fear reduction1,8. Although learning without explicit stimulus presentation has been previously demonstrated9–12, whether fear can be reduced while avoiding explicit exposure to CS+s remains largely unknown. One recently developed approach employs an implicit method to induce learning by reinforcing stimulus-specific neural representations using real-time decoding of multivariate functional magnetic resonance imaging (fMRI) signals13–15 in the absence of stimulus presentation; that is, pairing rewards with the occurrences of multi-voxel brain activity patterns matching a specific stimulus (decoded fMRI neurofeedback (DecNef)13,15). It has been shown that participants exhibit perceptual learning for a specific visual stimulus feature through DecNef, without being given any strategy for the induction of specific neural representations, and without awareness of the content of reinforced neural representations13. Here we examined whether a similar approach could be applied to counter-conditioning of fear. We show that we can reduce fear towards CS+s by pairing rewards with the activation patterns in visual cortex representing a CS+, while participants remain unaware of the content and purpose of the procedure. This procedure may be an initial step towards novel treatments for fear-related disorders such as phobia and PTSD, via unconscious processing.

Hierarchical rule switching in prefrontal cortex.

Most real-world decision-making problems involve consideration of numerous possible actions, and it is often impossible to evaluate all of them before settling on preferred strategy. In such situations, humans might explore actions more efficiently by searching only the most likely subspace of the whole action space. To study how the brain solves such action selection problems, we designed a Multi Feature Sorting Task in which the task rules defining an optimal action have a hierarchical structure and studied concurrent brain activity using it. The task consisted of two kinds of rule switches: a higher-order switch to search for a rule across different subspaces and a lower-order switch to change a rule within the same subspace. The results revealed that the left dorsolateral prefrontal cortex (DLPFC) was more active in the higher-order switching, and the right fronto-polar cortex (FPC) was significantly activated with the lower-order switching. We discuss a possible functional model in the prefrontal cortex where the left DLPFC encodes the hierarchical organization of behaviours and the right FPC maintains and updates multiple behavioural. This interpretation is highly consistent with the previous findings and current theories of hierarchical organization in the prefrontal functional network.

Model-based reinforcement learning: a computational model and an fMRI study

In this paper, we discuss an optimal decision-making problem in an unknown environment on the bases of both machine learning and brain learning. We present a model-based reinforcement learning (RL) in which the environment is directly estimated. Our RL performs action selection according to the detection of environmental changes and the current value function. In a partially-observable situation, in which the environment includes unobservable state variables, our RL incorporates estimation of unobservable variables. We propose a possible functional model of our RL, focusing on the prefrontal cortex and the anterior cingulate cortex. To test the model, we conducted a human imaging study during a sequential learning task, and found significant activations in the dorsolateral prefrontal cortex and the anterior cingulate cortex during RL. From a comparison of the mean activations in the earlier and later learning phases, we suggest that the dorsolateral prefrontal cortex maintains and manipulates the environmental model, while the anterior cingulate cortex is related to the uncertainty of action selection. These experimental results are consistent with our model.

Reconstruction of chaotic dynamics using a noise-robust embedding method

In this article, we discuss the reconstruction of chaotic dynamics in a partial observation situation. As a function approximator, we employ a normalized Gaussian network (NGnet), which is trained by an on-line EM algorithm. In order to deal with the partial observation, we propose a new embedding method based on smoothing filters, which is called integral embedding. The NGnet is trained to learn the dynamical system in the integral coordinate space. Experimental results show that the trained NGnet is able to reproduce a chaotic attractor that well approximates the complexity and instability of the original chaotic attractor, even when the data involve large noise. In comparison with our previous method using delay coordinate embedding, this new method is more robust to noise and faster in learning.

Thermosensory perceptual learning is associated with structural brain changes in parietal-opercular (SII) cortex

The location of a sensory cortex for temperature perception remains a topic of substantial debate. Both the parietal–opercular (SII) and posterior insula have been consistently implicated in thermosensory processing, but neither region has yet been identified as the locus of fine temperature discrimination. Using a perceptual learning paradigm in male and female humans, we show improvement in discrimination accuracy for subdegree changes in both warmth and cool detection over 5 d of repetitive training. We found that increases in discriminative accuracy were specific to the temperature (cold or warm) being trained. Using structural imaging to look for plastic changes associated with perceptual learning, we identified symmetrical increases in gray matter volume in the SII cortex. Furthermore, we observed distinct, adjacent regions for cold and warm discrimination, with cold discrimination having a more anterior locus than warm. The results suggest that thermosensory discrimination is supported by functionally and anatomically distinct temperature-specific modules in the SII cortex. SIGNIFICANCE STATEMENT We provide behavioral and neuroanatomical evidence that perceptual learning is possible within the temperature system. We show that structural plasticity localizes to parietal–opercular (SII), and not posterior insula, providing the best evidence to date resolving a longstanding debate about the location of putative “temperature cortex.” Furthermore, we show that cold and warm pathways are behaviorally and anatomically dissociable, suggesting that the temperature system has distinct temperature-dependent processing modules.

The control of tonic pain by active relief learning

Tonic pain after injury characterises a behavioural state that prioritises recovery. Although generally suppressing cognition and attention, tonic pain needs to allow effective relief learning to reduce the cause of the pain. Here, we describe a central learning circuit that supports learning of relief and concurrently suppresses the level of ongoing pain. We used computational modelling of behavioural, physiological and neuroimaging data in two experiments in which subjects learned to terminate tonic pain in static and dynamic escape-learning paradigms. In both studies, we show that active relief-seeking involves a reinforcement learning process manifest by error signals observed in the dorsal putamen. Critically, this system uses an uncertainty (‘associability’) signal detected in pregenual anterior cingulate cortex that both controls the relief learning rate, and endogenously and parametrically modulates the level of tonic pain. The results define a self-organising learning circuit that reduces ongoing pain when learning about potential relief.