Kevin Miller

Unconscious neural processing differs with method used to render stimuli invisible

Visual stimuli can be kept from awareness using various methods. The extent of processing that a given stimulus receives in the absence of awareness is typically used to make claims about the role of consciousness more generally. The neural processing elicited by a stimulus, however, may also depend on the method used to keep it from awareness, and not only on whether the stimulus reaches awareness. Here we report that the method used to render an image invisible has a dramatic effect on how category information about the unseen stimulus is encoded across the human brain. We collected fMRI data while subjects viewed images of faces and tools, that were rendered invisible using either continuous flash suppression (CFS) or chromatic flicker fusion (CFF). In a third condition, we presented the same images under normal fully visible viewing conditions. We found that category information about visible images could be extracted from patterns of fMRI responses throughout areas of neocortex known to be involved in face or tool processing. However, category information about stimuli kept from awareness using CFS could be recovered exclusively within occipital cortex, whereas information about stimuli kept from awareness using CFF was also decodable within temporal and frontal regions. We conclude that unconsciously presented objects are processed differently depending on how they are rendered subjectively invisible. Caution should therefore be used in making generalizations on the basis of any one method about the neural basis of consciousness or the extent of information processing without consciousness.

Dorsal hippocampus contributes to model-based planning

Planning can be defined as action selection that leverages an internal model of the outcomes likely to follow each possible action. Its neural mechanisms remain poorly understood. Here we adapt recent advances from human research for rats, presenting for the first time an animal task that produces many trials of planned behavior per session, making multitrial rodent experimental tools available to study planning. We use part of this toolkit to address a perennially controversial issue in planning: the role of the dorsal hippocampus. Although prospective hippocampal representations have been proposed to support planning, intact planning in animals with damaged hippocampi has been repeatedly observed. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide causal evidence directly linking dorsal hippocampus with planning behavior. Our results and methods open the door to new and more detailed investigations of the neural mechanisms of planning in the hippocampus and throughout the brain.

Value Representations in Orbitofrontal Cortex Drive Learning, but not Choice

Humans and animals make predictions about the rewards they expect to receive in different situations. In formal models of behavior, these predictions are known as value representations, and they play two very different roles. Firstly, they drive choice: the expected values of available options are compared to one another, and the best option is selected. Secondly, they support learning: expected values are compared to rewards actually received, and future expectations are updated accordingly. Whether these different functions are mediated by different neural representations remains an open question. Here we employ a recently-developed multi-step task for rats that computationally separates learning from choosing. We investigate the role of value representations in the rodent orbitofrontal cortex, a key structure for value-based cognition. Electrophysiological recordings and optogenetic perturbations indicate that these representations do not directly drive choice. Instead, they signal expected reward information to a learning process elsewhere in the brain that updates choice mechanisms.

Dorsal hippocampus plays a causal role in model-based planning

Planning can be defined as a process of action selection that leverages an internal model of the environment. Such models provide information about the likely outcomes that will follow each selected action, and their use is a key function underlying complex adaptive behavior. However, the neural mechanisms supporting this ability remain poorly understood. In the present work, we adapt for rodents recent advances from work on human planning, presenting for the first time a task for animals which produces many trials of planned behavior per session, allowing the experimental toolkit available for use in trial-by-trial tasks for rodents to be applied to the study of planning. We take advantage of one part of this toolkit to address a perennially controversial issue in planning research: the role of the dorsal hippocampus. Although prospective representations in the hippocampus have been proposed to support model-based planning, intact planning in hippocampally damaged animals has been observed in a number of assays. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide the first causal evidence directly linking dorsal hippocampus with planning behavior. The results reported, and the methods introduced, open the door to new and more detailed investigations of the neural mechanisms of planning, in the hippocampus and throughout the brain.Abstract Planning can be defined as a process of action selection that leverages an internal model of the environment. Such models provide information about the likely outcomes that will follow each selected action, and their use is a key function underlying complex adaptive behavior. However, the neural mechanisms supporting this ability remain poorly understood. In the present work, we adapt for rodents recent advances from work on human planning, presenting for the first time a task for animals which produces many trials of planned behavior per session, allowing the experimental toolkit available for use in trial-by-trial tasks for rodents to be applied to the study of planning. We take advantage of one part of this toolkit to address a perennially controversial issue in planning research: the role of the dorsal hippocampus. Although prospective representations in the hippocampus have been proposed to support model-based planning, intact planning in hippocampally damaged animals has been observed in a number of assays. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide the first causal evidence directly linking dorsal hippocampus with planning behavior. The results reported, and the methods introduced, open the door to new and more detailed investigations of the neural mechanisms of planning, in the hippocampus and throughout the brain.

Identifying Model-Based and Model-Free Patterns in Behavior on Multi-Step Tasks

Recent years have seen a surge of research into the neuroscience of planning. Much of this work has taken advantage of a two-step sequential decision task developed by Daw et al. (2011), which gives the ability to diagnose whether or not subjects’ behavior is the result of planning. Here, we present simulations which suggest that the techniques most commonly used to analyze data from this task may be confounded in important ways. We introduce a new analysis technique, which suffers from fewer of these issues. This technique also presents a richer view of behavior, making it useful for characterizing patterns in behavior in a theory-neutral manner. This allows it to provide an important check on the assumptions of more theory-driven analysis such as agent-based model-fitting.

Habits without Values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning (RL) mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally.

Re-aligning models of habitual and goal-directed decision-making

Abstract The classic dichotomy between habitual and goal-directed behavior is often mapped onto a dichotomy between model-free and model-based reinforcement learning (RL) algorithms, putatively implemented in segregated neuronal circuits. Despite significant heuristic value in motivating experimental investigations, several lines of evidence suggest that this mapping is in need of modification and/or realignment. First, whereas habitual and goal-directed behaviors have been shown to depend on cleanly separable neural circuitry, recent data suggest that model-based and model-free representations in the brain are largely overlapping. Second, habitual behaviors need not involve representations of expected reinforcement (i.e., need not involve RL, model-free, or otherwise) but may be based instead on simple stimulus–response associations. Finally, goal-directed decisions may not reflect a single model-based algorithm but rather a continuum of “model-basedness.” These lines of evidence thus suggest a possible reconceptualization of the distinction between model-free versus model-based RL—one in which both contribute to a single goal-directed system that is value-based, as opposed to distinct, habitual mechanisms that are value-free. In this chapter, we discuss new models that have extended the RL approach to modeling habitual and goal-directed behavior and assess how these have clarified our understanding of the underlying neural circuitry.

Walking bundles of habits (and response–outcome associations) (Commentary on Liljeholm et al.)

Distinct behaviors imply distinct neural processes. Goal-directed and habitual control have been repeatedly shown to have starkly different behavioral signatures and neural substrates. The distinction critically hinges on the necessary underlying representations: stimulus–response (S–R) associations subserve habits, while response–outcome (R–O) associations allow flexible, goal-directed control. But another separation remains relatively underexplored: that between acquisition and expression. Are the structures that learn S–R and R–O representations also responsible for acting on them? Lesion studies in animal models have fractionated cortical and striatal subregions based on separable contributions to learning and performance (Ostlund & Balleine, 2005; Atallah et al., 2007). However, while these studies offer temporal specificity and causal manipulation, they have necessarily been restricted to select brain regions. Studies employing lesions offer temporal specificity and causal manipulation, and have been conducted in many brain regions (for a review, see Balleine & O’Doherty, 2010), but suffer from logical limitations: the finding that a lesion disables expression does not exclude a structure’s role in acquisition. Previous human functional magnetic resonance imaging (fMRI) studies have examined goal-directed and habitual control systems across the whole brain, but were not designed to examine the acquisition/ expression distinction (Tricomi et al., 2009; Gl€ascher et al., 2010; Liljeholm et al., 2012). A new study by Liljeholm et al. (2015) attempts to fill this gap by studying human behavior and neural activity in a novel behavioral task while undergoing fMRI. Their task addresses a key confound in the typical practice of studying habit acquisition through overtraining (extensive experience with a given S–R pairing): overtraining conflates acquisition with the performance improvements that come with practice. Instead, the conditions in this experiment were designed to produce habits or goal-directed behaviors with minimal training. As expected based on the previous literature, following training habits proved less sensitive to a change in outcome contingencies in the task, referred to as devaluation. This design allowed the authors to distinguish neural correlates of S–R/R–O representations ‘early’ vs. ‘late’ in their development (putatively capturing acquisition vs. expression). Using this task, the authors were able to replicate and extend many previous findings. Specifically, correlates of habit acquisition were found in the posterior caudate and cerebellum; whereas the former is to be expected, the latter is rarely reported in these types of studies. Similarly, correlates of habit expression in the ventral striatum and subgenual anterior cingulate cortex (ACC) both affirmed established knowledge (in the former case) and offered tantalizing clues to new functional relationships (in the latter case). Ventral striatal involvement accords with previous reports of its centrality to habitual control, but the role of subgenual ACC in this process remains a matter of much debate. Here, the authors suggest it as a functional homolog to rodent infralimbic cortex, a structure that has been implicated in both learning and expression of habits (Smith et al., 2012; Smith & Graybiel, 2013). Indeed, the question of cross-species homology remains a persistent source of tension in the instrumental control literature. The authors found that putamen activity decreased across the experiment, interpreted as suggesting a role in acquisition rather than expression. This appears to conflict with reports from perturbations of the rodent homolog, dorsolateral striatum (DLS; Furlong et al., 2014). Both results are, however, consistent with the observation that DLS representations ‘sharpen’ with training (Smith & Graybiel, 2013) – and thus fMRI measures of population activity might decrease. An important question for future research is whether divergent results across species arise from interpretational nuances of different methodologies, or actual differences in the functional anatomy. As this design is novel in the field, further work is needed to confirm that it faithfully distinguishes habitual and goal-directed behavior. One potential concern is that the current task may generate these behaviors by biasing attention towards or away from devaluation-relevant stimuli rather than by distinctly training S–R vs. R–O associations. Moreover, unlike classical manifestations of habitual behavior in the devaluation literature, habits in the current study were not necessarily maladaptive: giving the incorrect (devalued) response required completing a three-button sequence, sequences that were begun but rarely completed when subjects performed a (devaluation-insensitive) habit. Whether these features had bearing on the desired distinction between habits and goal-directed actions can be tested in future studies, for instance by using shorter response sequences and decoupling the focus of attention from the devalued target. Nevertheless, that these conditions distinguished behaviors and brain networks largely along expected lines should be seen as promising, if not yet conclusive, evidence that the manipulation was successful. Finally, though this study begins to clarify a previously muddled distinction, it also hints at new divisions yet to be substantiated. In the computational reinforcement learning literature, the question arises whether goal-directed and habit systems operate largely in parallel, or