Vivian V. Valentin

Determining the Neural Substrates of Goal-Directed Learning in the Human Brain

Instrumental conditioning is considered to involve at least two distinct learning systems: a goal-directed system that learns associations between responses and the incentive value of outcomes, and a habit system that learns associations between stimuli and responses without any link to the outcome that that response engendered. Lesion studies in rodents suggest that these two distinct components of instrumental conditioning may be mediated by anatomically distinct neural systems. The aim of the present study was to determine the neural substrates of the goal-directed component of instrumental learning in humans. Nineteen human subjects were scanned with functional magnetic resonance imaging while they learned to choose instrumental actions that were associated with the subsequent delivery of different food rewards (tomato juice, chocolate milk, and orange juice). After training, one of these foods was devalued by feeding the subject to satiety on that food. The subjects were then scanned again, while being re-exposed to the instrumental choice procedure (in extinction). We hypothesized that regions of the brain involved in goal-directed learning would show changes in their activity as a function of outcome devaluation. Our results indicate that neural activity in one brain region in particular, the orbitofrontal cortex, showed a strong modulation in its activity during selection of a devalued compared with a nondevalued action. These results suggest an important contribution of orbitofrontal cortex in guiding goal-directed instrumental choices in humans.

FROST: A Distributed Neurocomputational Model of Working Memory Maintenance

Many studies suggest that the sustained activation underlying working memory (WM) maintenance is mediated by a distributed network that includes the prefrontal cortex and other structures (e.g., posterior parietal cortex, thalamus, globus pallidus, and the caudate nucleus). A computational model of WM, called FROST (short for FROntal-Striatal-Thalamic), is proposed in which the representation of items and spatial positions is encoded in the lateral prefrontal cortex. During delay intervals, activation in these prefrontal cells is sustained via parallel, prefrontal cortical-thalamic loops. Activation reverberates in these loops because prefrontal cortical excitation of the head of the caudate nucleus leads to disinhibition of the thalamus (via the globus pallidus). FROST successfully accounts for a wide variety of WM data, including single-cell recording data and human behavioral data.

MULTIPLE SYSTEMS OF PERCEPTUAL CATEGORY LEARNING: THEORY AND COGNITIVE TESTS

The first neurobiologically detailed theory of multiple systems in category learning, called COVIS, was originally conceived in 1998. COVIS, which is now well established, postulates two systems that compete throughout learning—a frontal-based declarative system that uses logical reasoning and depends on working memory and executive attention, and a basal ganglia-mediated system that uses procedural-learning. The procedural system can learn a wide variety of category structures, but it learns in a slow incremental fashion and is highly dependent on reliable and immediate feedback. In contrast, the declarative rule-based (RB) system can learn a fairly small set of category structures quickly—specifically, those structures that can be learned via a logical reasoning process. These two systems learn simultaneously, but as long as RB strategies lead to successful performance, the declarative system inhibits the procedural system. This theory is described in detail and a variety of cognitive behavioral and cognitive neuroscience experiments are reviewed that test some parameter-free a priori predictions made by COVIS.

A computational model of the temporal dynamics of plasticity in procedural learning: sensitivity to feedback timing

The evidence is now good that different memory systems mediate the learning of different types of category structures. In particular, declarative memory dominates rule-based (RB) category learning and procedural memory dominates information-integration (II) category learning. For example, several studies have reported that feedback timing is critical for II category learning, but not for RB category learning—results that have broad support within the memory systems literature. Specifically, II category learning has been shown to be best with feedback delays of 500 ms compared to delays of 0 and 1000 ms, and highly impaired with delays of 2.5 s or longer. In contrast, RB learning is unaffected by any feedback delay up to 10 s. We propose a neurobiologically detailed theory of procedural learning that is sensitive to different feedback delays. The theory assumes that procedural learning is mediated by plasticity at cortical-striatal synapses that are modified by dopamine-mediated reinforcement learning. The model captures the time-course of the biochemical events in the striatum that cause synaptic plasticity, and thereby accounts for the empirical effects of various feedback delays on II category learning.

Differential effects of dopamine-directed treatments on cognition

Dopamine, a prominent neuromodulator, is implicated in many neuropsychiatric disorders. It has wide-ranging effects on both cortical and subcortical brain regions and on many types of cognitive tasks that rely on a variety of different learning and memory systems. As neuroscience and behavioral evidence for the existence of multiple memory systems and their corresponding neural networks accumulated, so did the notion that dopamine’s role is markedly different depending on which memory system is engaged. As a result, dopamine-directed treatments will have different effects on different types of cognitive behaviors. To predict what these effects will be, it is critical to understand: which memory system is mediating the behavior; the neural basis of the mediating memory system; the nature of the dopamine projections into that system; and the time course of dopamine after its release into the relevant brain regions. Consideration of these questions leads to different predictions for how changes in brain dopamine levels will affect automatic behaviors and behaviors mediated by declarative, procedural, and perceptual representation memory systems.

Chapter 25 – MULTIPLE SYSTEMS OF PERCEPTUAL CATEGORY LEARNING: THEORY AND COGNITIVE TESTS

The first neurobiologically detailed theory of multiple systems in category learning, called COVIS, was originally conceived in 1998. COVIS, which is now well established, postulates two systems that compete throughout learning—a frontal-based declarative system that uses logical reasoning and depends on working memory and executive attention, and a basal ganglia-mediated system that uses procedural-learning. The procedural system can learn a wide variety of category structures, but it learns in a slow incremental fashion and is highly dependent on reliable and immediate feedback. In contrast, the declarative rule-based (RB) system can learn a fairly small set of category structures quickly—specifically, those structures that can be learned via a logical reasoning process. These two systems learn simultaneously, but as long as RB strategies lead to successful performance, the declarative system inhibits the procedural system. This theory is described in detail and a variety of cognitive behavioral and cognitive neuroscience experiments are reviewed that test some parameter-free a priori predictions made by COVIS.

Dopamine dependence in aggregate feedback learning: A computational cognitive neuroscience approach

Procedural learning of skills depends on dopamine-mediated striatal plasticity. Most prior work investigated single stimulus-response procedural learning followed by feedback. However, many skills include several actions that must be performed before feedback is available. A new procedural-learning task is developed in which three independent and successive unsupervised categorization responses receive aggregate feedback indicating either that all three responses were correct, or at least one response was incorrect. Experiment 1 showed superior learning of stimuli in position 3, and that learning in the first two positions was initially compromised, and then recovered. An extensive theoretical analysis that used parameter space partitioning found that a large class of procedural-learning models, which predict propagation of dopamine release from feedback to stimuli, and/or an eligibility trace, fail to fully account for these data. The analysis also suggested that any dopamine released to the second or third stimulus impaired categorization learning in the first and second positions. A second experiment tested and confirmed a novel prediction of this large class of procedural-learning models that if the to-be-learned actions are introduced one-by-one in succession then learning is much better if training begins with the first action (and works forwards) than if it begins with the last action (and works backwards).