Mimi Liljeholm

The integrative function of the basal ganglia in instrumental conditioning

Recent research in instrumental conditioning has focused on the striatum, particularly the role of the dorsal striatum in the learning processes that contribute to instrumental performance in rats. This research has found evidence of what appear to be parallel, functionally and anatomically distinct circuits involving dorsomedial striatum (DMS) and dorsolateral striatum (DLS) that contribute to two independent instrumental learning processes. Evidence suggests that the formation of the critical action-outcome associations mediating goal-directed action are localized to the dorsomedial striatum, whereas the sensorimotor connections that control the performance of habitual actions are localized to the dorsolateral striatum. In addition to the dorsal striatum, these learning processes appear to engage distinct cortico-striatal networks and to be embedded in a complex of converging and partially segregated loops that constitute the cortico-striatal thalamo-cortical feedback circuit. As the entry point for the basal ganglia, cortical circuits involving the dorsal striatum are clearly in a position to control a variety of motor functions but, as recent studies of various neurodegenerative disorders have made clear, they are also involved in a number of cognitive and executive functions including action selection, planning, and decision-making.

Contributions of the striatum to learning, motivation, and performance: an associative account

It has long been recognized that the striatum is composed of distinct functional sub-units that are part of multiple cortico-striatal-thalamic circuits. Contemporary research has focused on the contribution of striatal sub-regions to three main phenomena: learning of associations between stimuli, actions and rewards; selection between competing response alternatives; and motivational modulation of motor behavior. Recent proposals have argued for a functional division of the striatum along these lines, attributing, for example, learning to one region and performance to another. Here, we consider empirical data from human and animal studies, as well as theoretical notions from both the psychological and computational literatures, and conclude that striatal sub-regions instead differ most clearly in terms of the associations being encoded in each region.

Bayesian Generic Priors for Causal Learning

The article presents a Bayesian model of causal learning that incorporates generic priors--systematic assumptions about abstract properties of a system of cause-effect relations. The proposed generic priors for causal learning favor sparse and strong (SS) causes--causes that are few in number and high in their individual powers to produce or prevent effects. The SS power model couples these generic priors with a causal generating function based on the assumption that unobservable causal influences on an effect operate independently (P. W. Cheng, 1997). The authors tested this and other Bayesian models, as well as leading nonnormative models, by fitting multiple data sets in which several parameters were varied parametrically across multiple types of judgments. The SS power model accounted for data concerning judgments of both causal strength and causal structure (whether a causal link exists). The model explains why human judgments of causal structure (relative to a Bayesian model lacking these generic priors) are influenced more by causal power and the base rate of the effect and less by sample size. Broader implications of the Bayesian framework for human learning are discussed.

Neural Correlates of Specific and General Pavlovian-to-Instrumental Transfer within Human Amygdalar Subregions: A High-Resolution fMRI Study

It is widely held that the interaction between instrumental and Pavlovian conditioning induces powerful motivational biases. Pavlovian-Instrumental Transfer (PIT) is one of the key paradigms demonstrating this effect, which can further be decomposed into a general and specific component. Although these two forms of PIT have been studied at the level of amygdalar subregions in rodents, it is still unknown whether they involve different areas of the human amygdala. Using a high-resolution fMRI (hr-fMRI) protocol optimized for the amygdala in combination with a novel free operant task designed to elicit effects of both general and specific PIT, we demonstrate that a region of ventral amygdala within the boundaries of the basolateral complex and the ventrolateral putamen are involved in specific PIT, while a region of dorsal amygdala within the boundaries of the centromedial complex is involved in general PIT. These results add to a burgeoning literature indicating different functional contributions for these different amygdalar subregions in reward-processing and motivation.

Neural Correlates of Instrumental Contingency Learning: Differential Effects of Action–Reward Conjunction and Disjunction

Contingency theories of goal-directed action propose that experienced disjunctions between an action and its specific consequences, as well as conjunctions between these events, contribute to encoding the action–outcome association. Although considerable behavioral research in rats and humans has provided evidence for this proposal, relatively little is known about the neural processes that contribute to the two components of the contingency calculation. Specifically, while recent findings suggest that the influence of action–outcome conjunctions on goal-directed learning is mediated by a circuit involving ventromedial prefrontal, medial orbitofrontal cortex, and dorsomedial striatum, the neural processes that mediate the influence of experienced disjunctions between these events are unknown. Here we show differential responses to probabilities of conjunctive and disjunctive reward deliveries in the ventromedial prefrontal cortex, the dorsomedial striatum, and the inferior frontal gyrus. Importantly, activity in the inferior parietal lobule and the left middle frontal gyrus varied with a formal integration of the two reward probabilities, ΔP, as did response rates and explicit judgments of the causal efficacy of the action.

Characterizing the Associative Content of Brain Structures Involved in Habitual and Goal-Directed Actions in Humans: A Multivariate fMRI Study

While there is accumulating evidence for the existence of distinct neural systems supporting goal-directed and habitual action selection in the mammalian brain, much less is known about the nature of the information being processed in these different brain regions. Associative learning theory predicts that brain systems involved in habitual control, such as the dorsolateral striatum, should contain stimulus and response information only, but not outcome information, while regions involved in goal-directed action, such as ventromedial and dorsolateral prefrontal cortex and dorsomedial striatum, should be involved in processing information about outcomes as well as stimuli and responses. To test this prediction, human participants underwent fMRI while engaging in a binary choice task designed to enable the separate identification of these different representations with a multivariate classification analysis approach. Consistent with our predictions, the dorsolateral striatum contained information about responses but not outcomes at the time of an initial stimulus, while the regions implicated in goal-directed action selection contained information about both responses and outcomes. These findings suggest that differential contributions of these regions to habitual and goal-directed behavioral control may depend in part on basic differences in the type of information that these regions have access to at the time of decision making.

When Is a Cause the “Same”?: Coherent Generalization Across Contexts

Two competing psychological approaches to causal learning make different predictions regarding what aspect of perceived causality is generalized across contexts. Two experiments tested these predictions. In one experiment, the task required a judgment regarding the existence of a simple causal relation; in the other, the task required a judgment regarding the existence of an interaction between a candidate cause and unobserved background causes. The task materials did not mention assessments of causal strength. Results indicate that causal power (Cartwright, 1989; Cheng, 1997) is the mental construct that people carry from one context to another.

Differentiating neural systems mediating the acquisition vs. expression of goal-directed and habitual behavioral control

Considerable behavioral data indicate that operant actions can become habitual, as demonstrated by insensitivity to changes in the action–outcome contingency and in subjective outcome values. Notably, although several studies have investigated the neural substrates of habits, none has clearly differentiated the areas of the human brain that support habit formation from those that implement habitual control. We scanned participants with functional magnetic resonance imaging as they learned and performed an operant task in which the conditional structure of the environment encouraged either goal‐directed encoding of the consequences of actions, or a habit‐like mapping of actions to antecedent cues. Participants were also scanned during a subsequent assessment of insensitivity to outcome devaluation. We identified dissociable roles of the cerebellum and ventral striatum, across learning and test performance, in behavioral insensitivity to outcome devaluation. We also showed that the inferior parietal lobule (an area previously implicated in several aspects of goal‐directed action selection, including the attribution of intent and awareness of agency) predicted sensitivity to outcome devaluation. Finally, we revealed a potential functional homology between the human subgenual cortex and rodent infralimbic cortex in the implementation of habitual control. In summary, our findings suggested a broad systems division, at the cortical and subcortical levels, between brain areas mediating the encoding and expression of action–outcome and stimulus–response associations.

Neural Correlates of the Divergence of Instrumental Probability Distributions

Flexible action selection requires knowledge about how alternative actions impact the environment: a “cognitive map” of instrumental contingencies. Reinforcement learning theories formalize this map as a set of stochastic relationships between actions and states, such that for any given action considered in a current state, a probability distribution is specified over possible outcome states. Here, we show that activity in the human inferior parietal lobule correlates with the divergence of such outcome distributions–a measure that reflects whether discrimination between alternative actions increases the controllability of the future–and, further, that this effect is dissociable from those of other information theoretic and motivational variables, such as outcome entropy, action values, and outcome utilities. Our results suggest that, although ultimately combined with reward estimates to generate action values, outcome probability distributions associated with alternative actions may be contrasted independently of valence computations, to narrow the scope of the action selection problem.

Stimulus salience and retrospective revaluation.

The effect of conditioning or extinguishing the more salient element of a previously reinforced compound on responding to the less salient element of that compound was assessed in rats. Experiment 1 established that the 2 elements making up an audiovisual compound differed significantly in salience. In Experiment 2A, compound conditioning was followed by either reinforcement or extinction of either the less or more salient element. On test, evidence of retrospective revaluation of the less salient element was found but not of the more salient element. In Experiment 2B, extinction of the more salient element was found to be more effective than its reinforcement in producing retrospective revaluation of the less salient element. The implications of these results are discussed.