Lusha Zhu

Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning

Decision-making in the presence of other competitive intelligent agents is fundamental for social and economic behavior. Such decisions require agents to behave strategically, where in addition to learning about the rewards and punishments available in the environment, they also need to anticipate and respond to actions of others competing for the same rewards. However, whereas we know much about strategic learning at both theoretical and behavioral levels, we know relatively little about the underlying neural mechanisms. Here, we show using a multi-strategy competitive learning paradigm that strategic choices can be characterized by extending the reinforcement learning (RL) framework to incorporate agents’ beliefs about the actions of their opponents. Furthermore, using this characterization to generate putative internal values, we used model-based functional magnetic resonance imaging to investigate neural computations underlying strategic learning. We found that the distinct notions of prediction errors derived from our computational model are processed in a partially overlapping but distinct set of brain regions. Specifically, we found that the RL prediction error was correlated with activity in the ventral striatum. In contrast, activity in the ventral striatum, as well as the rostral anterior cingulate (rACC), was correlated with a previously uncharacterized belief-based prediction error. Furthermore, activity in rACC reflected individual differences in degree of engagement in belief learning. These results suggest a model of strategic behavior where learning arises from interaction of dissociable reinforcement and belief-based inputs.

Damage To Dorsolateral Prefrontal Cortex Affects Tradeoffs Between Honesty And Self-Interest

Substantial correlational evidence suggests that prefrontal regions are critical to honest and dishonest behavior, but causal evidence specifying the nature of this involvement remains absent. We found that lesions of the human dorsolateral prefrontal cortex (DLPFC) decreased the effect of honesty concerns on behavior in economic games that pit honesty motives against self-interest, but did not affect decisions when honesty concerns were absent. These results point to a causal role for DLPFC in honest behavior.

Dopamine Modulates Egalitarian Behavior in Humans

Egalitarian motives form a powerful force in promoting prosocial behavior and enabling large-scale cooperation in the human species [1]. At the neural level, there is substantial, albeit correlational, evidence suggesting a link between dopamine and such behavior [2, 3]. However, important questions remain about the specific role of dopamine in setting or modulating behavioral sensitivity to prosocial concerns. Here, using a combination of pharmacological tools and economic games, we provide critical evidence for a causal involvement of dopamine in human egalitarian tendencies. Specifically, using the brain penetrant catechol-O-methyl transferase (COMT) inhibitor tolcapone [4, 5], we investigated the causal relationship between dopaminergic mechanisms and two prosocial concerns at the core of a number of widely used economic games: (1) the extent to which individuals directly value the material payoffs of others, i.e., generosity, and (2) the extent to which they are averse to differences between their own payoffs and those of others, i.e., inequity. We found that dopaminergic augmentation via COMT inhibition increased egalitarian tendencies in participants who played an extended version of the dictator game [6]. Strikingly, computational modeling of choice behavior [7] revealed that tolcapone exerted selective effects on inequity aversion, and not on other computational components such as the extent to which individuals directly value the material payoffs of others. Together, these data shed light on the causal relationship between neurochemical systems and human prosocial behavior and have potential implications for our understanding of the complex array of social impairments accompanying neuropsychiatric disorders involving dopaminergic dysregulation.

Dissociable contribution of prefrontal and striatal dopaminergic genes to learning in economic games

Significance Game theory is used throughout the social and biological sciences to study behavior in social interactions. Recent research suggests an important role for the dopamine neurotransmitter system in these types of decisions. This study used a competitive game to study how people varied in their decision-making processes and related these differences in the set of genes that carry out biological functions required for dopaminergic functioning. We found that genes differentially expressed in separate brain regions influenced distinct components of people’s decision-making processes and that a surprising degree of consistency exists with what is known at the brain level about how people make decisions in social interactions. Game theory describes strategic interactions where success of players’ actions depends on those of coplayers. In humans, substantial progress has been made at the neural level in characterizing the dopaminergic and frontostriatal mechanisms mediating such behavior. Here we combined computational modeling of strategic learning with a pathway approach to characterize association of strategic behavior with variations in the dopamine pathway. Specifically, using gene-set analysis, we systematically examined contribution of different dopamine genes to variation in a multistrategy competitive game captured by (i) the degree players anticipate and respond to actions of others (belief learning) and (ii) the speed with which such adaptations take place (learning rate). We found that variation in genes that primarily regulate prefrontal dopamine clearance—catechol-O-methyl transferase (COMT) and two isoforms of monoamine oxidase—modulated degree of belief learning across individuals. In contrast, we did not find significant association for other genes in the dopamine pathway. Furthermore, variation in genes that primarily regulate striatal dopamine function—dopamine transporter and D2 receptors—was significantly associated with the learning rate. We found that this was also the case with COMT, but not for other dopaminergic genes. Together, these findings highlight dissociable roles of frontostriatal systems in strategic learning and support the notion that genetic variation, organized along specific pathways, forms an important source of variation in complex phenotypes such as strategic behavior.

Neuroeconomic Measures of Social Decision-Making Across the Lifespan

Social and decision-making deficits are often the first symptoms of a striking number of neurodegenerative disorders associated with aging. These includes not only disorders that directly impact dopamine and basal ganglia, such as Parkinson’s disorder, but also degeneration in which multiple neural pathways are affected over the course of normal aging. The impact of such deficits can be dramatic, as in cases of financial fraud, which disproportionately affect the elderly. Unlike memory and motor impairments, however, which are readily recognized as symptoms of more serious underlying neurological conditions, social and decision-making deficits often do not elicit comparable concern in the elderly. Furthermore, few behavioral measures exist to quantify these deficits, due in part to our limited knowledge of the core cognitive components or their neurobiological substrates. Here we probe age-related differences in decision-making using a game theory paradigm previously shown to dissociate contributions of basal ganglia and prefrontal regions to behavior. Combined with computational modeling, we provide evidence that age-related changes in elderly participants are driven primarily by an over-reliance in trial-and-error reinforcement learning that does not take into account the strategic context, which may underlie cognitive deficits that contribute to social vulnerability in elderly individuals.

Associability-modulated loss learning is increased in posttraumatic stress disorder

Disproportionate reactions to unexpected stimuli in the environment are a cardinal symptom of posttraumatic stress disorder (PTSD). Here, we test whether these heightened responses are associated with disruptions in distinct components of reinforcement learning. Specifically, using functional neuroimaging, a loss-learning task, and a computational model-based approach, we assessed the mechanistic hypothesis that overreactions to stimuli in PTSD arise from anomalous gating of attention during learning (i.e., associability). Behavioral choices of combat-deployed veterans with and without PTSD were fit to a reinforcement learning model, generating trial-by-trial prediction errors (signaling unexpected outcomes) and associability values (signaling attention allocation to the unexpected outcomes). Neural substrates of associability value and behavioral parameter estimates of associability updating, but not prediction error, increased with PTSD during loss learning. Moreover, the interaction of PTSD severity with neural markers of associability value predicted behavioral choices. These results indicate that increased attention-based learning may underlie aspects of PTSD and suggest potential neuromechanistic treatment targets.

Spatial gradient in activity within the insula reflects dissociable neural mechanisms underlying context-dependent advantageous and disadvantageous inequity aversion

Humans are capable of integrating social contextual information into decision-making processes to adjust their attitudes towards inequity. This context-dependency emerges both when individual is better off (i.e. advantageous inequity) and worse off (i.e. disadvantageous inequity) than others. It is not clear however, whether the context-dependent processing of advantageous and disadvantageous inequity rely on dissociable or shared neural mechanisms. Here, by combining an interpersonal interactive game that gave rise to interpersonal guilt and different versions of the dictator games that enabled us to characterize individual weights on aversion to advantageous and disadvantageous inequity, we investigated the neural mechanisms underlying the two forms of inequity aversion in the interpersonal guilt context. In each round, participants played a dot-estimation task with an anonymous co-player. The co-players received pain stimulation with 50% probability when anyone responded incorrectly. At the end of each round, participants completed a dictator game, which determined payoffs of him/herself and the co-player. Both computational model-based and model-free analyses demonstrated that when inflicting pain upon co-players (i.e., the guilt context), participants cared more about advantageous inequity and became less sensitive to disadvantageous inequity, compared with other social contexts. The contextual effects on two forms of inequity aversion are uncorrelated with each other at the behavioral level. Neuroimaging results revealed that the context-dependent representation of inequity aversion exhibited a spatial gradient in activity within the insula, with anterior parts predominantly involved in the aversion to advantageous inequity and posterior parts predominantly involved in the aversion to disadvantageous inequity. The dissociable mechanisms underlying the two forms of inequity aversion are further supported by the involvement of right dorsolateral prefrontal cortex and dorsomedial prefrontal cortex in advantageous inequity processing, and the involvement of right amygdala and dorsal anterior cingulate cortex in disadvantageous inequity processing. These results extended our understanding of decision-making processes involving inequity and the social functions of inequity aversion.

Distinguishing neural correlates of context-dependent advantageous- and disadvantageous-inequity aversion

Significance Despite extensive research on disadvantageous inequity, little is known about advantageous inequity and whether these two types of inequity involve differential neurocognitive mechanisms. We address these questions from the perspective of context dependency and suggest that these two types of inequity are associated with differential neurocognitive substrates, subserved by different brain regions and in particular by the spatial gradient in insular activity. Our findings shed light on how social contexts (i.e., interpersonal guilt) are integrated into social decision making and suggest that the resistance to unequal situations when individuals are in disadvantageous status may primarily stem from their emotional responses, whereas the resistance to unequal situations when individuals are in advantageous status may involve advanced cognitive functions such as mentalizing. Humans can integrate social contextual information into decision-making processes to adjust their responses toward inequity. This context dependency emerges when individuals receive more (i.e., advantageous inequity) or less (i.e., disadvantageous inequity) than others. However, it is not clear whether context-dependent processing of advantageous and disadvantageous inequity involves differential neurocognitive mechanisms. Here, we used fMRI to address this question by combining an interactive game that modulates social contexts (e.g., interpersonal guilt) with computational models that enable us to characterize individual weights on inequity aversion. In each round, the participant played a dot estimation task with an anonymous coplayer. The coplayer would receive pain stimulation with 50% probability when either of them responded incorrectly. At the end of each round, the participant completed a variant of dictator game, which determined payoffs for him/herself and the coplayer. Computational modeling demonstrated the context dependency of inequity aversion: when causing pain to the coplayer (i.e., guilt context), participants cared more about the advantageous inequity and became more tolerant of the disadvantageous inequity, compared with other conditions. Consistently, neuroimaging results suggested the two types of inequity were associated with differential neurocognitive substrates. While the context-dependent processing of advantageous inequity was associated with social- and mentalizing-related processes, involving left anterior insula, right dorsolateral prefrontal cortex, and dorsomedial prefrontal cortex, the context-dependent processing of disadvantageous inequity was primarily associated with emotion- and conflict-related processes, involving left posterior insula, right amygdala, and dorsal anterior cingulate cortex. These results extend our understanding of decision-making processes related to inequity aversion.

Predicting human behavior toward members of different social groups

Significance Societal disparities appear in domains including education, healthcare, and the labor market, and stereotypes have been widely hypothesized to play a role in these disparities. However, a mechanistic understanding of how stereotypes influence decision making has largely eluded prevailing models. By integrating economic and psychological approaches, we offer a computational framework providing robust explanatory and predictive power for treatment disparities. This framework generates psychological insights into the nature and force of stereotypes’ influence on behavior and generalizes from behavior in the laboratory to successfully predict naturalistic behavior in the field. Together, these findings show how societally shared assumptions about social groups can produce and reinforce societal disparities, opening the door to a common, quantitative framework to advance scientific understanding of discrimination. Disparities in outcomes across social groups pervade human societies and are of central interest to the social sciences. How people treat others is known to depend on a multitude of factors (e.g., others’ gender, ethnicity, appearance) even when these should be irrelevant. However, despite substantial progress, much remains unknown regarding (i) the set of mechanisms shaping people’s behavior toward members of different social groups and (ii) the extent to which these mechanisms can explain the structure of existing societal disparities. Here, we show in a set of experiments the important interplay between social perception and social valuation processes in explaining how people treat members of different social groups. Building on the idea that stereotypes can be organized onto basic, underlying dimensions, we first found using laboratory economic games that quantitative variation in stereotypes about different groups’ warmth and competence translated meaningfully into resource allocation behavior toward those groups. Computational modeling further revealed that these effects operated via the interaction of social perception and social valuation processes, with warmth and competence exerting diverging effects on participants’ preferences for equitable distributions of resources. This framework successfully predicted behavior toward members of a diverse set of social groups across samples and successfully generalized to predict societal disparities documented in labor and education settings with substantial precision and accuracy. Together, these results highlight a common set of mechanisms linking social group information to social treatment and show how preexisting, societally shared assumptions about different social groups can produce and reinforce societal disparities.

Cognitive neuroscience of honesty and deception: a signaling framework

Understanding the neural basis of human honesty and deception has enormous potential scientific and practical value. However, past approaches, largely developed out of studies with forensic applications in mind, are increasingly recognized as having serious methodological and conceptual shortcomings. Here we propose to address these challenges by drawing on so-called signaling games widely used in game theory and ethology to study behavioral and evolutionary consequences of information transmission and distortion. In particular, by separating and capturing distinct adaptive problems facing signal senders and receivers, signaling games provide a framework to organize the complex set of cognitive processes associated with honest and deceptive behavior. Furthermore, this framework provides novel insights into feasibility and practical challenges of neuroimaging-based lie detection.