Dharshan Kumaran

Human-level control through deep reinforcement learning

The theory of reinforcement learning provides a normative account, deeply rooted in psychological and neuroscientific perspectives on animal behaviour, of how agents may optimize their control of an environment. To use reinforcement learning successfully in situations approaching real-world complexity, however, agents are confronted with a difficult task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. Remarkably, humans and other animals seem to solve this problem through a harmonious combination of reinforcement learning and hierarchical sensory processing systems, the former evidenced by a wealth of neural data revealing notable parallels between the phasic signals emitted by dopaminergic neurons and temporal difference reinforcement learning algorithms. While reinforcement learning agents have achieved some successes in a variety of domains, their applicability has previously been limited to domains in which useful features can be handcrafted, or to domains with fully observed, low-dimensional state spaces. Here we use recent advances in training deep neural networks to develop a novel artificial agent, termed a deep Q-network, that can learn successful policies directly from high-dimensional sensory inputs using end-to-end reinforcement learning. We tested this agent on the challenging domain of classic Atari 2600 games. We demonstrate that the deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester across a set of 49 games, using the same algorithm, network architecture and hyperparameters. This work bridges the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks.

Using Imagination to Understand the Neural Basis of Episodic Memory

Functional MRI (fMRI) studies investigating the neural basis of episodic memory recall, and the related task of thinking about plausible personal future events, have revealed a consistent network of associated brain regions. Surprisingly little, however, is understood about the contributions individual brain areas make to the overall recollective experience. To examine this, we used a novel fMRI paradigm in which subjects had to imagine fictitious experiences. In contrast to future thinking, this results in experiences that are not explicitly temporal in nature or as reliant on self-processing. By using previously imagined fictitious experiences as a comparison for episodic memories, we identified the neural basis of a key process engaged in common, namely scene construction, involving the generation, maintenance and visualization of complex spatial contexts. This was associated with activations in a distributed network, including hippocampus, parahippocampal gyrus, and retrosplenial cortex. Importantly, we disambiguated these common effects from episodic memory-specific responses in anterior medial prefrontal cortex, posterior cingulate cortex and precuneus. These latter regions may support self-schema and familiarity processes, and contribute to the brains ability to distinguish real from imaginary memories. We conclude that scene construction constitutes a common process underlying episodic memory and imagination of fictitious experiences, and suggest it may partially account for the similar brain networks implicated in navigation, episodic future thinking, and the default mode. We suggest that additional brain regions are co-opted into this core network in a task-specific manner to support functions such as episodic memory that may have additional requirements.

Tracking the Emergence of Conceptual Knowledge during Human Decision Making

Summary Concepts lie at the very heart of intelligence, providing organizing principles with which to comprehend the world. Surprisingly little, however, is understood about how we acquire and deploy concepts. Here, we show that a functionally coupled circuit involving the hippocampus and ventromedial prefrontal cortex (vMPFC) underpins the emergence of conceptual knowledge and its effect on choice behavior. Critically, the hippocampus alone supported the efficient transfer of knowledge to a perceptually novel setting. These findings provide compelling evidence that the hippocampus supports conceptual learning through the networking of discrete memories and reveal the nature of its interaction with downstream valuation modules such as the vMPFC. Our study offers neurobiological insights into the remarkable capacity of humans to discover the conceptual structure of related experiences and use this knowledge to solve exacting decision problems.

A Genetically Mediated Bias in Decision Making Driven by Failure of Amygdala Control

Genetic variation at the serotonin transporter-linked polymorphic region (5-HTTLPR) is associated with altered amygdala reactivity and lack of prefrontal regulatory control. Similar regions mediate decision-making biases driven by contextual cues and ambiguity, for example the “framing effect.” We hypothesized that individuals hemozygous for the short (s) allele at the 5-HTTLPR would be more susceptible to framing. Participants, selected as homozygous for either the long (la) or s allele, performed a decision-making task where they made choices between receiving an amount of money for certain and taking a gamble. A strong bias was evident toward choosing the certain option when the option was phrased in terms of gains and toward gambling when the decision was phrased in terms of losses (the frame effect). Critically, this bias was significantly greater in the ss group compared with the lala group. In simultaneously acquired functional magnetic resonance imaging data, the ss group showed greater amygdala during choices made in accord, compared with those made counter to the frame, an effect not seen in the lala group. These differences were also mirrored by differences in anterior cingulate–amygdala coupling between the genotype groups during decision making. Specifically, lala participants showed increased coupling during choices made counter to, relative to those made in accord with, the frame, with no such effect evident in ss participants. These data suggest that genetically mediated differences in prefrontal–amygdala interactions underpin interindividual differences in economic decision making.

Match–Mismatch Processes Underlie Human Hippocampal Responses to Associative Novelty

The hippocampus has long been proposed to play a critical role in novelty detection through its ability to act as a comparator between past and present experience. A recent study provided evidence for this hypothesis by characterizing hippocampal responses to sequence novelty, a type of associative novelty where familiar items appear in a new temporal order. Here, we ask whether a hippocampal match–mismatch (i.e., comparator) mechanism operates selectively to identify the violation of predictions within the temporal domain or instead also underlies the processing of associative novelty in other domains (e.g., spatial). We used functional magnetic resonance imaging and a repetition paradigm in which subjects viewed sequences of objects presented in distinct locations on the screen and performed an incidental target detection task. The left hippocampus exhibited a pattern of activity consistent with that of an associative match–mismatch detector, with novelty signals generated only in conditions where one contextual component was novel and the other repeated. In contrast, right hippocampal activation signaled the presence of objects in familiar locations. Our results suggest that hippocampal match–mismatch computations constitute a general mechanism underpinning the processing of associative novelty. These findings support a model in which hippocampal mismatch signals rely critically on the recall of previous experience, a process that only occurs when novel sensory inputs overlap significantly with stored representations. More generally, the current study also offers insights into how the hippocampus automatically represents the spatiotemporal context of our experiences, a function that may relate to its role in episodic memory.

Novelty signals: a window into hippocampal information processing

The function of the human hippocampus is a contentious subject among neuroscientists. Theoreticians have long viewed the hippocampus as a computational device, with researchers in humans increasingly adopting this perspective, buoyed by recent reports that its role is not limited to declarative memory. Here, we set out a new strategy for discovering the nature of information processing within the human hippocampus. We argue that novelty responses, measured by functional magnetic resonance imaging, provide a window into the neural representations and computations sustained by the hippocampus. More generally, we suggest that a renewed emphasis on the information processing qualities of the human hippocampus offers the promise of a long awaited union between theoretical and empirical research across species.

Overcoming catastrophic forgetting in neural networks

Significance Deep neural networks are currently the most successful machine-learning technique for solving a variety of tasks, including language translation, image classification, and image generation. One weakness of such models is that, unlike humans, they are unable to learn multiple tasks sequentially. In this work we propose a practical solution to train such models sequentially by protecting the weights important for previous tasks. This approach, inspired by synaptic consolidation in neuroscience, enables state of the art results on multiple reinforcement learning problems experienced sequentially. The ability to learn tasks in a sequential fashion is crucial to the development of artificial intelligence. Until now neural networks have not been capable of this and it has been widely thought that catastrophic forgetting is an inevitable feature of connectionist models. We show that it is possible to overcome this limitation and train networks that can maintain expertise on tasks that they have not experienced for a long time. Our approach remembers old tasks by selectively slowing down learning on the weights important for those tasks. We demonstrate our approach is scalable and effective by solving a set of classification tasks based on a hand-written digit dataset and by learning several Atari 2600 games sequentially.

The dynamics of hippocampal activation during encoding of overlapping sequences.

Sequence disambiguation, the process by which overlapping sequences are kept separate, has been proposed to underlie a wide range of memory capacities supported by the hippocampus, including episodic memory and spatial navigation. We used functional magnetic resonance imaging (fMRI) to explore the dynamic pattern of hippocampal activation during the encoding of sequences of faces. Activation in right posterior hippocampus, only during the encoding of overlapping sequences but not nonoverlapping sequences, was found to correlate robustly with a subject-specific behavioral index of sequence learning. Moreover, our data indicate that hippocampal activation in response to elements common to both sequences in the overlapping sequence pair, may be particularly important for accurate sequence encoding and retrieval. Together, these findings support the conclusion that the human hippocampus is involved in the earliest stage of sequence disambiguation, when memory representations are in the process of being created, and provide empirical support for contemporary computational models of hippocampal function.

What Learning Systems do Intelligent Agents Need? Complementary Learning Systems Theory Updated

We update complementary learning systems (CLS) theory, which holds that intelligent agents must possess two learning systems, instantiated in mammalians in neocortex and hippocampus. The first gradually acquires structured knowledge representations while the second quickly learns the specifics of individual experiences. We broaden the role of replay of hippocampal memories in the theory, noting that replay allows goal-dependent weighting of experience statistics. We also address recent challenges to the theory and extend it by showing that recurrent activation of hippocampal traces can support some forms of generalization and that neocortical learning can be rapid for information that is consistent with known structure. Finally, we note the relevance of the theory to the design of artificial intelligent agents, highlighting connections between neuroscience and machine learning.

The Emergence and Representation of Knowledge about Social and Nonsocial Hierarchies

Summary Primates are remarkably adept at ranking each other within social hierarchies, a capacity that is critical to successful group living. Surprisingly little, however, is understood about the neurobiology underlying this quintessential aspect of primate cognition. In our experiment, participants first acquired knowledge about a social and a nonsocial hierarchy and then used this information to guide investment decisions. We found that neural activity in the amygdala tracked the development of knowledge about a social, but not a nonsocial, hierarchy. Further, structural variations in amygdala gray matter volume accounted for interindividual differences in social transitivity performance. Finally, the amygdala expressed a neural signal selectively coding for social rank, whose robustness predicted the influence of rank on participants’ investment decisions. In contrast, we observed that the linear structure of both social and nonsocial hierarchies was represented at a neural level in the hippocampus. Our study implicates the amygdala in the emergence and representation of knowledge about social hierarchies and distinguishes the domain-general contribution of the hippocampus.