Matthew M. Walsh

Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice

To behave adaptively, we must learn from the consequences of our actions. Studies using event-related potentials (ERPs) have been informative with respect to the question of how such learning occurs. These studies have revealed a frontocentral negativity termed the feedback-related negativity (FRN) that appears after negative feedback. According to one prominent theory, the FRN tracks the difference between the values of actual and expected outcomes, or reward prediction errors. As such, the FRN provides a tool for studying reward valuation and decision making. We begin this review by examining the neural significance of the FRN. We then examine its functional significance. To understand the cognitive processes that occur when the FRN is generated, we explore variables that influence its appearance and amplitude. Specifically, we evaluate four hypotheses: (1) the FRN encodes a quantitative reward prediction error; (2) the FRN is evoked by outcomes and by stimuli that predict outcomes; (3) the FRN and behavior change with experience; and (4) the system that produces the FRN is maximally engaged by volitional actions.

Modulation of the feedback-related negativity by instruction and experience

A great deal of research focuses on how humans and animals learn from trial-and-error interactions with the environment. This research has established the viability of reinforcement learning as a model of behavioral adaptation and neural reward valuation. Error-driven learning is inefficient and dangerous, however. Fortunately, humans learn from nonexperiential sources of information as well. In the present study, we focused on one such form of information, instruction. We recorded event-related potentials as participants performed a probabilistic learning task. In one experiment condition, participants received feedback only about whether their responses were rewarded. In the other condition, they also received instruction about reward probabilities before performing the task. We found that instruction eliminated participants’ reliance on feedback as evidenced by their immediate asymptotic performance in the instruction condition. In striking contrast, the feedback-related negativity, an event-related potential component thought to reflect neural reward prediction error, continued to adapt with experience in both conditions. These results show that, whereas instruction may immediately control behavior, certain neural responses must be learned from experience.

Learning from delayed feedback: neural responses in temporal credit assignment

When feedback follows a sequence of decisions, relationships between actions and outcomes can be difficult to learn. We used event-related potentials (ERPs) to understand how people overcome this temporal credit assignment problem. Participants performed a sequential decision task that required two decisions on each trial. The first decision led to an intermediate state that was predictive of the trial outcome, and the second decision was followed by positive or negative trial feedback. The feedback-related negativity (fERN), a component thought to reflect reward prediction error, followed negative feedback and negative intermediate states. This suggests that participants evaluated intermediate states in terms of expected future reward, and that these evaluations supported learning of earlier actions within sequences. We examine the predictions of several temporal-difference models to determine whether the behavioral and ERP results reflected a reinforcement-learning process.

Navigating complex decision spaces: Problems and paradigms in sequential choice

To behave adaptively, we must learn from the consequences of our actions. Doing so is difficult when the consequences of an action follow a delay. This introduces the problem of temporal credit assignment. When feedback follows a sequence of decisions, how should the individual assign credit to the intermediate actions that comprise the sequence? Research in reinforcement learning provides 2 general solutions to this problem: model-free reinforcement learning and model-based reinforcement learning. In this review, we examine connections between stimulus-response and cognitive learning theories, habitual and goal-directed control, and model-free and model-based reinforcement learning. We then consider a range of problems related to temporal credit assignment. These include second-order conditioning and secondary reinforcers, latent learning and detour behavior, partially observable Markov decision processes, actions with distributed outcomes, and hierarchical learning. We ask whether humans and animals, when faced with these problems, behave in a manner consistent with reinforcement learning techniques. Throughout, we seek to identify neural substrates of model-free and model-based reinforcement learning. The former class of techniques is understood in terms of the neurotransmitter dopamine and its effects in the basal ganglia. The latter is understood in terms of a distributed network of regions including the prefrontal cortex, medial temporal lobes, cerebellum, and basal ganglia. Not only do reinforcement learning techniques have a natural interpretation in terms of human and animal behavior but they also provide a useful framework for understanding neural reward valuation and action selection.

The Discovery of Processing Stages: Extension of Sternberg's Method

We introduce a method for measuring the number and durations of processing stages from the electroencephalographic signal and apply it to the study of associative recognition. Using an extension of past research that combines multivariate pattern analysis with hidden semi-Markov models, the approach identifies on a trial-by-trial basis where brief sinusoidal peaks (called bumps) are added to the ongoing electroencephalographic signal. We propose that these bumps mark the onset of critical cognitive stages in processing. The results of the analysis can be used to guide the development of detailed process models. Applied to the associative recognition task, the hidden semi-Markov models multivariate pattern analysis method indicates that the effects of associative strength and probe type are localized to a memory retrieval stage and a decision stage. This is in line with a previously developed the adaptive control of thought-rational process model, called ACT-R, of the task. As a test of the generalization of our method we also apply it to a data set on the Sternberg working memory task collected by Jacobs, Hwang, Curran, and Kahana (2006). The analysis generalizes robustly, and localizes the typical set size effect in a late comparison/decision stage. In addition to providing information about the number and durations of stages in associative recognition, our analysis sheds light on the event-related potential components implicated in the study of recognition memory. (PsycINFO Database Record

ERP profiles for face and word recognition are based on their status in semantic memory not their stimulus category

Previous research has suggested that faces and words are processed and remembered differently as reflected by different ERP patterns for the two types of stimuli. Specifically, face stimuli produced greater late positive deflections for old items in anterior compared to posterior regions, while word stimuli produced greater late positive deflections in posterior compared to anterior regions. Given that words have existing representations in subjects׳ long-term memories (LTM) and that face stimuli used in prior experiments were of unknown individuals, we conducted an ERP study that crossed face and letter stimuli with the presence or absence of a prior (stable or existing) memory representation. During encoding, subjects judged whether stimuli were known (famous face or real word) or not known (unknown person or pseudo-word). A surprise recognition memory test required subjects to distinguish between stimuli that appeared during the encoding phase and stimuli that did not. ERP results were consistent with previous research when comparing unknown faces and words; however, the late ERP pattern for famous faces was more similar to that for words than for unknown faces. This suggests that the critical ERP difference is mediated by whether there is a prior representation in LTM, and not whether the stimulus involves letters or faces.

Stages of processing in associative recognition: Evidence from behavior, eeg, and classification

In this study, we investigated the stages of information processing in associative recognition. We recorded EEG data while participants performed an associative recognition task that involved manipulations of word length, associative fan, and probe type, which were hypothesized to affect the perceptual encoding, retrieval, and decision stages of the recognition task, respectively. Analyses of the behavioral and EEG data, supplemented with classification of the EEG data using machine-learning techniques, provided evidence that generally supported the sequence of stages assumed by a computational model developed in the Adaptive Control of Thought-Rational cognitive architecture. However, the results suggested a more complex relationship between memory retrieval and decision-making than assumed by the model. Implications of the results for modeling associative recognition are discussed. The study illustrates how a classifier approach, in combination with focused manipulations, can be used to investigate the timing of processing stages.

The effects of probe similarity on retrieval and comparison processes in associative recognition

In this study, we investigated the information processing stages underlying associative recognition. We recorded EEG data while participants performed a task that involved deciding whether a probe word triple matched any previously studied triple. We varied the similarity between probes and studied triples. According to a model of associative recognition developed in the Adaptive Control of Thought-Rational cognitive architecture, probe similarity affects the duration of the retrieval stage: Retrieval is fastest when the probe is similar to a studied triple. This effect may be obscured, however, by the duration of the comparison stage, which is fastest when the probe is not similar to the retrieved triple. Owing to the opposing effects of probe similarity on retrieval and comparison, overall RTs provide little information about each stages duration. As such, we evaluated the model using a novel approach that decomposes the EEG signal into a sequence of latent states and provides information about the durations of the underlying information processing stages. The approach uses a hidden semi-Markov model to identify brief sinusoidal peaks (called bumps) that mark the onsets of distinct cognitive stages. The analysis confirmed that probe type has opposite effects on retrieval and comparison stages.

Deciding How to Act Is Not Achieved by Watching Mental Movies.

In the early days of research on visual imagery, it was believed that visual images are like pictures in ones head. Only as the field matured did it come to be appreciated that visual images do not bear a first-order isomorphic relation to visual percepts. Now that the early days of research on motor imagery are coming to an end, it is important to ask whether motor images bear a first-order isomorphic relation to movements. We asked whether they do by focusing on internal simulations for motor planning. Our participants indicated which of two possible actions they preferred either by performing the preferred action or by indicating which action they would prefer to perform. We reasoned that if internal simulations of physical actions bear a first-order isomorphic relation to actual physical actions, the choices would be the same in the two conditions. They were not. We discuss the reasons for this outcome, including the adaptive advantage of a representational system for action which, like the representational system for vision, does not bear a first-order isomorphic relation to its external analog.

Relationship of P3b single-trial latencies and response times in one, two, and three-stimulus oddball tasks.

The P300 is one of the most widely studied components of the human event-related potential. According to a longstanding view, the P300, and particularly its posterior subcomponent (i.e., the P3b), is driven by stimulus categorization. Whether the P3b relates to tactical processes involved in immediate responding or strategic processes that affect future behavior remains controversial, however. It is difficult to determine whether variability in P3b latencies relates to variability in response times because of limitations in the methods currently available to quantify the latency of the P3b during single trials. In this paper, we report results from the Psychomotor Vigilance Task (PVT), the Hitchcock Radar Task, and a 3-Stimulus Oddball Task. These represent variants of the one-, two-, and three-stimulus oddball paradigms commonly used to study the P3b. The PVT requires simple detection, whereas the Hitchcock Radar Task and the 3-Stimulus Task require detection and categorization. We apply a novel technique that combines hidden semi-Markov models and multi-voxel pattern analysis (HSMM-MVPA) to data from the three experiments. HSMM-MVPA revealed a processing stage in each task corresponding to the P3b. Trial-by-trial variability in the latency of the processing stage correlated with response times in the Hitchcock Radar Task and the 3-Stimulus Task, but not the PVT. These results indicate that the P3b reflects a stimulus categorization process, and that its latency is strongly associated with response times when the stimulus must be categorized before responding. In addition to those theoretical insights, the ability to detect the onset of the P3b and other components on a single-trial basis using HSMM-MVPA opens the door for new uses of mental chronometry in cognitive neuroscience.