Differentiating effort-related aspects of motivation from reinforcement learning: commentary on Soder et al. “Dose–response effects of d-amphetamine on effort-based decision-making and reinforcement learning”

Abstract

One of the most intensive areas of research and debate in neuropsychopharmacology over the last few decades has been the study of the behavioral functions of brain dopamine (DA) systems. Drugs that alter DA transmission such as DA transport (DAT) inhibitors or DA receptor antagonists are known to affect aspects of appetitive motivation and instrumental behavior. Nevertheless, researchers are still grappling with the details of this involvement, such as the specific aspects of motivation being regulated, and also are trying to determine the conceptual framework that is most useful for understanding this complex area of study. In contemplating this issue, it is vital to consider some of the contributions that the field of neuropsychology has provided to psychopharmacology research. It is important to recognize that global functional descriptors such as “reward”, “reinforcement”, “motivation”, and “motor control” are blunt instruments; these are global and complex processes that can be parsed into several distinct and dissociable aspects. Classical brain manipulations such as lesions or drugs, as well as more recently developed techniques such as optogenetics or chemogenetics, can dissociate these specific aspects from each other by demonstrating that it is possible to substantially alter one function while leaving another largely or completely intact. The Soder et al. [1] paper in this issue provides a compelling example of the role that neuropsychopharmacology research can play in dissociating aspects of appetitive motivation in humans. These studies were conducted to test the effects of different doses of d-amphetamine that are in the therapeutic range for adult ADHD (i.e., placebo, 10 and 20mg) on effort-related decision making and reinforcement learning in the same group of healthy human volunteers. In order to assess effort-related aspects of motivation, participants were tested on the Effort Expenditure for Reward Task (EEfRT) task, which is a procedure developed by Treadway et al. in 2009. The EEfRT task measures effort-related decision-making by offering participants the choice between more difficult and vigorous finger tapping to receive a larger monetary reward vs. less effortful finger tapping to receive less reward. Thus, it provides a measure of effort selection based upon the vigorous repetition of a response, though not specifically on force output. This procedure has been used in previous studies of the effort-related motivational effects of d-amphetamine, as well as assessments of these processes in people with depression, schizophrenia, and other psychiatric disorders (for review see ref. [2]). In addition, the same participants were tested on the Probabilistic Reward Task to provide a measure of reinforcement learning [3]. With this task, cartoon face images were presented, and discrimination of the lengths of facial features was differentially reinforced on a probabilistic basis, after which the results were analyzed by a signal detection approach that detected bias towards the more reinforced category. The authors also examined whether individual differences in baseline working memory and willingness to exert effort for monetary reward acted as moderators of the effects of d-amphetamine. Consistent with previous studies, damphetamine increased willingness to exert effort, especially at lowto-moderate expected levels of monetary reinforcement. Computational modeling analyses demonstrated that baseline effort expenditure and working memory emerged as moderators of the effect of d-amphetamine on exertion of effort. d-amphetamine increased selection of high-effort choices to a greater extent in people with lower working memory performance and lower baseline effort expenditure, and also had greater effects when expected values of reinforcement were moderate-to-low. In contrast to these clear effects on selection of high effort activity, the same doses of d-amphetamine had no significant effect on reinforcement learning as assessed by the Probabilistic Reward Task. These results indicate that the effects of d-amphetamine on selection of high-effort activity are not dependent upon actions of the drug on reinforcement learning processes per se, and that these two aspects of motivational and cognitive function can be dissociated. Moreover, they are consistent with the idea that the enhancement of DA transmission induced by d-amphetamine at the doses used is not having some broad or general effect on all aspects of “reward” processes, but rather, is producing differential actions on distinct aspects of appetitive motivation [2]. The Soder et al. results are consistent with rodent studies of effort-based choice involving genetic or pharmacological manipulations of the DAT. Knockdown of the DAT in mice enhanced selection of high effort lever pressing but did not affect reward learning [4]. In rats tested on effort-based choice lever pressing tasks, administration of selective DAT inhibitors consistently increases selection of high effort progressive ratio responding [5, 6], in contrast to drugs that block the norepinephrine and serotonin transporters [5]. Effort-based choice tasks in rodents are increasingly being used to model motivational dysfunctions in psychopathology, and develop potential drug treatments for