William R. Stauffer

Dopamine prediction error responses integrate subjective value from different reward dimensions

Significance Most real-world rewards have multiple dimensions, such as amount, risk, and type. Here we show that within a bounded set of such multidimensional rewards monkeys’ choice behavior fulfilled several core tenets of rational choice theory; namely, their choices were stochastically complete and transitive. As such, in selecting between rewards, the monkeys behaved as if they maximized value on a common scale. Dopamine neurons encoded prediction errors that reflected that scale. A particular reward dimension influenced dopamine activity only to the extent that it influenced choice. Thus, vastly different reward types such as juice and food activated dopamine neurons in accordance with subjective value derived from the different rewards. This neuronal signal could serve to update value signals for economic choice. Prediction error signals enable us to learn through experience. These experiences include economic choices between different rewards that vary along multiple dimensions. Therefore, an ideal way to reinforce economic choice is to encode a prediction error that reflects the subjective value integrated across these reward dimensions. Previous studies demonstrated that dopamine prediction error responses reflect the value of singular reward attributes that include magnitude, probability, and delay. Obviously, preferences between rewards that vary along one dimension are completely determined by the manipulated variable. However, it is unknown whether dopamine prediction error responses reflect the subjective value integrated from different reward dimensions. Here, we measured the preferences between rewards that varied along multiple dimensions, and as such could not be ranked according to objective metrics. Monkeys chose between rewards that differed in amount, risk, and type. Because their choices were complete and transitive, the monkeys chose “as if” they integrated different rewards and attributes into a common scale of value. The prediction error responses of single dopamine neurons reflected the integrated subjective value inferred from the choices, rather than the singular reward attributes. Specifically, amount, risk, and reward type modulated dopamine responses exactly to the extent that they influenced economic choices, even when rewards were vastly different, such as liquid and food. This prediction error response could provide a direct updating signal for economic values.

Dopamine Reward Prediction Error Responses Reflect Marginal Utility

Summary Background Optimal choices require an accurate neuronal representation of economic value. In economics, utility functions are mathematical representations of subjective value that can be constructed from choices under risk. Utility usually exhibits a nonlinear relationship to physical reward value that corresponds to risk attitudes and reflects the increasing or decreasing marginal utility obtained with each additional unit of reward. Accordingly, neuronal reward responses coding utility should robustly reflect this nonlinearity. Results In two monkeys, we measured utility as a function of physical reward value from meaningful choices under risk (that adhered to first- and second-order stochastic dominance). The resulting nonlinear utility functions predicted the certainty equivalents for new gambles, indicating that the functions’ shapes were meaningful. The monkeys were risk seeking (convex utility function) for low reward and risk avoiding (concave utility function) with higher amounts. Critically, the dopamine prediction error responses at the time of reward itself reflected the nonlinear utility functions measured at the time of choices. In particular, the reward response magnitude depended on the first derivative of the utility function and thus reflected the marginal utility. Furthermore, dopamine responses recorded outside of the task reflected the marginal utility of unpredicted reward. Accordingly, these responses were sufficient to train reinforcement learning models to predict the behaviorally defined expected utility of gambles. Conclusions These data suggest a neuronal manifestation of marginal utility in dopamine neurons and indicate a common neuronal basis for fundamental explanatory constructs in animal learning theory (prediction error) and economic decision theory (marginal utility).

Surface immobilization of neural adhesion molecule L1 for improving the biocompatibility of chronic neural probes: In vitro characterization

Silicon-based implantable neural electrode arrays are known to experience failure during long-term recording, partially due to host tissue responses. Surface modification and immobilization of biomolecules may provide a means to improve their biocompatibility and integration within the host brain tissue. Previously, the laminin biomolecule or laminin fragments have been used to modify the neural probes silicon surface to promote neuronal attachment and growth. Here we report the successful immobilization of the L1 biomolecule on a silicon surface. L1 is a neuronal adhesion molecule that can specifically promote neurite outgrowth and neuronal survival. Silane chemistry and the heterobifunctional coupling agent 4-maleimidobutyric acid N-hydroxysuccinimide ester (GMBS) were used to covalently bind these two biomolecules onto the surface of silicon dioxide wafers, which mimic the surface of silicon-based implantable neural probes. After covalent binding of the biomolecules, polyethylene glycol (PEG)-NH(2) was used to cap the unreacted GMBS groups. Surface immobilization was verified by goniometry, dual polarization interferometry, and immunostaining techniques. Primary murine neurons or astrocytes were used to evaluate the modified silicon surfaces. Both L1- and laminin-modified surfaces promoted neuronal attachment, while the L1-modified surface demonstrated significantly enhanced levels of neurite outgrowth (p<0.05). In addition, the laminin-modified surface promoted astrocyte attachment, while the L1-modified surface showed significantly reduced levels of astrocyte attachment relative to the laminin-modified surface and other controls (p<0.05). These results demonstrate the ability of the L1-immobilized surface to specifically promote neuronal growth and neurite extension, while inhibiting the attachment of astrocytes, one of the main cellular components of the glial sheath. Such unique properties present vast potentials to improve the biocompatibility and chronic recording performance of neural probes.

Dopamine Neuron-Specific Optogenetic Stimulation in Rhesus Macaques

Summary Optogenetic studies in mice have revealed new relationships between well-defined neurons and brain functions. However, there are currently no means to achieve the same cell-type specificity in monkeys, which possess an expanded behavioral repertoire and closer anatomical homology to humans. Here, we present a resource for cell-type-specific channelrhodopsin expression in Rhesus monkeys and apply this technique to modulate dopamine activity and monkey choice behavior. These data show that two viral vectors label dopamine neurons with greater than 95% specificity. Infected neurons were activated by light pulses, indicating functional expression. The addition of optical stimulation to reward outcomes promoted the learning of reward-predicting stimuli at the neuronal and behavioral level. Together, these results demonstrate the feasibility of effective and selective stimulation of dopamine neurons in non-human primates and a resource that could be applied to other cell types in the monkey brain.

Enhanced Differentiation of Embryonic and Neural Stem Cells to Neuronal Fates on Laminin Peptides Doped Polypyrrole

PPy is a conducting polymer material that has been widely investigated for biomedical applications. hESCs and adult rNSCs were grown on four PPy surfaces doped with PSS or peptide from laminin (p20, p31, and a mixture of p20 and p31) respectively. After 7 d, both PPy/p20 and PPy/p31 promoted neuroectoderm formation from hESCs. After 14 d of culture, surfaces containing p20 showed the highest percentage of neuronal differentiation from hESC, while the PPy/p31 surface showed better cell attachment and spreading. In rNSCs cultures, a higher percentage of neurons were found on the PPy/p20 surface than other surfaces at 7 and 14 d. For differentiated neurons, p20 promoted both the primary and total neurite outgrowth. Longer primary neurites were found on p20-containing surfaces and a longer total neurite length was found on PPy/p20 surface. These results demonstrated that, by doping PPy with different bioactive peptides, differentiation of stem cells seeded at different stages of development is affected.

Economic choices reveal probability distortion in macaque monkeys.

Economic choices are largely determined by two principal elements, reward value (utility) and probability. Although nonlinear utility functions have been acknowledged for centuries, nonlinear probability weighting (probability distortion) was only recently recognized as a ubiquitous aspect of real-world choice behavior. Even when outcome probabilities are known and acknowledged, human decision makers often overweight low probability outcomes and underweight high probability outcomes. Whereas recent studies measured utility functions and their corresponding neural correlates in monkeys, it is not known whether monkeys distort probability in a manner similar to humans. Therefore, we investigated economic choices in macaque monkeys for evidence of probability distortion. We trained two monkeys to predict reward from probabilistic gambles with constant outcome values (0.5 ml or nothing). The probability of winning was conveyed using explicit visual cues (sector stimuli). Choices between the gambles revealed that the monkeys used the explicit probability information to make meaningful decisions. Using these cues, we measured probability distortion from choices between the gambles and safe rewards. Parametric modeling of the choices revealed classic probability weighting functions with inverted-S shape. Therefore, the animals overweighted low probability rewards and underweighted high probability rewards. Empirical investigation of the behavior verified that the choices were best explained by a combination of nonlinear value and nonlinear probability distortion. Together, these results suggest that probability distortion may reflect evolutionarily preserved neuronal processing.

Dopamine neurons learn relative chosen value from probabilistic rewards

Economic theories posit reward probability as one of the factors defining reward value. Individuals learn the value of cues that predict probabilistic rewards from experienced reward frequencies. Building on the notion that responses of dopamine neurons increase with reward probability and expected value, we asked how dopamine neurons in monkeys acquire this value signal that may represent an economic decision variable. We found in a Pavlovian learning task that reward probability-dependent value signals arose from experienced reward frequencies. We then assessed neuronal response acquisition during choices among probabilistic rewards. Here, dopamine responses became sensitive to the value of both chosen and unchosen options. Both experiments showed also the novelty responses of dopamine neurones that decreased as learning advanced. These results show that dopamine neurons acquire predictive value signals from the frequency of experienced rewards. This flexible and fast signal reflects a specific decision variable and could update neuronal decision mechanisms. DOI: http://dx.doi.org/10.7554/eLife.18044.001

Components and characteristics of the dopamine reward utility signal.

Rewards are defined by their behavioral functions in learning (positive reinforcement), approach behavior, economic choices, and emotions. Dopamine neurons respond to rewards with two components, similar to higher order sensory and cognitive neurons. The initial, rapid, unselective dopamine detection component reports all salient environmental events irrespective of their reward association. It is highly sensitive to factors related to reward and thus detects a maximal number of potential rewards. It also senses aversive stimuli but reports their physical impact rather than their aversiveness. The second response component processes reward value accurately and starts early enough to prevent confusion with unrewarded stimuli and objects. It codes reward value as a numeric, quantitative utility prediction error, consistent with formal concepts of economic decision theory. Thus, the dopamine reward signal is fast, highly sensitive and appropriate for driving and updating economic decisions. J. Comp. Neurol. 524:1699–1711, 2016.

Rapid modulation of local neural activity by controlled drug release from polymer-coated recording microelectrodes

We demonstrate targeted perturbation of neuronal activity with controlled release of neurochemicals from conducting polymer-coated microelectrodes. Polymer coating and chemical incorporation are achieved through individually addressable electrodeposition, a process that does not compromise the recording capabilities of the electrodes. Release is realized by the application of brief voltage pulses that electrochemically reduce the polymer and dissociate incorporated neurochemicals; whereby they can diffuse away and achieve locally effective concentrations. Inhibition of evoked synaptic currents in neurons within 200 µm of a 6-cyano-7-nitroquinoxaline-2,3-dione releasing electrode lasts for several seconds. Spiking activity of neurons in local circuits recorded extracellularly near the releasing electrode is silenced for a similar duration following release. This methodology is compatible with many neuromodulatory chemicals and various recording electrodes, including in vitro and implantable neural electrode arrays, thus providing an inexpensive and accessible technique capable of achieving sophisticated patterned chemical modulation of neuronal circuits.

Utility functions predict variance and skewness risk preferences in monkeys

Significance Utility, the key decision variable underlying economic choices, should represent risk, which is inherent to real-life decisions. We studied two prevalent forms of risk that are characterized by the spread (variance-risk) and asymmetry (skewness-risk) of rewards. We show that monkeys preferred higher variance and positively skewed gambles. Importantly, empirically estimated utility functions predicted both of these risk preferences. Thus, the abstract concept of utility seemed to explain primates’ choices under common forms of risk. Utility is the fundamental variable thought to underlie economic choices. In particular, utility functions are believed to reflect preferences toward risk, a key decision variable in many real-life situations. To assess the validity of utility representations, it is therefore important to examine risk preferences. In turn, this approach requires formal definitions of risk. A standard approach is to focus on the variance of reward distributions (variance-risk). In this study, we also examined a form of risk related to the skewness of reward distributions (skewness-risk). Thus, we tested the extent to which empirically derived utility functions predicted preferences for variance-risk and skewness-risk in macaques. The expected utilities calculated for various symmetrical and skewed gambles served to define formally the direction of stochastic dominance between gambles. In direct choices, the animals’ preferences followed both second-order (variance) and third-order (skewness) stochastic dominance. Specifically, for gambles with different variance but identical expected values (EVs), the monkeys preferred high-variance gambles at low EVs and low-variance gambles at high EVs; in gambles with different skewness but identical EVs and variances, the animals preferred positively over symmetrical and negatively skewed gambles in a strongly transitive fashion. Thus, the utility functions predicted the animals’ preferences for variance-risk and skewness-risk. Using these well-defined forms of risk, this study shows that monkeys’ choices conform to the internal reward valuations suggested by their utility functions. This result implies a representation of utility in monkeys that accounts for both variance-risk and skewness-risk preferences.