Adrian M. Haith

Model-based and model-free mechanisms of human motor learning.

Motor learning can be framed theoretically as a problem of optimizing a movement policy in a potentially uncertain or changing environment. This is precisely the general problem studied in the field of reinforcement learning. Reinforcement learning theory proposes two distinct approaches to solving this general problem: Model-based approaches first identify the dynamics of the task or environment then use this knowledge to compute the optimal movement policy. Model-free approaches, by contrast, directly identify successful policies through a process of trial and error. Here, we review existing literature on motor control in the light of this distinction. Motor learning research in the last decade has been dominated by studies that elicit learning through adaptation paradigms and find the results to be consistent with a model-based framework. Studying the behavior of patients in such adaptation paradigms has implicated the cerebellum as prime candidate for the neural substrate of the internal models that sub serve model-based control. A growing body of experimental results, however, demonstrates that not all of motor learning in conventional paradigms can be explained within model-based frameworks, but can be understood in terms of an additional component of learning driven by model-free reinforcement of successful actions. We conclude that the brain maintains distinct model-based and model-free learning systems, with distinct neural substrates, which act in competitive balance to direct behavior.

Contributions of the cerebellum and the motor cortex to acquisition and retention of motor memories

We investigated the contributions of the cerebellum and the motor cortex (M1) to acquisition and retention of human motor memories in a force field reaching task. We found that anodal transcranial direct current stimulation (tDCS) of the cerebellum, a technique that is thought to increase neuronal excitability, increased the ability to learn from error and form an internal model of the field, while cathodal cerebellar stimulation reduced this error-dependent learning. In addition, cathodal cerebellar stimulation disrupted the ability to respond to error within a reaching movement, reducing the gain of the sensory-motor feedback loop. By contrast, anodal M1 stimulation had no significant effects on these variables. During sham stimulation, early in training the acquired motor memory exhibited rapid decay in error-clamp trials. With further training the rate of decay decreased, suggesting that with training the motor memory was transformed from a labile to a more stable state. Surprisingly, neither cerebellar nor M1 stimulation altered these decay patterns. Participants returned 24hours later and were re-tested in error-clamp trials without stimulation. The cerebellar group that had learned the task with cathodal stimulation exhibited significantly impaired retention, and retention was not improved by M1 anodal stimulation. In summary, non-invasive cerebellar stimulation resulted in polarity-dependent up- or down-regulation of error-dependent motor learning. In addition, cathodal cerebellar stimulation during acquisition impaired the ability to retain the motor memory overnight. Thus, in the force field task we found a critical role for the cerebellum in both formation of motor memory and its retention.

Overcoming Motor “Forgetting” Through Reinforcement Of Learned Actions

The human motor system rapidly adapts to systematic perturbations but the adapted behavior seems to be forgotten equally rapidly. The reason for this forgetting is unclear, as is how to overcome it to promote long-term learning. Here we show that adapted behavior can be stabilized by a period of binary feedback about success and failure in the absence of vector error feedback. We examined the time course of decay after adaptation to a visuomotor rotation through a visual error-clamp condition—trials in which subjects received false visual feedback showing perfect directional performance, regardless of the movements they actually made. Exposure to this error-clamp following initial visuomotor adaptation led to a rapid reversion to baseline behavior. In contrast, exposure to binary feedback after initial adaptation turned the adapted state into a new baseline, to which subjects reverted after transient exposure to another visuomotor rotation. When both binary feedback and vector error were present, some subjects exhibited rapid decay to the original baseline, while others persisted in the new baseline. We propose that learning can be decomposed into two components—a fast-learning, fast-forgetting adaptation process that is sensitive to vector errors and insensitive to task success, and a second process driven by success that learns more slowly but is less susceptible to forgetting. These two learning systems may be recruited to different degrees across individuals. Understanding this competitive balance and exploiting the long-term retention properties of learning through reinforcement is likely to be essential for successful neuro-rehabilitation.

Evidence for Hyperbolic Temporal Discounting of Reward in Control of Movements

Suppose that the purpose of a movement is to place the body in a more rewarding state. In this framework, slower movements may increase accuracy and therefore improve the probability of acquiring reward, but the longer durations of slow movements produce devaluation of reward. Here we hypothesize that the brain decides the vigor of a movement (duration and velocity) based on the expected discounted reward associated with that movement. We begin by showing that durations of saccades of varying amplitude can be accurately predicted by a model in which motor commands maximize expected discounted reward. This result suggests that reward is temporally discounted even in timescales of tens of milliseconds. One interpretation of temporal discounting is that the true objective of the brain is to maximize the rate of reward—which is equivalent to a specific form of hyperbolic discounting. A consequence of this idea is that the vigor of saccades should change as one alters the intertrial intervals between movements. We find experimentally that in healthy humans, as intertrial intervals are varied, saccade peak velocities and durations change on a trial-by-trial basis precisely as predicted by a model in which the objective is to maximize the rate of reward. Our results are inconsistent with theories in which reward is discounted exponentially. We suggest that there exists a single cost, rate of reward, which provides a unifying principle that may govern control of movements in timescales of milliseconds, as well as decision making in timescales of seconds to years.

The Influence of Movement Preparation Time on the Expression of Visuomotor Learning and Savings

Following a change in the environment or motor apparatus, human subjects are able to rapidly compensate their movements to recover accurate performance. This ability to adapt is thought to be achieved through multiple, qualitatively distinct learning processes acting in parallel. It is unclear, however, what the relative contributions of these multiple processes are during learning. In particular, long-term memories in such paradigms have been extensively studied through the phenomenon of savings—faster adaptation to a given perturbation the second time it is experienced—but it is unclear which components of learning contribute to this effect. Here we show that distinct components of learning in an adaptation task can be dissociated based on the amount of preparation time they require. During adaptation, we occasionally forced subjects to generate movements at very low preparation times. Early in learning, subjects expressed only a limited amount of their prior learning in these trials, though performance improved gradually with further practice. Following washout, subjects exhibited a strong and persistent aftereffect in trials in which preparation time was limited. When subjects were exposed to the same perturbation twice in successive days, they adapted faster the second time. This savings effect was, however, not seen in movements generated at low preparation times. These results demonstrate that preparation time plays a critical role in the expression of some components of learning but not others. Savings is restricted to those components that require prolonged preparation to be expressed and might therefore reflect a declarative rather than procedural form of memory.

Sensitivity to prediction error in reach adaptation

It has been proposed that the brain predicts the sensory consequences of a movement and compares it to the actual sensory feedback. When the two differ, an error signal is formed, driving adaptation. How does an error in one trial alter performance in the subsequent trial? Here we show that the sensitivity to error is not constant but declines as a function of error magnitude. That is, one learns relatively less from large errors compared with small errors. We performed an experiment in which humans made reaching movements and randomly experienced an error in both their visual and proprioceptive feedback. Proprioceptive errors were created with force fields, and visual errors were formed by perturbing the cursor trajectory to create a visual error that was smaller, the same size, or larger than the proprioceptive error. We measured single-trial adaptation and calculated sensitivity to error, i.e., the ratio of the trial-to-trial change in motor commands to error size. We found that for both sensory modalities sensitivity decreased with increasing error size. A reanalysis of a number of previously published psychophysical results also exhibited this feature. Finally, we asked how the brain might encode sensitivity to error. We reanalyzed previously published probabilities of cerebellar complex spikes (CSs) and found that this probability declined with increasing error size. From this we posit that a CS may be representative of the sensitivity to error, and not error itself, a hypothesis that may explain conflicting reports about CSs and their relationship to error.

Dual-process decomposition in human sensorimotor adaptation.

Multiple distinct learning processes are known to contribute to sensorimotor adaptation in humans. It is challenging to identify and characterize these multiple processes, however, because only their summed contribution can typically be observed. A general strategy for decomposing adaptation into its constituent components is to exploit their differential susceptibility to specific experimental manipulations. Several such approaches have recently emerged which, taken together, suggest that two fundamental systems operate together to achieve the adapted state: one system learns slowly, is implicit, is temporally stable over short breaks, is expressible at low reaction times, and its properties do not change based on experience. The second learns rapidly, is explicit, requires a long preparation time to be expressed, and exhibits long-term memory for prior learning.

Unlearning versus savings in visuomotor adaptation: comparing effects of washout, passage of time, and removal of errors on motor memory

Humans are able to rapidly adapt their movements when a visuomotor or other systematic perturbation is imposed. However, the adaptation is forgotten or unlearned equally rapidly once the perturbation is removed. The ultimate cause of this unlearning remains poorly understood. Unlearning is often considered to be a passive process due to inability to retain an internal model. However, we have recently suggested that it may instead be a process of reversion to habit, without necessarily any forgetting per se. We compared the timecourse and nature of unlearning across a variety of protocols where unlearning is known to occur: error-clamp trials, removal of visual feedback, removal of the perturbation, or simply a period of inactivity. We found that, in agreement with mathematical models, there was no significant difference in the rate of decay between subject who experienced zero-error clamp trials, and subjects who made movements with no visual feedback. Time alone did lead to partial unlearning (over the duration we tested), but the amount of unlearning was inconsistent across subjects. Upon re-exposure to the same perturbation, subjects who unlearned through time or by reverting to veridical feedback exhibited savings. By contrast, no savings was observed in subjects who unlearned by having visual feedback removed or by being placed in a series of error-clamp trials. Thus although these various forms of unlearning can all revert subjects back to baseline behavior, they have markedly different effects on whether long-term memory for the adaptation is spared or is also unlearned. On the basis of these and previous findings, we suggest that unlearning is not due to passive forgetting of an internal model, but is instead an active process whereby adapted behavior gradually reverts to baseline habits.

Hedging Your Bets: Intermediate Movements as Optimal Behavior in the Context of an Incomplete Decision

Existing theories of movement planning suggest that it takes time to select and prepare the actions required to achieve a given goal. These theories often appeal to circumstances where planning apparently goes awry. For instance, if reaction times are forced to be very low, movement trajectories are often directed between two potential targets. These intermediate movements are generally interpreted as errors of movement planning, arising either from planning being incomplete or from parallel movement plans interfering with one another. Here we present an alternative view: that intermediate movements reflect uncertainty about movement goals. We show how intermediate movements are predicted by an optimal feedback control model that incorporates an ongoing decision about movement goals. According to this view, intermediate movements reflect an exploitation of compatibility between goals. Consequently, reducing the compatibility between goals should reduce the incidence of intermediate movements. In human subjects, we varied the compatibility between potential movement goals in two distinct ways: by varying the spatial separation between targets and by introducing a virtual barrier constraining trajectories to the target and penalizing intermediate movements. In both cases we found that decreasing goal compatibility led to a decreasing incidence of intermediate movements. Our results and theory suggest a more integrated view of decision-making and movement planning in which the primary bottleneck to generating a movement is deciding upon task goals. Determining how to move to achieve a given goal is rapid and automatic.

Formation of a long-term memory for visuomotor adaptation following only a few trials of practice

The term savings refers to faster motor adaptation upon reexposure to a previously experienced perturbation, a phenomenon thought to reflect the existence of a long-term motor memory. It is commonly assumed that sustained practice during the first perturbation exposure is necessary to create this memory. Here we sought to test this assumption by determining the minimum amount of experience necessary during initial adaptation to a visuomotor rotation to bring about savings the following day. Four groups of human subjects experienced 2, 5, 10, or 40 trials of a counterclockwise 30° cursor rotation during reaching movements on one day and were retested the following day to assay for savings. Groups that experienced five trials or more of adaptation on day 1 showed clear savings on day 2. Subjects in all groups learned significantly more from the first rotation trial on day 2 than on day 1, but this learning rate advantage was maintained only in groups that had reached asymptote during the initial exposure. Additional experiments revealed that savings occurred when the magnitude, but not the direction, of the rotation differed across exposures, and when a 5-min break, rather than an overnight one, separated the first and second exposure. The overall pattern of savings we observe across conditions can be explained as rapid retrieval of the state of learning attained during the first exposure rather than as modulation of sensitivity to error. We conclude that a long-term memory for compensating for a perturbation can be rapidly acquired and rapidly retrieved.