Nicolas Schweighofer

Role of the cerebellum in reaching movements in humans. I. Distributed inverse dynamics control

This study focuses on the role of the motor cortex, the spinal cord and the cerebellum in the dynamics stage of the control of arm movement. Currently, two classes of models have been proposed for the neural control of movements, namely the virtual trajectory control hypothesis and the acquisition of internal models of the motor apparatus hypothesis. In the present study, we expand the virtual trajectory model to whole arm reaching movements. This expanded model accurately reproduced slow movements, but faster reaching movements deviated significantly from the planned trajectories, indicating that for fast movements, this model was not sufficient. These results led us to propose a new distributed functional model consistent with behavioural, anatomical and neurophysiological data, which takes into account arm muscles, spinal cord, motor cortex and cerebellum and is consistent with the view that the central nervous system acquires a distributed inverse dynamics model of the arm. Previous studies indicated that the cerebellum compensates for the interaction forces that arise during reaching movements. We show here how the cerebellum may increase the accuracy of reaching movements by compensating for the interaction torques by learning a portion of an inverse dynamics model that refines a basic inverse model in the motor cortex and spinal cord.

Role of the cerebellum in reaching movements in humans. II. A neural model of the intermediate cerebellum

The cerebellum is essential for the control of multijoint movements; when the cerebellum is lesioned, the performance error is more than the summed errors produced by single joints. In the companion paper ( Schweighofer et al. 1998 ), a functional anatomical model for visually guided arm movement was proposed. The model comprised a basic feedforward/feedback controller with realistic transmission delays and was connected to a two‐link, six‐muscle, planar arm. In the present study, we examined the role of the cerebellum in reaching movements by embedding a novel, detailed cerebellar neural network in this functional control model. We could derive realistic cerebellar inputs and the role of the cerebellum in learning to control the arm was assessed.

Reward Improves Long-Term Retention of a Motor Memory through Induction of Offline Memory Gains

In humans, training in which good performance is rewarded or bad performance punished results in transient behavioral improvements. The relative effects of reward and punishment on consolidation and long-term retention, critical behavioral stages for successful learning, are not known. Here, we investigated the effects of reward and punishment on these different stages of human motor skill learning. We studied healthy subjects who trained on a motor task under rewarded, punished, or neutral control conditions. Performance was tested before and immediately, 6 hr, 24 hr, and 30 days after training in the absence of reward or punishment. Performance improvements immediately after training were comparable in the three groups. At 6 hr, the rewarded group maintained performance gains, whereas the other two groups experienced significant forgetting. At 24 hr, the reward group showed significant offline (posttraining) improvements, whereas the other two groups did not. At 30 days, the rewarded group retained the gains identified at 24 hr, whereas the other two groups experienced significant forgetting. We conclude that training under rewarded conditions is more effective than training under punished or neutral conditions in eliciting lasting motor learning, an advantage driven by offline memory gains that persist over time.

Motor Learning Without Doing: Trial-by-Trial Improvement in Motor Performance During Mental Training

Although there is converging experimental and clinical evidences suggesting that mental training with motor imagery can improve motor performance, it is unclear how humans can learn movements through mental training despite the lack of sensory feedback from the body and the environment. In a first experiment, we measured the trial-by-trial decrease in durations of executed movements (physical training group) and mentally simulated movements (motor-imagery training group), by means of training on a multiple-target arm-pointing task requiring high accuracy and speed. Movement durations were significantly lower in posttest compared with pretest after both physical and motor-imagery training. Although both the posttraining performance and the rate of learning were smaller in motor-imagery training group than in physical training group, the change in movement duration and the asymptotic movement duration after a hypothetical large number of trials were identical. The two control groups (eye-movement training and rest groups) did not show change in movement duration. In the second experiment, additional kinematic analyses revealed that arm movements were straighter and faster both immediately and 24 h after physical and motor-imagery training. No such improvements were observed in the eye-movement training group. Our results suggest that the brain uses state estimation, provided by internal forward model predictions, to improve motor performance during mental training. Furthermore, our results suggest that mental practice can, at least in young healthy subjects and if given after a short bout of physical practice, be successfully substituted to physical practice to improve motor performance.

Dual adaptation supports a parallel architecture of motor memory.

Although our understanding of the mechanisms underlying motor adaptation has greatly benefited from previous computational models, the architecture of motor memory is still uncertain. On one hand, two-state models that contain both a fast-learning–fast-forgetting process and a slow-learning–slow-forgetting process explain a wide range of data on motor adaptation, but cannot differentiate whether the fast and slow processes are arranged serially or in parallel and cannot account for learning multiple tasks simultaneously. On the other hand, multiple parallel-state models learn multiple tasks simultaneously but cannot account for a number of motor adaptation data. Here, we investigated the architecture of human motor memory by systematically testing possible architectures via a combination of simulations and a dual visuomotor adaptation experimental paradigm. We found that only one parsimonious model can account for both previous motor adaptation data and our dual-task adaptation data: a fast process that contains a single state is arranged in parallel with a slow process that contains multiple states switched via contextual cues. Our result suggests that during motor adaptation, fast and slow processes are updated simultaneously from the same motor learning errors.

Meta-learning in reinforcement learning

Meta-parameters in reinforcement learning should be tuned to the environmental dynamics and the animal performance. Here, we propose a biologically plausible meta-reinforcement learning algorithm for tuning these meta-parameters in a dynamic, adaptive manner. We tested our algorithm in both a simulation of a Markov decision task and in a non-linear control task. Our results show that the algorithm robustly finds appropriate meta-parameter values, and controls the meta-parameter time course, in both static and dynamic environments. We suggest that the phasic and tonic components of dopamine neuron firing can encode the signal required for meta-learning of reinforcement learning.

Stroke rehabilitation reaches a threshold.

Motor training with the upper limb affected by stroke partially reverses the loss of cortical representation after lesion and has been proposed to increase spontaneous arm use. Moreover, repeated attempts to use the affected hand in daily activities create a form of practice that can potentially lead to further improvement in motor performance. We thus hypothesized that if motor retraining after stroke increases spontaneous arm use sufficiently, then the patient will enter a virtuous circle in which spontaneous arm use and motor performance reinforce each other. In contrast, if the dose of therapy is not sufficient to bring spontaneous use above threshold, then performance will not increase and the patient will further develop compensatory strategies with the less affected hand. To refine this hypothesis, we developed a computational model of bilateral hand use in arm reaching to study the interactions between adaptive decision making and motor relearning after motor cortex lesion. The model contains a left and a right motor cortex, each controlling the opposite arm, and a single action choice module. The action choice module learns, via reinforcement learning, the value of using each arm for reaching in specific directions. Each motor cortex uses a neural population code to specify the initial direction along which the contralateral hand moves towards a target. The motor cortex learns to minimize directional errors and to maximize neuronal activity for each movement. The derived learning rule accounts for the reversal of the loss of cortical representation after rehabilitation and the increase of this loss after stroke with insufficient rehabilitation. Further, our model exhibits nonlinear and bistable behavior: if natural recovery, motor training, or both, brings performance above a certain threshold, then training can be stopped, as the repeated spontaneous arm use provides a form of motor learning that further bootstraps performance and spontaneous use. Below this threshold, motor training is “in vain”: there is little spontaneous arm use after training, the model exhibits learned nonuse, and compensatory movements with the less affected hand are reinforced. By exploring the nonlinear dynamics of stroke recovery using a biologically plausible neural model that accounts for reversal of the loss of motor cortex representation following rehabilitation or the lack thereof, respectively, we can explain previously hard to reconcile data on spontaneous arm use in stroke recovery. Further, our threshold prediction could be tested with an adaptive train–wait–train paradigm: if spontaneous arm use has increased in the “wait” period, then the threshold has been reached, and rehabilitation can be stopped. If spontaneous arm use is still low or has decreased, then another bout of rehabilitation is to be provided.

Computational motor control in humans and robots

Computational models can provide useful guidance in the design of behavioral and neurophysiological experiments and in the interpretation of complex, high dimensional biological data. Because many problems faced by the primate brain in the control of movement have parallels in robotic motor control, models and algorithms from robotics research provide useful inspiration, baseline performance, and sometimes direct analogs for neuroscience.

Humans Can Adopt Optimal Discounting Strategy under Real-Time Constraints

Critical to our many daily choices between larger delayed rewards, and smaller more immediate rewards, are the shape and the steepness of the function that discounts rewards with time. Although research in artificial intelligence favors exponential discounting in uncertain environments, studies with humans and animals have consistently shown hyperbolic discounting. We investigated how humans perform in a reward decision task with temporal constraints, in which each choice affects the time remaining for later trials, and in which the delays vary at each trial. We demonstrated that most of our subjects adopted exponential discounting in this experiment. Further, we confirmed analytically that exponential discounting, with a decay rate comparable to that used by our subjects, maximized the total reward gain in our task. Our results suggest that the particular shape and steepness of temporal discounting is determined by the task that the subject is facing, and question the notion of hyperbolic reward discounting as a universal principle.

Cerebellar learning of accurate predictive control for fast-reaching movements

Abstract. Long conduction delays in the nervous system prevent the accurate control of movements by feedback control alone. We present a new, biologically plausible cerebellar model to study how fast arm movements can be executed in spite of these delays. To provide a realistic test-bed of the cerebellar neural model, we embed the cerebellar network in a simulated biological motor system comprising a spinal cord model and a six-muscle two-dimensional arm model. We argue that if the trajectory errors are detected at the spinal cord level, memory traces in the cerebellum can solve the temporal mismatch problem between efferent motor commands and delayed error signals. Moreover, learning is made stable by the inclusion of the cerebello-nucleo-olivary loop in the model. It is shown that the cerebellar network implements a nonlinear predictive regulator by learning part of the inverse dynamics of the plant and spinal circuit. After learning, fast accurate reaching movements can be generated.