Emanuel Todorov

Optimal feedback control as a theory of motor coordination

A central problem in motor control is understanding how the many biomechanical degrees of freedom are coordinated to achieve a common goal. An especially puzzling aspect of coordination is that behavioral goals are achieved reliably and repeatedly with movements rarely reproducible in their detail. Existing theoretical frameworks emphasize either goal achievement or the richness of motor variability, but fail to reconcile the two. Here we propose an alternative theory based on stochastic optimal feedback control. We show that the optimal strategy in the face of uncertainty is to allow variability in redundant (task-irrelevant) dimensions. This strategy does not enforce a desired trajectory, but uses feedback more intelligently, correcting only those deviations that interfere with task goals. From this framework, task-constrained variability, goal-directed corrections, motor synergies, controlled parameters, simplifying rules and discrete coordination modes emerge naturally. We present experimental results from a range of motor tasks to support this theory.

Direct cortical control of muscle activation in voluntary arm movements: a model.

What neural activity in motor cortex represents and how it controls ongoing movement remain unclear. Suggestions that cortex generates low-level muscle control are discredited by correlations with higher-level parameters of hand movement, but no coherent alternative exists. I argue that the view of low-level control is in principle correct, and that seeming contradictions result from overlooking known properties of the motor periphery. Assuming direct motor cortical activation of muscle groups and taking into account the state dependence of muscle-force production and multijoint mechanics, I show that cortical population output must correlate with hand kinematics in quantitative agreement with experimental observations. The model reinterprets the ‘neural population vector’ to afford unified control of posture, movement and force production.

A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems

We present an iterative linear-quadratic-Gaussian method for locally-optimal feedback control of nonlinear stochastic systems subject to control constraints. Previously, similar methods have been restricted to deterministic unconstrained problems with quadratic costs. The new method constructs an affine feedback control law, obtained by minimizing a novel quadratic approximation to the optimal cost-to-go function. Global convergence is guaranteed through a Levenberg-Marquardt method; convergence in the vicinity of a local minimum is quadratic. Performance is illustrated on a limited-torque inverted pendulum problem, as well as a complex biomechanical control problem involving a stochastic model of the human arm, with 10 state dimensions and 6 muscle actuators. A Matlab implementation of the new algorithm is availabe at www.cogsci.ucsd.edu//spl sim/todorov.

Evidence for the Flexible Sensorimotor Strategies Predicted by Optimal Feedback Control

Everyday movements pursue diverse and often conflicting mixtures of task goals, requiring sensorimotor strategies customized for the task at hand. Such customization is mostly ignored by traditional theories emphasizing movement geometry and servo control. In contrast, the relationship between the task and the strategy most suitable for accomplishing it lies at the core of our optimal feedback control theory of coordination. Here, we show that the predicted sensitivity to task goals affords natural explanations to a number of novel psychophysical findings. Our point of departure is the little-known fact that corrections for target perturbations introduced late in a reaching movement are incomplete. We show that this is not simply attributable to lack of time, in contradiction with alternative models and, somewhat paradoxically, in agreement with our model. Analysis of optimal feedback gains reveals that the effect is partly attributable to a previously unknown trade-off between stability and accuracy. This yields a testable prediction: if stability requirements are decreased, then accuracy should increase. We confirm the prediction experimentally in three-dimensional obstacle avoidance and interception tasks in which subjects hit a robotic target with programmable impedance. In additional agreement with the theory, we find that subjects do not rely on rigid control strategies but instead exploit every opportunity for increased performance. The modeling methodology needed to capture this extra flexibility is more general than the linear-quadratic methods we used previously. The results suggest that the remarkable flexibility of motor behavior arises from sensorimotor control laws optimized for composite cost functions.

Stochastic Optimal Control and Estimation Methods Adapted to the Noise Characteristics of the Sensorimotor System

Optimality principles of biological movement are conceptually appealing and straightforward to formulate. Testing them empirically, however, requires the solution to stochastic optimal control and estimation problems for reasonably realistic models of the motor task and the sensorimotor periphery. Recent studies have highlighted the importance of incorporating biologically plausible noise into such models. Here we extend the linear-quadratic-gaussian frameworkcurrently the only framework where such problems can be solved efficientlyto include control-dependent, state-dependent, and internal noise. Under this extended noise model, we derive a coordinate-descent algorithm guaranteed to converge to a feedback control law and a nonadaptive linear estimator optimal with respect to each other. Numerical simulations indicate that convergence is exponential, local minima do not exist, and the restriction to nonadaptive linear estimators has negligible effects in the control problems of interest. The application of the algorithm is illustrated in the context of reaching movements. A Matlab implementation is available at www.cogsci.ucsd.edu/todorov.

Efficient computation of optimal actions

Optimal choice of actions is a fundamental problem relevant to fields as diverse as neuroscience, psychology, economics, computer science, and control engineering. Despite this broad relevance the abstract setting is similar: we have an agent choosing actions over time, an uncertain dynamical system whose state is affected by those actions, and a performance criterion that the agent seeks to optimize. Solving problems of this kind remains hard, in part, because of overly generic formulations. Here, we propose a more structured formulation that greatly simplifies the construction of optimal control laws in both discrete and continuous domains. An exhaustive search over actions is avoided and the problem becomes linear. This yields algorithms that outperform Dynamic Programming and Reinforcement Learning, and thereby solve traditional problems more efficiently. Our framework also enables computations that were not possible before: composing optimal control laws by mixing primitives, applying deterministic methods to stochastic systems, quantifying the benefits of error tolerance, and inferring goals from behavioral data via convex optimization. Development of a general class of easily solvable problems tends to accelerate progress—as linear systems theory has done, for example. Our framework may have similar impact in fields where optimal choice of actions is relevant.

MuJoCo: A physics engine for model-based control

We describe a new physics engine tailored to model-based control. Multi-joint dynamics are represented in generalized coordinates and computed via recursive algorithms. Contact responses are computed via efficient new algorithms we have developed, based on the modern velocity-stepping approach which avoids the difficulties with spring-dampers. Models are specified using either a high-level C++ API or an intuitive XML file format. A built-in compiler transforms the user model into an optimized data structure used for runtime computation. The engine can compute both forward and inverse dynamics. The latter are well-defined even in the presence of contacts and equality constraints. The model can include tendon wrapping as well as actuator activation states (e.g. pneumatic cylinders or muscles). To facilitate optimal control applications and in particular sampling and finite differencing, the dynamics can be evaluated for different states and controls in parallel. Around 400,000 dynamics evaluations per second are possible on a 12-core machine, for a 3D homanoid with 18 dofs and 6 active contacts. We have already used the engine in a number of control applications. It will soon be made publicly available.

Synthesis and stabilization of complex behaviors through online trajectory optimization

We present an online trajectory optimization method and software platform applicable to complex humanoid robots performing challenging tasks such as getting up from an arbitrary pose on the ground and recovering from large disturbances using dexterous acrobatic maneuvers. The resulting behaviors, illustrated in the attached video, are computed only 7 × slower than real time, on a standard PC. The video also shows results on the acrobot problem, planar swimming and one-legged hopping. These simpler problems can already be solved in real time, without pre-computing anything.

Structured variability of muscle activations supports the minimal intervention principle of motor control

Numerous observations of structured motor variability indicate that the sensorimotor system preferentially controls task-relevant parameters while allowing task-irrelevant ones to fluctuate. Optimality models show that controlling a redundant musculo-skeletal system in this manner meets task demands while minimizing control effort. Although this line of inquiry has been very productive, the data are mostly behavioral with no direct physiological evidence on the level of muscle or neural activity. Furthermore, biomechanical coupling, signal-dependent noise, and alternative causes of trial-to-trial variability confound behavioral studies. Here we address those confounds and present evidence that the nervous system preferentially controls task-relevant parameters on the muscle level. We asked subjects to produce vertical fingertip force vectors of prescribed constant or time-varying magnitudes while maintaining a constant finger posture. We recorded intramuscular electromyograms (EMGs) simultaneously from all seven index finger muscles during this task. The experiment design and selective fine-wire muscle recordings allowed us to account for a median of 91% of the variance of fingertip forces given the EMG signals. By analyzing muscle coordination in the seven-dimensional EMG signal space, we find that variance-per-dimension is consistently smaller in the task-relevant subspace than in the task-irrelevant subspace. This first direct physiological evidence on the muscle level for preferential control of task-relevant parameters strongly suggest the use of a neural control strategy compatible with the principle of minimal intervention. Additionally, variance is nonnegligible in all seven dimensions, which is at odds with the view that muscle activation patterns are composed from a small number of synergies.

Discovery of complex behaviors through contact-invariant optimization

We present a motion synthesis framework capable of producing a wide variety of important human behaviors that have rarely been studied, including getting up from the ground, crawling, climbing, moving heavy objects, acrobatics (hand-stands in particular), and various cooperative actions involving two characters and their manipulation of the environment. Our framework is not specific to humans, but applies to characters of arbitrary morphology and limb configuration. The approach is fully automatic and does not require domain knowledge specific to each behavior. It also does not require pre-existing examples or motion capture data. At the core of our framework is the contact-invariant optimization (CIO) method we introduce here. It enables simultaneous optimization of contact and behavior. This is done by augmenting the search space with scalar variables that indicate whether a potential contact should be active in a given phase of the movement. These auxiliary variables affect not only the cost function but also the dynamics (by enabling and disabling contact forces), and are optimized together with the movement trajectory. Additional innovations include a continuation scheme allowing helper forces at the potential contacts rather than the torso, as well as a feature-based model of physics which is particularly well-suited to the CIO framework. We expect that CIO can also be used with a full physics model, but leave that extension for future work.