Gabriel A. D. Lopes

Automated gait adaptation for legged robots

Gait parameter adaptation on a physical robot is an error-prone, tedious and time-consuming process. In this paper we present a system for gait adaptation in our RHex series of hexapedal robots that renders this arduous process nearly autonomous. The robot adapts its gait parameters by recourse to a modified version of Nelder-Mead descent, while managing its self-experiments and measuring the outcome by visual servoing within a partially engineered environment The resulting performance gains extend considerably beyond what we have managed with hand tuning. For example, the best hand tuned alternating tripod gaits never exceeded 0.8 m/s nor achieved specific resistance below 2.0. In contrast, Nelder-Mead based tuning has yielded alternating tripod gaits at 2.7 m/s (well over 5 body lengths per second) and reduced specific resistance to 0.6 while requiring little human intervention at low and moderate speeds. Comparable gains have been achieved on the much larger ruggedized version of this machine.

A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients

Policy-gradient-based actor-critic algorithms are amongst the most popular algorithms in the reinforcement learning framework. Their advantage of being able to search for optimal policies using low-variance gradient estimates has made them useful in several real-life applications, such as robotics, power control, and finance. Although general surveys on reinforcement learning techniques already exist, no survey is specifically dedicated to actor-critic algorithms in particular. This paper, therefore, describes the state of the art of actor-critic algorithms, with a focus on methods that can work in an online setting and use function approximation in order to deal with continuous state and action spaces. After starting with a discussion on the concepts of reinforcement learning and the origins of actor-critic algorithms, this paper describes the workings of the natural gradient, which has made its way into many actor-critic algorithms over the past few years. A review of several standard and natural actor-critic algorithms is given, and the paper concludes with an overview of application areas and a discussion on open issues.

Visual Servoing for Nonholonomically Constrained Three Degree of Freedom Kinematic Systems

This paper addresses problems of robot navigation with nonholonomic motion constraints and perceptual cues arising from onboard visual servoing in partially engineered environments. A general hybrid procedure is proposed that adapts to the constrained motion setting the standard feedback controller arising from a navigation function in the fully actuated case. This is accomplished by switching back and forth between moving “down” and “across” the associated gradient field toward the stable manifold it induces in the constrained dynamics. Guaranteed to avoid obstacles in all cases, conditions are provided under which the new procedure brings initial configurations to within an arbitrarily small neighborhood of the goal. Simulation results are given for a sample of visual servoing problems with a few different perceptual models. The empirical effectiveness of the proposed algorithm is documented by reporting results of its application to outdoor autonomous visual registration experiments with the robot RHex guided by engineered beacons.

Spatial reconstruction of human motion by means of a single camera and a biomechanical model

The value of the results of the inverse dynamic analysis procedures used in the study of human tasks is dependent on the quality of the kinematic and kinetic data supplied to the biomechanical model that supports it. The kinematic data, containing the position, velocity and acceleration of all anatomical segments of the biomechanical model, result from the reconstruction of human spatial motion by means of the evaluation of the anatomic points positions that enable to uniquely define the position of all anatomical segments. Furthermore, the motion data must be kinematically consistent with the structure of the biomechanical model used in the analysis. The traditional photogrammetric methodologies used for the spatial reconstruction of the human motion require images of two or more calibrated and synchronized cameras. This is due to the fact that the projection of each anatomical point is described by two linear equations relating its three spatial coordinates with the two coordinates of the projected point. The need for the image of another camera arises from the fact that a third equation is necessary to find the original spatial position of the anatomical point. The methodology proposed here substitutes the projection equations of the second camera with the kinematic constraint equations associated with a biomechanical model in the motion reconstruction process. In the formulation the system of equations arising from the point projections and biomechanical model kinematic constraints, representing the constant length of the anatomical segments, are solved simultaneously. Because the system of equations has multiple solutions for each image, a strategy based on the minimization of a cost function associated to the smoothness of the reconstructed motion is devised. It is shown how the process is implemented computationally avoiding any operator intervention during the motion reconstruction for a given time period. This leads to an automated computer procedure that ensures the uniqueness of the reconstructed motion. The result of the reconstruction process is a set of data that is kinematically consistent with the biomechanical model used. Through applications of the proposed methodology to several sports exercises its benefits and shortcomings are discussed.

Spatial reconstruction of the human motion based on images of a single camera

The inverse dynamic analysis procedures used in the study of the human gait require that the kinematics of the supporting biomechanical model is known beforehand. The first step to obtain the kinematic data is the reconstruction of human spatial motion, i.e., the evaluation of the anatomic points positions that enables to uniquely define the position of all anatomical segments. In photogrammetry, the projection of each anatomical point is described by two linear equations relating its three spatial coordinates with the two coordinates of the projected point. The need for the image of two cameras arises from the fact that three equations are necessary to find the original spatial position of the anatomical point. It is shown here that the kinematic constraint equations associated with a biomechanical model can be used as the extra set of equations required for the reconstruction process, instead of the equations associated with the second camera. With this methodology, the system of equations arising from the point projections and biomechanical model kinematic constraints are solved simultaneously. Since the system of equations has multiple solutions for each image, a strategy based on the minimization of the cost function associated to the smoothness of the reconstructed motion is devised, leading to an automated computer procedure enabling a unique reconstruction.

Design and Performance Evaluation of a Bio-Inspired and Single-Motor-Driven Hexapod Robot With Dynamical Gaits

Over its lifetime, the hexapedal robot RHex has shown impressive performance. Combining preflexes with a range of control schemes, various behaviors such as leaping, running, bounding, as well as running on rough terrain have been exhibited. In order to better determine the extent to which the passive and mechanical aspects of the design contribute to performance, a new version of the hexapedal spring-loaded inverted pendulum (SLIP)-based runner with a novel minimal control scheme is developed and tested. A unique drive mechanism is utilized to allow for operation (including steering) of the robot with only two motors. The simplified robot operates robustly and it exhibits walking, SLIP-like running, or high-speed motion profiles depending only on the actuation frequency. In order to better capture the critical nonlinear properties of the robot’s legs, a more detailed dynamic model termed R2-SLIP is presented. The performance of the robot is compared to the basic SLIP, the R-SLIP, and this new R2-SLIP model. Furthermore, these results suggest that, in the future, the R2-SLIP model can be used to tune/improve the design of the leg compliance and noncircular gears to optimize performance.

Modeling and Control of Legged Locomotion via Switching Max-Plus Models

We present a gait generation framework for multi-legged robots based on max-plus algebra that is endowed with intrinsically safe gait transitions. The time schedule of each foot liftoff and touchdown is modeled by sets of max-plus linear equations. The resulting discrete-event system is translated to continuous time via piecewise constant leg phase velocities; thus, it is compatible with traditional central pattern generator approaches. Different gaits and gait parameters are interleaved by utilizing different max-plus system matrices. We present various gait transition schemes and show that optimal transitions, in the sense of minimizing the stance time variation, allow for constant acceleration and deceleration on legged platforms. The framework presented in this paper relies on a compact representation of the gait space, provides guarantees regarding the transient and steady-state behavior, and results in simple implementations on legged robotic platforms.

Optimal model-free output synchronization of heterogeneous systems using off-policy reinforcement learning

This paper considers optimal output synchronization of heterogeneous linear multi-agent systems. Standard approaches to output synchronization of heterogeneous systems require either the solution of the output regulator equations or the incorporation of a p-copy of the leaders dynamics in the controller of each agent. By contrast, in this paper neither one is needed. Moreover, here both the leaders and the followers dynamics are assumed to be unknown. First, a distributed adaptive observer is designed to estimate the leaders state for each agent. The output synchronization problem is then formulated as an optimal control problem and a novel model-free off-policy reinforcement learning algorithm is developed to solve the optimal output synchronization problem online in real time. It is shown that this optimal distributed approach implicitly solves the output regulation equations without actually doing so. Simulation results are provided to verify the effectiveness of the proposed approach.

Visual registration and navigation using planar features

This paper addresses the problem of registering the hexapedal robot, RHex, relative to a known set of beacons, by real-time visual servoing. A suitably constructed navigation function represents the task, in the sense that for a completely actuated machine in the horizontal plane, the gradient dynamics guarantee convergence to the visually cued goal without ever losing sight of the beacons that define it. Since the horizontal plane behavior of RHex can be represented as a unicycle, feeding back the navigation function gradient avoids loss of beacons, but does not yield an asymptotically stable goal. We address new problems arising from the configuration of the beacons and present preliminary experimental results that illustrate the discrepancies between the idealized and physical robot actuation capabilities.

Reinforcement learning for sequential composition control

Sequential composition is an effective strategy for addressing complex control specifications and complex dynamical systems by partitioning the problem in time and space. Traditionally, sequential composition controllers are synthesized offline given a control task and a static environment with possible constraints. Dynamical environments may require redesigning the entire sequential composition controller, which may be time costly and inefficient. In this paper we introduce a learning strategy to augment online a pre-designed sequential composition controller based on reinforcement learning. By interpreting the sequential composition controller as an automaton, we add and delete nodes in the graph online, based on newly acquired knowledge via learning. We present simulation and experimental results for a nonlinear motion-control system.