Said Ghani Khan

Reinforcement learning and optimal adaptive control: An overview and implementation examples

Abstract This paper provides an overview of the reinforcement learning and optimal adaptive control literature and its application to robotics. Reinforcement learning is bridging the gap between traditional optimal control, adaptive control and bio-inspired learning techniques borrowed from animals. This work is highlighting some of the key techniques presented by well known researchers from the combined areas of reinforcement learning and optimal control theory. At the end, an example of an implementation of a novel model-free Q-learning based discrete optimal adaptive controller for a humanoid robot arm is presented. The controller uses a novel adaptive dynamic programming (ADP) reinforcement learning (RL) approach to develop an optimal policy on-line. The RL joint space tracking controller was implemented for two links (shoulder flexion and elbow flexion joints) of the arm of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) torso. The constrained case (joint limits) of the RL scheme was tested for a single link (elbow flexion) of the BERT II arm by modifying the cost function to deal with the extra nonlinearity due to the joint constraints.

Safe Adaptive Compliance Control of a Humanoid Robotic Arm with Anti-Windup Compensation and Posture Control

Safety is very important for physical human-robot interaction. Compliance control can solve an important aspect of the safety problem by dealing with impact and other forces arising during close contact between humans and robots.An adaptive compliance model reference controller was implemented in real-time on a 4 degrees of freedom (DOF) humanoid robotic arm in Cartesian space. The robot manipulator has been controlled in such a way as to follow the compliant passive behaviour of a reference mass-spring-damper system model subject to an externally sensed force. The redundant DOF were used to control the robot motion in a human-like pattern via minimization of effort, a function of gravity. Associated actuator saturation issues were addressed by incorporating a novel anti-windup (AW) compensator originally developed for a neural network scheme. The controller was simulated for a robotic arm representing the Bristol-Elumotion-Robotic-Torso II (BERT II) and then tested on the real BERT II arm. BERT II has been developed in collaboration by Bristol Robotics Laboratory and Elumotion Ltd.

A novel robust adaptive control algorithm with finite-time online parameter estimation of a humanoid robot arm

A novel robust adaptive control algorithm is proposed and implemented in real-time on two degrees-of-freedom (DOF) of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) arm in joint-space. In addition to having a significant robustness property for the tracking, the algorithm also features a sliding-mode term based adaptive law that captures directly the parameter estimation error. An auxiliary filtered regression vector and filtered computed torque is introduced. This allows the definition of another auxiliary matrix, a filtered regression matrix, which facilitates the introduction of a sliding mode term into the adaptation law. Parameter error convergence to zero can be guaranteed within finite-time with a Persistent-Excitation (PE) condition or Sufficient Richness condition for the demand. The proposed scheme also exhibits robustness both in the tracking and parameter estimation errors to any bounded additive disturbance. This theoretical result is then exemplified for the BERT II robot arm in simulation and for experiments.

Adaptive multi-dimensional compliance control of a humanoid robotic arm with anti-windup compensation

An adaptive multi-dimensional compliance model reference controller was implemented in real-time on a 4 degrees of freedom (DOF) of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) arm in Cartesian space. The robot manipulator has been controlled in such a way as to follow the compliant passive behaviour of a reference mass-spring-damper system model subject to externally sensed forces/torques in all DOF. The relevant reference model converts all measured torques into their equivalent forces at the end-effector and reacts accordingly. The suggested control scheme takes in particular account of the multi-variable aspect and the problem of body own torques when measuring external torques. The redundant DOF were used to control the robot motion in a human-like pattern via effort minimization. Associated actuator saturation issues were addressed by incorporating a novel anti-windup (AW) compensator.

A Novel Adaptive Control Algorithm in application to a humanoid robot arm

A novel adaptive control algorithm was proposed and implemented in real-time on 2 degrees-of-freedom (DOF) of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) arm in joint-space. The algorithm features a sliding-mode term based adaptive law that captures directly the parameter estimation error. An auxiliary filtered regression vector and filtered computed torque is introduced. This allows the definition of another auxiliary matrix, a filtered regression matrix, which facilitates the introduction of the sliding term into the adaptation law. Parameter error convergence to zero can be guaranteed within finite-time with a Persistent-Excitation (PE) or Sufficient Richness condition for the demand. This theoretical result is then exemplified for the BERT II robot arm in simulation and experiment.

Estimation of the shear force in Transverse Dynamic Force Microscopy using a sliding mode observer

This paper concerns the application of a sliding mode observer to the problem of estimation of the shear force affecting the cantilever dynamics of a Transverse Dynamic Force Microscope (TDFM). The oscillated cantilever in proximity to a specimen permits the investigation of the specimen topography at nano-metre precision. The oscillation amplitude, but also in particular the shear forces, are a measure of distance to the specimen, and therefore the estimation of the shear force is of significance when attempting to construct TDFM images at submolecular accuracy. For estimation of the shear forces, an approximate model of the cantilever is derived using the method of lines. Model order reduction and sliding mode techniques are employed to reconstruct the unknown shear force affecting the cantilever dynamics based on only tip position measurements. Simulations are presented to illustrate the proposed scheme, which is to be implemented on the TDFM set up at the Centre for NSQI at Bristol.

Robotic hand posture and compliant grasping control using operational space and integral sliding mode control

This paper establishes a novel approach of robotic hand posture and grasping control. For this purpose, the control uses the operational space approach. This permits the consideration of the shape of the object to be grasped. Thus, the control is split into a task control and a particular optimizing posture control. The task controller employs Cylindrical and Spherical coordinate systems due to their simplicity and geometric suitability. This is achieved by using an integral sliding mode controller (ISMC) as task controller. The ISMC allows us to introduce a model reference approach where a virtual mass-spring-damper system can be used to design a compliant trajectory tracking controller. The optimizing posture controller together with the task controller creates a simple approach to obtain pre-grasping/object approach hand postures. The experimental results show that target trajectories can be easily followed by the task control despite the presence of friction and stiction. When the object is grasped, the compliant control will automatically adjust to a specific compliance level due to an augmented compliance parameter adjustment algorithm. Once a specific compliance model has been achieved, the fixed compliance controller can be tested for a specific object grasp scenario. The experimental results prove that the Bristol Elumotion robot hand (BERUL) can automatically and successfully attain different compliance levels for a particular object via the ISMC.

Compliance Control and Human–Robot Interaction: Part II — Experimental Examples

Compliance control is highly relevant to human safety in human–robot interaction (HRI). This paper presents multi-dimensional compliance control of a humanoid robot arm. A dynamic model-free adaptive controller with an anti-windup compensator is implemented on four degrees of freedom (DOF) of a humanoid robot arm. The paper is aimed to compliment the associated review paper on compliance control. This is a model reference adaptive compliance scheme which employs end-effector forces (measured via joint torque sensors) as a feedback. The robots body-own torques are separated from external torques via a simple but effective algorithm. In addition, an experiment of physical human robot interaction is conducted employing the above mentioned adaptive compliance control along with a speech interface. The experiment is focused on passing an object (a cup) between a human and a robot. Compliance is providing an immediate layer of safety for this HRI scenario by avoiding pushing, pulling or clamping and minimizing the effect of collisions with the environment.

A novel Q-learning based adaptive optimal controller implementation for a humanoid robotic arm

Abstract This paper presents the implementation of a novel model-free Q-learning based discrete adaptive optimal controller for a humanoid robotic arm. The controller uses a novel adaptive dynamic programming (ADP) reinforcement learning (RL) approach to develop an optimal policy on-line. This is in contrast with the other optimal control design techniques which are carried out off-line and need full information of the system dynamics. The RL tracking controller was implemented for two links (shoulder flexion and elbow flexion joints) of the arm of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) torso. The constrained case (joint limits) of the RL scheme was tested for a single link (elbow flexion) of the BERT II arm by modifying the cost function to deal with the extra nonlinearity due to the joint constraint.

Compliance Control and Human–Robot Interaction: Part 1 — Survey

Compliance control is highly relevant to human safety in human–robot interaction (HRI). This paper presents a review of various compliance control techniques. The paper is aimed to provide a good background knowledge for new researchers and highlight the current hot issues in compliance control research. Active compliance, passive compliance, adaptive and reinforcement learning-based compliance control techniques are discussed. This paper provides a comprehensive literature survey of compliance control keeping in view physical human robot interaction (pHRI) e.g., passing an object, such as a cup, between a human and a robot. Compliance control may eventually provide an immediate and effective layer of safety by avoiding pushing, pulling or clamping in pHRI. Emerging areas such as soft robotics, which exploit the deformability of biomaterial as well as hybrid approaches which combine active and passive compliance are also highlighted.