Tony Pipe

Reinforcement learning and optimal adaptive control: An overview and implementation examples

Abstract This paper provides an overview of the reinforcement learning and optimal adaptive control literature and its application to robotics. Reinforcement learning is bridging the gap between traditional optimal control, adaptive control and bio-inspired learning techniques borrowed from animals. This work is highlighting some of the key techniques presented by well known researchers from the combined areas of reinforcement learning and optimal control theory. At the end, an example of an implementation of a novel model-free Q-learning based discrete optimal adaptive controller for a humanoid robot arm is presented. The controller uses a novel adaptive dynamic programming (ADP) reinforcement learning (RL) approach to develop an optimal policy on-line. The RL joint space tracking controller was implemented for two links (shoulder flexion and elbow flexion joints) of the arm of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) torso. The constrained case (joint limits) of the RL scheme was tested for a single link (elbow flexion) of the BERT II arm by modifying the cost function to deal with the extra nonlinearity due to the joint constraints.

Towards a Platform-Independent Cooperative Human Robot Interaction System: III An Architecture for Learning and Executing Actions and Shared Plans

Robots should be capable of interacting in a cooperative and adaptive manner with their human counterparts in open-ended tasks that can change in real-time. An important aspect of the robot behavior will be the ability to acquire new knowledge of the cooperative tasks by observing and interacting with humans. The current research addresses this challenge. We present results from a cooperative human-robot interaction system that has been specifically developed for portability between different humanoid platforms, by abstraction layers at the perceptual and motor interfaces. In the perceptual domain, the resulting system is demonstrated to learn to recognize objects and to recognize actions as sequences of perceptual primitives, and to transfer this learning, and recognition, between different robotic platforms. For execution, composite actions and plans are shown to be learnt on one robot and executed successfully on a different one. Most importantly, the system provides the ability to link actions into shared plans, that form the basis of human-robot cooperation, applying principles from human cognitive development to the domain of robot cognitive systems.

Prokaryotic Bio-Inspired Model for Embryonics

This paper is presented in conjunction with, and forms the first part of, the paper entitled “Prokaryotic Bio-Inspired Systems.” In this part we propose and investigate a novel prokaryotic cell-based bio-inspired model suitable to implement self-healing bio-inspired systems. A key feature of our model is that system reliability can be increased with a minimal amount of hardware overhead. It also offers a bio-inspired compression/decompression technique that exploits the intimate relationship between different genes. Distributed DNA, highly dynamic and flexible routing resources and optimized self-repair characteristics (using Block and cell elimination) are some of the other advantages of the proposed model.

Safe Adaptive Compliance Control of a Humanoid Robotic Arm with Anti-Windup Compensation and Posture Control

Safety is very important for physical human-robot interaction. Compliance control can solve an important aspect of the safety problem by dealing with impact and other forces arising during close contact between humans and robots.An adaptive compliance model reference controller was implemented in real-time on a 4 degrees of freedom (DOF) humanoid robotic arm in Cartesian space. The robot manipulator has been controlled in such a way as to follow the compliant passive behaviour of a reference mass-spring-damper system model subject to an externally sensed force. The redundant DOF were used to control the robot motion in a human-like pattern via minimization of effort, a function of gravity. Associated actuator saturation issues were addressed by incorporating a novel anti-windup (AW) compensator originally developed for a neural network scheme. The controller was simulated for a robotic arm representing the Bristol-Elumotion-Robotic-Torso II (BERT II) and then tested on the real BERT II arm. BERT II has been developed in collaboration by Bristol Robotics Laboratory and Elumotion Ltd.

TACTIP - tactile fingertip device, texture analysis through optical tracking of skin features

In this paper we present texture analysis results for TACTIP, a versatile tactile sensor and artificial fingertip which exploits compliant materials and optical tracking. In comparison to previous MEMS sensors, the TACTIP device is especially suited to tasks for which humans use their fingertips; examples include object manipulation, contact sensing, pressure sensing and shear force detection. This is achieved whilst maintaining a high level of robustness. Previous development of the TACTIP device has proven the devices capability to measure force interaction and identify shape through edge detection. Here we present experimental results which confirm the ability to also identify textures. This is achieved by measuring the vibration of the in-built human-like skin features in relation to textured profiles. Modifications to the mechanical design of the TACTIP are explored to increase the sensitivity to finer textured profiles. The results show that a contoured outer skin, similar to a finger print, increases the sensitivity of the device.

SCRATCHbot: active tactile sensing in a whiskered mobile robot

The rodent vibrissal (whisker) system is one of the most widely investigated model sensory systems in neuroscience owing to its discrete organisation from the sensory apparatus (the whisker shaft) all the way to the sensory cortex, its ease of manipulation, and its presence in common laboratory animals. Neurobiology shows us that the brain nuclei and circuits that process vibrissal touch signals, and that control the positioning and movement of the whiskers, form a neural architecture that is a good model of how the mammalian brain, in general, coordinates sensing with action. In this paper we describe SCRATCHbot, a biomimetic robot based on the rat whisker system, and show how this robot is providing insight into the operation of neural systems underlying vibrissal control, and is helping us to understand the active sensing strategies that animals employ in order to boost the quality and quantity of information provided by their sensory organs.

Seeing by Touch: Evaluation of a Soft Biologically-Inspired Artificial Fingertip in Real-Time Active Touch

Effective tactile sensing for artificial platforms remains an open issue in robotics. This study investigates the performance of a soft biologically-inspired artificial fingertip in active exploration tasks. The fingertip sensor replicates the mechanisms within human skin and offers a robust solution that can be used both for tactile sensing and gripping/manipulating objects. The softness of the optical sensors contact surface also allows safer interactions with objects. High-level tactile features such as edges are extrapolated from the sensors output and the information is used to generate a tactile image. The work presented in this paper aims to investigate and evaluate this artificial fingertip for 2D shape reconstruction. The sensor was mounted on a robot arm to allow autonomous exploration of different objects. The sensor and a number of human participants were then tested for their abilities to track the raised perimeters of different planar objects and compared. By observing the technique and accuracy of the human subjects, simple but effective parameters were determined in order to evaluate the artificial systems performance. The results prove the capability of the sensor in such active exploration tasks, with a comparable performance to the human subjects despite it using tactile data alone whereas the human participants were also able to use proprioceptive cues.

Prokaryotic Bio-Inspired System

This paper presents a novel bio-inspired artificial system that is based on biological prokaryotic organisms and their artificial model, and proposes a new type of fault tolerant, self-healing architecture. The system comprises of a sea of bio-inspired cells, arranged in a rectangular array with a topology that is similar to that employed by FPGAs. A key feature of the array is its high level of fault tolerance, achieved with only minimal amount of hardware overhead. Inspired by similar biological processes, the technique is based on direct-correlated redundancy, where the redundant (stand-by) configuration bits, as extrinsic experience, are shared between blocks and cells of a colony in the artificial system. Bio-inspired array implementation is particularly advantageous in applications where the system is subject to extreme environmental conditions such as temperature, radiation, SEU (Single Event Upset) etc. and where fault tolerance is of particular importance.

Adaptive multi-dimensional compliance control of a humanoid robotic arm with anti-windup compensation

An adaptive multi-dimensional compliance model reference controller was implemented in real-time on a 4 degrees of freedom (DOF) of the humanoid Bristol-Elumotion-Robotic-Torso II (BERT II) arm in Cartesian space. The robot manipulator has been controlled in such a way as to follow the compliant passive behaviour of a reference mass-spring-damper system model subject to externally sensed forces/torques in all DOF. The relevant reference model converts all measured torques into their equivalent forces at the end-effector and reacts accordingly. The suggested control scheme takes in particular account of the multi-variable aspect and the problem of body own torques when measuring external torques. The redundant DOF were used to control the robot motion in a human-like pattern via effort minimization. Associated actuator saturation issues were addressed by incorporating a novel anti-windup (AW) compensator.

Bio-inspired self-test for evolvable fault tolerant hardware systems

This paper presents a novel bio-inspired self-test technique for the implementation of evolvable fault tolerant systems based on the structure, behavior and processes observed in prokaryote unicellular organisms. Such Unitronic (unicellular electronic) artificial systems are implemented by FPGA-like bio-inspired cellular arrays and made up of structurally identical cells. All cells possess self-diagnostic and self-healing capability. Our underlying conceptual postulation is: if it can be guaranteed that during the test phase a cell, the internal functionality of which is configured with a complementary input sequence, demonstrates the same functionality, as that with the original sequence during its normal mode of operation, then the cell is fault free, otherwise it is faulty. Our proposed self-test can evaluate all stuck-at-zero and stuck-at-one faults of the system if at any time only one fault exists. Hardware redundancy is optimised because the same hardware, by simple reconfiguration is able to test itself and thus eliminates the need of duplicated, triplicated hardware.