Markus Nowak

Stable myoelectric control of a hand prosthesis using non-linear incremental learning

Stable myoelectric control of hand prostheses remains an open problem. The only successful human–machine interface is surface electromyography, typically allowing control of a few degrees of freedom. Machine learning techniques may have the potential to remove these limitations, but their performance is thus far inadequate: myoelectric signals change over time under the influence of various factors, deteriorating control performance. It is therefore necessary, in the standard approach, to regularly retrain a new model from scratch. We hereby propose a non-linear incremental learning method in which occasional updates with a modest amount of novel training data allow continual adaptation to the changes in the signals. In particular, Incremental Ridge Regression and an approximation of the Gaussian Kernel known as Random Fourier Features are combined to predict finger forces from myoelectric signals, both finger-by-finger and grouped in grasping patterns. We show that the approach is effective and practically applicable to this problem by first analyzing its performance while predicting single-finger forces. Surface electromyography and finger forces were collected from 10 intact subjects during four sessions spread over two different days; the results of the analysis show that small incremental updates are indeed effective to maintain a stable level of performance. Subsequently, we employed the same method on-line to teleoperate a humanoid robotic arm equipped with a state-of-the-art commercial prosthetic hand. The subject could reliably grasp, carry and release everyday-life objects, enforcing stable grasping irrespective of the signal changes, hand/arm movements and wrist pronation and supination.

Low-cost wearable multichannel surface EMG acquisition for prosthetic hand control

Prosthetic hand control based on the acquisition and processing of surface electromyography signals (sEMG) is a well-established method that makes use of the electric potentials evoked by the physiological contraction processes of one or more muscles. Furthermore intelligent mobile medical devices are on the brink of introducing safe and highly sophisticated systems to help a broad patient community to regain a considerable amount of life quality. The major challenges which are inherent in such integrated systems design are mainly to be found in obtaining a compact system with a long mobile autonomy, capable of delivering the required signal requirements for EMG based prosthetic control with up to 32 simultaneous acquisition channels and - with an eye on a possible future exploitation as a medical device - a proper perspective on a low priced system. Therefore, according to these requirements we present a wireless, mobile platform for acquisition and communication of sEMG signals embedded into a complete mobile control system structure. This environment further includes a portable device such as a laptop providing the necessary computational power for the control and a commercially available robotic hand-prosthesis. Means of communication among those devices are based on the Bluetooth standard. We show, that the developed low cost mobile device can be used for proper prosthesis control and that the device can rely on a continuous operation for the usual daily life usage of a patient.

Online Bimanual Manipulation Using Surface Electromyography and Incremental Learning

The paradigm of simultaneous and proportional myocontrol of hand prostheses is gaining momentum in the rehabilitation robotics community. As opposed to the traditional surface electromyography classification schema, in simultaneous and proportional control the desired force/torque at each degree of freedom of the hand/wrist is predicted in real-time, giving to the individual a more natural experience, reducing the cognitive effort and improving his dexterity in daily-life activities. In this study we apply such an approach in a realistic manipulation scenario, using 10 non-linear incremental regression machines to predict the desired torques for each motor of two robotic hands. The prediction is enforced using two sets of surface electromyography electrodes and an incremental, non-linear machine learning technique called Incremental Ridge Regression with Random Fourier Features. Nine able-bodied subjects were engaged in a functional test with the aim to evaluate the performance of the system. The robotic hands were mounted on two hand/wrist orthopedic splints worn by healthy subjects and controlled online. An average completion rate of more than 95% was achieved in single-handed tasks and 84% in bimanual tasks. On average, 5 min of retraining were necessary on a total session duration of about 1 h and 40 min. This work sets a beginning in the study of bimanual manipulation with prostheses and will be carried on through experiments in unilateral and bilateral upper limb amputees thus increasing its scientific value.

The LET Procedure for Prosthetic Myocontrol: Towards Multi-DOF Control Using Single-DOF Activations.

Simultaneous and proportional myocontrol of dexterous hand prostheses is to a large extent still an open problem. With the advent of commercially and clinically available multi-fingered hand prostheses there are now more independent degrees of freedom (DOFs) in prostheses than can be effectively controlled using surface electromyography (sEMG), the current standard human-machine interface for hand amputees. In particular, it is uncertain, whether several DOFs can be controlled simultaneously and proportionally by exclusively calibrating the intended activation of single DOFs. The problem is currently solved by training on all required combinations. However, as the number of available DOFs grows, this approach becomes overly long and poses a high cognitive burden on the subject. In this paper we present a novel approach to overcome this problem. Multi-DOF activations are artificially modelled from single-DOF ones using a simple linear combination of sEMG signals, which are then added to the training set. This procedure, which we named LET (Linearly Enhanced Training), provides an augmented data set to any machine-learning-based intent detection system. In two experiments involving intact subjects, one offline and one online, we trained a standard machine learning approach using the full data set containing single- and multi-DOF activations as well as using the LET-augmented data set in order to evaluate the performance of the LET procedure. The results indicate that the machine trained on the latter data set obtains worse results in the offline experiment compared to the full data set. However, the online implementation enables the user to perform multi-DOF tasks with almost the same precision as single-DOF tasks without the need of explicitly training multi-DOF activations. Moreover, the parameters involved in the system are statistically uniform across subjects.

A virtual piano-playing environment for rehabilitation based upon ultrasound imaging

In this paper we evaluate ultrasound imaging as a human-machine interface in the context of rehabilitation. Ultrasound imaging can be used to estimate finger forces in real-time with a short and easy calibration procedure. Forces are individually predicted using a transducer fixed on the forearm, which leaves the hand completely free to operate. In this application, a standard ultrasound machine is connected to a virtual-reality environment in which a human operator can play a dynamic harmonium over two octaves, using either finger (including the thumb). The interaction in the virtual environment is managed via a fast collision detection algorithm and a physics engine. Ten human subjects have been engaged in two games of increasing difficulty. Our experimental results, both objective and subjective, clearly show that both tasks could be accomplished to the required degree of precision and that the subjects underwent a typical learning curve. The learning happened uniformly, irrespective of the required finger, force or note. Such a system could be made portable, and has potential applications as rehabilitation device for amputees and muscle impaired, even at home.

Online tactile myography for simultaneous and proportional hand and wrist myocontrol

Tactile myography is a promising method for dexterous myocontrol. It stems from the idea of detecting muscle activity, and hence the desired actions to be performed by a prosthesis, via the muscle deformations induced by said activity, using a tactile sensor on the stump. Tactile sensing is high-resolution force / pressure sensing; such a technique promises to yield a rich flow of information about an amputated subject’s intent. In this work we propose a preliminary comparison between tactile myography and surface electromyography enforcing simultaneous and proportional control during an online target-reaching experiment. Six intact subjects and a trans-radial amputee were engaged in repeated hand opening / closing, wrist flexion / extension and wrist pronation / supination, to various degrees of activation. Albeit limited, the results we show indicate that tactile myography enforces an almost uniformly better performance than sEMG.

Wrist and grasp myocontrol: Simplifying the training phase

The term “myocontrol” denotes, in the assistive robotics / machine learning community, the feed-forward control of a dexterous prosthetic device enforced by a disabled human subject, typically an amputee, using the activation of remnant muscles. Myocontrol relies on a human-machine interface (HMI), which converts muscle activation signals of diverse nature into control commands for the prosthetic device. Although novel kinds of HMIs are being explored, the traditional basis for myocontrol is surface electromyography (sEMG), a technique which records the electrical field emitted by the muscles when contracting. Due to the complexity of the HMI, it is desirable to shorten the calibration procedure as much as possible whenever the prosthetic device has two or more degrees of freedom (DOFs). In this paper we extend the Linearly Enhanced Training (LET) procedure, already employed in myocontrol of single fingers and their combinations, to myocontrol of two DOFs of the wrist plus the action of grasping (hand opening and closing). The LET principle, according to which combined simultaneous activation of more than one DOF are artificially modelled using a simple linear combination of single-DOF activations, was tested on six intact subjects engaged in wrist flexion, extension, pronation and grasping. The experimental results show that LET can solve this problem with a similar level of accuracy as in the case of single fingers. As well, the LET hyperparameters are shown to be invariant across subjects.

Multi-modal myocontrol: Testing combined force- and electromyography

Myocontrol, that is control of prostheses using bodily signals, has proved in the decades to be a surprisingly hard problem for the scientific community of assistive and rehabilitation robotics. In particular, traditional surface electromyography (sEMG) seems to be no longer enough to guarantee dexterity (i.e., control over several degrees of freedom) and, most importantly, reliability. Multi-modal myocontrol is concerned with the idea of using novel signal gathering techniques as a replacement of, or alongside, sEMG, to provide high-density and diverse signals to improve dexterity and make the control more reliable. In this paper we present an offline and online assessment of multi-modal sEMG and force myography (FMG) targeted at hand and wrist myocontrol. A total number of twenty sEMG and FMG sensors were used simultaneously, in several combined configurations, to predict opening/closing of the hand and activation of two degrees of freedom of the wrist of ten intact subjects. The analysis was targeted at determining the optimal sensor combination and control parameters; the experimental results indicate that sEMG sensors alone perform worst, yielding a nRMSE of 9.1%, while mixing FMG and sEMG or using FMG only reduces the nRMSE to 5.2–6.6%. To validate these results, we engaged the subject with median performance in an online goal-reaching task. Analysis of this further experiment reveals that the online behaviour is similar to the offline one.

A preliminary study towards automatic detection of failures in myocontrol

Reliability is still the main issue in myocontrol: enforcing (dexterous) grasping, releasing and moving exactly and only when the wearer desires it. One specific path towards the solution of this problem is incremental machine learning, leading to interactive myocontrol, in which unreliability is taken care of via on-demand model updates, requested by the experimenter and/or the subject herself/himself. One natural drawback of this approach is that an “oracle” is needed at all times, stopping the prediction and calling for an update whenever this is deemed to be the case; an automated oracle, as reliable as possible, is therefore very desirable. This work shows the results of a preliminary study in which we tried to find features of the control signals and predictions, as well as environmental information (inertial sensors and motor currents) to automatically identify the failures of the myocontrol system. The outcome is promising, showing that a classifier can match the observer’s judgement with an overall average accuracy of slightly more than 75%.

Exploiting Knowledge Composition to Improve Real-Life Hand Prosthetic Control

In myoelectric prosthesis control, one of the hottest topics nowadays is enforcing simultaneous and proportional (s/p) control over several degrees of freedom. This problem is particularly hard and the scientific community has so far failed to provide a stable and reliable s/p control, effective in daily-life activities. In order to improve the reliability of this form of control, in this paper we propose on-the-fly knowledge composition, thereby reducing the burden of matching several patterns at the same time, and simplifying the task of the system. In particular, we show that using our method it is possible to dynamically compose a model by juxtaposing subsets of previously gathered (sample, target) pairs in real-time, rather than composing a single model in the beginning and then hoping it can reliably distinguish all patterns. Fourteen intact subjects participated in an experiment, where repetitive daily-life tasks (e.g. ironing a cloth) were performed using a commercially available dexterous prosthetic hand mounted on a splint and wirelessly controlled using a machine learning method. During the experiment, the subjects performed these tasks using myocontrol with and without knowledge composition and the results demonstrate that employing knowledge composition allowed better performance, i.e. reducing the overall task completion time by 30%.