Patrick M. Pilarski

Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning

As a contribution toward the goal of adaptable, intelligent artificial limbs, this work introduces a continuous actor-critic reinforcement learning method for optimizing the control of multi-function myoelectric devices. Using a simulated upper-arm robotic prosthesis, we demonstrate how it is possible to derive successful limb controllers from myoelectric data using only a sparse human-delivered training signal, without requiring detailed knowledge about the task domain. This reinforcement-based machine learning framework is well suited for use by both patients and clinical staff, and may be easily adapted to different application domains and the needs of individual amputees. To our knowledge, this is the first my-oelectric control approach that facilitates the online learning of new amputee-specific motions based only on a one-dimensional (scalar) feedback signal provided by the user of the prosthesis.

Proceedings of the first workshop on peripheral machine interfaces: Going beyond traditional surface electromyography

One of the hottest topics in rehabilitation robotics is that of proper control of prosthetic devices. Despite decades of research, the state of the art is dramatically behind the expectations. To shed light on this issue, in June, 2013 the first international workshop on Present and future of non-invasive peripheral nervous system (PNS)–Machine Interfaces (MI; PMI) was convened, hosted by the International Conference on Rehabilitation Robotics. The keyword PMI has been selected to denote human–machine interfaces targeted at the limb-deficient, mainly upper-limb amputees, dealing with signals gathered from the PNS in a non-invasive way, that is, from the surface of the residuum. The workshop was intended to provide an overview of the state of the art and future perspectives of such interfaces; this paper represents is a collection of opinions expressed by each and every researcher/group involved in it.

Model-Free reinforcement learning with continuous action in practice

Reinforcement learning methods are often considered as a potential solution to enable a robot to adapt to changes in real time to an unpredictable environment. However, with continuous action, only a few existing algorithms are practical for real-time learning. In such a setting, most effective methods have used a parameterized policy structure, often with a separate parameterized value function. The goal of this paper is to assess such actor-critic methods to form a fully specified practical algorithm. Our specific contributions include 1) developing the extension of existing incremental policy-gradient algorithms to use eligibility traces, 2) an empirical comparison of the resulting algorithms using continuous actions, 3) the evaluation of a gradient-scaling technique that can significantly improve performance. Finally, we apply our actor-critic algorithm to learn on a robotic platform with a fast sensorimotor cycle (10ms). Overall, these results constitute an important step towards practical real-time learning control with continuous action.

A Genome-Wide Aberrant RNA Splicing in Patients with Acute Myeloid Leukemia Identifies Novel Potential Disease Markers and Therapeutic Targets

Purpose: Despite new treatments, acute myeloid leukemia (AML) remains an incurable disease. More effective drug design requires an expanded view of the molecular complexity that underlies AML. Alternative splicing of RNA is used by normal cells to generate protein diversity. Growing evidence indicates that aberrant splicing of genes plays a key role in cancer. We investigated genome-wide splicing abnormalities in AML and based on these abnormalities, we aimed to identify novel potential biomarkers and therapeutic targets. Experimental Design: We used genome-wide alternative splicing screening to investigate alternative splicing abnormalities in two independent AML patient cohorts [Dana-Farber Cancer Institute (DFCI) (Boston, MA) and University Hospital de Nantes (UHN) (Nantes, France)] and normal donors. Selected splicing events were confirmed through cloning and sequencing analysis, and than validated in 193 patients with AML. Results: Our results show that approximately 29% of expressed genes genome-wide were differentially and recurrently spliced in patients with AML compared with normal donors bone marrow CD34+ cells. Results were reproducible in two independent AML cohorts. In both cohorts, annotation analyses indicated similar proportions of differentially spliced genes encoding several oncogenes, tumor suppressor proteins, splicing factors, and heterogeneous-nuclear-ribonucleoproteins, proteins involved in apoptosis, cell proliferation, and spliceosome assembly. Our findings are consistent with reports for other malignances and indicate that AML-specific aberrations in splicing mechanisms are a hallmark of AML pathogenesis. Conclusions: Overall, our results suggest that aberrant splicing is a common characteristic for AML. Our findings also suggest that splice variant transcripts that are the result of splicing aberrations create novel disease markers and provide potential targets for small molecules or antibody therapeutics for this disease. Clin Cancer Res; 20(5); 1135–45. ©2013 AACR.

Inherited and acquired variations in the hyaluronan synthase 1 (HAS1) gene may contribute to disease progression in multiple myeloma and Waldenstrom macroglobulinemia.

To characterize genetic contributions toward aberrant splicing of the hyaluronan synthase 1 (HAS1) gene in multiple myeloma (MM) and Waldenstrom macroglobulinemia (WM), we sequenced 3616 bp in HAS1 exons and introns involved in aberrant splicing, from 17 patients. We identified a total of 197 HAS1 genetic variations (GVs), a range of 3 to 24 GVs/patient, including 87 somatic GVs acquired in splicing regions of HAS1. Nearly all newly identified inherited and somatic GVs in MM and/or WM were absent from B chronic lymphocytic leukemia, nonmalignant disease, and healthy donors. Somatic HAS1 GVs recurred in all hematopoietic cells tested, including normal CD34(+) hematopoietic progenitor cells and T cells, or as tumor-specific GVs restricted to malignant B and plasma cells. An in vitro splicing assay confirmed that HAS1 GVs direct aberrant HAS1 intronic splicing. Recurrent somatic GVs may be enriched by strong mutational selection leading to MM and/or WM.

Adaptive artificial limbs: a real-time approach to prediction and anticipation

Predicting the future has long been regarded as a powerful means to improvement and success. The ability to make accurate and timely predictions enhances our ability to control our situation and our environment. Assistive robotics is one prominent area in which foresight of this kind can bring improved quality of life. In this article, we present a new approach to acquiring and maintaining predictive knowledge during the online ongoing operation of an assistive robot. The ability to learn accurate, temporally abstracted predictions is shown through two case studies: 1) able-bodied myoelectric control of a robot arm and 2) an amputees interactions with a myoelectric training robot. To our knowledge, this research is the first demonstration of a practical method for real-time prediction learning during myoelectric control. Our approach therefore represents a fundamental tool for addressing one major unsolved problem: amputee-specific adaptation during the ongoing operation of a prosthetic device. The findings in this article also contribute a first explicit look at prediction learning in prosthetics as an important goal in its own right, independent of its intended use within a specific controller or system. Our results suggest that real-time learning of predictions and anticipations is a significant step toward more intuitive myoelectric prostheses and other assistive robotic devices.

Tuning-free step-size adaptation

Incremental learning algorithms based on gradient descent are effective and popular in online supervised learning, reinforcement learning, signal processing, and many other application areas. An oft-noted drawback of these algorithms is that they include a step-size parameter that needs to be tuned for best performance, which may require manual intervention and significant domain knowledge or additional data. In many cases, an entire vector of step-size parameters (e.g., one for each input feature) needs to be tuned in order to attain the best performance of the algorithm. To address this, several methods have been proposed for adapting step sizes online. For example, Suttons IDBD method can find the best vector step size for the LMS algorithm, and Schraudolphs ELK1 method, an extension of IDBD to neural networks, has proven effective on large applications, such as 3D hand tracking. However, to date all such step-size adaptation methods have included a tunable step-size parameter of their own, which we call the meta-step-size parameter. In this paper we show that the performance of existing step-size adaptation methods are strongly dependent on the choice of their meta-step-size parameter and that their meta-step-size parameter cannot be set reliably in a problem-independent way. We introduce a series of modifications and normalizations to the IDBD method that together eliminate the need to tune the meta-step-size parameter to the particular problem. We show that the resulting overall algorithm, called Autostep, performs as well or better than the existing step-size adaptation methods on a number of idealized and robot prediction problems and does not require any tuning of its meta-step-size parameter. The ideas behind Autostep are not restricted to the IDBD method and the same principles are potentially applicable to other incremental learning settings, such as reinforcement learning.

NOTCH2 and FLT3 gene mis-splicings are common events in patients with acute myeloid leukemia (AML): new potential targets in AML

Our previous studies revealed an increase in alternative splicing of multiple RNAs in cells from patients with acute myeloid leukemia (AML) compared with CD34(+) bone marrow cells from normal donors. Aberrantly spliced genes included a number of oncogenes, tumor suppressor genes, and genes involved in regulation of apoptosis, cell cycle, and cell differentiation. Among the most commonly mis-spliced genes (>70% of AML patients) were 2, NOTCH2 and FLT3, that encode myeloid cell surface proteins. The splice variants of NOTCH2 and FLT3 resulted from complete or partial exon skipping and utilization of cryptic splice sites. Longitudinal analyses suggested that NOTCH2 and FLT3 aberrant splicing correlated with disease status. Correlation analyses between splice variants of these genes and clinical features of patients showed an association between NOTCH2-Va splice variant and overall survival of patients. Our results suggest that NOTCH2 and FLT3 mis-splicing is a common characteristic of AML and has the potential to generate transcripts encoding proteins with altered function. Thus, splice variants of these genes might provide disease markers and targets for novel therapeutics.

Real-time prediction learning for the simultaneous actuation of multiple prosthetic joints

Integrating learned predictions into a prosthetic control system promises to enhance multi-joint prosthesis use by amputees. In this article, we present a preliminary study of different cases where it may be beneficial to use a set of temporally extended predictions - learned and maintained in real time - within an engineered or learned prosthesis controller. Our study demonstrates the first successful combination of actor-critic reinforcement learning with real-time prediction learning. We evaluate this new approach to control learning during the myoelectric operation of a robot limb. Our results suggest that the integration of real-time prediction and control learning may speed control policy acquisition, allow unsupervised adaptation in myoelectric controllers, and facilitate synergies in highly actuated limbs. These experiments also show that temporally extended prediction learning enables anticipatory actuation, opening the way for coordinated motion in assistive robotic devices. Our work therefore provides initial evidence that realtime prediction learning is a practical way to support intuitive joint control in increasingly complex prosthetic systems.

Aberrant Splicing, Hyaluronan Synthases and Intracellular Hyaluronan as Drivers of Oncogenesis and Potential Drug Targets

Current evidence suggests a significant role of aberrant splicing in the development and maintenance of malignancy. This multistep, tightly regulated epigenetic process leads to the production of abnormal proteins with abnormal functions contributing to underlying mechanisms of malignant transformation. Splicing patterns in malignant cells can be altered not only by the mutations detected on the aberrantly spliced gene, but also by the mutations detected on the genes encoding splicing factors. For example, aberrant pre-mRNA splicing, leading to intracellular or extracellular HA synthesis by HASs, contributes to the initiation and progression of various types of cancer. The influence of intracellular HA appears to be particularly significant and is promoted by aberrant splicing. In this review we report a model describing oncogenic potential of aberrant splicing, with a focus on HAS1 and intracellular HA. We also suggest that the influence of splicing mutations on malignant disease is likely multifactorial. For the triple axis of HA, HAS1 and RHAMM, mutations in HAS1 provide an indicator that these aberrations contribute to the events that lead to malignancy through increased risk and predisposition. Here, we also summarize the impact of splicing abnormalities on cancer and the possible oncogenic impact of aberrantly spliced HAS1. In conclusion, we emphasize that specific gene splice variants and the splicing process itself offer potential targets for novel drug treatment strategies.