Jeremy H. Gillula

Design of guaranteed safe maneuvers using reachable sets: Autonomous quadrotor aerobatics in theory and practice

For many applications, the control of a complex nonlinear system can be made easier by modeling the system as a collection of simplified hybrid modes, each representing a particular operating regime. An example of this is the decomposition of complex aerobatic flights into sequences of discrete maneuvers, an approach that has proven very successful for both human piloted and autonomously controlled aircraft. However, a critical step when designing such control systems is to ensure the safety and feasibility of transitions between these maneuvers. This work presents a hybrid dynamics framework for the design of guaranteed safe switching regions and is applied to a quadrotor helicopter performing an autonomous backflip. The regions are constructed using reachable sets calculated via a Hamilton-Jacobi differential game formulation, and experimental results are presented from flight tests on the STARMAC quadrotor platform.

Applications of hybrid reachability analysis to robotic aerial vehicles

The control of complex non-linear systems can be aided by modeling each system as a collection of simplified hybrid modes, with each mode representing a particular operating regime defined by the system dynamics or by a region of the state space in which the system operates. Guarantees on the safety and performance of such hybrid systems can still be challenging to generate, however. Reachability analysis using a dynamic game formulation with Hamilton—Jacobi methods provides a useful way to generate these types of guarantees, and the technique is flexible enough to analyze a wide variety of systems. This paper presents two applications of reachable sets, both focused on guaranteeing the safety and performance of robotic aerial vehicles. In the first example, reachable sets are used to design and implement a backflip maneuver for a quadrotor helicopter. In the second, reachability analysis is used to design a decentralized collision avoidance algorithm for multiple quadrotors. The theory for both examples is explained, and successful experimental results are presented from flight tests on the STARMAC quadrotor helicopter platform.

Reachability-based safe learning with Gaussian processes

Reinforcement learning for robotic applications faces the challenge of constraint satisfaction, which currently impedes its application to safety critical systems. Recent approaches successfully introduce safety based on reachability analysis, determining a safe region of the state space where the system can operate. However, overly constraining the freedom of the system can negatively affect performance, while attempting to learn less conservative safety constraints might fail to preserve safety if the learned constraints are inaccurate. We propose a novel method that uses a principled approach to learn the systems unknown dynamics based on a Gaussian process model and iteratively approximates the maximal safe set. A modified control strategy based on real-time model validation preserves safety under weaker conditions than current approaches. Our framework further incorporates safety into the reinforcement learning performance metric, allowing a better integration of safety and learning. We demonstrate our algorithm on simulations of a cart-pole system and on an experimental quadrotor application and show how our proposed scheme succeeds in preserving safety where current approaches fail to avoid an unsafe condition.

Guaranteed Safe Online Learning via Reachability: tracking a ground target using a quadrotor

While machine learning techniques have become popular tools in the design of autonomous systems, the asymptotic nature of their performance guarantees means that they should not be used in scenarios in which safety and robustness are critical for success. By pairing machine learning algorithms with rigorous safety analyses, such as Hamilton-Jacobi-Isaacs (HJI) reachability, this limitation can be overcome. Guaranteed Safe Online Learning via Reachability (GSOLR) is a framework which combines HJI reachability with general machine learning techniques, allowing for the design of robotic systems which demonstrate both high performance and guaranteed safety. In this paper we show how the GSOLR framework can be applied to a target tracking problem, in which an observing quadrotor helicopter must keep a target ground vehicle with unknown (but bounded) dynamics inside its field of view at all times, while simultaneously attempting to build a motion model of the target. The resulting algorithm was implemented on board the Stanford Testbed of Autonomous Rotorcraft for Multi-Agent Control, and was compared to a naive safety-only algorithm and a learning-only algorithm. Experimental results illustrate the success of the GSOLR algorithm, even under scenarios in which the machine learning algorithm performed poorly (and would otherwise lead to unsafe actions), thus demonstrating the power of this technique.

Design and Analysis of Hybrid Systems, with Applications to Robotic Aerial Vehicles

Decomposing complex, highly nonlinear systems into aggregates of simpler hybrid modes has proven to be a very successful way of designing and controlling autonomous vehicles. Examples include the use of motion primitives for robotic motion planning and equivalently the use of discrete maneuvers for aggressive aircraft trajectory planning. In all of these approaches, it is extremely important to verify that transitions between modes are safe. In this paper, we present the use of a Hamilton-Jacobi differential game formulation for finding continuous reachable sets as a method of generating provably safe transitions through a sequence of modes for a quadrotor performing a backflip maneuver.

Hybrid Systems in Robotics

Robotics has provided the motivation and inspiration for many innovations in planning and control. From nonholonomic motion planning [1] to probabilistic road maps [2], from capture basins [3] to preimages [4] of obstacles to avoid, and from geometric nonlinear control [5], [6] to machine-learning methods in robotic control [7], there is a wide range of planning and control algorithms and methodologies that can be traced back to a perceived need or anticipated benefit in autonomous or semiautonomous systems.

Stanford Testbed of Autonomous Rotorcraft for Multi-Agent Control

The Stanford Testbed of Autonomous Rotorcraft for Multi-Agent Control, a fleet of quadrotor helicopters, has been developed as a testbed for novel algorithms that enable autonomous operation of aerial vehicles. The testbed has been used to validate multiple algorithms such as reactive collision avoidance, collision avoidance through Nash Bargaining, path planning, cooperative search and aggressive maneuvering. This article briefly describes the algorithms presented and provides references for a more in-depth formulation, and the accompanying movie shows the demonstration of the algorithms on the testbed.

Reducing Conservativeness in Safety Guarantees by Learning Disturbances Online: Iterated Guaranteed Safe Online Learning

Reinforcement learning has proven itself to be a powerful technique in robotics, however it has not often been employed to learn a controller in a hardware-in-the-loop environment due to the fact that spurious training data could cause a robot to take an unsafe (and potentially catastrophic) action. One approach to overcoming this limitation is known as Guaranteed Safe Online Learning via Reachability (GSOLR), in which the controller being learned is wrapped inside another controller based on reachability analysis that seeks to guarantee safety against worst-case disturbances. This paper proposes a novel improvement to GSOLR which we call Iterated Guaranteed Safe Online Learning via Reachability (IGSOLR), in which the worst-case disturbances are modeled in a state-dependent manner (either parametrically or nonparametrically), this model is learned online, and the safe sets are periodically recomputed (in parallel with whatever machine learning is being run online to learn how to control the system). As a result the safety of the system automatically becomes neither too liberal nor too conservative, depending only on the actual system behavior. This allows the machine learning algorithm running in parallel the widest possible latitude in performing its task while still guaranteeing system safety. In addition to explaining IGSOLR, we show how it was used in a real-world example, namely that of safely learning an altitude controller for a quadrotor helicopter. The resulting controller, which was learned via hardware-inthe-loop reinforcement learning, out-performs our original handtuned controller while still maintaining safety. To our knowledge, this is the first example in the robotics literature of an algorithm in which worst-case disturbances are learned online in order to guarantee system safety.

Sampling-based approximation of the viability kernel for high-dimensional linear sampled-data systems

Proving that systems satisfy hard input and state constraints is frequently desirable when designing cyber-physical systems. One method for doing so is to compute the viability kernel, the subset of the state space for which a control signal exists that is guaranteed to keep the system within the constraints over some time horizon. In this paper we present a novel method for approximating the viability kernel for linear sampled-data systems using a sampling-based algorithm, which by its construction offers a direct trade-off between scalability and accuracy. We also prove that the algorithm is correct, that its convergence properties are optimal, and demonstrate it on a simple example. We conclude by briefly describing additional results which are omitted due to space constraints.

Verification and control of hybrid systems using reachability analysis with machine learning

This talk will present reachability analysis as a tool for model checking and controller synthesis for dynamic systems. We will consider the problem of guaranteeing reachability to a given desired subset of the state space while satisfying a safety property defined in terms of state constraints. We allow for nonlinear and hybrid dynamics, and possibly non-convex state constraints. We use these results to synthesize controllers that ensure safety and reachability properties under bounded model disturbances that vary continuously. We also consider the effects of sampling and quantization. The resulting control policy is an explicit feedback law involving both a selection of continuous inputs and discrete switching commands at each time instant, based upon measurement of system state. We discuss real time implementations of this, and present several examples from multiple aerial vehicle control, human-robot interaction, and multi-stage games. Finally, we show how reachability techniques can be used to guarantee safety in robotics systems that use machine learning to generate dynamic models on-the-fly.