Ivana Palunko

Learning swing-free trajectories for UAVs with a suspended load

Attaining autonomous flight is an important task in aerial robotics. Often flight trajectories are not only subject to unknown system dynamics, but also to specific task constraints. This paper presents a motion planning method for generating trajectories with minimal residual oscillations (swing-free) for rotorcraft carrying a suspended loads. We rely on a finite-sampling, batch reinforcement learning algorithm to train the system for a particular load. We find criteria that allow the trained agent to be transferred to a variety of models, state and action spaces and produce a number of different trajectories. Through a combination of simulations and experiments, we demonstrate that the inferred policy is robust to noise and the unmodeled dynamics of the system. The contributions of this work are 1) applying reinforcement learning to solve the problem of finding swing-free trajectories for rotorcraft, 2) designing a problem-specific feature vector for value function approximation, 3) giving sufficient conditions for successful learning transfer to different models, state and action spaces, and 4) verification of the resulting trajectories in both simulation and autonomous control of quadrotors with suspended loads.

A reinforcement learning approach towards autonomous suspended load manipulation using aerial robots

In this paper, we present a problem where a suspended load, carried by a rotorcraft aerial robot, performs trajectory tracking. We want to accomplish this by specifying the reference trajectory for the suspended load only. The aerial robot needs to discover/learn its own trajectory which ensures that the suspended load tracks the reference trajectory. As a solution, we propose a method based on least-square policy iteration (LSPI) which is a type of reinforcement learning algorithm. The proposed method is verified through simulation and experiments.

Automated Aerial Suspended Cargo Delivery through Reinforcement Learning

Abstract Cargo-bearing unmanned aerial vehicles (UAVs) have tremendous potential to assist humans by delivering food, medicine, and other supplies. For time-critical cargo delivery tasks, UAVs need to be able to quickly navigate their environments and deliver suspended payloads with bounded load displacement. As a constraint balancing task for joint UAV-suspended load system dynamics, this task poses a challenge. This article presents a reinforcement learning approach for aerial cargo delivery tasks in environments with static obstacles. We first learn a minimal residual oscillations task policy in obstacle-free environments using a specifically designed feature vector for value function approximation that allows generalization beyond the training domain. The method works in continuous state and discrete action spaces. Since planning for aerial cargo requires very large action space (over 106 actions) that is impractical for learning, we define formal conditions for a class of robotics problems where learning can occur in a simplified problem space and successfully transfer to a broader problem space. Exploiting these guarantees and relying on the discrete action space, we learn the swing-free policy in a subspace several orders of magnitude smaller, and later develop a method for swing-free trajectory planning along a path. As an extension to tasks in environments with static obstacles where the load displacement needs to be bounded throughout the trajectory, sampling-based motion planning generates collision-free paths. Next, a reinforcement learning agent transforms these paths into trajectories that maintain the bound on the load displacement while following the collision-free path in a timely manner. We verify the approach both in simulation and in experiments on a quadrotor with suspended load and verify the methods safety and feasibility through a demonstration where a quadrotor delivers an open container of liquid to a human subject. The contributions of this work are two-fold. First, this article presents a solution to a challenging, and vital problem of planning a constraint-balancing task for an inherently unstable non-linear system in the presence of obstacles. Second, AI and robotics researchers can both benefit from the provided theoretical guarantees of system stability on a class of constraint-balancing tasks that occur in very large action spaces.

Cooperative suspended object manipulation using reinforcement learning and energy-based control

Cooperative dynamic object manipulation can extend the manipulation capabilities of robot-robot and human-robot teams. In order to be able to inject energy into various suspended objects of unknown parameters, in this paper we propose an adaptive controller which combines reinforcement learning with energy based swing-up control. The proposed controller is successfully verified in a single robot and human-robot experimental setup for different types of suspended objects.

State estimation, robust control and obstacle avoidance for multicopter in cluttered environments: EuRoC experience and results

This paper reports the results of the UNIZG-FER team in the third European Robotics Challenge (EuRoC). More precisely, the results of the 1st qualifying stage of the challenge where a micro aerial vehicle (MAV) is tested in realistic simulation scenarios. The paper presents the entire controller setup, starting from the power distribution level, low level cascade controllers with wind disturbance rejection to high level obstacle avoidance algorithms. The proposed controllers were tested in realistic simulations environments where their effectiveness was evaluated based on objective criteria set by the challenge organizers.

Small Helicopter Control Design Based on Model Reduction and Decoupling

In this paper a complete nonlinear mathematical model of a small scale helicopter is derived. A coupling between input and output variables, revealed by the model, is investigated. The influences that particular inputs have on particular outputs are examined, and their dependence on flying conditions is shown. In order to demonstrate this dependence, the model is linearized in various operating points, and linear, direct and decoupling, controllers are determined. Simulation results, presented at the end of the paper, confirm that the proposed control structure could be successfully used for gain scheduling or switching control of a small scale helicopter in order to provide acrobatic flight by using simple linear controllers.

Decentralized trust-based self-organizing cooperative control

In this paper, we propose a mechanism for decentralized trust-based self-organizing cooperative control. The driving force of this work is the idea that not all agents are equally adept to take the role of a group leader at a given moment. Depending on the application, agents with the most desirable sensing, communication or processing capabilities should take the leading role. Hence, our main objective is to establish a group hierarchy in a decentralized fashion and reconfigure the underlying communication topology accordingly. The hierarchy is established upon trust values towards each agent in the group. These trust values reflect the reputation that each agent enjoys within the group. Initially, we construct observed trust values of each agent by exploiting its local observations and observations received from the neighbors. Next, these observed trust values are passed on to the mechanism that favors more accomplished agents by assigning those agents higher trust values. Since this mechanism is decentralized, scenarios with unequal trusts values toward an agent need to be handled with caution in order not to compromise the collective behavior. To that end, we employ adaptive control concepts when negotiating trust values. The proposed mechanism is illustrated through a decentralized formation control case study.

Human-in-the-loop control of multi-agent aerial systems

This paper presents a framework for human-in-the-loop control of multi-agent systems (MASs), comprised of different unmanned aerial vehicles (UAVs), with realistic communication channels that give rise to intermittent, corrupted and delayed information. We propose a novel design of a human-machine interface (HMI) that allows a human to become a supervisor, when necessary, instead of a single unit operator. The proposed framework allows the supervisor to interact with its surroundings by deploying a dexterous aerial robot within MASs. Experimental results, presented at the end of the paper, verify functionality of the proposed framework by bringing together a heterogeneous team of aerial robots, each with its particular capabilities, and a human operator in order to close a valve in a disaster stricken industrial environment.

Multi-agent control in degraded communication environments

The goal of this paper is to compute transmission rates and delays that provably stabilize Multi-Agent Systems (MASs) in the presence of disturbances and noise. In other words, given the existing information delay among the agents and the underlying communication topology, we determine rates at which information between the agents need to be exchanged such that the MAS of interest is ℒp-stable with bias, where this bias accounts for distorted/noisy data. In an effort to consider MASs characterized by sets of equilibrium points, the notions of ℒp-stability (with bias) and ℒp-detectability with respect to a set are utilized. Using Lyapunov-Razumikhin type of arguments, we are able to consider delays that are greater than the transmission intervals. The present methodology is applicable to general (not merely to single- and double-integrator) heterogeneous linear agents, directed topologies and output feedback. The computed transmission rates are experimentally verified employing a group of off-the-shelf quadcopters.

Decentralized Cooperative Control in Degraded Communication Environments

Abstract We compute transmission rates and delays that provably stabilize multiagent systems (MASs) in the presence of disturbances and noise. Namely, given the existing information delay among the agents and the underlying communication topology, we determine the rates at which information between the agents needs to be exchanged such that the MAS of interest is L p -stable with bias, where this bias accounts for noisy data. To consider MASs characterized by sets of equilibrium points, the notions of L p -stability (with bias) and L p -detectability with respect to a set are employed. Using arguments of the Lyapunov-Razumikhin type, we are able to consider delays greater than the transmission intervals. Our method is applicable to general (not merely to single- and double-integrator) heterogeneous linear agents, directed topologies, and output feedback. The computed transmission rates are experimentally verified by use of a group of off-the-shelf quadcopters.