N. Kemal Ure

Decentralized control of partially observable Markov decision processes

The focus of this paper is on solving multi-robot planning problems in continuous spaces with partial observability. Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) are general models for multi-robot coordination problems, but representing and solving Dec-POMDPs is often intractable for large problems. To allow for a high-level representation that is natural for multi-robot problems and scalable to large discrete and continuous problems, this paper extends the Dec-POMDP model to the Decentralized Partially Observable Semi-Markov Decision Process (Dec-POSMDP). The Dec-POSMDP formulation allows asynchronous decision-making by the robots, which is crucial in multi-robot domains. We also present an algorithm for solving this Dec-POSMDP which is much more scalable than previous methods since it can incorporate closed-loop belief space macro-actions in planning. These macro-actions are automatically constructed to produce robust solutions. The proposed methods performance is evaluated on a complex multi-robot package delivery problem under uncertainty, showing that our approach can naturally represent multi-robot problems and provide high-quality solutions for large-scale problems.

Integration of Path/Maneuver Planning in Complex Environments for Agile Maneuvering UCAVs

In this work, we consider the problem of generating agile maneuver profiles for Unmanned Combat Aerial Vehicles in 3D Complex environments. This problem is complicated by the fact that, generation of the dynamically and geometrically feasible flight trajectories for agile maneuver profiles requires search of nonlinear state space of the aircraft dynamics. This work suggests a two layer feasible trajectory/maneuver generation system. Integrated Path planning (considers geometrical, velocity and acceleration constraints) and maneuver generation (considers saturation envelope and attitude continuity constraints) system enables each layer to solve its own reduced order dimensional feasibility problem, thus simplifies the problem and improves the real time implement ability. In Trajectory Planning layer, to solve the time depended path planning problem of an unmanned combat aerial vehicles, we suggest a two step planner. In the first step, the planner explores the environment through a randomized reachability tree search using an approximate line segment model. The resulting connecting path is converted into flight way points through a line-of-sight segmentation. In the second step, every consecutive way points are connected with B-Spline curves and these curves are repaired probabilistically to obtain a geometrically and dynamically feasible path. This generated feasible path is turned in to time depended trajectory with using time scale factor considering the velocity and acceleration limits of the aircraft. Maneuver planning layer is constructed upon multi modal control framework, where the flight trajectories are decomposed to sequences of maneuver modes and associated parameters. Maneuver generation algorithm, makes use of mode transition rules and agility metric graphs to derive feasible maneuver parameters for each mode and overall sequence. Resulting integrated system; tested on simulations for 3D complex environments, gives satisfactory results and promises successful real time implementation.

An automated battery management system to enable persistent missions with multiple aerial vehicles

This paper presents the development and hardware implementation of an autonomous battery maintenance mechatronic system that significantly extends the operational time of battery powered small-scaled unmanned aerial vehicles (UAVs). A simultaneous change and charge approach is used to overcome the significant downtime experienced by existing charge-only approaches. The automated system quickly swaps a depleted battery of a UAV with a replenished one while simultaneously recharging several other batteries. This results in a battery maintenance system with low UAV downtime, arbitrarily extensible operation time, and a compact footprint. Hence, the system can enable multi-agent UAV missions that require persistent presence. This capability is illustrated by developing and testing in flight a centralized autonomous planning and learning algorithm that incorporates a probabilistic health model dependent on vehicle battery health that is updated during the mission, and replans to improve the performance based on the improved model. Flight test results are presented for a 3-h-long persistent mission with three UAVs that each has an endurance of 8-10 min on a single battery charge (more than 100 battery swaps).

Adaptive planning for markov decision processes with uncertain transition models via incremental feature dependency discovery

Solving large scale sequential decision making problems without prior knowledge of the state transition model is a key problem in the planning literature. One approach to tackle this problem is to learn the state transition model online using limited observed measurements. We present an adaptive function approximator (incremental Feature Dependency Discovery (iFDD)) that grows the set of features online to approximately represent the transition model. The approach leverages existing feature-dependencies to build a sparse representation of the state transition model. Theoretical analysis and numerical simulations in domains with state space sizes varying from thousands to millions are used to illustrate the benefit of using iFDD for incrementally building transition models in a planning framework.

Design and flight testing of an autonomous variable-pitch quadrotor

This video submission presents a design concept of an autonomous variable-pitch quadrotor with constant motor speed. The main aim of this work is to increase the maneuverability of the quadrotor vehicle concept while largely maintaining its mechanical simplicity. This added maneuverability will allow autonomous agile maneuvers like inverted hover and flip. A custom in lab built quadrotor with onboard attitude stabilization is developed and tested in the ACLs (Aerospace Controls Laboratory) RAVEN (Real-time indoor Autonomous Vehicle test ENvironment). Initial flight results show that the quadrotor is capable of waypoint tracking and hovering both upright and inverted.

Health Aware Stochastic Planning For Persistent Package Delivery Missions Using Quadrotors

In persistent missions, taking systems health and capability degradation into account is an essential factor to predict and avoid failures. The state space in health-aware planning problems is often a mixture of continuous vehicle-level and discrete mission-level states. This in particular poses a challenge when the mission domain is partially observable and restricts the use of computationally expensive forward search methods. This paper presents a method that exploits a structure that exists in many health-aware planning problems and performs a two-layer planning scheme. The lower layer exploits the local linearization and Gaussian distribution assumption over vehicle-level states while the higher layer maintains a non-Gaussian distribution over discrete mission-level variables. This two-layer planning scheme allows us to limit the expensive online forward search to the mission-level states, and thus predict systems behavior over longer horizons in the future. We demonstrate the performance of the method on a long duration package delivery mission using a quadrotor in a partially-observable domain in the presence of constraints and health/capability degradation.

A Probabilistic Algorithm for Mode Based Motion Planning of Agile Unmanned Air Vehicles in Complex Environments

Abstract In this work, we consider the design of a probabilistic trajectory planner for a highly maneuverable unmanned air vehicle flying in a dense and complex city-like environment. Our design hinges on the decomposition of the problem into a) flight controls of fundamental agile-maneuvering flight modes and b) trajectory planning using these controlled flight modes from which almost any aggressive maneuver (or a combination of) can be created. This allows significant decreases in control input space and thus search dimensions, resulting in a natural way to design controllers and implement trajectory planning using the closed-form flight modes. Focusing on the trajectory planning part, we provide a three-step probabilistic trajectory planner. In the first step, the algorithm rapidly explores the environment through a randomized reachability tree search using an approximate line segment models. The resulting connecting path is converted into flight milestones through a line-of-sight segmentation. This path and the corresponding milestones are refined with a single-query Probabilistic Road Map (PRM) implementation that creates dynamically feasible flight paths with distinct flight mode selections. We address the problematic issue of narrow passages through non-uniform distributed capture regions, which prefer state solutions that align the vehicle to enter the milestone region in line with the next milestone to come. Numerical simulations in 3D and 2D demonstrate the ability of the method to provide real-time solutions in dense and complex environments.

Planning for large-scale multiagent problems via hierarchical decomposition with applications to UAV health management

This paper introduces a novel hierarchical decomposition approach for solving Multiagent Markov Decision Processes (MMDPs) by exploiting coupling relationships in the reward function. MMDP is a natural framework for solving stochastic multi-stage multiagent decision-making problems, such as optimizing mission performance of Unmanned Aerial Vehicles (UAVs) with stochastic health dynamics. However, computing the optimal solutions is often intractable because the state-action spaces scale exponentially with the number of agents. Approximate solution techniques do exist, but they typically rely on extensive domain knowledge. This paper presents the Hierarchically Decomposed MMDP (HD-MMDP) algorithm, which autonomously identifies different degrees of coupling in the reward function and decomposes the MMDP into a hierarchy of smaller MDPs that can be solved separately. Solutions to the smaller MDPs are embedded in an autonomously constructed tree structure to generate an approximate solution to the original problem. Simulation results show HD-MMDP obtains more cumulative reward than that of the existing algorithm for a ten-agent Persistent Search and Track (PST) mission, which is a cooperative multi-UAV mission with more than 1019 states, stochastic fuel consumption model, and health progression model.

The development of a Software and Hardware-in-the-Loop Test System for ITU-PSAT II nano satellite ADCS

In this work, we present the operational concept of ITU-PSAT II, the reconfigurable fault-tolerant ADCS architecture and the associated Software and Hardware-in-the-Loop Test System for three-axis active control. ADCS of ITU PSAT II consists of three distinct hardware layers integrating sensors, actuators, and ADCS computer over the CAN bus. A multi mode control algorithm which acts over different operation modes and actuator / sensor failure scenarios, along with simulation results, is provided. For testing operational properties of ADCS, a Software and Hardware-in-the-Loop Test system were developed. The system includes direct satellite-bus emulation, integration of satellite hardware, an air bearing table and the Helmholtz Coil magnetic field emulator.12

Distributed Learning for Planning Under Uncertainty Problems with Heterogeneous Teams

This paper considers the problem of multiagent sequential decision making under uncertainty and incomplete knowledge of the state transition model. A distributed learning framework, where each agent learns an individual model and shares the results with the team, is proposed. The challenges associated with this approach include choosing the model representation for each agent and how to effectively share these representations under limited communication. A decentralized extension of the model learning scheme based on the Incremental Feature Dependency Discovery (Dec-iFDD) is presented to address the distributed learning problem. The representation selection problem is solved by leveraging iFDD’s property of adjusting the model complexity based on the observed data. The model sharing problem is addressed by having each agent rank the features of their representation based on the model reduction error and broadcast the most relevant features to their teammates. The algorithm is tested on the multi-agent block building and the persistent search and track missions. The results show that the proposed distributed learning scheme is particularly useful in heterogeneous learning setting, where each agent learns significantly different models. We show through large-scale planning under uncertainty simulations and flight experiments with state-dependent actuator and fuel-burn- rate uncertainty that our planning approach can outperform planners that do not account for heterogeneity between agents.