James H. Neilan

Collaborating with Autonomous Agents

With the anticipated increase of small unmanned aircraft systems (sUAS) entering into the National Airspace System, it is highly likely that vehicle operators will be teaming with fleets of small autonomous vehicles. The small vehicles may consist of sUAS, which are 55 pounds or less that typically will y at altitudes 400 feet and below, and small ground vehicles typically operating in buildings or defined small campuses. Typically, the vehicle operators are not concerned with manual control of the vehicle; instead they are concerned with the overall mission. In order for this vision of high-level mission operators working with fleets of vehicles to come to fruition, many human factors related challenges must be investigated and solved. First, the interface between the human operator and the autonomous agent must be at a level that the operator needs and the agents can understand. This paper details the natural language human factors e orts that NASA Langleys Autonomy Incubator is focusing on. In particular these e orts focus on allowing the operator to interact with the system using speech and gestures rather than a mouse and keyboard. With this ability of the system to understand both speech and gestures, operators not familiar with the vehicle dynamics will be able to easily plan, initiate, and change missions using a language familiar to them rather than having to learn and converse in the vehicles language. This will foster better teaming between the operator and the autonomous agent which will help lower workload, increase situation awareness, and improve performance of the system as a whole.

Operating in "Strange New Worlds" and Measuring Success - Test and Evaluation in Complex Environments

Software tools are being developed by the Autonomy Incubator at NASAs Langley Research Center that will provide an integrated and scalable capability to support research and non-research flight operations across several flight domains, including urban and mixed indoor-outdoor operations. These tools incorporate a full range of data products to support mission planning, approval, flight operations, and post-flight review. The system can support a number of different operational scenarios that can incorporate live and archived data streams for UAS operators, airspace regulators, and other important stakeholders. Example use cases are described that illustrate how the tools will benefit a variety of users in nominal and off-nominal operational scenarios. An overview is presented for the current state of the toolset, including a summary of current demonstrations that have been completed. Details of the final, fully operational capability are also presented, including the interfaces that will be supported to ensure compliance with existing and future airspace operations environments.

A Flexible Flight Control System for Rapid GNC and Distributed Control Deployment

The NASA Langley Autonomy Incubator focuses on enabling autonomous, cooperative operations of multiple small Unmanned Aerial Vehicles (UAV) and, more generally, creating autonomous system technologies that change the way people and goods are moved from place to place. To enable rapid test and deployment of autonomous algorithms an avionics system is under development. The system, called the Autonomous Entity Operations Network Flight Control System (AEON-FCS), is being developed to be capable of implementing both classic Guidance, Navigation, and Control (GNC) law algorithms in a monolithic system architecture and also a network connected fully distributed control system. AEON-FCS is a subset of the overall AEON system that includes the Avionics System for Remotely Operated Vehicles LiTe (ASROV-LT) codebase and additional data centric tools for distributed control implementation and rapid simulation and testing. The ASROV-LT codebase provides utilities for hard real-time flight control loop processing and serialized sensor parsing. To enable rapid testing of autonomous algorithms, AEON-FCS provides seamless integration between simulation and hardware by utilizing a data centric inter-process communication approach and a global data bus available on the network. A goal for the AEON-FCS is to enable implementation of fully distributed control. Processing locations may be paired with sensors and distributed across either an airframe or across different air and/or ground vehicles on the network connected system. AEON-FCS aims to enable agile, high performance, robust operation of single or multiple cooperative UAV performing challenging missions like search and rescue under the forest canopy, delivery in rapidly changing unstructured environments (rescue in burning building), and ensemble collaboration for robust resilient system operations in hostile environments. The current state of the structure, function, and novel features of the AEON-FCS are described herein.

Reinforcement Learning with Autonomous Small Unmanned Aerial Vehicles in Cluttered Environments - "After all these years among humans, you still haven't learned to smile."

We present ongoing work in the Autonomy Incubator at NASA Langley Research Center (LaRC) exploring the efficacy of a data set aggregation approach to reinforcement learning for small unmanned aerial vehicle (sUAV) flight in dense and cluttered environments with reactive obstacle avoidance. The goal is to learn an autonomous flight model using training experiences from a human piloting a sUAV around static obstacles. The training approach uses video data from a forward-facing camera that records the human pilots flight. Various computer vision based features are extracted from the video relating to edge and gradient information. The recorded human-controlled inputs are used to train an autonomous control model that correlates the extracted feature vector to a yaw command. As part of the reinforcement learning approach, the autonomous control model is iteratively updated with feedback from a human agent who corrects undesired model output. This data driven approach to autonomous obstacle avoidance is explored for simulated forest environments furthering autonomous flight under the tree canopy research. This enables flight in previously inaccessible environments which are of interest to NASA researchers in Earth and Atmospheric sciences.

Towards an Open, Distributed Software Architecture for UxS Operations - "It's difficult to work in groups when you're omnipotent," - Q Star Trek: The Next Generation:: Deja Q (1990)

To address the growing need to evaluate, test, and certify an ever expanding ecosystem of UxS platforms in preparation of cultural integration, NASA Langley Research Centers Autonomy Incubator (AI) has taken on the challenge of developing a software framework in which UxS platforms developed by third parties can be integrated into a single system which provides evaluation and testing, mission planning and operation, and out-of-the-box autonomy and data fusion capabilities. This software framework, named AEON (Autonomous Entity Operations Network), has two main goals. The first goal is the development of a cross-platform, extensible, onboard software system that provides autonomy at the mission execution and course-planning level, a highly configurable data fusion framework sensitive to the platforms available sensor hardware, and plug-and-play compatibility with a wide array of computer systems, sensors, software, and controls hardware. The second goal is the development of a ground control system that acts as a test-bed for integration of the proposed heterogeneous fleet, and allows for complex mission planning, tracking, and debugging capabilities. The ground control system should also be highly extensible and allow plug-and-play interoperability with third party software systems. In order to achieve these goals, this paper proposes an open, distributed software architecture which utilizes at its core the Data Distribution Service (DDS) standards, established by the Object Management Group (OMG), for inter-process communication and data flow. The design decisions proposed herein leverage the advantages of existing robotics software architectures and the DDS standards to develop software that is scalable, high-performance, fault tolerant, modular, and readily interoperable with external platforms and software.

Using Multimodal Input for Autonomous Decision Making for Unmanned Systems - “What it needs in order to evolve, is a human quality. Our capacity to leap beyond logic.” - Capt. Kirk, Star Trek: The Motion Picture

Autonomous decision making in the presence of uncertainly is a deeply studied problem space particularly in the area of autonomous systems operations for land, air, sea, and space vehicles. Various techniques ranging from single algorithm solutions to complex ensemble classifier systems have been utilized in a research context in solving mission critical flight decisions. Realized systems on actual autonomous hardware, however, is a difficult systems integration problem, constituting a majority of applied robotics development timelines. The ability to reliably and repeatedly classify objects during a vehicles mission execution is vital for the vehicle to mitigate both static and dynamic environmental concerns such that the mission may be completed successfully and have the vehicle operate and return safely. In this paper, the Autonomy Incubator proposes and discusses an ensemble learning and recognition system planned for our autonomous framework, AEON, in selected domains, which fuse decision criteria, using prior experience on both the individual classifier layer and the ensemble layer to mitigate environmental uncertainty during operation.