Testing Artificial Intelligence in High-Performance, Tactical Aircraft

Abstract

The United States Air Force Test Pilot School (USAF TPS), in partnership with Lockheed Martin Skunk Works, developed and tested a suite of algorithms and artificial intelligence (AI) agents flown on an F-16D. The system was designed to autonomously locate and navigate to an emitter and optically recognize its associated vehicle under the project Have SPI-DERs. This project was the first to explore the use of AI agents trained in simulation using deep reinforcement learning and tested on a high-performance aircraft in the real world performing routing and computer vision tasks. During flight test, the Have SPIDERS project set out to demonstrate the SUT s ability to find and fix the emitter through three phases. The first phase passively detected the emitter using a small antenna array and routed the aircraft using a neural network to reduce the uncertainty volume of the emitter s location. The second phase routed the aircraft using a second neural network to an image point where the emitter could be optically identified. The final phase used a third neural network to optically recognize a vehicle collocated with the emitter. The first two phases each used two feedforward neural networks functioning as actor-critic algorithms. The third phase used a RetinaNet architecture. This paper uses the Have SPIDERS project as a vehicle to discuss lessons learned that can be applied to future AI flight test. These lessons focus on six areas for consideration during system development, test planning, and execution: understanding system under test (SUT) development, instrumenting the AI to understand its behavior, planning for unpredictability, simulation-to-real-world generalization, hardware choices, and manned surrogate testing considerations.