Robert D. Oaks

Implementation and Metrics for a Trajectory Prediction Validation Methodology

At the heart of every air traffic decision support tool’s functionality is its trajectory prediction, where a trajectory is defined as the 4-dimensional path of an aircraft. This paper presents a comprehensive implementation for measuring the accuracy of a trajectory prediction in support of a validation methodology. The process includes four main processing areas: (1) parsing and checking the actual positional data of an aircraft (i.e., the aircraft’s actual trajectory), (2) parsing the trajectory predictions, (3) comparing the actual and predicted aircraft trajectory by sampling and measuring, and (4) analyzing the results. This paper presents detailed descriptions of the sampling process and metrics used to measure the accuracy of a predicted trajectory. Several aspects of the analysis and implementation are provided as well, such as inferential statistical approaches and graphical user interfaces to examine individual flights.

Methodology for evaluating and regression testing a conflict probe

A conflict probe is an air traffic management decision support tool that predicts aircraft-to-aircraft and aircraft-to-airspace conflicts. In order to achieve the confidence of the air traffic controllers who are provided this tool, a conflict probe must accurately predict these events. To ensure their continued confidence, the accuracy should not only be assessed in the laboratory before the probe is deployed but continue to be reassessed as the system undergoes upgrades and software changes. Furthermore, it is desirable to use recorded air traffic data to test these tools in order to preserve real-world errors that affect their performance. This paper utilizes a proven approach that modifies surveillance radar track data in time to create traffic scenarios containing conflicts with characteristic properties similar to those encountered in actual air traffic operations. It is these time shifted traffic scenarios that are used to evaluate the conflict probe. This paper describes the detailed process of evaluating the missed and false conflict predictions, the calculation of the corresponding error probabilities, and a regression testing methodology to examine two runs of the conflict probe to determine if the conflict prediction accuracy has improved or degraded over time. A detailed flight example is presented which illustrates the specific processing involved in conflict accuracy analysis. Next using a scenario of many flights, a methodology utilizing categorical data analysis techniques is applied to determine if a new version of the conflict probes software significantly improved or degraded in conflict prediction accuracy.

Analysis of the Aircraft to Aircraft Conflict Properties in the National Airspace System

*† ‡ The primary function of administering the United States’ National Airspace System (NAS) is the air traffic controller task of actively monitoring assigned aircraft and resolving the conflicts (i.e. losses of minimum separations between aircraft) anticipated some time in the future. To mitigate the safety risks of increased traffic growth and effectively designing automation to aid in the separation management task, knowledge of the characteristics or properties of the conflicts is required. This paper reports on a comprehensive study that has examined these properties by collecting traffic data from all 20 NAS en route centers, developing software models to determine these events, implementing experimental design techniques to calibrate them, validating the models by comparing to advanced operational systems, and presenting detailed graphical and statistical analysis of the results. I. Introduction In the United States, the overall system of managing and controlling air traffic is known as the National Airspace System (NAS), which is administered by the Federal Aviation Administration (FAA). Detailed procedures involving restrictions on routing, speeds, and altitudes are an integral part of the NAS. These restrictions severely reduce the amount of aircraft traffic that NAS can accommodate, yet are needed to ensure the high level of safety required. At the heart of these operations is the human air traffic controller who must synthesize many pieces of timely information including radar surveillance information and flight data. Their fundamental responsibility is to ensure the safety of the aircraft flying within their regions of airspace in the most efficient means possible. To accomplish this, air traffic controllers actively monitor their aircraft and then resolve any conflicts (i.e., loss of minimum separation between aircraft or restricted airspace) predicted some time into the future. Furthermore, these resolutions are administered by air traffic controller voice instructions via radio transmissions to the aircraft. In the current system, there are automation systems that aid the air traffic controller mainly in the monitoring part of the task such as the ground based tactical and strategic conflict probes. In the en route centers, typically managing the aircraft above 18,000 feet, the Host Computer System’s (HCS) Conflict Alert function provides tactical alerts. The upgrade to the HCS, still under development, called the En Route Automation Modernization (ERAM), replaces Conflict Alert with several categories of alerts with the basic function requiring a minimum of 75 seconds warning. The User Request Evaluation Tool (URET), developed by MITRE Corporation’s Center for Advanced Aviation System Development, is an example of a strategic conflict probe in operation in the en route centers. It predicts conflicts up to 20 minutes in the future with typically a minimum warning of five minutes. Even with the aid of ground-based conflict probes, the task of separating aircraft will become increasingly difficult, since most air traffic service providers in the United States and Europe anticipate significant growth in air traffic. The growth is expected to out pace the capacity limits of the aviation systems, resulting in greater congestion and inefficiency. The interagency Joint Development Planning Office (JPDO) in the United States foresees a traffic demand increase by 2025 up to three times the number of flights of today’s traffic. 1 Given the need for enhanced safety and efficiency, broad categories of advances in ground and airborne automation are required. The JDPO, as established in their charter under the “Vision-100” legislation (Public Law 108-176) signed by President G. W. Bush in December 2003, has mandated a next generation operational concept of the NAS for 2025. 1 This next generation

Determination of Lateral Flight Adherence in Recorded Air Traffic Data

When using recorded air traffic data to measure the accuracy of air traffic management decision support tools that use intent-based trajectory modelers, it is necessary to determine whether or not an aircraft is adhering to its known flight plan clearance. This paper defines what is meant by adherence and presents metrics that can be used to define lateral flight adherence. The paper describes an algorithm that is currently being applied that uses a subset of these metrics. The paper then presents a number of examples obtained from recorded air traffic data, which show instances where aircraft deviate from their known lateral clearance. The paper then presents a number of alternative approaches that could be used to implement a better algorithm for determining whether or not an aircraft was in lateral adherence based on recorded air traffic data.

Prototype Implementation and Concept Validation of a 4-D Trajectory Fuel Burn Model Application

As new technologies are implemented to enhance the National Airspace System it is important to consider their impact on the environment and on the cost of operations. One of the major considerations for both is the amount of fuel consumed by aircraft. This paper describes a unique methodology that can be used to estimate the total fuel consumed by aircraft based on their 4-D trajectories and the weather through which they flew. This methodology is based on the European Organisation for the Safety of Air Navigation (EUROCONTROL) BADA (Base of Aircraft Data) fuel burn model. The paper then describes how the BADA fuel burn model and weather data provided by the National Weather Service were implemented in a prototype application and how this prototype was validated using actual data recorded during a flight for a specific aircraft. The paper concludes with a summary that includes suggestions for improvements that should be incorporated when implementing the production version of the application.

TIME SHIFTING AIR TRAFFIC DATA FOR QUANTITATIVE EVALUATION OF A CONFLICT PROBE

A conflict probe is an air traffic management decision support tool that predicts aircraft-to-aircraft and aircraft-to-airspace conflicts. In order to achieve the confidence of the air traffic controllers who are provided this tool, a conflict probe must accurately predict these conflicts. This paper discusses how a conflict probes quantitative accuracy requirements can be tested using hypothesis testing techniques. The paper also asserts that air traffic scenarios based on recorded field data are essential to the evaluation of a conflict probe and states that time shifting these scenarios can create data samples necessary to perform the hypothesis testing. This paper then compares three time shifting techniques: time compression, random time adjustment, and an implementation of a genetic algorithm.

A Study on the Feasibility of Using a Genetic Algorithm to Generate Realistic Air Traffic Scenarios Based on Recorded Field Data

Air traffic scenarios based on recorded live data are used for the development and analysis of decision support tools used by air traffic controllers. Frequently these scenarios need to be modified in order to create aircraft-toaircraft encounters that are not present in the live data. This paper shows that a genetic algorithm can be used to time shift the flights in a scenario in order to create encounters with specific constrained characteristics. For this study these constraints were the distributions of the closest point of approach and the encounter angle of the encounters. The paper first describes how a genetic algorithm was implemented and then presents the results of two series of tests. The first series of tests were designed to determine if the implementation of a genetic algorithm could solve the problem. The second series of tests were designed to assess the time it took the implementation to solve the problem. The results of the study showed that a genetic algorithm could solve the problem in a reasonable time.

Methodology for the generation of air traffic scenarios based on recorded traffic data

This article presents a methodology to generate air traffic scenarios for the support of the testing and analysis of air traffic management decision support tools. This methodology was developed by and is currently being used by the Conflict Probe Assessment Team, a workgroup within the Federal Aviation Administrations Simulation and Modeling Group located at the Federal Aviation Administrations William J. Hughes Technical Center. The individual flights within the scenarios generated by this methodology follow realistic flight routes, yet the air traffic in these scenarios contains aircraft-to-aircraft conflicts and encounters that do not exist in the field. Scenarios with these characteristics are necessary to evaluate decision support tools that predict conflicts. The paper describes each of the three steps that comprise the methodology. The first step is data extraction during which traffic data, available from various sources and recorded in different formats, is extracted and placed into a set of relational database tables. The second step is data modification, where the data in the tables may be manipulated for test purposes. This manipulation may consist of simply culling undesirable flights, or it may involve using a genetic algorithm to time-shift the flights to induce conflicts or encounters. The third step is scenario generation, where scenarios are created based on the traffic data retrieved from the modified database tables. During this final step, the scenarios may be formatted for various target systems. The paper then describes how the Conflict Probe Assessment Team has used this methodology to generate scenarios that have been used for accuracy and risk reduction testing and for a study assessing the effect of weather forecast errors.

Evaluation of a genetic algorithm that modifies air traffic data for conflict probe testing

Air traffic scenarios based on recorded live data are essential for the development, testing and evaluation of air traffic automation. It is often desirable to modify the live data to introduce additional encounters and conflicts because live data generally contains no conflicts. To this end a genetic algorithm (GA) has been developed that time shifts individual flights in the scenario based on a set of characteristics. Although this approach has been successful, larger scenarios and additional constraints dramatically increase the time for the GA to reach an acceptable solution. This paper first introduces an enhanced GA technique that distributes the processing over several computers. Following the biological metaphor further, this approach is called the Island Model because each instance of the running algorithm represents an island where periodic sharing of information represents migration between islands. The paper concludes with an evaluation of the Island Model utilizing a set of designed experiments. The evaluation will not only consider time to solve but the quality of the solution produced relative to the specified constraints, with the goal of identifying important factors and optimal settings for those factors.

Time shifting beacon radar reports in recorded air traffic data

The Federal Aviation Administrationpsilas National Airspace System is a network of air navigation facilities, air traffic control facilities, and airports, along with the technologies and the rules and regulations to operate the system. One of these technologies is a conflict probe, which is used by air traffic controllers to predict aircraft-to-aircraft and aircraft-to-airspace conflicts. To be effective the conflict probe must predict potential conflicts accurately. Although functional testing can be performed to evaluate a conflict probe, it is essential that the probe also be tested using scenarios that preserve the real-world errors that affect the probepsilas accuracy. This requires that test scenarios be developed that are based on recorded air traffic data. However, recorded air traffic data generally does not contain actual conflicts. This requires that the flights be shifted in time so that aircraft-to-aircraft conflicts are created in the test scenarios. This paper describes a technique in which recorded beacon radar reports can be time shifted in order to create aircraft-to-aircraft conflicts for conflict probe testing.