Hollis F. Ryan

Review of Trajectory Accuracy Methodology and Comparison of Error Measurement Metrics

The aircraft trajectory is a prediction of the aircraft’s anticipated flight path. Trajectory accuracy is measured by comparing the trajectory prediction to the actual flight path of the aircraft. Trajectory accuracy analysis starts with a sampling process that selects the actual and predicted trajectory positions for measurement and concludes with the application of statistical and graphical analysis methods. At the base of a study are the error measurements and how they are defined. Their definitions are the focus of this paper. Two spatial metrics and two time metrics were specifically defined and compared in the horizontal dimension. Both detailed synthetic flight examples and a large traffic sample were employed to evaluate these error metrics. The second spatial metric and first time error metric proved to be superior methods in turns and approximately equivalent elsewhere. From the traffic sample, the standard deviation of the differences between the presented spatial and time error metrics was significant, ranging from 0.6 to 4.2 nautical miles and 9 to 55 seconds, respectively. Therefore, it is concluded that it is essential to clearly define the particular error measurement technique applied in a trajectory accuracy study to not only be relevant, but also for the results to be extensible and cross comparable with other studies.

Methodology for evaluating and regression testing a conflict probe

A conflict probe is an air traffic management decision support tool that predicts aircraft-to-aircraft and aircraft-to-airspace conflicts. In order to achieve the confidence of the air traffic controllers who are provided this tool, a conflict probe must accurately predict these events. To ensure their continued confidence, the accuracy should not only be assessed in the laboratory before the probe is deployed but continue to be reassessed as the system undergoes upgrades and software changes. Furthermore, it is desirable to use recorded air traffic data to test these tools in order to preserve real-world errors that affect their performance. This paper utilizes a proven approach that modifies surveillance radar track data in time to create traffic scenarios containing conflicts with characteristic properties similar to those encountered in actual air traffic operations. It is these time shifted traffic scenarios that are used to evaluate the conflict probe. This paper describes the detailed process of evaluating the missed and false conflict predictions, the calculation of the corresponding error probabilities, and a regression testing methodology to examine two runs of the conflict probe to determine if the conflict prediction accuracy has improved or degraded over time. A detailed flight example is presented which illustrates the specific processing involved in conflict accuracy analysis. Next using a scenario of many flights, a methodology utilizing categorical data analysis techniques is applied to determine if a new version of the conflict probes software significantly improved or degraded in conflict prediction accuracy.

Prototype Implementation and Concept Validation of a 4-D Trajectory Fuel Burn Model Application

As new technologies are implemented to enhance the National Airspace System it is important to consider their impact on the environment and on the cost of operations. One of the major considerations for both is the amount of fuel consumed by aircraft. This paper describes a unique methodology that can be used to estimate the total fuel consumed by aircraft based on their 4-D trajectories and the weather through which they flew. This methodology is based on the European Organisation for the Safety of Air Navigation (EUROCONTROL) BADA (Base of Aircraft Data) fuel burn model. The paper then describes how the BADA fuel burn model and weather data provided by the National Weather Service were implemented in a prototype application and how this prototype was validated using actual data recorded during a flight for a specific aircraft. The paper concludes with a summary that includes suggestions for improvements that should be incorporated when implementing the production version of the application.

Comparison of host radar positions to Global Positioning Satellite positions

The Federal Aviation Administration (FAA) air traffic control system relies directly on aircraft locations provided by the long range en route surveillance radars. The accuracy of the radars is an important factor in determining the overall performance of the system. To support the planned modernization of the air traffic control system a study was conducted to measure the accuracy of the radar tracking function of the current system. The aircraft radar tracks were compared to the positions produced by the Global Positioning Satellite System (GPS), which was considered the true aircraft position. The GPS data was available from the FAAs Reduced Vertical Separation Minimum Certification Program. Utilizing the host air traffic management data distribution system at each air route traffic control center that captures the radar tracking data, 265 flights of radar tracking data were compared to their GPS positions. Three distance metrics were used. The time coincident straight line distance, referred to as the horizontal track error, and its two orthogonal components: cross track error (side to side error) and along track error (longitudinal error) were calculated. A total of 54,170 measurements were taken. This resulted in an average horizontal error of 0.69 nautical miles, an average (unsigned) cross track error of 0.12 nautical miles, and an average (unsigned) along track error of 0.67 nautical miles.

HEURISTIC METHODS TO POST PROCESS AIRCRAFT RADAR TRACK DATA

The Conflict Probe Assessment Team of the Federal Aviation Administration (FAA) has evaluated several decision support tools within the National Airspace System. The evaluation required knowledge of the actual flight paths flown by the aircraft in the test scenarios used. The aircraft flight paths are determined from radar surveillance data as processed by the FAA’s Host mainframe computers. The tracking data obtained from the FAA’s Host computers contains errors and noise and must be corrected before the evaluation software can use it. This paper presents a set of carefully crafted heuristic data processing methods that have been developed and used to clean up the radar tracking data as it is supplied by the Host computers. The methods have been applied to numerous air traffic scenarios containing thousands of aircraft flights. The methods are described and representative statistical results are presented. For a typical air traffic five-hour scenario it was necessary to discard 2% of the aircraft radar tracks, 2.4% of the track position reports, and to correct 1.3% of the reports. The post processing of the track reports enabled the downstream software tools to satisfactorily process the radar data and establish ground truth for the testing and evaluation of the air traffic control systems.

Aircraft conflict probe sensitivity to weather forecasts errors

This study investigated the user request evaluation tools (URET) prediction sensitivity to weather forecast error. A quantitative experiment was designed and performed by the Federal Aviation Administrations Conflict Probe Assessment Team (CPAT) to evaluate the impact of weather forecast errors on URET trajectory and conflict predictions. The experiment used approximately two hours of traffic data recorded at the Indianapolis en route center in May 1999. The flights were time shifted to generate a sufficient number of test conflicts using a genetic algorithm technique developed by CPAT. The resulting scenario was input into the URET prototype system. To induce weather forecast error, the weather input file (rapid update cycle, RUC) was altered by adding 20 or 60 knots to the wind magnitude, 45 or 90 degrees to the wind direction, and 5 or 15 degrees Kelvin to the air temperature. This produced seven URET runs for the experiment -the unaltered control run and six treatment runs. The analysis compared the control run against the treatment runs. A methodology was developed to compare the trajectory and conflict prediction accuracy of these runs. A statistical analysis provided evidence that the forecast errors in wind magnitude and direction had significant effect on the longitudinal trajectory error and a modest impact on retracted false alerts, which caused at most an increase in the false alert probability by six percent. It also showed that the air temperature runs did not have a significant effect. Based on this experiment, a controller suspecting errors in the input wind forecast should expect only a modest impact on URET predictions. The impact would mainly be a moderate increase in the number of retractions of its conflict predictions (defined in this study as a retracted false alert). If the controller notices an increase in retractions, it may be symptomatic of inaccurate wind forecasts, which should be investigated.

Performance Metrics for Tactical Aircraft to Aircraft Conflicts

An air traffic control system’s main function is to separate aircraft. The computer supporting the system assists the air traffic controllers by generating a conflict alert whenever it predicts that two aircraft are about to get too close to each other. The performance of the conflict alert function is a key element to the overall functioning of the air traffic control system. A set of metrics has been designed to measure the conflict alerting performance of an aircraft traffic control system. The key factors are the missed alert rate and the false alert rate. However there are several secondary factors that are essential to measuring the performance, especially in a simulation environment. This paper describes a set of metrics that have been developed to evaluate the performance of the FAA’s en route aircraft traffic control system. They have been applied to the existing system, the Host Computer System, and will be used to establish metrics for the new system now under development, the En Route Automation Modernization system. The metrics are calculated by post processing recorded data that has been produced by running a real time simulation of the air traffic system without controllers, using as input field recorded aircraft data that has been time shifted to induce aircraft-to-aircraft conflicts.