Mike Paglione

Assessing trajectory prediction performance - metrics definition

As one of the central components of air traffic management (ATM) automation tools, trajectory predictors have a significant impact on the performance of ATM automation and hence the ATM system. Building on prior efforts, this paper applies a framework to assist in the development of performance metrics for trajectory predictors. Key performance areas are defined for trajectory prediction based upon existing trajectory predictor (TP) performance evaluations. Basic metrics are described within the accuracy key performance area for both input and output metrics. The basic metrics rely on precise definitions of events. Issues associated with the definition of events are discussed and approaches are provided for dealing with these. Specifiers are defined to further refine the definition of metrics in all performance areas. The application of basic metrics, events and specifiers is illustrated by drawing upon examples from the literature. An example of the impact of trajectory prediction accuracy on conflict probe performance is provided to illustrate considerations that must be given to the impact on higher-level systems when developing metrics for TP performance.

Methodology for Generating Conflict Scenarios by Time Shifting Recorded Traffic Data

A methodology is presented for generating conflict scenarios that can be used as test cases to estimate the operational performance of a conflict probe. Recorded air traffic data is time shifted to create traffic scenarios featuring conflicts with characteristic properties similar to those encountered in typical air traffic operations. First, a reference set of conflicts is obtained from trajectories that are computed using birth points and nominal flight plans extracted from recorded traffic data. Distributions are obtained for several primary properties (e.g., encounter angle) that are most likely to affect the performance of a conflict probe. A genetic algorithm is then utilized to determine the values of time shifts for the recorded track data so that the primary properties of conflicts generated by the time shifted data match those of the reference set. This methodology is successfully demonstrated using recorded traffic data for the Memphis Air Route Traffic Control Center; a key result is that the required time shifts are less than 5 min for 99% of the tracks. It is also observed that close matching of the primary properties used in this study additionally provides a good match for some other secondary properties.

Implementation and Metrics for a Trajectory Prediction Validation Methodology

At the heart of every air traffic decision support tool’s functionality is its trajectory prediction, where a trajectory is defined as the 4-dimensional path of an aircraft. This paper presents a comprehensive implementation for measuring the accuracy of a trajectory prediction in support of a validation methodology. The process includes four main processing areas: (1) parsing and checking the actual positional data of an aircraft (i.e., the aircraft’s actual trajectory), (2) parsing the trajectory predictions, (3) comparing the actual and predicted aircraft trajectory by sampling and measuring, and (4) analyzing the results. This paper presents detailed descriptions of the sampling process and metrics used to measure the accuracy of a predicted trajectory. Several aspects of the analysis and implementation are provided as well, such as inferential statistical approaches and graphical user interfaces to examine individual flights.

Academia-academia-industry collaborations on software engineering projects using local-remote teams

It is widely recommended by both academia and industry that todays technology and software engineering students be well prepared for industry before graduation, especially given global outsourcing and other trends. Various methods have been developed to ensure student readiness, including co-ops and capstone courses. These approaches increasingly use real-world projects for their benefits to industry and often to the community at large. In this paper, we argue that students can be prepared to effectively join industry and keep the US technology workforce competitive through a curriculum that includes a theoretical software engineering course with real-world projects and the collaboration of paired teams across two or more universities. We present a case study of a successful teaching experience that features these aspects, and describe the outcome along with the unique perspective of a participating student.

Review of Trajectory Accuracy Methodology and Comparison of Error Measurement Metrics

The aircraft trajectory is a prediction of the aircraft’s anticipated flight path. Trajectory accuracy is measured by comparing the trajectory prediction to the actual flight path of the aircraft. Trajectory accuracy analysis starts with a sampling process that selects the actual and predicted trajectory positions for measurement and concludes with the application of statistical and graphical analysis methods. At the base of a study are the error measurements and how they are defined. Their definitions are the focus of this paper. Two spatial metrics and two time metrics were specifically defined and compared in the horizontal dimension. Both detailed synthetic flight examples and a large traffic sample were employed to evaluate these error metrics. The second spatial metric and first time error metric proved to be superior methods in turns and approximately equivalent elsewhere. From the traffic sample, the standard deviation of the differences between the presented spatial and time error metrics was significant, ranging from 0.6 to 4.2 nautical miles and 9 to 55 seconds, respectively. Therefore, it is concluded that it is essential to clearly define the particular error measurement technique applied in a trajectory accuracy study to not only be relevant, but also for the results to be extensible and cross comparable with other studies.

A collaborative approach to trajectory modeling validation

Air service providers view the growth in future air traffic demand exceeding that of capacity, making it increasingly difficult to maintain yet alone improve the current levels of safety and efficiency. Advanced air traffic management (ATM) and flight deck decision support tool (DST) capabilities are seen as the functional enablers of the future ATM concepts needed to increase capacity by two-threefold. Such automation will provide support in flight data, metering, and conflict prediction/resolution functions to name a few. DST capabilities depend directly on the performance of the underlying trajectory predictor(s) (TP) that provide the anticipated future path of the aircraft. The accuracy of the TP is critical to the success of these DST functions. A common TP validation strategy has been developed for universal application to each element of the TP structure. The TP validation strategy is complemented by a broad database of actual trajectory recordings posted on a Web site and formatted in the extensible markup language (XML). The methodology presented here provides the process for any developer to utilize this database to validate and improve their TPs performance. This paper outlines the TP validation strategy, describes the various types of validation data provided, XML format, and tools developed.

Methodology for evaluating and regression testing a conflict probe

A conflict probe is an air traffic management decision support tool that predicts aircraft-to-aircraft and aircraft-to-airspace conflicts. In order to achieve the confidence of the air traffic controllers who are provided this tool, a conflict probe must accurately predict these events. To ensure their continued confidence, the accuracy should not only be assessed in the laboratory before the probe is deployed but continue to be reassessed as the system undergoes upgrades and software changes. Furthermore, it is desirable to use recorded air traffic data to test these tools in order to preserve real-world errors that affect their performance. This paper utilizes a proven approach that modifies surveillance radar track data in time to create traffic scenarios containing conflicts with characteristic properties similar to those encountered in actual air traffic operations. It is these time shifted traffic scenarios that are used to evaluate the conflict probe. This paper describes the detailed process of evaluating the missed and false conflict predictions, the calculation of the corresponding error probabilities, and a regression testing methodology to examine two runs of the conflict probe to determine if the conflict prediction accuracy has improved or degraded over time. A detailed flight example is presented which illustrates the specific processing involved in conflict accuracy analysis. Next using a scenario of many flights, a methodology utilizing categorical data analysis techniques is applied to determine if a new version of the conflict probes software significantly improved or degraded in conflict prediction accuracy.

Analysis of the Aircraft to Aircraft Conflict Properties in the National Airspace System

*† ‡ The primary function of administering the United States’ National Airspace System (NAS) is the air traffic controller task of actively monitoring assigned aircraft and resolving the conflicts (i.e. losses of minimum separations between aircraft) anticipated some time in the future. To mitigate the safety risks of increased traffic growth and effectively designing automation to aid in the separation management task, knowledge of the characteristics or properties of the conflicts is required. This paper reports on a comprehensive study that has examined these properties by collecting traffic data from all 20 NAS en route centers, developing software models to determine these events, implementing experimental design techniques to calibrate them, validating the models by comparing to advanced operational systems, and presenting detailed graphical and statistical analysis of the results. I. Introduction In the United States, the overall system of managing and controlling air traffic is known as the National Airspace System (NAS), which is administered by the Federal Aviation Administration (FAA). Detailed procedures involving restrictions on routing, speeds, and altitudes are an integral part of the NAS. These restrictions severely reduce the amount of aircraft traffic that NAS can accommodate, yet are needed to ensure the high level of safety required. At the heart of these operations is the human air traffic controller who must synthesize many pieces of timely information including radar surveillance information and flight data. Their fundamental responsibility is to ensure the safety of the aircraft flying within their regions of airspace in the most efficient means possible. To accomplish this, air traffic controllers actively monitor their aircraft and then resolve any conflicts (i.e., loss of minimum separation between aircraft or restricted airspace) predicted some time into the future. Furthermore, these resolutions are administered by air traffic controller voice instructions via radio transmissions to the aircraft. In the current system, there are automation systems that aid the air traffic controller mainly in the monitoring part of the task such as the ground based tactical and strategic conflict probes. In the en route centers, typically managing the aircraft above 18,000 feet, the Host Computer System’s (HCS) Conflict Alert function provides tactical alerts. The upgrade to the HCS, still under development, called the En Route Automation Modernization (ERAM), replaces Conflict Alert with several categories of alerts with the basic function requiring a minimum of 75 seconds warning. The User Request Evaluation Tool (URET), developed by MITRE Corporation’s Center for Advanced Aviation System Development, is an example of a strategic conflict probe in operation in the en route centers. It predicts conflicts up to 20 minutes in the future with typically a minimum warning of five minutes. Even with the aid of ground-based conflict probes, the task of separating aircraft will become increasingly difficult, since most air traffic service providers in the United States and Europe anticipate significant growth in air traffic. The growth is expected to out pace the capacity limits of the aviation systems, resulting in greater congestion and inefficiency. The interagency Joint Development Planning Office (JPDO) in the United States foresees a traffic demand increase by 2025 up to three times the number of flights of today’s traffic. 1 Given the need for enhanced safety and efficiency, broad categories of advances in ground and airborne automation are required. The JDPO, as established in their charter under the “Vision-100” legislation (Public Law 108-176) signed by President G. W. Bush in December 2003, has mandated a next generation operational concept of the NAS for 2025. 1 This next generation

Determination of Lateral Flight Adherence in Recorded Air Traffic Data

When using recorded air traffic data to measure the accuracy of air traffic management decision support tools that use intent-based trajectory modelers, it is necessary to determine whether or not an aircraft is adhering to its known flight plan clearance. This paper defines what is meant by adherence and presents metrics that can be used to define lateral flight adherence. The paper describes an algorithm that is currently being applied that uses a subset of these metrics. The paper then presents a number of examples obtained from recorded air traffic data, which show instances where aircraft deviate from their known lateral clearance. The paper then presents a number of alternative approaches that could be used to implement a better algorithm for determining whether or not an aircraft was in lateral adherence based on recorded air traffic data.

A structured approach for validation and verification of aircraft trajectory predictors

A generic trajectory predictor (TP) structure is presented including the decomposition of the TP into individual services. This TP structure allows us to describe specific elements of the TP that are being validated during different stages of validation. A TP validation strategy is developed by first considering the portion of the ATM system being modeled by the TP and decomposing the ATM system into elements that can be individually validated. This can be accomplished using different classes of data that have been identified: aircraft performance model data, FMS/simulation data, aircraft flight data recordings, and ATC operational data recordings. For each data class, different control loops can be modeled from the innermost to the outermost loops. We seek to verify the TP by first verifying the inner loops then gradually working outwards as confidence in the core model develops.