Edward Balaban

Metrics for evaluating performance of prognostic techniques

Prognostics is an emerging concept in condition based maintenance (CBM) of critical systems. Along with developing the fundamentals of being able to confidently predict Remaining Useful Life (RUL), the technology calls for fielded applications as it inches towards maturation. This requires a stringent performance evaluation so that the significance of the concept can be fully exploited. Currently, prognostics concepts lack standard definitions and suffer from ambiguous and inconsistent interpretations. This lack of standards is in part due to the varied end-user requirements for different applications, time scales, available information, domain dynamics, etc. to name a few issues. Instead, the research community has used a variety of metrics based largely on convenience with respect to their respective requirements. Very little attention has been focused on establishing a common ground to compare different efforts. This paper surveys the metrics that are already used for prognostics in a variety of domains including medicine, nuclear, automotive, aerospace, and electronics. It also considers other domains that involve prediction-related tasks, such as weather and finance. Differences and similarities between these domains and health maintenance have been analyzed to help understand what performance evaluation methods may or may not be borrowed. Further, these metrics have been categorized in several ways that may be useful in deciding upon a suitable subset for a specific application. Some important prognostic concepts have been defined using a notational framework that enables interpretation of different metrics coherently. Last, but not the least, a list of metrics has been suggested to assess critical aspects of RUL predictions before they are fielded in real applications.

Modeling, Detection, and Disambiguation of Sensor Faults for Aerospace Applications

Sensor faults continue to be a major hurdle for systems health management to reach its full potential. At the same time, few recorded instances of sensor faults exist. It is equally difficult to seed particular sensor faults. Therefore, research is underway to better understand the different fault modes seen in sensors and to model the faults. The fault models can then be used in simulated sensor fault scenarios to ensure that algorithms can distinguish between sensor faults and system faults. The paper illustrates the work with data collected from an electromechanical actuator in an aerospace setting, equipped with temperature, vibration, current, and position sensors. The most common sensor faults, such as bias, drift, scaling, and dropout were simulated and injected into the experimental data, with the goal of making these simulations as realistic as feasible. A neural network-based classifier was then created and tested on both experimental data and the more challenging randomized data sequences. Additional studies were also conducted to determine sensitivity of detection and disambiguation efficacy with respect to severity of fault conditions.

A diagnostic approach for electro-mechanical actuators in aerospace systems

Electro-mechanical actuators (EMA) are finding increasing use in aerospace applications, especially with the trend towards all all-electric aircraft and spacecraft designs. However, electro-mechanical actuators still lack the knowledge base accumulated for other fielded actuator types, particularly with regard to fault detection and characterization. This paper presents a thorough analysis of some of the critical failure modes documented for EMAs and describes experiments conducted on detecting and isolating a subset of them. The list of failures has been prepared through an extensive Failure Modes and Criticality Analysis (FMECA) reference, literature review, and accessible industry experience. Methods for data acquisition and validation of algorithms on EMA test stands are described. A variety of condition indicators were developed that enabled detection, identification, and isolation among the various fault modes. A diagnostic algorithm based on an artificial neural network is shown to operate successfully using these condition indicators and furthermore, robustness of these diagnostic routines to sensor faults is demonstrated by showing their ability to distinguish between them and component failures. The paper concludes with a roadmap leading from this effort towards developing successful prognostic algorithms for electromechanical actuators.

Propulsion IVHM technology experiment overview

NASA researchers recently demonstrated successful real-time fault detection and isolation of a virtual reusable launch vehicle main propulsion system. Using a detailed simulation of a vehicle propulsion system to produce synthesized sensor readings, the NASA team demonstrated that advanced diagnostic algorithms, running on flight-like computers, can, in real time, successfully diagnose the presence and cause of faults. This demonstration was conducted as part of the NASA Propulsion IVHM Technology Experiment, or P E X .

Experimental Validation of a Prognostic Health Management System for Electro-Mechanical Actuators

Electro-Mechanical Actuators (EMA) are gaining prominent roles in the next generation fly-by-wire aircraft and spacecraft. With these roles often being safety-critical (control surface or landing gear actuation, for instance), the key to faster adoption of EMA in aerospace applications is development of accurate and reliable prognostic health management (PHM) systems that not only detect and identify faults, but also predict how the identified they affect the remaining useful life (RUL) of both the faulty component and the actuator as a whole. Such information can be invaluable to pilots, controllers, and maintenance personnel in assessing how to complete or re-plan the desired mission with a sufficient safety margin. A team consisting of members of NASA Ames Diagnostic & Prognostic Group has developed a prototype PHM system for EMA that provides coverage for a number of faults modes typical to this type of actuators. The diagnostic portion of the system is implemented using a hybrid approach which utilizes both qualitative (bond graph, model-based) and quantitative (data-driven) reasoners to achieve low false positive and false negative detection rates and a high accuracy of diagnostic output. Once a fault has been diagnosed, the prognostic component, which is implemented using Gaussian Process Regression (GPR) principles, estimates the RUL of the component that is faulted. Experiments were conducted both in laboratory and flight conditions to validate the PHM system using an innovative Flyable Electromechanical Actuator (FLEA) test stand. The test stand allows experimental actuators to be subjected to environmental and operating conditions similar to what actuators on the host aircraft are experiencing, while providing researchers with the capability to safely inject and monitor propagation of various fault modes. Prognostic run-to-failure experiments were done in laboratory conditions on ballscrew jam and motor winding short faults. Flight experiments (albeit not run-to-failure) were conducted in collaboration with the US Army on UH-60 Blackhawk helicopters. The paper describes these experiments in detail and presents the results obtained from the PHM system with regard to the estimation of the RUL of the actuator.

Experimental and analytical development of health management for Electro-Mechanical Actuators

Expanded deployment of Electro-Mechanical Actuators (EMAs) in critical applications has created much interest in EMA Prognostic Health Management (PHM), a key enabling technology of Condition Based Maintenance (CBM). As such, Impact Technologies, LLC is collaborating with the NASA Ames Research Center to perform a number of research efforts in support of NASAs Integrated Vehicle Health Management (IVHM) initiatives. These efforts have combined experimental test stand development, laboratory seeded fault testing, and physical model-based health monitoring in a comprehensive PHM system development strategy. This paper discusses two closely related EMA research programs being conducted by Impact and NASA Ames. The first of these efforts resulted in the creation of an electro-mechanical actuator test stand for the Prognostics Center of Excellence at the NASA Ames Research Center. The second effort is ongoing and is utilizing physics-based modeling techniques to develop an algorithm and software package toolset for PHM of aircraft EMA systems using a hybrid (virtual sensor) approach.

Addressing the Real-World Challenges in the Development of Propulsion IVHM Technology Experiment (PITEX)

‡‡ The Propulsion IVHM Technology Experiment (PITEX) has been an on-going research effort conducted over several years. PITEX has developed and applied a model-based diagnostic system for the main propulsion system of the X -34 reusable launch vehicle, a space-launch technology demonstrator. The application was simulation-based using detailed models of the propulsion subsystem to generate nominal and failure scenarios during captive carry, which is the most safety-critical portion of the X-34 flight. Since no system-level testing of the X-34 Main Propulsion System (MPS) was performed, these simulated data were used to verify and validate the software system. Advanced diagnostic and signal processing algorithms were developed and tested in real -time on flight-like hardware. In an attempt to expose potential performance problems, these PITEX algorithms were subject to numerous real-world effects in the simulated data includ ing noise, sensor resolution, command/valve talkback information, and nominal build variations. The current research has demonstrated the potential benefits of model-based diagnostics, defined the performance metrics required to evaluate the diagnostic system, and s tudied the impact of real-world challenges encountered when monitoring propulsion subsystems.

Predicting Real-Time Safety of the National Airspace System

Situation awareness is necessary for operators to make informed decisions regarding avoidance of airspace hazards. To this end, each operator must consolidate operationsrelevant information from disparate sources and apply extensive domain knowledge to correctly interpret the current state of the NAS as well as forecast its (combined) evolution over the duration of the NAS operation. This timeand workload-intensive process is periodically repeated throughout the operation so that changes can be managed in a timely manner. The imprecision, inaccuracy, inconsistency, and incompleteness of the incoming data further challenges the process. To facilitate informed decision making, this paper presents a model-based framework for the automated real-time monitoring and prediction of possible effects of airspace hazards on the safety of the National Airspace System (NAS). First, hazards to flight are identified and transformed into safety metrics, that is, quantities of interest that could be evaluated based on available data and are predictive of an unsafe event. The safety metrics and associated thresholds that specify when an event transitions from safe to unsafe are combined with models of airspace operations and aircraft dynamics. The framework can include any hazard to flight that can be modeled quantitatively. Models can be detailed and complex, or they can be considerably simplifed, as appropriate to the application. Real-time NAS safety monitoring and prediction begins with an estimate of the state of the NAS using the dynamic models. Given the state estimate and a probability distribution of future inputs to the NAS, we can then predict the evolution of the NAS the future state and the occurrence of hazards and unsafe events. The entire probability distribution of airspace safety metrics is computed, not just point estimates, without significant assumptions regarding the distribution type and/or parameters. We demonstrate our overall approach through a simulated scenario in which we predict the occurrence of some unsafe events and show how these predictions evolve in time as flight operations progress. Predictions accounting for common sources of uncertainty are included and it is shown how the predictions improve in time, become more confident, and change dynamically as new information is made available to the prediction algorithm.

Autonomous Decision Making for Planetary Rovers Using Diagnostic and Prognostic Information

Abstract Rover missions typically involve visiting a set of predetermined waypoints to perform science functions, such as sample collection. Given the communication delay between Earth and the rover, and the possible occurrence of faults, an autonomous decision making system is essential to ensure that the rover maximizes the scientific operations performed without damaging itself further or stalling. This paper presents a modular software architecture for autonomous decision making for rover operations that uses diagnostic and prognostic information to influence mission planning and decision making to maximize the completion of mission objectives. The decision making system consists of separate modules that perform the functions of control, diagnosis, prognosis, and decision making. We demonstrate our implementation of this architecture on a simulated rover testbed.

Model-Based Fault Detection and Diagnosis System for NASA Mars Subsurface Drill Prototype

The Drilling Automation for Mars Environment (DAME) project, led by NASA Ames Research Center, is aimed at developing a lightweight, low-power drill prototype that can be mounted on a Mars lander and be capable of drilling down several meters below the Mars surface for conducting geology and astrobiology research. The DAME drill system incorporates a large degree of autonomy -from quick diagnosis of system state and fault conditions to taking the appropriate recovery actions -while also striving to achieve as many of the operational objectives as possible.