Russell Shannon

Data mining navy flight and maintenance data to affect repair

A large portion of Naval aircraft maintenance is driven by avionics-related deficiencies. Military avionics systems rely heavily on built-in test (BIT) to troubleshoot discrepancies during unscheduled maintenance. One study found that analyzing BIT codes for trends (both at the aircraft level and at the squadron level) and scheduling maintenance accordingly, increased aircraft operational availability (Ao) by twenty-two percent within a single squadron [1]. The study focused on F/A-18C aircraft over a forty-four month period. While this is a substantial increase, it takes into account only organizational level (O-level) BIT data. It does not include all available information, such as historical maintenance data, operating environment, and past repair history. The Integrated Diagnostics and Automated Test Systems (IDATS) team at Naval Air Systems Command (NAVAIR) Lakehurst is investigating the use of data mining to mitigate ambiguity within Naval avionics maintenance, with the intent of reducing costly and inefficient maintenance practices [2]. This includes building models from both aircraft historical maintenance data and BIT codes recorded to an aircrafts memory unit (MU) during a flight in order to identify trends in the data that would not be obvious or trivial to a maintainer. The IDATS team has utilized a commercial off-the-shelf (COTS) data mining software package called ThinkAnalytics in combination with custom software tools to find meaningful trends within the F/A-18 BIT and maintenance datasets. ThinkAnalytics is a real-time enterprise data mining tool that provided the necessary data mining functionality. Trends identified by the software are currently being validated by the user and system Subject Matter Experts (SMEs) to ensure that they are accurate, novel and/or non-trivial. Through the use of data mining, in combination with knowledge about how the system operates and communicates with other systems, BIT can be augmented to improve maintenance efficiency at all levels of maintenance.

Realizing Net-Centric Diagnostics Within the Naval Maintenance System

The U.S. Navys current implementation of the three-level maintenance philosophy achieves mission readiness. However, there are inefficiencies in the maintenance system which waste resources and manpower. These inefficiencies may be overcome by facilitating improved testing capabilities and data sharing between the Organizational level of maintenance (O-level) and the Intermediate level (l-level) and/or Depot (D-level). The Integrated Diagnostics and Automated Test Systems (IDATS) team at Naval Air Systems Command (NAVAIR) Lakehurst has investigated the realization of net-centric avionics diagnostics within the current Navy maintenance system. This paper describes efforts by the IDATS team to realize a proof-of-concept demonstration involving many aircraft maintenance technologies. These include data mining, diagnostic reasoning, and the transmission of diagnostic and maintenance data between levels of maintenance in order to reduce diagnostic ambiguities and testing time for an avionics system. Cost savings resulting from increased aircraft operational availability (Ao) can be achieved by implementing a net-centric approach to diagnostics Navy maintenance system. within the current Navy maintenance system.

Enabling net-centric diagnostics in the F/A-18 automated maintenance environment

For the past eight years, the AN/USM-681 Electro-Optics Pallet/Pod Tester (EOPT) system has been the main on-aircraft support equipment (SE) item used to test and troubleshoot the AN/ASD-10 Advanced Tactical Aerial Reconnaissance System (ATARS), the AN/ASD-12 Shared Reconnaissance Pod (SHARP), and the AN/ASQ-228 Advanced Targeting Forward Looking Infrared (ATFLIR) Pod. Parts obsolescence issues have caused significant increases in the cost and time to repair and/or procure an EOPT. Developing a replacement for the EOPT was an opportunity to reduce maintenance costs and implement net-centric diagnostics within the F/A-18 automated maintenance environment (AME). This paper describes the first year of a two-year Technology Transition Initiative (TTI) in which the tester hardware was redesigned to mitigate obsolescence by leveraging Local Area Network eXtensions for Instrumentation (LXI) technology, and a net-centric diagnostics framework (NCDF) was designed to enable the bi-directional exchange of test data and maintenance data for smarter testing in both the on-aircraft and off-aircraft maintenance environments, leveraging existing technologies, such as Smart TPS.

Achieving optimum test by applying standardization

Over the past two decades, the US Department of Defense (DoD) has seen the introduction of weapon systems that do not meet their diagnostic requirements when initially fielded. Some suffer false alarm rates over eighty percent [1]. During a products lifecycle, the ability to determine how well it performs is based on the capability to test and the evaluation of those tests. Different tests and evaluations are required throughout the products lifecycle. These tests must, therefore, have the standards that define parameters, techniques and procedures for measurements that present an accurate and precise communication of information. In order to effectively accomplish this task, the development of quality tests capable of supporting these tasks must be defined and documented for the design, production, and operations-and-support phases of a systems lifecycle. As a product proceeds through its life cycle, the information collected at each phase must be used for the support of subsequent phases.

Lessons learned in implementing a net-centric diagnostic solution for the F/A-18 maintenance environment

The AN/USM-681 Electro-Optics Pallet/Pod Tester (EOPT) system has been the main on-aircraft support equipment (SE) item used to test and troubleshoot the AN/ASD-10 Advanced Tactical Aerial Reconnaissance System (ATARS), the AN/ASD-12 Shared Reconnaissance Pod (SHARP), and the AN/ASQ-228 Advanced Targeting Forward Looking Infrared (ATFLIR) Pod. Because most of the technologies used by the EOPT are now obsolete, an SE item was needed to replace the EOPT system. As previously reported, developing a replacement for the EOPT was an opportunity to implement net-centric diagnostics within the F/A-18 automated maintenance environment (AME). This paper describes the second year of a two-year Technology Transition Initiative (TTI) in which the goals were to redesign the tester hardware to mitigate obsolescence by leveraging Local Area Network eXtensions for Instrumentation (LXI) technology, and to design a net-centric diagnostics framework (NCDF) to enable the bi-directional exchange of test data and maintenance data for smarter testing in both the on-aircraft and off-aircraft maintenance environments. This paper will focus on the practical realities of implementing a net-centric diagnostic solution in todays Navy/Marine Corps maintenance environment. This includes a report of the results of first article testing (FAT) as well as descriptions of the numerous design challenges, configuration management challenges, network challenges, and security challenges that were overcome, and those yet to be overcome, in order to move this technology from a working prototype to a fielded piece of support equipment.

Navigating the diagnostics career labyrinth

Integrated diagnostics is a career field for which there currently exists no standard set of basic qualifications, few educational opportunities to study at the university level, no clear processes within most organizations for practicing integrated diagnostics as a systems engineering activity and often no uniform method of sharing techniques and lessons learned with new employees. Several authors have stated the importance of increased training in test ((M. Choi et al., 2003), (M. Katara, 2005), (W. Moorhead and S. Demidenko, 2002),(L.Y. Ungar, 1997), (L.Y. Ungar, 2000), (S. Demidenko and W. Moorhead, 2006). Studies have found that the majority of test engineer training is on-the-job, rather than knowledge acquired as part of a higher education degree program or a formal training process (L.Y. Ungar, 1997). Few authors, however, have focused on the need for a well-defined career path which leads from higher education, or formal training, to a career in diagnostics. This paper is an attempt to describe what such a career path may look like. First, the need for a formal career path will be established from published literature and interviews conducted with engineers from industry, academia and United States (US) government. This paper will then describe actions that are necessary first steps in creating and maintaining a career path that is required for the sustainment of a knowledgeable and well-trained pool of diagnostic engineers needed to meet future technology challenges. A case study of engineers from the Naval Air Systems Commands (NAVAIRs) Integrated Diagnostics and Automated Test Systems (IDATS) team in Lakehurst, New Jersey is presented to illustrate the very diverse and dissimilar backgrounds and training of diagnostic engineers within one group in one organization.

A novel device for fault isolation at aircraft

For certain avionics systems, built-in test (BIT) alone cannot accurately fault isolate to a single avionics box, or Weapons Replaceable Assembly (WRA). This is evident in the number of WRAs that are incorrectly removed and replaced at the aircraft. Such false remove and replace maintenance actions are realized in terms of increased support cost and decreased aircraft operational availability, AO. A patent-pending prototype device has been developed to break diagnostic ambiguities at the aircraft by inserting test access points between WRAs. This prototype includes a customized cylindrical Military Specification (MIL-SPEC) connector, which routes avionics bus signals to a handheld tester while the aircraft is powered on. The handheld tester analyzes data bus traffic between two functioning WRAs and provides a red, yellow, or green Light Emitting Diode (LED) output to indicate the health of the connected WRAs. The device can be reprogrammed to detect a wide range of fault indicators as they are discovered through the analysis of fielded avionics systems. This novel device provides diagnostics information at the aircraft for improved fault isolation, which may result in a significant reduction in the unnecessary cost associated with false removal and replacement of healthy WRAs.

A system to automatically generate test program sets

This paper describes a novel technique for creating the test program sets (TPSs) that are used by automatic test equipment (ATE) to test electronic circuits and devices. This paper presents an architecture consisting of a genetic algorithm (GA) test proposer and a pattern classifier test evaluator. This architecture has been shown to produce optimized test sequences without human intervention. In contrast to the above, the current method of developing TPSs and test sequences is an analytical process involving the building of fault trees using circuit diagrams of the unit under test and industrial-strength circuit simulation models. Since TPS software currently is coded manually, it can cost millions of dollars and take 12 months to 18 months of lead time to produce. In the prototype system, both the GA and the test evaluator are used to optimize input stimuli, significantly reducing the labor hours of a human TPS developer. The outputs of the process are a stimulus signals specification and a diagnostics reasoning system that could be deployed to ATE. The system has been demonstrated on a small scale using a band-pass filter circuit with twenty components, and in simulation on circuits with up to three hundred components. Success was shown by automatically generating a TPS that provided full fault detection and full fault isolation of the band-pass filter circuit. The next steps in the development process include demonstrating the technology on Fleet asset circuit cards, improving the scalability using high-performance computing assets, and implementing external interfaces using Automatic Test Markup Language (ATML).

Inhibiting factors in design for testability higher education

Many engineering students are not graduating with the necessary knowledge or experience in design for testability (DFT), automatic test equipment (ATE), or diagnostics in order to work in these fields. They typically do not demonstrate a consistent understanding of integrated diagnostics, or have an appreciation of the need. These same “fresh out” engineers will ultimately derive the low-level requirements for developing diagnostic systems, and this lack of knowledge of testing environments will have a significant impact. Failure to adequately address the integrated diagnostics and testing needs of a system greatly impacts its supportability and, consequently, the cost of that system throughout its life cycle. Integrated diagnostics is a career field for which there currently exists no standard set of basic qualifications, few educational opportunities to study at the university level, no clear processes within most organizations for practicing integrated diagnostics as a systems engineering activity, and no uniform method of sharing techniques and lessons learned with new employees. Studies have found that the majority of test engineer training is on-the-job, rather than knowledge acquired as part of a higher education degree program, or a formal training process [1]-[7]. As a result, it requires two to three years for any recent graduate to become competent in the field of test engineering. There are three main inhibiting factors to teaching design for testability as part of post-secondary education. The first factor is cost. The high cost, and quick obsolescence, of many ATE systems is a barrier to entry to any small- or medium-sized colleges engineering department budget. Even accounting for corporate donations, there are hidden costs, such as facilities and equipment maintenance, which make ATE prohibitively expensive. Moreover, in the United States, all engineering curricula must be accredited by the Accreditation Board for Engineering and Technology (ABET). It is an arduous process, even for such well-worn topics as electrical engineering or mechanical engineering. A department chair is unlikely to risk the departments accreditation, or prolong the accreditation process, by including an exotic topic such as DFT or diagnostics. Finally, it is the goal of most institutions that their students will obtain employment upon graduation. To that end, curricula are often tailored to the demands of local employers. If surrounding industry is not asking for skilled diagnostic or DFT engineers, then there is no incentive for an engineering department to include it in a degree curriculum. This paper explores each of these factors in depth, and provides mitigations for overcoming the challenges that each presents.

Power law behavior in Navy and Marine Corps avionics systems

A power law distribution is a mathematically lopsided probability distribution in which one quantity varies as a power of another. Probability distributions, in general, describe the percentage of items that have a particular value in a data set. Empirical examples of power law distributions typically involve a small group of bad actors within a population that cause the tail of the distribution to skew away from that of a straight line. Avionics failures within the Naval Air Enterprise (NAE) also typically involve a small population of bad actors which account for a large portion of failure conditions. Therefore, the authors were eager to investigate whether Navy and Marine Corps avionics systems failures fit a power law distribution. If so, it would suggest that radical change is needed to current maintenance practices, including a change in investments in automatic test equipment. To address this question, the top five hundred avionics degraders from across the NAE were analyzed using a method laid out by Clauset et al. [1]. First, data was gathered and candidate systems were identified using visual inspection of the data. Data from candidates was then analyzed such that the tail of the distribution could be compared to the power law distribution. Finally, a goodness of fit calculation was performed to find whether or not the power law distribution appropriately described the behavior of the candidate system. All power law candidates were then compared to other types of distributions. This information was used to determine if the power law distribution truly is the favored distribution for a given data set. Analysis was done on two data sets: one from 2000 to 2010 and one with data from 2010 to mid-2015. It was found that the results from this comparison favored other types of distributions over the power law in almost every case. Furthermore, it was found that no-faultfound maintenance actions can create the illusion of power law failure behavior in some systems, where none actually exists.