Michael R. Lowry

Combining unit-level symbolic execution and system-level concrete execution for testing nasa software

We describe an approach to testing complex safety critical software that combines unit-level symbolic execution and system-level concrete execution for generating test cases that satisfy user-specified testing criteria. We have developed Symbolic Java PathFinder, a symbolic execution framework that implements a non-standard bytecode interpreter on top of the Java PathFinder model checking tool. The framework propagates the symbolic information via attributes associated with the program data. Furthermore, we use two techniques that leverage system-level concrete program executions to gather information about a units input to improve the precision of the unit-level test case generation. We applied our approach to testing a prototype NASA flight software component. Our analysis helped discover a serious bug that resulted in design changes to the software. Although we give our presentation in the context of a NASA project, we believe that our work is relevant for other critical systems that require thorough testing.

Formal analysis of a space-craft controller using SPIN

The paper documents an application of the finite state model checker SPIN to formally analyze a multithreaded plan execution module. The plan execution module is one component of NASAs New Millennium Remote Agent, an artificial intelligence-based spacecraft control system architecture which launched in October of 1998 as part of the DEEP SPACE 1 mission. The bottom layer of the plan execution module architecture is a domain specific language, named ESL (Executive Support Language), implemented as an extension to multithreaded COMMON LISP. ESL supports the construction of reactive control mechanisms for autonomous robots and spacecraft. For the case study, we translated the ESL services for managing interacting parallel goal-and-event driven processes into the PROMELA input language of SPIN. A total of five previously undiscovered concurrency errors were identified within the implementation of ESL. According to the Remote Agent programming team, the effort has had a major impact, locating errors that would not have been located otherwise and, in one case, identifying a major design flaw. In fact, in a different part of the system, a concurrency bug identical to one discovered by this study escaped testing and caused a deadlock during an in-flight experiment, 96 million kilometers from Earth. The work additionally motivated the introduction of procedural abstraction in terms of inline procedures into SPIN.

Deductive Composition of Astronomical Software from Subroutine Libraries

Automated deduction techniques are being used in a system called Amphion to derive, from graphical specifications, programs composed from a subroutine library. The system has been applied to construct software for the planning and analysis of interplanetary missions. The library for that application is a collection of subroutines written in FORTRAN-77 at JPL to perform computations in solar-system kinematics. An application domain theory has been developed that describes the procedures in a portion of the library, as well as some basic properties of solar-system astronomy, in the form of first-order axioms.

Combining test case generation and runtime verification

Software testing is typically an ad hoc process where human testers manually write test inputs and descriptions of expected test results, perhaps automating their execution in a regression suite. This process is cumbersome and costly. This paper reports results on a framework to further automate this process. The framework consists of combining automated test case generation based on systematically exploring the input domain of the program with runtime verification, where execution traces are monitored and verified against properties expressed in temporal logic. Capabilities also exist for analyzing traces for concurrency errors, such as deadlocks and data races. The input domain of the program is explored using a model checker extended with symbolic execution. Properties are formulated in an expressive temporal logic. A methodology is advocated that automatically generates properties specific to each input rather than formulating properties uniformly true for all inputs. The paper describes an application of the technology to a NASA rover controller.

Experimental Evaluation of Verification and Validation Tools on Martian Rover Software

We report on a study to determine the maturity of different verification and validation technologies (V&V) applied to a representative example of NASA flight software. The study consisted of a controlled experiment where three technologies (static analysis, runtime analysis and model checking) were compared to traditional testing with respect to their ability to find seeded errors in a prototype Mars Rover controller. What makes this study unique is that it is the first (to the best of our knowledge) controlled experiment to compare formal methods based tools to testing on a realistic industrial-size example, where the emphasis was on collecting as much data on the performance of the tools and the participants as possible. The paper includes a description of the Rover code that was analyzed, the tools used, as well as a detailed description of the experimental setup and the results. Due to the complexity of setting up the experiment, our results cannot be generalized, but we believe it can still serve as a valuable point of reference for future studies of this kind. It confirmed our belief that advanced tools can outperform testing when trying to locate concurrency errors. Furthermore, the results of the experiment inspired a novel framework for testing the next generation of the Rover.

A formal approach to domain-oriented software design environments

This paper describes a formal approach to domain-oriented software design environments, based on declarative domain theories, formal specifications, and deductive program synthesis. A declarative domain theory defines the semantics of a domain-oriented specification language and its relationship to implementation-level subroutines. Formal specification development and reuse is made accessible to users through an intuitive graphical interface that guides them in creating diagrams denoting formal specifications. Deductive program synthesis ensures that specifications are correctly implemented. This approach has been implemented in AMPHION, a generic KBSE system that targets scientific subroutine libraries. AMPHION has been applied to the domain of solar system kinematics. AMPHION enables space scientists to develop, modify, and reuse specifications an order of magnitude more rapidly than manual program development. Program synthesis is efficient and completely automatic.<<ETX>>

AMPHION: Automatic Programming for Scientific Subroutine Libraries

This paper describes Amphion, a knowledge-based software engineering (KBSE) system that guides a user in developing a formal specification of a problem and then implements this specification as a program consisting of calls to subroutines from a library. Amphion is domain independent and is specialized to an application domain through a declarative domain theory. A user is guided in creating a diagram that represents the formal specification through menus based upon the domain theory and the current state of the specification. The diagram also serves to document the specification. Program synthesis is based upon constructive theorem proving, and is efficient and totally automatic.

META-AMPHION: synthesis of efficient domain-specific program synthesis systems

Amphion is a real-world knowledge-based software engineering (KBSE) system whose program synthesis subsystem is based on deductive synthesis. Amphion is designed to automate use of software component libraries—the programs it generates are compositions of software components. Amphion has a domain-independent generic architecture that is specialized to an application domain and component library through a declarative domain theory. Up to now, program synthesis has been made efficient and automatic through manual tuning of theorem-proving strategies and tactics, and careful formulation of domain theories. The Meta-Amphion system is being developed to empower domain experts to develop, maintain, and evolve their own Amphion applications. Meta-Amphion is intended to be the knowledge-based analogue of application-generator generator technology. This paper describes an essential part of Meta-Amphion—technology for automatically transforming declarative domain theories into efficient domain-specific program synthesis systems.

Towards a theory for integration of mathematical verification and empirical testing

From the viewpoint of a project manager who is responsible for the verification and validation (V&V) of a software system, mathematical verification techniques provide a potentially valuable addition to otherwise standard empirical testing. However, the value they add, both in terms of coverage and in fault detection, has been difficult to quantify. Potential cost savings from replacing testing with mathematical techniques cannot be realized until the tradeoffs can be quantified. This paper first describes a framework for a theory of software fault detection that is based on software reliability and formalized fault models. The novelty of this approach is that it takes into account the relative utility of the various tools for fault detection. Second, the paper describes a utility model for integrating mathematical and empirical techniques with respect to fault detection and coverage analysis for software. Third, the paper discusses how to determine the optimal combination of black-box testing, white-box (structural) testing and formal methods in V&V of a software system. Finally, a demonstration of how this utility model can be used in practice is offered using a case study from a NASA software system.

Model Based Analysis and Test Generation for Flight Software

We describe a framework for model-based analysis and test case generation in the context of a heterogeneous model-based development paradigm that uses and combines MathWorks and UML 2.0 models and the associated code generation tools. This paradigm poses novel challenges to analysis and test case generation that, to the best of our knowledge, have not been addressed before. The framework is based on a common intermediate representation for different modeling formalisms and leverages and extends model checking and symbolic execution tools for model analysis and test case generation, respectively. We discuss the application of our framework to software models for a NASA flight mission.