John Rushby

Using model checking to help discover mode confusions and other automation surprises

Abstract Automation surprises occur when an automated system behaves differently than its operator expects. If the actual system behavior and the operators ‘mental model’ are both described as finite state transition systems, then mechanized techniques known as ‘model checking’ can be used automatically to discover any scenarios that cause the behaviors of the two descriptions to diverge from one another. These scenarios identify potential surprises and pinpoint areas where design changes, or revisions to training materials or procedures, should be considered. The mental models can be suggested by human factors experts, or can be derived from training materials, or can express simple requirements for ‘consistent’ behavior. The approach is demonstrated by applying the Muro state exploration system to a ‘kill-the-capture’ surprise in the MD-88 autopilot. This approach does not supplant the contributions of those working in human factors and aviation psychology, but rather provides them with a tool to examine properties of their models using mechanized calculation. These calculations can be used to explore the consequences of alternative designs and cues, and of systematic operator error, and to assess the cognitive complexity of designs. The description of model checking is tutorial and is hoped to be accessible to those from the human factors community to whom this technology may be new.

PVS: Combining Specification, Proof Checking, and Model Checking

PVS (Prototype Verification System) is an environment for constructing clear and precise specifications and for developing readable proofs that have been mechanically verified. It is designed to exploit the synergies between language and deduction, automation and interaction, and theorem proving and model checking. For example, the type system of PVS requires the use of theorem proving to establish type correctness, and conversely, type information is used extensively during a proof. Similarly, decision procedures are heavily used in order to simplify the tedious and obvious steps in a proof leaving the user to interactively supply the high-level steps in a verification. Model checking is one such decision procedure that is used to discharge temporal properties of specific finite-state systems. A variety of examples from functional programming, fault tolerance, and real time computing have been verified using PVS [7]. The most substantial use of PVS has been in the verification of the microcode for selected instructions of a commercial-scale microprocessor called AAMP5 designed by Rockwell-Collins and containing about 500,000 transistors [5]. Most recently, PVS has been applied to the verification of the design of an SRT divider [9]. The key elements of the PVS design are described below in greater detail below.

Critical system properties: survey and taxonomy

Abstract Computer systems are increasingly employed in circumstances where their failure (or even their correct operation, if they are built to flawed requirements) can have serious consequences. There is a surprising diversity of opinion concerning the properties that such ‘critical systems’ should possess, and the best methods to develop them. The dependability approach grew out of the tradition of ultra-reliable and fault-tolerant systems, while the safety approach grew out of the tradition of hazard analysis and system safety engineering. Yet another tradition is found in the security community, and there are further specialized approaches in the tradition of real-time systems. In this article are examined the critical properties considered in each approach, and the techniques that have been developed to specify them and to ensure their satisfaction. Since systems are now being constructed that must satisfy several of these critical system properties simultaneously, there is particular interest in the extent to which techniques from one tradition support or conflict with those of another, and in whether certain critical system properties are fundamentally compatible or incompatible with each other. As a step toward improved understanding of these issues, it is suggested that a taxonomy, based on Perrows analysis (Perrow, C. Normal Accidents: Living with High Risk Technologies. Basic Books, New York, 1984), that considers the complexity of component interactions and tightness of coupling as primary factors, is used.

Generating efficient test sets with a model checker

It is well-known that counterexamples produced by model checkers can provide a basis for automated generation of test cases. However when this approach is used to meet a coverage criterion, it generally results in very inefficient test sets having many tests and much redundancy. We describe an improved approach that uses model checkers to generate efficient test sets. Furthermore, the generation is itself efficient, and is able to reach deep regions of the statespace. We have prototyped the approach using the model checkers of our SAL system and have applied it to model-based designs developed in Stateflow. In one example, our method achieves complete state and transition coverage in a Stateflow model for the shift scheduler of a 4-speed automatic transmission with a single test case.

An operational semantics for Stateflow

We present a formal operational semantics for Stateflow, the graphical Statecharts-like language of the Matlab/Simulink tool suite that is widely used in model-based development of embedded systems. Stateflow has many tricky features but our operational treatment yields a surprisingly simple semantics for the subset that is generally recommended for industrial applications. We have validated our semantics by developing an interpreter that allows us to compare its behavior against the Matlab simulator. We have used the semantics as a foundation for developing prototype tools for formal analysis of Stateflow designs.

Systematic formal verification for fault-tolerant time-triggered algorithms

Many critical real-time applications are implemented as time-triggered systems. We present a systematic way to derive such time-triggered implementations from algorithms specified as functional programs (in which form their correctness and fault-tolerance properties can be formally and mechanically verified with relative ease). The functional program is first transformed into an untimed synchronous system and, then, to its time-triggered implementation. The first step is specific to the algorithm concerned, but the second is generic and we prove its correctness. This proof has been formalized and mechanically checked with the PVS verification system. The approach provides a methodology that can ease the formal specification and assurance of critical fault-tolerant systems.

Formal Methods and their Role in the Certification of Critical Systems

This article describes the rationale for formal methods and considers the benefits, weaknesses, and difficulties in applying these methods to digital systems used in critical applications. It suggests factors for consideration when formal methods are offered in support of certification in a context such as DO-178B (the guidelines for software used on board civil aircraft) [40]. The presentation is intended for those to whom these topics are new. A more technical discussion of formal methods is available as a technical report [42].

A Formally Verified Algorithm for Interactive Consistency Under a Hybrid Fault Model

P. Thambidurai and Y.-K. Park (1988) have proposed an algorithm for interactive consistency that retains resilience to the arbitrary (or Byzantine) fault mode, while tolerating more faults of simpler kinds than standard Byzantine-resilient algorithms. Unfortunately, and despite a published proof of correctness, their algorithm is flawed. The authors detected this while undertaking a formal verification of the algorithm. They present a corrected algorithm that has been subjected to mechanically checked formal verification. Because informal proofs seem unreliable in this domain, and the consequences of failure could be catastrophic, the authors believe formal verification should become standard for algorithms intended for safety-critical applications.

The MILS component integration approach to secure information sharing

To achieve the vision of information superiority, secure and timely sharing of information is needed between geographically separated platforms and users. However, often the producers and consumers of the information, as well as the information itself are separated in different security domains. A COTS marketplace of composable, high assurance components would not only make the vision of cross-domain information sharing achievable, but could also help to make it much more affordable than is currently possible. As part of the Multiple Independent Levels of Security/Safety initiative, AFRLs multi-year High Assurance Middleware for Embedded Systems (HAMES) program is conducting research in integrating trusted components in such a way that the security properties of the system can be predicted. MILS is characterized by a two-level approach to secure system design. At the policy level, a decomposition to a virtual architecture is performed while identifying the trusted components, the local policies and the communications channels. This is done in a way that minimizes complexity of trusted components and their policies. At the resource sharing level, implementation of components is considered, which includes the allocation of components to shared physical resources. MILS provides an implementation technology that enables virtual components of various types, and their intercommunication channels, to share physical resources without compromising the integrity of the policy level. Security is seldom identified with a single, simple policy; the two-level approach of MILS was introduced as a rational way to organize the multiple cooperating components and sub-policies that realize a complete secure system. A MILS system needs to provide assurance that this design and implementation strategy and, in particular, the separate sub-policies of its components and the resource-sharing properties of its physical subsystems, compose to guarantee the security policy required of the overall system. This paper will describe the progress made so far in our research and some of the remaining challenges.

An Operational Semantics for Stateflow

We present a formal operational semantics for Stateflow, the graphical Statecharts-like language of the Matlab/Simulink tool suite that is widely used in model-based development of embedded systems. Stateflow has many tricky features but our operational treatment yields a surprisingly simple semantics for the subset that is generally recommended for industrial applications. We have validated our semantics by developing an interpreter that allows us to compare its behavior against the Matlab simulator. We have used the semantics as a foundation for developing prototype tools for formal analysis of Stateflow designs.