Nancy G. Leveson

A New Accident Model for Engineering Safer Systems

New technology is making fundamental changes in the etiology of accidents and is creating a need for changes in the explanatory mechanisms used. We need better and less subjective understanding of why accidents occur and how to prevent future ones. The most effective models will go beyond assigning blame and instead help engineers to learn as much as possible about all the factors involved, including those related to social and organizational structures. This paper presents a new accident model founded on basic systems theory concepts. The use of such a model provides a theoretical foundation for the introduction of unique new types of accident analysis, hazard analysis, accident prevention strategies including new approaches to designing for safety, risk assessment techniques, and approaches to designing performance monitoring and safety metrics.

An investigation of the Therac-25 accidents

Between June 1985 and January 1987, the Therac-25 medical electron accelerator was involved in six massive radiation overdoses. As a result, several people died and others were seriously injured. A detailed investigation of the factors involved in the software-related overdoses and attempts by users, manufacturers, and government agencies to deal with the accidents is presented. The authors demonstrate the complex nature of accidents and the need to investigate all aspects of system development and operation in order to prevent future accidents. The authors also present some lessons learned in terms of system engineering, software engineering, and government regulation of safety-critical systems containing software components.<<ETX>>

Requirements specification for process-control systems

The paper describes an approach to writing requirements specifications for process-control systems, a specification language that supports this approach, and an example application of the approach and the language on an industrial aircraft collision avoidance system (TCAS II). The example specification demonstrates: the practicality of writing a formal requirements specification for a complex, process-control system; and the feasibility of building a formal model of a system using a specification language that is readable and reviewable by application experts who are not computer scientists or mathematicians. Some lessons learned in the process of this work, which are applicable both to forward and reverse engineering, are also presented. >

Software safety: why, what, and how

Software safety issues become important when computers are used to control real-time, safety-critical processes. This survey attempts to explain why there is a problem, what the problem is, and what is known about how to solve it. Since this is a relatively new software research area, emphasis is placed on delineating the outstanding issues and research topics.

Safety Analysis Using Petri Nets

The application of Time Petri net modeling and analysis techniques to safety-critical real-time systems is explored and procedures described which allow analysis of safety, recoverability, and fault-tolerance.

Completeness and consistency in hierarchical state-based requirements

This paper describes methods for automatically analyzing formal, state-based requirements specifications for some aspects of completeness and consistency. The approach uses a low-level functional formalism, simplifying the analysis process. State-space explosion problems are eliminated by applying the analysis at a high level of abstraction; i.e., instead of generating a reachability graph for analysis, the analysis is performed directly on the model. The method scales up to large systems by decomposing the specification into smaller, analyzable parts and then using functional composition rules to ensure that verified properties hold for the entire specification. The analysis algorithms and tools have been validated on TCAS II, a complex, airborne, collision-avoidance system required on all commercial aircraft with more than 30 passengers that fly in U.S. Airspace.

Software requirements analysis for real-time process-control systems

A set of criteria is defined to help find errors in, software requirements specifications. Only analysis criteria that examine the behavioral description of the computer are considered. The behavior of the software is described in terms of observable phenomena external to the software. Particular attention is focused on the properties of robustness and lack of ambiguity. The criteria are defined using an abstract state-machine model for generality. Using these criteria, analysis procedures can be defined for particular state-machine modeling languages to provide semantic analysis of real-time process-control software requirements. >

Moving Beyond Normal Accidents and High Reliability Organizations: A Systems Approach to Safety in Complex Systems

In this century society faces increasingly large-scale accidents and risks emerging from our own wondrous technologies. Two prominent organizational approaches to safety, Normal Accident Theory and High Reliability Organizations, have focused attention on a variety of industries that deal with hazardous situations, developed concepts to explicate organizational structure and culture, and debated whether accidents are inevitable in complex systems. We outline these approaches and identify some limitations, including narrow definitions, ambiguity about key concepts, confusion of reliability and safety, and overly pessimistic or optimistic conclusions. We believe that the debate between NAT and HRO can become a more productive three-way conversation by including a systems approach to safety emerging from engineering disciplines. The more comprehensive systems approach clarifies the strengths and weaknesses of NAT and HRO and offers a more powerful repertoire of analytic tools and intervention strategies to manage and control post modern risk in complex, high-tech, systems with their potential for catastrophic disruptions and losses.

Safety verification of Ada programs using software fault trees

The software fault-tree analysis technique is explained. It is then extended to allow its use on a more complex language involving such features as concurrency and exception handling. Ada is used as the example language because many safety-critical projects are using or planning to use Ada. It also contains complex, real-time programming facilities found in other languages used in these types of projects. Software fault-tree analysis uses failure-mode templates to generate the fault tree. The templates provided can be used to define the procedures for applying the technique to programs written in most other declarative languages. To explain the use of the templates an example Ada program, for a traffic-light-control system, is analyzed. The cost and practicality of the method and its implications for software reuse are assessed. The application of the safety analysis procedures to requirements modeling and specification languages is considered.<<ETX>>

Analysis of faults in an N-version software experiment

The authors have conducted a large-scale experiment in N-version programming. A total of 27 versions of a program were prepared independently from the same specification at two universities. The results of executing the versions revealed that the versions were individually extremely reliable but that the number of input cases in which more than one failed was substantially more than would be expected if they were statistically independent. After the versions had been executed, the failures of each version were examined and the associated faults located. It appears that minor differences in the software development environment would not have a major impact in reducing the incidence of faults that cause correlated failures. >