Carmen J. Trammell

Statistical testing of software based on a usage model

In statistical testing, a model is developed to characterize the population of uses of the software, and the model is used to generate a statistically correct sample of all uses of the software. A software ‘usage model’ characterizes the population of intended uses of the software in the intended environment. Statistical testing based on a software usage model ensures that the failures that will occur most frequently in operational use will be found early in the testing cycle. The usage model is based on the software specification. The model can be developed in parallel with the software, thus shortening the elapsed time required to develop the deliver software.

Adopting Cleanroom software engineering with a phased approach

Cleanroom software engineering is a theory-based, team-oriented engineering process for developing very high quality software under statistical quality control. The Cleanroom process combines formal methods of object-based box structure specification and design, function-theoretic correctness verification, and statistical usage testing for reliability certification to produce software approaching zero defects. Management of the Cleanroom process is based on a life cycle of development and certification of a pipeline of user-function increments that accumulate into the final product. Teams in IBM and other organizations that use the process are achieving remarkable quality results with high productivity. A phased implementation of the Cleanroom process enables quality and productivity improvements with an increased control of change. An introductory implementation involves the application of Cleanroom principles without the full formality of the process; full implementation involves the comprehensive use of formal Cleanroom methods; and advanced implementation optimizes the process through additional formal methods, reuse, and continual improvement. The AOEXPERT/MVS™ project, the largest IBM Cleanroom effort to date, successfully applied an introductory level of implementation. This paper presents both the implementation strategy and the project results.

Quantifying the reliability of software: statistical testing based on a usage model

When a population is too large for study, as is the case for all possible uses of a software system, a statistically correct sample must be drawn as a basis for inferences about the population. In statistical testing of software based on a Markov chain usage model, the rich body of analytical results available for Markov chains provides numerous insights that can be used in test planning. Further, the connection between Markov chains and operations research techniques permits a Markov usage model to be expressed as a system of constraints, with mathematical programming used to generate the optimal model for a particular objective function. Since a software usage model is based on the specification, all analyses may be performed early in the development cycle and used as a quantitative basis for management decisions. These techniques have been reduced to engineering practice and used in large projects by IBM, Ericsson, all branches of the US military, and others. In this paper, statistical experiments, Markov models, and optimization techniques are shown to provide a sound theoretical and practical basis far quantifying the reliability of software.

Application of statistical science to testing and evaluating software intensive systems

Defense systems are becoming increasingly software intensive. While software enhances the effectiveness and flexibility of these systems, it also introduces vulnerabilities related to inadequacies in software design, maintenance, and configuration control. Effective testing of these systems must take into account the special vulnerabilities introduced by software. The software testing problem is complex because of the astronomical number of scenarios and states of use. The domain of testing is large and complex beyond human intuition. Because the software testing problem is so complex, statistical principles must be used to guide testing strategy in order to get the best information for the resources invested in testing. From a statistical point of view, all the topics in this paper follow sound problem solving principles and are direct applications of well established theory and methodology. From a software testing point of view the application of statistical science is relatively new and rapidly evolving, as an increasing range of statistical principles is applied to a growing variety of systems. Statistical testing is used in pockets of industry and agencies of government, including the DoD, on both experimental and routine bases. This paper is a composite of what is in hand and within reasonable reach in the application of statistical science to software testing.

The incremental development process in cleanroom software engineering

Abstract The objective of this paper is to present the theoretical basis and practical application of incremental development in the Cleanroom software engineering process. Incremental development is based on the mathematical principle of referential transparency. Cleanroom uses incremental development to build systems in a succession of cumulative subsets of user function. The increments accumulate top-down into the final product in a development and certification pipeline. Increment planning occurs after top-level specification, and results in a construction plan for the software. Factors determining the composition of increments include clarity of requirements, usage probability of user functions, reliability requirements for subsystems, coordination with the hardware development schedule, dependencies between functions, complexity, reuse, or other factors that pose risks to the project. Each increment involves a complete development and certification cycle. The first increment is a minimal system, and the final increment is the complete system. User feedback on each increment is a gauge on whether the right system is being built, and quality measures in each increment are a gauge on whether the system is being built right. Benefits of incremental development include customer feedback on the evolving system, intellectual control of the technical work, and management control of the schedule and budget. While incremental development may be used with other development methods, it is particularly effective when used with the formal methods in the Cleanroom process.

Cleanroom Software Engineering: Theory and Practice

Cleanroom software engineering is a rigorous engineering discipline for the development and certification of high-reliability software systems under statistical quality control (Mills, 1992; Linger, 1993, 1994). The Cleanroom name is borrowed from hardware cleanrooms, with their emphasis on process control and focus on defect prevention rather than defect removal. Cleanroom combines mathematically-based methods of software specification, design, and correctness verification with statistical usage testing to certify software fitness for use.

A group process for defining local software quality: field applications and validation experiments

‘Global’ measures of software quality are generally not used by practitioners because they have not been calibrated for local operating environments. A participative process for defining local software quality has been defined, applied in the field, and evaluated in formal validation experiments. In four field applications of the process, a ‘jury’ of seasoned practitioners ranked a sample of modules from the organizations inventory of code and produced a ‘software quality rule set’. The results of each field application were tested in a formal validation experiment to determine whether the obtained rule set was a valid representation of the organizations sense of quality, i.e. whether the locally‐derived rules could enable non‐organizational programmers to make judgments about quality that were similar to those of the jury. In each experiment, non‐organizational programmers ranked the same set of modules that were ranked by the organizations jury, using either the locally‐derived rules, generic (textbook) rules, placebo (non‐helpful) rules, or no rules. The correlation between the judgments of non‐organizational programmers using local rules and the jury was statistically significant at p < 0–005; all other correlations were significant at p < 0–005. The results support a conclusion that the group process for defining local software quality is reliable in producing valid rule sets.

Integrating software development technology and management: Cleanroom software engineering and the CMM for software

The Software Engineering Institutes (SEI) Capability Maturity Model for software (CMM) provides a well defined paradigm for software process improvement. Cleanroom software engineering provides well defined theoretical foundations and practices for software specification, development, testing and certification. The principal focus of the CMM is on management and organization. The principal focus of Cleanroom is on the technology and engineering discipline. The SEI has completed a study that maps Cleanroom into the CMM. The mapping shows that Cleanroom and the CMM are fully compatible, and that the technical practices of Cleanroom can be effectively integrated into the management practices of the CMM. The engineering discipline embodied in Cleanroom provides a firm foundation for project management, as illustrated in a software testing example.