Gardner C. Patton

The AETG system: an approach to testing based on combinatorial design

This paper describes a new approach to testing that uses combinatorial designs to generate tests that cover the pairwise, triple, or n-way combinations of a systems test parameters. These are the parameters that determine the systems test scenarios. Examples are system configuration parameters, user inputs and other external events. We implemented this new method in the AETG system. The AETG system uses new combinatorial algorithms to generate test sets that cover all valid n-way parameter combinations. The size of an AETG test set grows logarithmically in the number of test parameters. This allows testers to define test models with dozens of parameters. The AETG system is used in a variety of applications for unit, system, and interoperability testing. It has generated both high-level test plans and detailed test cases. In several applications, it greatly reduced the cost of test plan development.

Model-based testing in practice

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

The combinatorial design approach to automatic test generation

The combinatorial design method substantially reduces testing costs. The authors describe an application in which the method reduced test plan development from one month to less than a week. In several experiments, the method demonstrated good code coverage and fault detection ability.

The Automatic Efficient Test Generator (AETG) system

Software testing is expensive, tedious and time consuming. Thus, the problem of making testing more efficient and mechanical, without losing its effectiveness, is very important. The Automatic Efficient Test Generator (AETG) is a new tool that mechanically generates efficient test sets from user defined test requirements. It is based on algorithms that use ideas from statistical experimental design theory to minimize the number of tests needed for a specific level of test coverage of the input test space. The savings due to AETG are substantial when compared to exhaustive testing or other methods of testing. AETG has been used in Bellcore for screen testing, interoperability testing and for protocol conformance testing. The paper describes the current system and it constructs and reports some preliminary results obtained during initial trials.<<ETX>>

AETG/sup SM/ Web: a Web based service for automatic efficient test generation from functional requirements

AETG/sup SM/ Web is a Web based service developed by Bellcore researchers for enabling model based testing. In model based testing, the functional test requirements of the system are first modeled and then test cases based on this model are created or generated. AETG/sup SM/ Web employs a Web based user interface to model the functional requirements of the system under test, and automatically generates test cases for the system using the AETG/sup SM/ test case generation technology. A brief review of modeling using AETG/sup SM/ Web and the AETG/sup SM/ test generation paradigm is explained.