Christopher M. Lott

Model-based testing in practice

Model-based testing is a new and evolving technique for generating a suite of test cases from requirements. Testers using this approach concentrate on a data model and generation infrastructure instead of hand-crafting individual tests. Several relatively small studies have demonstrated how combinatorial test generation techniques allow testers to achieve broad coverage of the input domain with a small number of tests. We have conducted several relatively large projects in which we applied these techniques to systems with millions of lines of code. Given the complexity of testing, the model-based testing approach was used in conjunction with test automation harnesses. Since no large empirical study has been conducted to measure efficacy of this new approach, we report on our experience with developing tools and methods in support of model-based testing. The four case studies presented here offer details and results of applying combinatorial test-generation techniques on a large scale to diverse applications. Based on the four projects, we offer our insights into what works in practice and our thoughts about obstacles to transferring this technology into testing organizations.

Repeatable software engineering experiments for comparing defect-detection techniques

Techniques for detecting defects in source code are fundamental to the success of any software development approach. A software development organization therefore needs to understand the utility of techniques such as reading or testing in its own environment. Controlled experiments have proven to be an effective means for evaluating software engineering techniques and gaining the necessary understanding about their utility. This paper presents a characterization scheme for controlled experiments that evaluate defect-detection techniques. The characterization scheme permits the comparison of results from similar experiments and establishes a context for cross-experiment analysis of those results. The characterization scheme is used to structure a detailed survey of four experiments that compared reading and testing techniques for detecting defects in source code. We encourage educators, researchers, and practitioners to use the characterization scheme in order to develop and conduct further instances of this class of experiments. By repeating this experiment we expect the software engineering community will gain quantitative insights about the utility of defect-detection techniques in different environments.

Model-based testing of a highly programmable system

The paradigm of model based testing shifts the focus of testing from writing individual test cases to developing a model from which a test suite can be generated automatically. We report on our experience with model based testing of a highly programmable system that implements intelligent telephony services in the US telephone network. Our approach used automatic test case generation technology to develop sets of self checking test cases based on a machine readable specification of the messages in the protocol under test. The AETG/sup TM/ software system selected a minimal number of test data tuples that covered pairwise combinations of tuple elements. We found the combinatorial approach of covering pairwise interactions between input fields to be highly effective. Our tests revealed failures that would have been difficult to detect using traditional test designs. However, transferring this technology to the testing organization was difficult. Automatic generation of cases represents a significant departure from conventional testing practice due to the large number of tests and the amount of software development involved.

Modeling requirements for combinatorial software testing

The combinatorial approach to software testing uses models to generate a minimal number of test inputs so that selected combinations of input values are covered. The most common coverage criteria is two-way, or pairwise coverage of value combinations, though for higher confidence three-way or higher coverage may be required. This paper presents example system requirements and corresponding models for applying the combinatorial approach to those requirements. These examples are intended to serve as a tutorial for applying the combinatorial approach to software testing. Although this paper focuses on pairwise coverage, the discussion is equally valid when higher coverage criteria such as three-way (triples) are used. We use terminology and modeling notation from the AETG1 system to provide concrete examples.

Lessons learned from automating tests for an operations support system

We present experience gained in automating tests for an operations support system. A major portion of the effort was devoted to extending a commercial test tool so that testers could easily manipulate graphical user interface (GUI) objects on two implementations of the application. For this purpose, we developed a test automation library as support infrastructure for writing tests. The challenges and tradeoffs are discussed such as simplicity/complexity for a tester versus a library developer, hiding/exposing window hierarchy to the tester, providing common methods for different types of GUI objects, transparently manipulating custom GUI widgets, and coping with data‐dependent test cases. We discuss the requirements of test code reusability, maintainability, and portability, and describe the solutions we found. In addition, we offer observations about benefits and pitfalls of test automation, our recommendations for maximizing return on investment, and results from automating a variety of tests. Copyright

A Controlled Experiment to Evaluate On-Line Process Guidance

Process-centered software engineering environments are expected to improve an individuals comprehension of work activities, as well as improve communication and reduce conflicts among teams of software developers. We chose to investigate individual responses when using such an environment before looking for a group response. A controlled experiment compared off-line and on-line implementations of measurement-based process guidance to test three hypotheses: first, individuals accomplish their work more efficiently when using on-line process guidance as compared to off-line guidance; second, individuals are willing to use an on-line system for guidance; and third, individuals adjust their behavior based on quantitative quality models. The subjects worked alone on two testing exercises. Subjects used either an off-line or an on-line guidance technique during the first exercise, and the other technique during the second exercise. The results did not confirm the hypotheses. On average, subjects worked more efficiently when using off-line guidance, preferred off-line guidance, and ignored quantitative quality models. Post hoc analysis identified a strong correlation between subject experience level and preference for the type of guidance, a result that deserves further investigation.

XML-based requirements engineering for an electronic clearinghouse

We present methods and tools to support XML-based requirements engineering for an electronic clearinghouse that connects trading partners in the telecommunications area. The original semi-structured requirements, locally known as business rules, were written as message specifications in a non-standardized and error-prone format using MS Word. To remedy the resulting software failures and faults, we first formalized the requirements by designing an W3C XML Schema for the precise definition of the requirements structure. The schema allows a highly structured representation of the essential information in eXtensible Markup Language (XML). Second, to offer the requirements engineers the ability to edit the XML documents in a friendly way while preserving their information structure, we developed a custom editor called XLEdit. Third, by developing a converter from MS Word to the target XML format, we helped the requirements engineers to migrate the existing business rules. Fourth, we developed translators from the structured requirements to schema languages, which enabled automated generation of message-validation code. The increase in customer satisfaction and clearinghouse-service efficiency are primary gains from the investment in the technology for structured requirements editing and validation.