Nicolas Kicillof

Model-based quality assurance of protocol documentation: tools and methodology

Microsoft is producing interoperability documentation for Windows client–server and server–server protocols. The Protocol Engineering Team in the Windows organization is responsible for verifying the documentation to ensure that it is of the highest quality. Various test‐driven methods are being applied including, when appropriate, a model‐based approach. This paper describes core aspects of the quality assurance process and tools that were put in place, and specifically focuses on model‐based testing (MBT). Experience so far confirms that MBT works and that it scales, provided it is accompanied by sound tool support and clear methodological guidance. Copyright

Model-Based Quality Assurance of Windows Protocol Documentation

Microsoft is producing high-quality documentation for Windows client-server and server-server protocols. Our group in the Windows organization is responsible for verifying the documentation to ensure it is of the highest quality. We are applying various test-driven methods including, when appropriate, a model-based approach. This paper describes certain aspects of the quality assurance process we put in place, and specifically focuses on model-based testing (MBT). Our experiences so far confirm that MBT works and that it scales, provided it is accompanied by sound tool support and clear methodological guidance.

Interaction Coverage Meets Path Coverage by SMT Constraint Solving

We present a novel approach for generating interaction combinations based on SMT constraint resolution. Our approach can generate maximal interaction coverage in the presence of general constraints as supported by the underlying solver. It supports seeding with general predicates, which allows us to combine it with path exploration such that both interaction and path coverage goals can be met. Our approach is motivated by the application to behavioral model-based testing in the Spec Explorer tool, where parameter combinations must be generated such that all path conditions of a model action have at least one combination which enables the path. It is applied in a large-scale project for model-based quality assurance of interoperability documentation at Microsoft.

Action Machines: a Framework for Encoding and Composing Partial Behaviors

We describe action machines, a framework for encoding and composing partial behavioral descriptions. Action machines encode behavior as a variation of labeled transition systems where the labels are observable activities of the described artifact and the states capture full data models. Labels may also have structure, and both labels and states may be partial with a symbolic representation of the unknown parts. Action machines may stem from software models or programs, and can be composed in a variety of ways to synthesize new behaviors. The composition operators described here include synchronized and interleaving parallel composition, sequential composition, and alternating simulation. We use action machines in analysis processes such as model checking and model-based testing. The current main application is in the area of model-based conformance testing, where our approach addresses practical problems users at Microsoft have in applying model-based testing technology.

Assuring the model evolution of protocol software specifications by regression testing process improvement

Model‐based testing helps test engineers automate their testing tasks so that they are more cost‐effective. When the model is changed because of the evolution of the specification, it is important to maintain the test suites up to date for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in the industry, including Microsoft. To handle specification evolution effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis. We also develop a test suite augmentation technique to generate new test cases to cover the change‐related parts of the new model. The experiment on four large protocol document testing projects shows that our technique can successfully identify a high percentage of reusable test cases and generate low‐redundancy new test cases. When compared with a complete regeneration of the whole test suite, our technique significantly reduces regression testing time while maintaining the stability of requirement coverage over the evolution of requirements specifications. Copyright

Regression Testing Process Improvement for Specification Evolution of Real-World Protocol Software

Model-based testing helps test engineers automate their testing tasks so that they can be more cost-effective. When the model is changed due to the evolution of the specification, it is important to maintain the test suites up to date for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in prac¬tice. To handle specification evolution effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis, so that we can generate new test cases to cover only the change-related parts of the new model. Our experiment on four large protocol docu¬ment testing projects shows that the technique can significantly reduce regression testing time when compared with complete re¬generation of the test suites.

Discretizing Technical Documentation for End-to-End Traceability Tests

This paper describes a technique by which English prose protocol standards were transcribed into individually testable assertions and traced from original protocol specifications, to lists of requirements, to models and test cases, and finally into test logs and network captures. Annotating statements to make them stand alone, the handling of optional behavior, and using the requirements to guide discovery of missing information from the standard is also described. The approach allows us to have both a fast feedback loop for debugging the specification of the protocol and a way to estimate the coverage degree of the generated tests with respect to requirements. We discuss tool support developed specifically for the approach and exemplify with excerpts from the application of this technique in the process of testing hundreds of Microsoft’s open protocol specifications.

Model-Based Quality Assurance of the SMB2 Protocol Documentation

Microsoft is producing technical documentation for Windows client-server and server-server protocols to enable licensees to produce interoperable server products. This paper describes certain aspects of a new quality assurance process for technical documents as it is in place at Microsoft. We are applying various test methods including, when appropriate, a model-based approach. The paper uses the Server Message Block Protocol Version 2 (SMB2) as a running example to illustrate the process.