Victor V. Kuliamin

UniTesK Test Suite Architecture

The article presents the main components of the test suite architecture underlying UniTesK test development technology, an automated specification based test development technology for use in industrial testing of general-purpose software. The architecture presented contains such elements as automatically generated oracles, components to monitor formally defined test coverage criteria, and test scenario specifications for test sequence generation with the help of an automata based testing mechanism. This work stems from the ISP RAS results of academic research and 7-years experience in industrial application of formal testing techniques [1].

A survey of methods for constructing covering arrays

The paper presents a survey of methods for constructing covering arrays used in generation of tests for interfaces with a great number of parameters. The application domain of these methods and algorithms used in them are analyzed. Specific characteristics of the methods, including time complexity and estimates of the required memory, are presented. Various—direct, recursive, optimization, genetic, and backtracking—algorithms used for constructing covering arrays are presented. Heuristics are presented that allow one to reduce arrays without loss of completeness, and application domains of these heuristics are outlined.

The UniTesK Approach to Designing Test Suites

Principles of the UniTesK test development technology based on the use of formal models of target software are presented. This technology was developed by the RedVerst group in the Institute for System Programming, Russian Academy of Sciences (ISPRAS) [1], which obtained rich experience in testing and verification of complex commercial software.

Irredundant Algorithms for Traversing Directed Graphs: The Nondeterministic Case

Problems of testing program systems modeled by deterministic finite automata are considered. The necessary (and, sometimes, sufficient) component of such testing is a traversal of the graph of the automaton state transitions. The main attention is given to the so-called irredundant traversal algorithms (algorithms for traversing unknown graphs, or on-line algorithms), which do not require an a priori knowledge of the total graph structure.

Integration of Functional and Timed Testing of Real-Time and Concurrent Systems

The article presents an approach to model based testing of complex systems based on a generalization of finite state machines (FSM) and input output state machines (IOSM). The approach presented is used in the context of UniTesK specification based test development method. The results of its practical applications are also discussed. Practical experience demonstrates the applicability of the approach for model based testing of protocol implementations, distributed and concurrent systems, and real-time systems. This work stems from ISPRAS results of academic research and industrial application of formal techniques in verification and testing [1].

Formal Conformance Testing of Systems with Refused Inputs and Forbidden Actions

The article introduces an extension of the well-known conformance relation ioco on labeled transition systems (LTS) with refused inputs and forbidden actions. This extension helps to apply the usual formal testing theory based on LTS models to incompletely specified systems, which are often met in practice. Another topic concerned in the article is compositional conformance. More precisely, we try to define a completion operation that turns any LTS into input-enabled one having the same set of ioco-conforming implementations. Such a completion enforces preservation of ioco conformance by parallel composition operation on LTSes.

Formalization of test experiments

Formal methods for testing conformance of the system under examination to its specification are examined. The operational interaction semantics is specified by a special testing machine that formally determines the testing capabilities. A set of theoretically powerful and practically important capabilities is distinguished that can be reduced to the observation of external actions and refusals (the absence of external actions). The novelties are as follows. (1) Parameterization of the semantics by the families of observable and not observable refusals, which makes it possible to take into account various constraints on the (correct) interactions. (2) Destruction as a forbidden action, which is possible but should not be performed in the case of a correct interaction. (3) Modeling of the divergence by the Δ-action, which also should be avoided in the case of a correct interaction. On the basis of this semantics, the concept of safe testing, the implementation safety hypothesis, and the safe conformance relation are proposed. The safe conformance relation corresponds to the principle of independent observations: a behavior of an implementation is correct or incorrect independently of its other possible behaviors. For a more narrow class of interactions, another version of the semantics based on the ready traces may be used along with the corresponding conformance relation. Some propositions concerning the relationships between the conformance relations under various semantics are formulated. The completion transformation that solves the problem of the conformance relation reflexivity and a monotone transformation that solves the monotonicity problem (preservation of the conformance under composition) are defined.

Practical approach to specification and conformance testing of distributed network applications

Standardization of infrastructure and services in distributed applications and frameworks requires ground methodological base. Design by Contract approach looks very promising as a candidate. It helps to obtain component-wise design, to separate concerns between developers accurately, and makes development of high quality complex systems a manageable process. Unfortunately, in its classic form it can hardly be applied to distributed network applications because of lack of adequate means to describe nondeterministic asynchronous events. We extend Design by Contract with capabilities to describe callbacks and asynchronous communication between components. The resulting method was used to specify distributed applications and to develop conformance test suites for them in automated manner. Specifications are developed in an extension of C language that makes them clear and useful for industrial developers and decreases greatly test construction effort. Practical results of numerous successful applications of the method are described. More information on the applications of the method can be found at the site of RedVerst group of ISP RAS [1].

The use of contract specifications for representing requirements and for functional testing of hardware models

Contract specifications in the form of pre-and postconditions are widely used in software engineering for formal description of interfaces of software components. On the one hand, such specifications are convenient for the developers since they can easily be attached to the system architecture. On the other hand, test oracles verifying conformance of the behavior of the target system to the specifications can automatically be generated from them. In the paper, it is suggested to use contract specifications for representing requirements and for functional testing of hardware models developed in languages such as VHDL, Verilog, SystemC, System Verilog, etc. An approach to specification of such systems is proposed and compared with the existing methods of hardware specification. An experience of its practical use is described. The approach is based on the UniTESK testing technology developed at the Institute for System Programming.

Formal methods in industrial software standards enforcement

The article presents an approach to development of software standards usage infrastructure. The approach is based on formalization of standards and automated conformance test derivation from the resulting formal specifications. Strong technological support of such a process in its engineering aspects makes it applicable to software standards of real-life complexity. This is illustrated by its application to Linux Standard Base. The work stands in line with goals of international initiative Grand Challenge 6: Dependable Systems Evolution [1].