Manoranjan Satpathy

Randomized directed testing (REDIRECT) for Simulink/Stateflow models

The Simulink/Stateflow (SL/SF) environment from Math-works is becoming the de facto standard in industry for model based development of embedded control systems. Many commercial tools are available in the market for test case generation from SL/SF designs; however, we have observed that these tools do not achieve satisfactory coverage in cases when designs involve nonlinear blocks and Stateflow blocks occur deep inside the Simulink blocks. The recent past has seen the emergence of several novel techniques for testing large C, C++ and Java programs; prominent among them are directed automated random testing (DART), hybrid concolic testing and feedback-directed random testing. We believe that some of these techniques could be lifted to testing of SL/SF based designs; REDIRECT (RandomizEd DIRECted Testing), the proposed testing method of this paper, is an attempt towards this direction. Specifically, REDIRECT uses a careful combination of the above techniques, and in addition, the method uses a set of pattern-guided heuristics for tackling nonlinear blocks. A prototype tool has been developed and the tool has been applied to many industrial strength case studies. Our experiments indicate that a careful choice of heuristics and certain combinations of random and directed testing achieve better coverages as compared to the existing commercial tools. 1

ProTest: An Automatic Test Environment for B Specifications

We present ProTest, an automatic test environment for B specifications. B is a model-oriented notation where systems are specified in terms of abstract states and operations on abstract states. ProTest first generates a state coverage graph of a B specification through exhaustive model checking, and the coverage graph is traversed to generate a set of test cases, each being a sequence of B operations. For the model checking to be exhaustive, some transformations are applied to the sets used in the B machine. The approach also works if it is not exhaustive; one can stop at any point in time during the state space exploration and generate test cases from the coverage graph obtained so far. ProTest then simultaneously performs animation of the B machine and the execution of the corresponding implementation in Java, and assigns verdicts on the test results. With some restrictions imposed on the B operations, the whole of the testing process is performed mechanically. We demonstrate the efficacy of our test environment by performing a small case study from industry. Furthermore, we present a solution to the problem of handling non-determinism in B operations.

A Generic Model for Assessing Process Quality

Process assessment and process improvement are both very difficult tasks since we are either assessing or improving a concept rather than an object. A quality process is expected to produce quality products efficiently. Most of the existing models such as CMM, ISO 9001/9000-3 etc. intend to enhance the maturity or the quality of an organization with the assumption that a matured organization will put its processes in place which in turn will produce matured products. However, matured processes do not necessarily produce quality products [21, 6]. The primary reasons are: (i) In the process quality models, the relationship between the process quality and product quality is far from clear, and (ii) many of the process models take a monolithic view of the whole life-cycle process, and as a result, the idiosyncrasies of the individual processes do not receive proper attention. In this paper, we first define an internal process model in a formal manner. Next, we define a generic quality model whose scope covers all the development processes and most of the supporting processes associated with the development phase. The generic quality model is a parametric template and could be instantiated in a systematic manner to produce the quality model for any individual process. We then show such a customization for the formal specification process and use this customized model to formulate a GQM-based measurement plan for the same process. We then discuss how the generic model would be useful in process assessment and process improvement.

An integrated test generation tool for enhanced coverage of simulink/stateflow models

Simulink/Stateflow (SL/SF) is the primary modeling notation for the development of control systems in automotive and aerospace industries. In model based testing, test cases derived from a design model are used to show model-code conformance. Safety standards such as ISO 26262 recommend model based testing to show the conformance of a software with the corresponding model. From our experiments with various test generation techniques, we have observed that their coverage capabilities are complementary in nature. With this observation in mind, we have developed a new tool called SmartTestGen which integrates different test generation techniques. In this paper, we discuss SmartTestGen and the different test generation techniques utilized - random testing, constraint solving, model checking and heuristics. We experimented with 20 production-quality SL/SF models and compared the performance of our tool with that of two prominent commercial tools.

Automatic testing from formal specifications

In this article, we consider model oriented formal specification languages. We generate test cases by performing symbolic execution over a model, and from the test cases obtain a Java program. This Java program acts as a test driver and when it is run in conjunction with the implementation then testing is performed in an automatic manner. Our approach makes the testing cycle fully automatic. The main contribution of our work is that we perform automatic testing even when the models are non-deterministic.

Cross-layer analysis, testing and verification of automotive control software

Automotive architectures today consist of up to 100 electronic control units (ECUs) that communicate via one or more FlexRay and CAN buses. Multiple control applications - like cruise control, brake control, etc. - are specified as Simulink/Stateflow models, from which code is generated and mapped onto the different ECUs. In addition, scheduling policies and parameters, both for the ECUs and the buses, need to be specified. Code generation/optimization from the Simulink/Stateflow models, task partitioning and mapping decisions, as well as the parameters chosen for the schedulers - all of these impact the execution times and timing behaviour of the control tasks and control messages. These in turn affect control performance, such as stability and steady-/transient-state behaviour. This paper discusses different aspects of this multi-layered design flow and the associated research challenges. The emphasis is on model-based code generation, analysis, testing and verification of control software for automotive architectures, as well as on architecture or platform configuration to ensure that the required control performance requirements are satisfied.

A model-based regression test selection approach for embedded applications

Regression test selection techniques for embedded programs have scarcely been reported in the literature. In this paper, we propose a model-based regression test selection technique for embedded programs. Our proposed model, in addition to capturing the data and control dependence aspects, also represents several additional program features that are important for regression test case selection of embedded programs. These features include control flow, exception handling, message paths, task priorities, state information and object relations. We select a regression test suite based on slicing our proposed graph model. We also propose a genetic algorithm-based technique to select an optimal subset of test cases from the set of regression test cases selected after slicing our proposed model.

Effective Software Project Management Education through Simulation Models: An Externally Replicated Experiment

It is an undeniable fact that software project managers need reliable techniques and robust tool support to be able to exercise a fine control over the development process so that products can be delivered in time and within budget. Therefore, managers need to be trained so that they could learn and use new techniques and be aware of their possible impacts. In this context, effective learning is an issue. A small number of empirical studies have been carried out to study the impact of software engineering education. One such study is by Pfahl et al [11] in which they have performed a controlled experiment to evaluate the learning effectiveness of using a process simulation model for educating computer science students in software project management. The experimental group applied a Systems Dynamics simulation model while the control group used the COCOMO model as a predictive tool for project planning. The results indicated that students using the simulation model gain a better understanding about typical behaviour patterns of software development projects. Experiments need to be externally replicated to both verify and generalise original results. In this paper, we will discuss an externally replicated experiment in which we keep the design and the goal of the above experiment intact. We then analyse our results in relation to the original experiment and another externally replicated experiment, discussed in [12].

Efficient coverage of parallel and hierarchical stateflow models for test case generation

This paper is concerned with test case generation from Simulink/Stateflow (SL/SF) models with a focus on coverage of SF model elements. Coverage of the SF component in a model is a difficult task because of two primary reasons: (i) the SF component itself may lie deep in the SL/SF model in which case, inputs have to pass through a complex chain of SL blocks to reach the SF block and (ii) nonlinear constraints in the model are difficult to solve using constraint solvers. Hierarchy and parallelism in the SF model add further complexity to the problem. The existing approaches flatten such SF elements, and generate test cases from the flattened finite state machines. Handling of issues (i) and (ii) has already been discussed in earlier research. In this paper, we present a method of covering SF components, which does not require to flatten any hierarchy or parallelism in the components. This not only makes the test case generation problem efficient but also addresses the problem of scalability. We have implemented this method and performed a number of medium‐sized case studies. The results show improved performance over the results obtained by some commercial tools. Copyright

Test case generation from formal models through abstraction refinement and model checking

In this paper, we adapt the abstraction refinement techniques of Clarke et al. [7] and Gulavani et al. [15] for systematic generation of model based test cases. Formal models are in general infinite state machines. We approximate such a formal model by a more abstract finite state machine. From the finite model, we obtain probable test cases through model checking, and then a guided symbolic execution is performed over the given formal model to check if this is a real test case. In case of failure, the finite abstract model is refined and the cycle is repeated. The two main results of this paper are: (a) better specification coverage is achieved, and (b) we obtain a more accurate coverage estimate. We explain our method by considering models in the specification language B.