Hans-Gerhard Gross

Prioritizing tests for fault localization through ambiguity group reduction

In practically all development processes, regression tests are used to detect the presence of faults after a modification. If faults are detected, a fault localization algorithm can be used to reduce the manual inspection cost. However, while using test case prioritization to enhance the rate of fault detection of the test suite (e.g., statement coverage), the diagnostic information gain per test is not optimal, which results in needless inspection cost during diagnosis. We present RAPTOR, a test prioritization algorithm for fault localization, based on reducing the similarity between statement execution patterns as the testing progresses. Unlike previous diagnostic prioritization algorithms, RAPTOR does not require false negative information, and is much less complex. Experimental results from the Software Infrastructure Repositorys benchmarks show that RAPTOR is the best technique under realistic conditions, with average cost reductions of 40% with respect to the next best technique, with negligible impact on fault detection capability.

Prioritizing Tests for Software Fault Localization

Test prioritization techniques select test cases that maximize the confidence on the correctness of the system when the resources for quality assurance (QA) are limited. In the event of a test failing, the fault at the root of the failure has to be localized, adding an extra debugging cost that has to be taken into account as well. However, test suites that are prioritized for failure detection can reduce the amount of useful information for fault localization. This deteriorates the quality of the diagnosis provided, making the subsequent debugging phase more expensive, and defeating the purpose of the test cost minimization. In this paper we introduce a new test case prioritization approach that maximizes the improvement of the diagnostic information per test. Our approach minimizes the loss of diagnostic quality in the prioritized test suite. When considering QA cost as the combination of testing cost and debugging cost, on the Siemens set, the results of our test case prioritization approach show up to a 53% reduction of the overall QA cost, compared with the next best technique.

An industrial case study in reconstructing requirements views

Requirements views, such as coverage and status views, are an important asset for monitoring and managing software development projects. We have developed a method that automates the process of reconstructing these views, and we have built a tool, ReqAnalyst, that supports this method. This paper presents an investigation as to which extent requirements views can be automatically generated in order to monitor requirements in industrial practice. The paper focuses on monitoring the requirements in test categories and test cases. In order to retrieve the necessary data, an information retrieval technique, called Latent Semantic Indexing, was used. The method was applied in an industrial study. A number of requirements views were defined and experiments were carried out with different reconstruction settings for generating these views. Finally, we explored how these views can help the developers during the software development process.

Web API growing pains: Stories from client developers and their code

Web APIs provide a systematic and extensible approach for application-to-application interaction. Developers using web APIs are forced to accompany the API providers in their software evolution tasks. In order to understand the distress caused by this imposition on web API client developers we perform a semi-structured interview with six such developers. We also investigate how major web API providers organize their API evolution, and we explore how this affects source code changes of their clients. Our exploratory study of the Twitter, Google Maps, Facebook and Netflix web APIs analyzes the state of web API evolution practices and provides insight into the impact of service evolution on client software. Our study is complemented with a set of observations regarding best practices for web API evolution.

A Model for the Measurement of the Runtime Testability of Component-Based Systems

Runtime testing is emerging as the solution for the integration and validation of software systems where traditional development-time integration testing cannot be performed, such as Systems of Systems or Service Oriented Architectures.However, performing tests during deployment or in-service time introduces interference problems, such as undesired side-effects in the state of the system or the outside world.This paper presents a qualitative model of runtime testability that complements Binders classical testability model, and a generic measurement framework for quantitatively assessing the degree of runtime testabilityof a system based on the ratio of what can be tested at runtime vs. what would have been tested during development time.A measurement is devised for the concrete case of architecture-based test coverage, by using a graph model of the systems architecture. Concretely, two testabilitystudies are performed for two component based systems, showing how to measure the runtime testability of a system.

Applying a Model-based Approach for Embedded System Development

Model-based and component-oriented software development approaches are slowly superseding traditional ways of developing embedded systems. For investigating to which extent model-based development is feasible for embedded system development, we conducted a case study in which a small embedded system is developed using the MARMOT approach. In order to evaluate the degree of reuse that might be achieved, the components of the case study are used in the context of different small projects. Several aspects of reuse, application size, ease of adaptation, and development effort are quantified. This analysis reveals that model-based and component-oriented development performs well for small embedded systems, and it leads to the conclusion that model-driven/component-based development of embedded systems enables adaptable applications with higher- than-normal reuse rate.

Component-Based Software Development for Embedded Systems

Component-Based Software Development for Embedded Systems - An Introduction.- Component-Based Software Development for Embedded Systems - An Introduction.- Specification and Verification.- Specification and Verification of Applications Based on Function Blocks.- A Model-Based Approach to Formal Specification and Verification of Embedded Systems Using Colored Petri Nets.- Modular Verification of Reconfigurable Components.- Component Compatibility.- Behavioral Types for Embedded Software - A Survey.- Assessing Real-Time Component Contracts Through Built-in Evolutionary Testing.- Component Architectures, Implementation and Tool Support.- Platform-Independent Specification of Component Architectures for Embedded Real-Time Systems Based on an Extended UML.- Model Driven Software Development in the Context of Embedded Component Infrastructures.- A Component Framework for Consumer Electronics Middleware.- Connecting Embedded Devices Using a Component Platform for Adaptable Protocol Stacks.- CoConES: An Approach for Components and Contracts in Embedded Systems.- Adopting a Component-Based Software Architecture for an Industrial Control System - A Case Study.- Non-functional Properties.- Specification and Evaluation of Safety Properties in a Component-Based Software Engineering Process.- Performance Evaluation Approaches for Software Architects.- Component-Based Engineering of Distributed Embedded Control Software.- Component-Based Development of Dependable Systems with UML.

Structural performance measure of evolutionary testing applied to worst-case timing of real-time systems

Evolutionary testing is a new testing technique for automatically generating test cases which satisfy a given test criterion. For best or worst-case execution time assessment of real-time systems it can be used to generate test cases which minimise or maximise execution times or possibly violate the timing specification of the system. As a typical search or optimisation technique, evolutionary testing cannot guarantee to find test cases according to the test objective. The only outcome of such a search process is the time found, but there is no information on how close the result comes to the actual minimal or maximal time. Experiments with this testing technique established a relationship between the complexity of a test object and the success of the search process to find optimal or near optimal solutions. The paper can be seen as an initial attempt to define a predictive complexity measure which is able to indicate the degree of how successfully an evolutionary search might have performed on a test object. The measure is simple and easy to retrieve as it is based on a programs source code. It is extensible, which is important for a further improvement in accuracy. The application of the new measure has shown to be successful for many example test programs but also revealed weaknesses on test objects whose complexity is difficult to capture.

Prioritizing tests for software fault diagnosis

During regression testing, test prioritization techniques select test cases that maximize the confidence on the correctness of the system when the resources for quality assurance (QA) are limited. In the event of a test failing, the fault at the root of the failure has to be localized, adding an extra debugging cost that has to be taken into account as well. However, test suites that are prioritized for failure detection can reduce the amount of useful information for fault localization. This deteriorates the quality of the diagnosis provided, making the subsequent debugging phase more expensive, and defeating the purpose of the test cost minimization. In this paper we introduce a new test case prioritization approach that maximizes the improvement of the diagnostic information per test. Our approach minimizes the loss of diagnostic quality in the prioritized test suite. When considering QA cost as a combination of testing cost and debugging cost, on our benchmark set, the results of our test case prioritization approach show reductions of up to 60% of the overall combined cost of testing and debugging, compared with the next best technique. Copyright

Self-certification and trust in component procurement

Component-based software engineering is typically perceived as application development in which existing individual software components are assembled and integrated in order to make up the final product. The main recent technological advances in this field therefore mainly focus on the integration step. This encompasses the syntactic and semantic mapping between components, the development of component wrappers and adapters, and the validation of all pair-wise component interactions. Additionally, prior to integration, components have to be located on a component market place, evaluated for their fitness for the purpose, and selected according to nonfunctional requirements. These activities are typically referred to as component procurement. Component brokerage platforms provide the support for these early phases of component assembly, and they are indispensable for strengthening the software component market. Although such platforms are good at the provision of components, they are not so good at their certification.This article proposes the combination of two contrasting technologies, component brokerage at one end of the component technology spectrum, and built-in contract testing at its other end, that combined, may alleviate the efforts involved in component certification. This is achieved through the access mechanisms that built-in contract testing provides for components, and additional tester components through which customers can themselves assess the quality of a candidate component that is coming from a broker. Every such extended component is added to the certification according to well-defined standards, that are provided by a third party on behalf of the supplier.