Michael Felderer

A taxonomy of risk-based testing

Software testing has often to be done under severe pressure due to limited resources and a challenging time schedule facing the demand to assure the fulfillment of the software requirements. In addition, testing should unveil those software defects that harm the mission-critical functions of the software. Risk-based testing uses risk (re-)assessments to steer all phases of the test process to optimize testing efforts and limit risks of the software-based system. Due to its importance and high practical relevance, several risk-based testing approaches were proposed in academia and industry. This paper presents a taxonomy of risk-based testing providing a framework to understand, categorize, assess, and compare risk-based testing approaches to support their selection and tailoring for specific purposes. The taxonomy is aligned with the consideration of risks in all phases of the test process and consists of the top-level classes risk drivers, risk assessment, and risk-based test process. The taxonomy of risk-based testing has been developed by analyzing the work presented in available publications on risk-based testing. Afterwards, it has been applied to the work on risk-based testing presented in this special section of the International Journal on Software Tools for Technology Transfer.

Integrating Manual and Automatic Risk Assessment for Risk-Based Testing

In this paper we define a model-based risk assessment procedure that integrates automatic risk assessment by static analysis, semi-automatic risk assessment and guided manual risk assessment. In this process probability and impact criteria are determined by metrics which are combined to estimate the risk of specific system development artifacts. The risk values are propagated to the assigned test cases providing a prioritization of test cases. This supports to optimize the allocation of limited testing time and budget in a risk-based testing methodology. Therefore, we embed our risk assessment process into a generic risk-based testing methodology. The calculation of probability and impact metrics is based on system and requirements artifacts which are formalized as model elements. Additional time metrics consider the temporal development of the system under test and take for instance the bug and version history of the system into account. The risk assessment procedure integrates several stakeholders and is explained by a running example.

Integrating risk-based testing in industrial test processes

Risk-based testing has a high potential to improve the software development and test process as it helps to optimize the allocation of resources and provides decision support for the management. But for many organizations, its integration into an existing test process is a challenging task. In this article, we provide a comprehensive overview of existing work and present a generic testing methodology enhancing an established test process to address risks. On this basis, we develop a procedure on how risk-based testing can be introduced in a test process and derive a stage model for its integration. We then evaluate our approach for introducing risk-based testing by means of an industrial study and discuss benefits, prerequisites and challenges to introduce it. Potential benefits of risk-based testing identified in the studied project are faster detection of defects resulting in an earlier release, a more reliable release quality statement as well as the involved test-process optimization. As necessary prerequisites for risk-based testing, we identified an inhomogeneous distribution of risks associated with the various parts of the tested software system as well as consolidated technical and business views on it. Finally, the identified challenges of introducing risk-based testing are reliable risk assessment in the context of complex systems, the availability of experts for risk assessment as well as established tool supports for test management.

Querying UML Models using OCL and Prolog: A Performance Study

The size of unified modeling language (UML) models used in practice is very large and ranges up to hundreds and thousands of classes. Querying of these models is used to support their quality assessment by information filtering and aggregating. For both, human cognition and automated analysis, there is a need for fast querying. In this context performance of model queries becomes an important issue. We investigated performance characteristics of two different querying engines: one using the object constraint language (OCL) and the other using prolog. Our comparison is based on equivalent queries in both languages. We applied the queries to 118 models of a size up to 10000 classes to analyze model load and evaluation time. Our preliminary results show that if execution time of queries is linear then prolog is faster. For one of the presented cases, the execution time in prolog was nonlinear and thus higher. Further studies should focus on a performance analysis reflecting expressiveness aspects. Our experimental material is accessible to enable future replications of this study.

Hybrid software and system development in practice: waterfall, scrum, and beyond

Software and system development faces numerous challenges of rapidly changing markets. To address such challenges, companies and projects design and adopt specific development approaches by combining well-structured comprehensive methods and flexible agile practices. Yet, the number of methods and practices is large, and available studies argue that the actual process composition is carried out in a fairly ad-hoc manner. The present paper reports on a survey on hybrid software development approaches. We study which approaches are used in practice, how different approaches are combined, and what contextual factors influence the use and combination of hybrid software development approaches. Our results from 69 study participants show a variety of development approaches used and combined in practice. We show that most combinations follow a pattern in which a traditional process model serves as framework in which several fine-grained (agile) practices are plugged in. We further show that hybrid software development approaches are independent from the company size and external triggers. We conclude that such approaches are the results of a natural process evolution, which is mainly driven by experience, learning, and pragmatism.

A multiple case study on risk-based testing in industry

In many development projects, testing has to be conducted under severe pressure due to limited resources and a challenging time schedule. Risk-based testing, which utilizes identified risks of the system for testing purposes, has a high potential to improve testing as it helps to optimize the allocation of resources and provides decision support for management. But for many organizations, the integration of a risk-based approach into established testing activities is a challenging task, and there are several options to do so. In this article, we analyze how risk is defined, assessed, and applied to support and improve testing activities in projects, products, and processes. We investigate these questions empirically by a multiple case study of currently applied risk-based testing activities in industry. The case study is based on three cases from different backgrounds, i.e., a test project in context of the extension of a large Web-based information system, product testing of a measurement and diagnostic equipment for the electrical power industry, as well as a test process of a system integrator of telecommunication solutions. By analyzing and comparing these different industrial cases, we draw conclusions on the state of risk-based testing and discuss possible improvements.

Model-based security testing: a taxonomy and systematic classification

Model‐based security testing relies on models to test whether a software system meets its security requirements. It is an active research field of high relevance for industrial applications, with many approaches and notable results published in recent years. This article provides a taxonomy for model‐based security testing approaches. It comprises filter criteria (i.e. model of system security, security model of the environment and explicit test selection criteria) as well as evidence criteria (i.e. maturity of evaluated system, evidence measures and evidence level). The taxonomy is based on a comprehensive analysis of existing classification schemes for model‐based testing and security testing. To demonstrate its adequacy, 119 publications on model‐based security testing are systematically extracted from the five most relevant digital libraries by three researchers and classified according to the defined filter and evidence criteria. On the basis of the classified publications, the article provides an overview of the state of the art in model‐based security testing and discusses promising research directions with regard to security properties, coverage criteria and the feasibility and return on investment of model‐based security testing. Copyright

A generic platform for model-based regression testing

Model-based testing has gained widespread acceptance in the last few years. Models enable the platform independent analysis and design of tests in an early phase of software development resulting in effort reduction in terms of time and money. Furthermore, test models are easier to maintain than test code when software systems evolve due to their platform independence and traceability support. Nevertheless, most regression testing approaches, which ensure that system evolution does not introduce unintended effects, are solely code-based. Additionally, many model-based testing approaches do not consider regression testing when applied in practice, mainly due to the lack of appropriate tool support. Therefore, in this paper we present a generic tool platform for model-based regression testing based on the model versioning and evolution framework MoVE. Our approach enhances existing model-based testing approaches with regression testing capabilities aiming at better tool support for model-based regression testing. In a case study, we apply our platform to the model-based testing approaches UML Testing Profile and Telling TestStories.

Naming the pain in requirements engineering

Requirements Engineering (RE) has received much attention in research and practice due to its importance to software project success. Its interdisciplinary nature, the dependency to the customer, and its inherent uncertainty still render the discipline difficult to investigate. This results in a lack of empirical data. These are necessary, however, to demonstrate which practically relevant RE problems exist and to what extent they matter. Motivated by this situation, we initiated the Naming the Pain in Requirements Engineering (NaPiRE) initiative which constitutes a globally distributed, bi-yearly replicated family of surveys on the status quo and problems in practical RE. In this article, we report on the qualitative analysis of data obtained from 228 companies working in 10 countries in various domains and we reveal which contemporary problems practitioners encounter. To this end, we analyse 21 problems derived from the literature with respect to their relevance and criticality in dependency to their context, and we complement this picture with a cause-effect analysis showing the causes and effects surrounding the most critical problems. Our results give us a better understanding of which problems exist and how they manifest themselves in practical environments. Thus, we provide a first step to ground contributions to RE on empirical observations which, until now, were dominated by conventional wisdom only.

Using Defect Taxonomies to Improve the Maturity of the System Test Process: Results from an Industrial Case Study

Defect taxonomies collect and organize the domain knowledge and project experience of experts and are a valuable instrument of system testing for several reasons. They provide systematic backup for the design of tests, support decisions for the allocation of testing resources and are a suitable basis for measuring the product and test quality. In this paper, we propose a method of system testing based on defect taxonomies and investigate how these can systematically improve the efficiency and effectiveness, i.e. the maturity of requirements-based testing. The method is evaluated via an industrial case study based on two projects from a public health insurance institution by comparing one project with defect taxonomy-supported testing and one without. Empirical data confirm that system testing supported by defect taxonomies (1) reduces the number of test cases, and (2) increases of the number of identified failures per test case.