Carina Andersson

What do we know about defect detection methods? [software testing]

A survey of defect detection studies comparing inspection and testing techniques yields practical recommendations: use inspections for requirements and design defects, and use testing for code. Evidence-based software engineering can help software practitioners decide which methods to use and for what purpose. EBSE involves defining relevant questions, surveying and appraising avail able empirical evidence, and integrating and evaluating new practices in the target environment. This article helps define questions regarding defect detection techniques and presents a survey of empirical studies on testing and inspection techniques. We then interpret the findings in terms of practical use. The term defect always relates to one or more underlying faults in an artifact such as code. In the context of this article, defects map to single faults

Verification and validation in industry - a qualitative survey on the state of practice

Verification and validation activities take a substantial share of project budgets and need improvements. This is an accepted truth, but the current practices are seldom assessed and analyzed. In this paper we present a qualitative survey of the verification and validation processes at 11 Swedish companies. The purpose was to exchange experience between the companies and to lay out a foundation for further research on the topic. The survey is conducted through workshop and interview sessions, loosely guided by a questionnaire scheme. It is concluded from the survey that there are substantial differences between small and large companies. In large companies, the documented process is emphasized while in small companies, single key, persons have a dominating impact on the procedures. Large companies use commercial tools while small companies make in-house tools or use shareware. Common to all the surveyed companies is that verification and validation is considered important, and thus having rather high status. Information exchange between companies during the survey was considered to be very valuable to the involved subjects.

Test processes in software product evolution: a qualitative survey on the state of practice

In order to understand the state of test process practices in the software industry, we have conducted a qualitative survey, covering software development departments at 11 companies in Sweden of different sizes and application domains. The companies develop products in an evolutionary manner, which means either new versions are released regularly, or new product variants under new names are released. The survey was conducted through workshop and interview sessions, loosely guided by a questionnaire scheme.The main conclusions of the survey are that the documented development process is emphasized by larger organizations as a key asset, while smaller organizations tend to lean more on experienced people. Further, product evolution is performed primarily as new product variants for embedded systems, and as new versions for packaged software. The development is structured using incremental development or a daily build approach; increments are used among more process-focused organizations, and daily build is more frequently utilized in less process-focused organizations. Test automation is performed using scripts for products with focus on functionality, and recorded data for products with focus on nonfunctional properties. Test automation is an issue which most organizations want to improve; handling the legacy parts of the product and related documentation presents a common problem in improvement efforts for product evolution.

A replicated empirical study of a selection method for software reliability growth models

Replications are commonly considered to be important contributions to investigate the generality of empirical studies. By replicating an original study it may be shown that the results are either valid or invalid in another context, outside the specific environment in which the original study was launched. The results of the replicated study show how much confidence we could possibly have in the original study. We present a replication of a method for selecting software reliability growth models to decide whether to stop testing and release software. We applied the selection method in an empirical study, conducted in a different development environment than the original study. The results of the replication study show that with the changed values of stability and curve fit, the selection method works well on the empirical system test data available, i.e., the method was applicable in an environment that was different from the original one. The application of the SRGMs to failures during functional testing resulted in predictions with low relative error, thus providing a useful approach in giving good estimates of the total number of failures to expect during functional testing.

Evaluation of Usage-Based Reading—Conclusions after Three Experiments

Software inspections have been introduced in software engineering in order to detect faults before testing is performed. Reading techniques provide reviewers in software inspections with guidelines on how they should check the documents under inspection. Several reading techniques with different purposes have been introduced and empirically evaluated. In this paper, we describe a reading technique with the special aim to detect faults that are severe from a user’s point of view. The reading technique is named usage-based reading (UBR) and it can be used to inspect all software artefacts. In the series of experiments, a high-level design document is used. The main focus of the paper is on the third experiment, which investigates the information needed for UBR in the individual preparation and the meeting of software inspections. Hence, the paper discusses (1) the series of three experiments of UBR, (2) the individual preparation of the third experiment, and (3) the meeting part of the third experiment. For each of these three parts, results are produced. The main results are (1) UBR is an efficient and effective reading technique that can be used for user-focused software inspections, (2) UBR is more efficient and effective if the information used for UBR is developed prior to, instead of during the individual preparation, and (3) the meeting affects the UBR inspection in terms of increased effectiveness and decreased efficiency. In summary, the empirical evidence shows that UBR is an efficient and effective reading technique to be used by software organizations that produce software for which the user perceived quality is important.

An experimental evaluation of inspection and testing for detection of design faults

The two most common strategies for verification and validation, inspection and testing, are in a controlled experiment evaluated in terms of their fault detection capabilities. These two techniques are in the previous work compared applied to code. In order to compare the efficiency and effectiveness of these techniques on a higher abstraction level than code, this experiment investigates inspection of design documents and testing of the corresponding program, to detect faults originating from the design document. Usage-based reading (UBR) and usage-based testing (UBT) were chosen for inspections and testing, respectively. These techniques provide similar aid to the reviewers as to the testers. The purpose of both fault detection techniques is to focus the inspection and testing from a users viewpoint. The experiment was conducted with 51 Masters students in a two-factor blocked design; each student applied each technique once, each application on different versions of the same program. The two versions contained different sets of faults, including 13 and 14 faults, respectively. The general results from this study show that when the two groups of subjects are combined, the efficiency and effectiveness are significantly higher for usage-based reading and that testing tends to require more learning. Rework is not taken into account, thus the experiment indicates strong support for design inspection over testing.

A Spiral Process Model for Case Studies on Software Quality Monitoring - Method and Metrics

This article presents a spiral process model for an iterative case study on quality monitoring, conducted in an industrial environment. In a highly iterative project, everything seems to happen at the same time: analysis, design and testing. We propose a spiral process model for case studies, and present a study conducted according to the proposed process. In the study, metrics collected from three software development projects are analysed to investigate which characteristics are stable across projects and feature groups of the product. The contribution of the article is multi-fold, detailing the case study methodology used with its sub-goals and procedures. Furthermore, the article presents the metrics collected and the results as such from the case study, which gives insights into a complex development environment and the trends of the retrieved data. The analysed data serve as feedback to the project staff to facilitate identification of software process improvement. The data have also been used for defect prediction. (Less)

A replicated experiment of usage-based and checklist-based reading

Software inspection is an effective method to detect faults in software artefacts. Several empirical studies have been performed on reading techniques, which are used in the individual preparation phase of software inspections. Besides new experiments, replications are needed to increase the body of knowledge in software inspections. We present a replication of an experiment, which compares usage-based and checklist-based reading. The results of the original experiment show that reviewers applying usage-based reading are more efficient and effective in detecting the most critical faults from a users point of view than reviewers using checklist-based reading. We present the data of the replication together with the original experiment and compares the experiments. The main result of the replication is that it confirms the result of the original experiment. This replication strengthens the evidence that usage-based reading is an efficient reading technique.

How much information is needed for usage-based reading? A series of experiments

Software inspections are regarded as an important technique to detect faults throughout the software development process. The individual preparation phase of software inspections has enlarged its focus from only comprehension to also include fault searching. Hence, reading techniques to support the reviewers on fault detection are needed. Usage-based reading (UBR) is a reading technique, which focuses on the important parts of a software document by using prioritized use cases. This paper presents a series of three UBR experiments on design specifications, with focus on the third. The first experiment evaluates the prioritization of UBR and the second compares UBR against checklist-based reading. The third experiment investigates the amount of information needed in the use cases and whether a more active approach helps the reviewers to detect more faults. The third study was conducted at two different places with a total of 82 subjects. The general result from the experiments is that UBR works as intended and is efficient as well as effective in guiding reviewers during the preparation phase of software inspections. Furthermore, the results indicate that use cases developed in advance are preferable compared to developing them as part of the preparation phase of the inspection.

Understanding software processes through system dynamics simulation: a case study

This paper presents a study with the intent to examine the opportunities provided by creating and using simulation models of software development processes. A model of one software development project was created through means of system dynamics, with data collected from documents, interviews and observations. The model was simulated in a commercial simulation tool. The simulation runs indicate that increasing the effort spent on the requirements phase, to a certain extent, will decrease the lead time and increase the quality in similar projects. The simulation model visualizes relations in the software process, and can be used by project managers when planning future projects. The study indicates that this type of simulation is a feasible way of modelling the process dynamically although the study calls for further investigations as to how project or process managers can benefit the most from using system dynamics simulations.