Oliver Laitenberger

The empirical investigation of perspective-based reading

We consider reading techniques a fundamental means of achieving high quality software. Due to the lack of research in this area, we are experimenting with the application and comparison of various reading techniques. This paper deals with our experiences with a family of reading techniques known as Perspective-Based Reading (PBR), and its application to requirements documents. The goal of PBR is to provide operational scenarios where members of a review team read a document from a particular perspective, e.g., tester, developer, user. Our assumption is that the combination of different perspectives provides better coverage of the document, i.e., uncovers a wider range of defects, than the same number of readers using their usual technique.To test the effectiveness of PBR, we conducted a controlled experiment with professional software developers from the National Aeronautics and Space Administration/Goddard Space Flight Center (NASA/GSFC) Software Engineering Laboratory (SEL). The subjects read two types of documents, one generic in nature and the other from the NASA domain, using two reading techniques, a PBR technique and their usual technique. The results from these experiments, as well as the experimental design, are presented and analyzed. Teams applying PBR are shown to achieve significantly better coverage of documents than teams that do not apply PBR.We thoroughly discuss the threats to validity so that external replications can benefit from the lessons learned and improve the experimental design if the constraints are different from those posed by subjects borrowed from a development organization.

An encompassing life cycle centric survey of software inspection

Abstract This paper contributes an integrated survey of the work in the area of software inspection. It consists of two main sections. The first one introduces a detailed description of the core concepts and relationships that together define the field of software inspection. The second one elaborates a taxonomy that uses a generic development life-cycle to contextualize software inspection in detail. After Fagans seminal work presented in 1976, the body of work in software inspection has greatly increased and reached measured maturity. Yet, there is still no encompassing and systematic view of this research body driven from a life-cycle perspective. This perspective is important since inspection methods and refinements are most often aligned to particular life-cycle artifacts. It also provides practitioners with a roadmap available in their terms. To provide a systematic and encompassing view of the research and practice body in software inspection, the contribution of this survey is, in a first step, to introduce in detail the core concepts and relationships that together embody the field of software inspection. This lays out the field key ideas and benefits and elicits a common vocabulary. There, we make a strong effort to unify the relevant vocabulary used in available literature sources. In a second step, we use this vocabulary to build a contextual map of the field in the form of a taxonomy indexed by the different development stages of a generic process. This contextual map can guide practitioners and focus their attention on the inspection work most relevant to the introduction or development of inspections at the level of their particular development stage; or to help motivate the use of software inspection earlier in their development cycle. Our work provides three distinct, practical benefits: First, the index taxonomy can help practitioners identify inspection experience directly related to a particular life-cycle stage. Second, our work allows structuring of the large amount of published inspection work. Third, such taxonomy can help researchers compare and assess existing inspection methods and refinements to identify fruitful areas of future work.

An internally replicated quasi-experimental comparison of checklist and perspective based reading of code documents

The basic premise of software inspections is that they detect and remove defects before they propagate to subsequent development phases where their detection and correction cost escalates. To exploit their full potential, software inspections must call for a close and strict examination of the inspected artifact. For this, reading techniques for defect detection may be helpful since these techniques tell inspection participants what to look for and, more importantly, how to scrutinize a software artifact in a systematic manner. Recent research efforts investigated the benefits of scenario-based reading techniques. A major finding has been that these techniques help inspection teams find more defects than existing state-of-the-practice approaches, such as, ad-hoc or checklist-based reading (CBR). We experimentally compare one scenario-based reading technique, namely, perspective-based reading (PBR), for defect detection in code documents with the more traditional CBR approach. The comparison was performed in a series of three studies, as a quasi experiment and two internal replications, with a total of 60 professional software developers at Bosch Telecom GmbH. Meta-analytic techniques were applied to analyze the data.

An experimental comparison of reading techniques for defect detection in UML design documents

Abstract The basic motivation for software inspections is to detect and remove defects before they propagate to subsequent development phases where their detection and removal become more expensive. To maximize this potential, the examination of the artefact under inspection must be as thorough and detailed as possible. This implies the need for systematic reading techniques that tell inspection participants what to look for and, more importantly, how to scrutinize a software document. Recent research efforts have investigated the benefits of scenario-based reading techniques for defect detection in functional requirements and functional code documents. A major finding has been that these techniques help inspection teams find more defects than existing state-of-the-art approaches, such as, ad hoc or checklist-based reading (CBR). In this paper, we describe and experimentally compare one scenario-based reading technique, namely perspective-based reading (PBR), for defect detection in object-oriented design documents using the notation of the unified modeling language (UML) to the more traditional CBR approach. The comparison was performed in a controlled experiment with 18 practitioners as subjects. Our results indicate that PBR teams discovered, on average, 58% of the defects and had an average cost per defect ratio of 56 min per defect. In this way, PBR is more effective than CBR (i.e., it resulted in inspection teams detecting, on average, 41% more unique defects than CBR). Moreover, the cost of defect detection using PBR is significantly lower than CBR (i.e., PBR exhibits, on average, a 58% cost per defect improvement over CBR). This study therefore provides evidence demonstrating the efficacy of PBR scenarios for defect detection in UML design documents. In addition, it demonstrates that a PBR inspection is a promising approach for improving the quality of models developed using the UML notation.

Software reviews, the state of the practice

A 2002 survey found that many companies use software reviews unsystematically, creating a mismatch between expected outcomes and review implementations. This suggests that many software practitioners understand basic review concepts but often fall to exploit their full potential.

Perspective-based reading of code documents at Robert Bosch GmbH

Despite dramatic changes in software development in the two decades since the term software engineering was coined, software quality deficiencies and cost overruns continue to afflict the software industry. Inspections, developed at IBM by Fagan in the early 1970s 1], can be used to improve upon these problems because they allow the detection and removal of defects after each phase of the software development process. But, in most published inspection processes, individuals performing defect detection are not systematically supported. There, defect detection depends heavily upon factors like chance or experience. Further, there is an ongoing debate in the literature whether or not defect detection is more effective when performed as a group activity and hence should be conducted in meetings 5,11,13,14]. In this article we introduce Perspective-based Reading (PBR) for code documents, a systematic technique to support individual defect detection. PBR offers guidance to individual inspectors for defect detection. This guidance is embodied within perspective-based algorithmic scenarios which makes individual defect detection independent of experience. To test this assumption, we tailored and introduced PBR in the inspection process at Robert Bosch GmbH. We conducted two training sessions in the form of a 2 × 3 fractional-factorial experiment in which 11 professional software developers reviewed code documents from three different perspectives. The experimental results are: (1) Perspectivebased Reading and the type of document have an influence on individual defect detection, (2) multi-individual inspection meetings were not very useful to detect defects, (3) the overlap of detected defects among inspectors using different perspectives is low, and (4) there are no significant differences with respect to defect detection between inspectors having experiences in the programming language and/or the application domain and those that do not.

Evaluating capture-recapture models with two inspectors

Capture-recapture (CR) models have been proposed as an objective method for controlling software inspections. CR models were originally developed to estimate the size of animal populations. In software, they have been used to estimate the number of defects in an inspected artifact. This estimate can be another source of information for deciding whether the artifact requires a reinspection to ensure that a minimal inspection effectiveness level has been attained. Little evaluative research has been performed thus far on the utility of CR models for inspections with two inspectors. We report on an extensive Monte Carlo simulation that evaluated capture-recapture models suitable for two inspectors assuming a code inspections context. We evaluate the relative error of the CR estimates as well as the accuracy of the reinspection decision made using the CR model. Our results indicate that the most appropriate capture-recapture model for two inspectors is an estimator that allows for inspectors with different capabilities. This model always produces an estimate (i.e., does not fail), has a predictable behavior (i.e., works well when its assumptions are met), will have a relatively high decision accuracy, and will perform better than the default decision of no reinspections. Furthermore, we identify the conditions under which this estimator will perform best.

Evaluating the learning effectiveness of using simulations in software project management education: results from a twice replicated experiment

Abstract The increasing demand for software project managers in industry requires strategies for the development of management-related knowledge and skills of the current and future software workforce. Although several educational approaches help to develop the necessary skills in a university setting, few empirical studies are currently available to characterise and compare their effects. This paper presents the results of a twice replicated experiment that evaluates the learning effectiveness of using a process simulation model for educating computer science students in software project management. While the experimental group applied a System Dynamics simulation model, the control group used the well-known COCOMO model as a predictive tool for project planning. The results of each empirical study indicate that students using the simulation model gain a better understanding about typical behaviour patterns of software development projects. The combination of the results from the initial experiment and the two replications with meta-analysis techniques corroborates this finding. Additional analysis shows that the observed effect can mainly be attributed to the use of the simulation model in combination with a web-based role-play scenario. This finding is strongly supported by information gathered from the debriefing questionnaires of subjects in the experimental group. They consistently rated the simulation-based role-play scenario as a very useful approach for learning about issues in software project management.

Cost-effective Detection of Software Defects through Perspective-basedInspections

Software organizations must deliver high quality products on time and within budget to remain competitive in the marketplace. However, the reliable and predictable development of high quality software continues to be a major problem, largely due to the inadequate and late removal of defects. One of the proposed solutions for early detection and removal of defects is software inspection (Fagan, 1976), (Gilb and Graham, 1993). A software inspection requires a team of qualified persons to scrutinize software documents for defects. It applies to any development methodology and to any phase in the software development process. The value of an inspection stems from the fact that it improves quality and saves defect costs, since it minimizes the time between insertion of a defect and its discovery. This is important because the cost of finding and fixing a defect increases significantly each time it propagates to the next development phase. To fully achieve the expected benefits, two practical questions related to software inspection need to be answered: how to organize a particular inspection in a development project with a large volume of documentation and how to provide technical support for the defect detection activity of an inspection with reading techniques (Basili, 1997). This research presents a new strategy for inspection organization. The basis for this strategy is the distinction between logical entities and their documentation. This distinction is necessary because, in contrast to other engineering disciplines, the entities in software engineering are invisible and cannot be inspected per se. It is only the documentation of the entities that can be scrutinized for defects. The strategy, dubbed architecture-centric inspection organization, suggests partitioning and grouping the documentation to be inspected according to logical entities rather than characteristics of the documentation, such as their type or size. Following this strategy, the most appropriate unit to inspect can be determined. This unit refers to documentation that contains relevant information about one or several logical entities. Organizing an inspection in this manner avoids a decline in the cost-effectiveness of inspection because of fatigue effects or loss of motivation. Although the architecture-centric approach is generally applicable, it is particularly valuable

The application of subjective estimates of effectiveness to controlling software inspections

Abstract One of the recently proposed tools for controlling software inspections is capture–recapture models. These are models that can be used to estimate the number of remaining defects in a software document after an inspection. Based on this information one can decide whether to reinspect a document to ensure that it is below a prespecified defect density threshold, and that the inspection process itself has attained a minimal level of effectiveness. This line of work has also recently been extended with other techniques, such as the detection profile method (DPM). In this paper, we investigate an alternative approach: the use of subjective estimates of effectiveness by the inspectors for making the reinspection decision. We performed a study with 30 professional software engineers and found that the median relative error of the engineers’ subjective estimates of defect content to be zero, and that the reinspection decision based on that estimate is consistently more correct than the default decision of never reinspecting. This means that subjective estimates provide a good basis for ensuring product quality and inspection process effectiveness during software inspections. Since a subjective estimation procedure can be easily integrated into existing inspection processes, it represents a good starting point for practitioners before introducing more objective decision making criteria by means of capture–recapture models or the defect detection profile method.