Murray Wood

Further Experiences with Scenarios and Checklists

Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. This replication is broadly supportive of the results from the original experiment, namely, that the Scenario approach is superior to the Checklist approach; and that the meeting component of a software inspection is not an effective defect detection mechanism. This experiment also tentatively proposes additional relationships between general academic performance and individual inspection performance; and between meeting loss and group inspection performance.

Comparing and combining software defect detection techniques: a replicated empirical study

This report describes an empirical study comparing three defect detection techniques: a) code reading by stepwise abstraction, b) functional testing using equivalence partitioning and boundary value analysis, and c) structural testing using branch coverage. It is a replication of a study that has been carried out at least four times previously over the last 20 years. This study used 47 student subjects to apply the techniques to small C programs in a fractional factorial experimental design. The major findings of the study are: a) that the individual techniques are of broadly similar effectiveness in terms of observing failures and finding faults, b) that the relative effectiveness of the techniques depends on the nature of the program and its faults, c) these techniques are consistently much more effective when used in combination with each other. These results contribute to a growing body of empirical evidence that supports generally held beliefs about the effectiveness of defect detection techniques in software engineering.

Statistical power and its subcomponents — missing and misunderstood concepts in empirical software engineering research

Abstract Recently we have witnessed a welcomed increase in the amount of empirical evaluation of Software Engineering methods and concepts. It is hoped that this increase will lead to establishing Software Engineering as a well-defined subject with a sound scientifically proven underpinning rather than a topic based upon unsubstantiated theories and personal belief. For this to happen the empirical work must be of the highest standard. Unfortunately producing meaningful empirical evaluations is a highly hazardous activity, full of uncertainties and often unseen difficulties. Any researcher can overlook or neglect a seemingly innocuous factor, which in fact invalidates all of the work. More serious is that large sections of the community can overlook essential experimental design guidelines, which bring into question the validity of much of the work undertaken to date. In this paper, the authors address one such factor — Statistical Power Analysis. It is believed, and will be demonstrated, that any body of research undertaken without considering statistical power as a fundamental design parameter is potentially fatally flawed. Unfortunately the authors are unaware of much Software Engineering research which takes this parameter into account. In addition to introducing Statistical Power, the paper will attempt to demonstrate the potential difficulties of applying it to the design of Software Engineering experiments and concludes with a discussion of what the authors believe is the most viable method of incorporating the evaluation of statistical power within the experimental design process.

A novel software visualisation model to support software comprehension

Current software visualisation tools do not address the full range of software comprehension requirements. This paper proposes a novel software visualisation model for supporting object-oriented software comprehension that is intended to address the shortcomings of existing tools. We discuss the previous work that prompted us to develop this model. An initial model is then presented, based on multiple levels of abstraction, multiple perspectives of the software system, and the integration of statically and dynamically extracted information. We review the evaluation tasks used in our previous work and those from the software visualisation and comprehension literature to produce a refined set of evaluation tasks. We then use these tasks to perform an initial assessment of the proposed model. The refined model is then defined more formally. Finally, a concrete example of the use of the model to generate abstraction hierarchies is discussed. We conclude that a visualisation model incorporating a hierarchy of interrelated abstraction levels, combined with structural and behavioural perspectives of the software, will provide effective support for software comprehension.

The Java system dependence graph

The program dependence graph was introduced by Ottenstein and Ottenstein in 1984. It was suggested to be a suitable internal program representation for monolithic programs, for the purpose of carrying out certain software engineering operations such as slicing and the computation of program metrics. Since then, Horwitz et al. have introduced the multiprocedural equivalent system dependence graph. Several authors have proposed object-oriented dependence graph construction approaches. Every approach provides its own benefits, some of which are language specific. We present a Java system dependence graph which draws on the strengths of a range of earlier works and adapts them, if necessary, to the Java language. It also provides guidance on the construction of the graph, identifies potential research topics based on it and shows a completed graph with a slice highlighted for a small, but realistic example.

An empirical evaluation of defect detection techniques

This report describes an empirical study comparing three defect detection techniques: (a) code reading by stepwise abstraction, (b) functional testing using equivalence partitioning and boundary value analysis, and (c) structural testing using branch coverage. It is a replication of a study that has been carried out at least four times previously over the last 20 years. This study used 47 student subjects to apply the techniques to small C programs in a fractional factorial experimental design. The major findings of the study are: (a) that the individual techniques are of broadly similar effectiveness in terms of observing failures and finding faults, (b) that the relative effectiveness of the techniques depends on the nature of the program and its faults, (c) these techniques are consistently much more effective when used in combination with each other. These results contribute to a growing body of empirical evidence that supports generally held beliefs about the effectiveness of defect detection techniques in software engineering.

A review of tool support for software inspection

Inspection is widely believed to be the most cost-effective method for detecting defects in documents produced during the software development lifecycle. However, it is by its very nature a labour intensive process. This has led to work on computer support for the process which should increase the efficiency and effectiveness beyond what is currently possible with a solely manual process. In this paper we describe the scope for tool support for the inspection process and review currently available products. We conclude that no single tool available fills all the identified needs of inspection.<<ETX>>

The development and evaluation of three diverse techniques for object-oriented code inspection

We describe the development and evaluation of a rigorous approach aimed at the effective and efficient inspection of object-oriented (OO) code. Since the time that inspections were developed they have been shown to be powerful defect detection strategies. However, little research has been done to investigate their application to OO systems, which have very different structural and execution models compared to procedural systems. This suggests that inspection techniques may not be currently being deployed to their best effect in the context of large-scale OO systems. Work to date has revealed three significant issues that need to be addressed - the identification of chunks of code to be inspected, the order in which the code is read, and the resolution of frequent nonlocal references. Three techniques are developed with the aim of addressing these issues: one based on a checklist, one focused on constructing abstract specifications, and the last centered on the route that a use case takes through a system. The three approaches are evaluated empirically and, in this instance, it is suggested that the checklist is the most effective approach, but that the other techniques also have potential strengths. For the best results in a practical situation, a combination of techniques is recommended, one of which should focus specifically on the characteristics of OO.

Replication's Role in Software Engineering

We provide motivation for researchers to replicate experiments in software engineering. The ideology of replication is discussed. We address the question: Is an experiment worth repeating? The current lack of replication studies is highlighted. We make clear that exact replication is unattainable and we draw on our first experience of performing an external replication. To categorise various kinds of replication, we propose a simple extension to Basili et als framework for experimentation in software engineering. We present guidance as to the level of reported detail required to enable others perform a replication. Our conclusion is that there is only one route for empirical software engineering to follow: to make available laboratory packages of experimental materials to facilitate internal and external replications, especially the latter, which have greater confirming power.

The effect of inheritance on the maintainability of object-oriented software: an empirical study

The empirical study was undertaken as part of a programme of research to explore unsupported claims about the object-oriented paradigm: a series of experiments tested the effect of inheritance on the maintainability of object-oriented software. Subjects were asked to modify object-oriented software with a hierarchy of 3 levels of inheritance depth and equivalent object-based software with no inheritance. The collected timing data showed that subjects maintaining object-oriented software using inheritance performed the modification tasks, on average, approximately 20% quicker than those maintaining equivalent object-based software with no inheritance. An initial inductive analysis revealed that 2 out of 3 subjects performed faster when maintaining the object-oriented software with inheritance. The findings are sufficiently important that attempts to verify the results should be made by independent researchers. Subsequent studies should seek to scale up the findings to the maintenance of more complex software by professional programmers.