David Godwin

The vital few versus the trivial many: examining the Pareto principle for software

This paper discusses the Pareto principle as it relates to the distribution of software defects in code. We look at evidence in the context of both the software test team, and users of the software. We also investigate two related principles. The first principle is that the distribution of defects in code relates to the distribution of complexity in code. The second principle is that how we define complexity relates to the distribution of defects in code. We present this work as an empirical study of three general hypotheses investigated for large production-level software; we show that the essence of the principle holds, while precise percentages do not.

Characteristics of multiple-component defects and architectural hotspots: a large system case study

The architecture of a large software system is widely considered important for such reasons as: providing a common goal to the stakeholders in realising the envisaged system; helping to organise the various development teams; and capturing foundational design decisions early in the development. Studies have shown that defects originating in system architectures can consume twice as much correction effort as that for other defects. Clearly, then, scientific studies on architectural defects are important for their improved treatment and prevention. Previous research has focused on the extent of architectural defects in software systems. For this paper, we were motivated to ask the following two complementary questions in a case study: (i) How do multiple-component defects (MCDs)—which are of architectural importance—differ from other types of defects in terms of (a) complexity and (b) persistence across development phases and releases? and (ii) How do highly MCD-concentrated components (the so called, architectural hotspots) differ from other types of components in terms of their (a) interrelationships and (b) persistence across development phases and releases? Results indicate that MCDs are complex to fix and are persistent across phases and releases. In comparison to a non-MCD, a MCD requires over 20 times more changes to fix it and is 6 to 8 times more likely to cross a phase or a release. These findings have implications for defect detection and correction. Results also show that 20% of the subject system’s components contain over 80% of the MCDs and that these components are 2–3 times more likely to persist across multiple system releases than other components in the system. Such MCD-concentrated components constitute architectural “hotspots” which management can focus upon for preventive maintenance and architectural quality improvement. The findings described are from an empirical study of a large legacy software system of size over 20 million lines of code and age over 17 years.

All code coverage is not created equal: a case study in prioritized code coverage

The increasing size and complexity of software systems has led to increased challenges in evaluating code coverage. In parallel, development projects for large complex software are increasingly limited in schedule and resources. Common code coverage analysis provides a complete measurement of testing breadth, but does not give guidance for coverage in a project setting. We propose a prioritized coverage approach that is indigenous to the development organization and provides capabilities for evaluating code coverage and setting priorities for testing. This paper presents an approach to code coverage analysis for large software that considers modern project constraints.

An iterative, multi-level, and scalable approach to comparing execution traces

In this paper, we overview a new approach to comparing execution traces. Such comparison can be useful for purposes such as improving test coverage and profiling systems users. In our approach, traces are compressed into different levels of compaction and are then compared iteratively from highest to lowest levels, rejecting dissimilar traces in the process and eventually leaving residual, similar traces. These residual traces form an important feedback for improvement or analysis goals. The preliminary results show that the approach is scalable for industrial use.

Analysis of pervasive multiple-component defects in a large software system

Certain software defects require corrective changes repeatedly in a few components of the system. One type of such defects spans multiple components of the system, and we call such defects pervasive multiple-component defects (PMCDs). In this paper, we describe an empirical study of six releases of a large legacy software system (of approx. size 20 million physical lines of code) to analyze PMCDs with respect to: (1) the complexity of fixing such defects and (2) the persistence of defect-prone components across phases and releases. The overall hypothesis in this study is that PMCDs inflict a greater negative impact than do other defects on defect-correction efficacy. Our findings show that the average number of changes required for fixing PMCDs is 20–30 times as much as the average for all defects. Also, over 80% of PMCD-contained defect-prone components still remain defect-prone in successive phases or releases. These findings support the overall hypothesis strongly. We compare our results, where possible, to those of other researchers and discuss the implications on maintenance processes and tools.

Focused iterative testing: a test automation case study

Timing-related defects are among the most difficult types of defects to catch while testing software. They are by definition difficult to reproduce and hence difficult to debug. Not all components of a software system have timing-related defects. For example, either a parser can analyze an input or it cannot. However, systems that have concurrent threads such as database systems are prone to timing-related defects. As a result, software developers must tailor testing to exploit vulnerabilities that occur because of threading. This paper presents the Focused Iterative Testing (FIT) approach, which uses a repetitive and iterative approach to find timing-related defects and target product areas with multithreaded characteristics by executing system tests with a multi-user test suite.