David M. Weiss

A Methodology for Collecting Valid Software Engineering Data

An effective data collection method for evaluating software development methodologies and for studying the software development process is described. The method uses goal-directed data collection to evaluate methodologies with respect to the claims made for them. Such claims are used as a basis for defining the goals of the data collection, establishing a list of questions of interest to be answered by data analysis, defining a set of data categorization schemes, and designing a data collection form. The data to be collected are based on the changes made to the software during development, and are obtained when the changes are made. To ensure accuracy of the data, validation is performed concurrently with software development and data collection. Validation is based on interviews with those people supplying the data. Results from using the methodology show that data validation is a necessary part of change data collection. Without it, as much as 50 percent of the data may be erroneous. Feasibility of the data collection methodology was demonstrated by applying it to five different projects in two different environments. The application showed that the methodology was both feasible and useful.

The Modular Structure of Complex Systems

This paper discusses the organization of software that is inherently complex because of very many arbitrary details that must be precisely right for the software to be correct. We show how the software design technique known as information hiding, or abstraction, can be supplemented by a hierarchically structured document, which we call a module guide. The guide is intended to allow both designers and maintainers to identify easily the parts of the software that they must understand, without reading irrelevant details about other parts of the software. The paper includes an extract from a software module guide to illustrate our proposals.

Commonality and variability in software engineering

The article describes how to perform domain engineering by identifying the commonalities and variabilities within a family of products. Through interesting examples dealing with reuse libraries, design patterns, and programming language design, the authors suggest a systematic scope, commonalities, and variabilities approach to formal analysis. Their SCV analysis has been an integral part of the FAST (Family-oriented Abstraction, Specification, and Translation) technology applied to over 25 domains at Lucent Technologies.

Predicting risk of software changes

Reducing the number of software failures is one of the most challenging problems of software production. We assume that software development proceeds as a series of changes and model the probability that a change to software will cause a failure. We use predictors based on the properties of a change itself. Such predictors include size in lines of code added, deleted, and unmodified; diffusion of the change and its component subchanges, as reflected in the number of files, modules, and subsystems touched, or changed; several measures of developer experience; and the type of change and its subchanges (fault fixes or new code). The model is built on historic information and is used to predict the risk of new changes. In this paper we apply the model to 5ESS® software updates and find that change diffusion and developer experience are essential to predicting failures. The predictive model is implemented as a Web-based tool to allow timely prediction of change quality. The ability to predict the quality of change enables us to make appropriate decisions regarding inspection, testing, and delivery. Historic information on software changes is recorded in many commercial software projects, suggesting that our results can be easily and widely applied in practice.

Architecture reviews: practice and experience

Architecture reviews have evolved over the past decade to become a critical part of our continuing efforts to improve the state of affairs. We use them to identify project problems before they become costly to fix and to provide timely information to upper management so that they can make better-informed decisions. It provides the foundation for reuse, using commercially available software, and getting to the marketplace fast. The reviews also help identify best practices to projects and socialize such practices across the organization, thereby improving the organizations quality and operations.

Evaluating Software Development by Analysis of Changes: Some Data from the Software Engineering Laboratory

An effective data collection methodology for evaluating software development methodologies was applied to five different software development projects. Results and data from three of the projects are presented. Goals of the data collection included characterizing changes, errors, projects, and programmers, identifying effective error detection and correction techniques, and investigating ripple effects.

Understanding and predicting effort in software projects

We set out to answer a question we were asked by software project management: how much effort remains to be spent on a specific software project and how will that effort be distributed over time? To answer this question we propose a model based on the concept that each modification to software may cause repairs at some later time and investigate its theoretical properties and application to several projects in Avaya to predict and plan development resource allocation. Our model presents a novel unified framework to investigate and predict effort, schedule, and defects of a software project. The results of applying the model confirm a fundamental relationship between the new feature and defect repair changes and demonstrate its predictive properties.

A Survey of Coverage-Based Testing Tools

Test coverage is sometimes used to measure how thoroughly software is tested and developers and vendors sometimes use it to indicate their confidence in the readiness of their software. This survey studies and compares 17 coverage-based testing tools primarily focusing on, but not restricted to, coverage measurement. We also survey features such as program prioritization for testing, assistance in debugging, automatic generation of test cases and customization of test reports. Such features make tools more useful and practical, especially for large-scale, commercial software applications. Our initial motivations were both to understand the available test coverage tools and to compare them to a tool that we have developed, called eXVantage (a tool suite that includes code coverage testing, debugging, performance profiling and reporting). Our study shows that each tool has some unique features tailored to its application domains. The readers may use this study to help pick the right coverage testing tools for their needs and environment. This paper is also valuable to those who are new to the practice and the art of software coverage testing, as well as those who want to understand the gap between industry and academia.

Software product lines: a case study

A software product line is a family of products that share common features to meet the needs of a market area. Systematic processes have been developed to dramatically reduce the cost of a product line. Such product‐line engineering processes have proven practical and effective in industrial use, but are not widely understood. The Family‐Oriented Abstraction, Specification and Translation (FAST) process has been used successfully at Lucent Technologies in over 25 domains, providing productivity improvements of as much as four to one. In this paper, we show how to use FAST to document precisely the key abstractions in a domain, exploit design patterns in a generic product‐line architecture, generate documentation and Java code, and automate testing to reduce costs. The paper is based on a detailed case study covering all aspects from domain analysis through testing. Copyright

A survey of coverage based testing tools

Test coverage is sometimes used as a way to measure how thoroughly software is tested. Coverage is used by software developers and sometimes by vendors to indicate their confidence in the readiness of their software. This survey studies and compares 17 coverage-based testing tools focusing on, but not restricted to coverage measurement. We also survey additional features, including program prioritization for testing, assistance in debugging, automatic generation of test cases, and customization of test reports. Such features make tools more useful and practical, especially for large-scale, real-life commercial software applications. Our initial motivations were both to understand the available test coverage tools and to compare them to a tool that we have developed, called eXVantage1 (a tool suite that includes code coverage testing, debugging, performance profiling, and reporting). Our study shows that each tool has its unique features tailored to its application domains. Therefore this study can be used to pick the right coverage testing tools depending on various requirements.