Barbara Unger

A controlled experiment in maintenance: comparing design patterns to simpler solutions

Software design patterns package proven solutions to recurring design problems in a form that simplifies reuse. We are seeking empirical evidence whether using design patterns is beneficial. In particular, one may prefer using a design pattern even if the actual design problem is simpler than that solved by the pattern, i.e., if not all of the functionality offered by the pattern is actually required. Our experiment investigates software maintenance scenarios that employ various design patterns and compares designs with patterns to simpler alternatives. The subjects were professional software engineers. In most of our nine maintenance tasks, we found positive effects from using a design pattern: either its inherent additional flexibility was achieved without requiring more maintenance time or maintenance time was reduced compared to the simpler alternative. In a few cases, we found negative effects: the alternative solution was less error-prone or required less maintenance time. Overall, we conclude that, unless there is a clear reason to prefer the simpler solution, it is probably wise to choose the flexibility provided by the design pattern because unexpected new requirements often appear. We identify several questions for future empirical research.

An experiment measuring the effects of personal software process (PSP) training

The personal software process is a process improvement methodology aimed at individual software engineers. It claims to improve software quality (in particular defect content), effort estimation capability, and process adaptation and improvement capabilities. We have tested some of these claims in an experiment comparing the performance of participants who had just previously received a PSP course to a different group of participants who had received other technical training instead. Each participant of both groups performed the same task. We found the following positive effects: the PSP group estimated their productivity (though not their effort) more accurately, made fewer trivial mistakes, and their programs performed more careful error-checking; further, the performance variability was smaller in the PSP group in various respects. However, the improvements are smaller than the PSP proponents usually assume, possibly due to the low actual usage of PSP techniques in the PSP group. We conjecture that PSP training alone does not automatically realize the PSPs potential benefits (as seen in some industrial PSP success stories) when programmers are left alone with motivating themselves to actually use the PSP techniques.

A controlled experiment on inheritance depth as a cost factor for code maintenance

In two controlled experiments we compare the performance on code maintenance tasks for three equivalent programs with 0, 3, and 5 levels of inheritance. For the given tasks, which focus on understanding effort more than change effort, programs with less inheritance were faster to maintain. Daly et al. previously reported similar experiments on the same question with quite different results. They found that the 5-level program tended to be harder to maintain than the 0-level program, while the 3-level program was significantly easier to maintain than the 0-level program. We describe the design and setup of our experiment, the differences to the previous ones, and the results obtained. Ours and the previous experiments are different in several ways: We used a longer and more complex program, made an inheritance diagram available to the subjects, and added a second kind of maintenance task.When taken together, the previous results plus ours suggest that there is no such thing as usefulness or harmfulness of a certain inheritance depth as such. Code maintenance effort is hardly correlated with inheritance depth, but rather depends on other factors (partly related to inheritance depth). Using statistical modeling, we identify the number of relevant methods to be one such factor. We use it to build an explanation model of average code maintenance effort that is much more powerful than a model relying on inheritance depth.

Replication of the first controlled experiment on the usefulness of design patterns: Detailed description and evaluation

Advocates of software design patterns claim that using design patterns improves communication between software developers. The controlled experiment that we describe in this report tests the hypotheses that software maintainers of well-structured, well-documented software containing design patterns can make changes (1) faster and (2) with less errors if the use of patterns is explicitly documented in the software. The experiment was performed with 22 participants of a university course on C++ and design patterns; it is similar to a previous experiment performed in Karlsruhe. For one of the two experiment tasks the experiment finds that both hypotheses appear to be true. For the other task the results are inconclusive, presumably because the task was too difficult for the given experience level of the subjects.

Methodik und Ergebnisse einer Experimentreihe ¨ uber Entwurfsmuster

Zusammenfassung. Software-Entwurfsmuster sind eine intuitiv einleuchtende Idee, die viele Befürworter hat. Als Forscher dürfen wir die behaupteten Wirkungen von Entwurfsmustern jedoch nicht einfach glauben, sondern müssen sie gründlich prüfen. Dieser Artikel beschreibt die wichtigsten Behauptungen und entwickelt daraus zugehörige Forschungsfragen. Wir diskutieren die Methodik für ein Forschungsprogramm, das die Fragen beantworten soll, und skizzieren praktische Beschränkungen, unter denen diese Forschung ablaufen muß. Es folgt eine kurze Beschreibung von drei kontrollierten Experimenten und den Hauptfolgerungen aus ihren Ergebnissen. So sollte man zum Beispiel die Benutzung von Entwurfsmustern in einem Entwurf genau dokumentieren und Entwurfsmuster nicht einsetzen, ohne alternative Entwürfe zu prüfen. Abschließend diskutieren wir ein im Rahmen des Forschungsprogramms geplantes viertes Experiment. Der Beitrag dieses Artikels liegt vorrangig in einer Beschreibung und Diskussion wichtiger methodischer Aspekte kontrollierter Experimente in der Softwaretechnik.Abstract. Software design patterns are an idea that is intuitively appealing and has found many advocates. However, as scientists we must be concerned about gathering hard evidence for the claims of beneficial consequences of design patterns. This article describes the major claims and derives the corresponding research questions. It discusses the methodology of a research programme for investigating these questions and sketches the practical constraints that make this research difficult. It then shortly summarizes three controlled experiments that were successfully carried out within these constraints and lists the main results and their consequences, such as: One should document design patterns when they are used and one must not apply design patterns without judgement of alternatives. Finally, design considerations of a fourth experiment are discussed. The contribution of this paper is a description of important methodological aspects of practical experimental work and how these relate to the results obtained. Understanding these relations will be important in future empirical software engineering research.