Marcus Ciolkowski

Software reviews, the state of the practice

A 2002 survey found that many companies use software reviews unsystematically, creating a mismatch between expected outcomes and review implementations. This suggests that many software practitioners understand basic review concepts but often fall to exploit their full potential.

Software inspection benchmarking-a qualitative and quantitative comparative opportunity

Software inspections are commonly used throughout the software industry, but there are still open questions about the relationship between inspection processes and inspection effectiveness. For example, which techniques work best in various environments? Are requirements specifications inspections and code inspections different in terms of effectiveness? What is the effectiveness in inspections for different group sizes? Benchmarking provides an opportunity to address such issues. This paper discusses how benchmarking may be applied for software inspections. The discussion is illustrated with an empirical study. It is shown how the data can be used to plan and manage software inspections. It is concluded that software inspections are well suited for benchmarking and that software practitioners as well as researchers can learn valuable lessons.

Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering

A survey is an empirical research strategy for the collection of information from heterogeneous sources. In this way, survey results often exhibit a high degree of external validity. It is complementary to other empirical research strategies such as controlled experiments, which usually have their strengths in the high internal validity of the findings. While there is a growing number of (quasi-)controlled experiments reported in the software engineering literature, few results of large scale surveys have been reported there. Hence, there is still a lack of knowledge on how to use surveys in a systematic manner for software engineering empirical research.

Towards evidence in software engineering

The aggregation of studies is of growing interest for the empirical software engineering community, since the numbers of studies steadily grow. We discuss challenges with the aggregation of studies into a common body of knowledge, based on a quantitative and qualitative evaluation of experience from the Experimental Software Engineering Network, ESERNET. Challenges are that the number of studies available is usually low, and the studies that exist are often too scattered and diverse to allow systematic aggregation as a means for generating evidence. ESERNET therefore attempted to coordinate studies and thus create research synergies to achieve a sufficiently large number of comparable studies to allow for aggregation; however, the coordination approach of ESERNET proved to be insufficient. Based on some lessons learned from ESERNET, a four-step procedure for evolving Empirical Software Engineering towards the generation of evidence is proposed. This consists of (1) developing a methodology for aggregating different kinds of empirical results, (2) establishing guidelines for performing, analyzing, and reporting studies as well as for aggregating the results for every kind of empirical study, (3) extract evidence, that is, apply the methodology to different areas of software engineering, and (4) package the extracted evidence into guidelines for practice.

Software inspections, reviews and walkthroughs

Presents some of the history of software inspections, walkthroughs and reviews. This shows that inspections are related to research efforts back in the 1970s. An example success story is briefly described to illustrate how research has impacted industrial software development practice in this area. The success story was the result of researchers and practitioners working closely together. Finally, challenges and questions as well as areas for further work are outlined. As a summary statement, one can say that, in the inspection area, research did have and still has impact on industrial practice.

Impact of research on practice in the field of inspections, reviews and walkthroughs: learning from successful industrial uses

Software inspections, reviews, and walkthroughs have become a standard process component in many software development domains. Maturity level 3 of the CMM-I requires establishment of peer reviews [12] and substantial sustained improvements in quality and productivity have been reported as a result of using reviews ([16], [21], [22], [27]). The NSF Impact project identifies the degree to which these industrial success cases have been instigated and improved by research in software engineering. This research identifies that there is widespread adoption of inspections, reviews or walkthroughs but that companies do not generally exploit their full potential. However there exist sustained industrial success cases with respect to the wide-spread and measurably successful application of them. It also identifies research in software engineering that can be credibly documented as having influenced the industrial success cases. Credible documentation may exist in the form of publications or documented reports by witnesses. Due to the semi-formal nature of inspections, reviews, and walkthroughs, a specific focus is given to empirical research results as motivators for adoption. Through the examination of one detailed case study, it is shown that software engineering research has had a significant impact on practice and that the impact can be traced in this case from research to that practice. The case study chosen provides evidence of both success and failure regarding sustained application in practice. Thus the analysis of historic impact chains of research reveals a clear impact of software engineering research on sustained industrial success for inspections, reviews and walkthroughs. More importantly, in impact chains where the empirical results have not been established, we conclude that success has not been achieved or has not been sustained. The paper closes with (1) lessons learned for creating the sustained use and impact of semi-formal software engineering processes, (2) a request for researchers and practitioners to further consider how their work can improve the effectiveness of research and practice, and (3) a request to contribute additional success cases and impact factors to the authors database for future enhancements of this paper.

Comparing code reading techniques applied to object-oriented software frameworks with regard to effectiveness and defect detection rate

This paper first reasons on understanding software frameworks for defect detection, and then presents an experimental research for comparing the effectiveness and defect detection rate of code-reading techniques, once applied to C++ coded object-oriented frameworks. We present and discuss the functionality-based approach to framework understanding. Then, we present an experiment that compared three reading techniques for inspection of software frameworks. Two of those reading techniques, namely checklist-based reading, and systematic order-based reading, were adopted from scientific literature, while the third one, namely functionality-based reading, was derived from the functionality-based approach. The results of the experiment are that (1) functionality-based reading is much more effective and efficient than checklist based reading. (2) Functionality-based Reading is significantly more effective and efficient than systematic order-based reading. (3) Systematic order-based reading performs significantly better than checklist based reading for what concerns defect detection rate. However, because we used checklist-based reading and systematic order-based reading quite as they are, with limited adaptation to frameworks, it is too early to draw strong conclusions from the experiment results and improving and replicating this study is strongly recommended.

Using academic courses for empirical validation of software development processes

Software process improvement needs sound empirical data gathered from a range of empirical studies such as controlled experiments or case studies. However, conducting empirical studies is often cost-intensive; in particular in industrial environments. Therefore, we need to find a way to balance cost and value of empirical findings. In This work, we discuss a way of gathering required empirical data in software engineering education programs. Although an academic setting is not representative of industrial organizations, the information gathered there can help organizations to decide upon alternatives to improve current software engineering processes. We describe how a practical software engineering course for graduate students at the University of Kaiserslautern was used as a platform for validating software development approaches. As there are specific restrictions and constraints defined by this context, we discuss how they influence the design of empirical studies. In addition, we present a concrete example of comparing product line approaches.

Investigating the active guidance factor in reading techniques for defect detection

Inspections are an established quality assurance technique. In order to optimize the inspection performance, different reading techniques, such as checklist-based reading and scenario-based reading have been proposed. Various experiments have been conducted to evaluate which of these techniques produces better inspection results (i.e., which finds more defects with less effort). However, results of these empirical investigations are not conclusive yet. Thus, the success factors of the different reading approaches need to be further analyzed. In this paper, we report on a preliminary empirical study that examined the influence of the active guidance factor (provided by scenario-based approaches) when inspecting requirements specification documents. First results show that active guidance is accepted with favor by inspectors and suggest that it is better suited for larger and more complex documents.

Assessing the impact of active guidance for defect detection: a replicated experiment

Scenario-based reading (SBR) techniques have been proposed as an alternative to checklists to support the inspectors throughout the reading process in the form of operational scenarios. Many studies have been performed to compare these techniques regarding their impact on the inspector performance. However, most of the existing studies have compared generic checklists to a set of specific reading scenarios, thus confounding the effects of two SBR key factors: separation of concerns and active guidance. In a previous work we have preliminarily conducted a repeated case study at the University of Kaiserslautern to evaluate the impact of active guidance on inspection performance. Specifically, we compared reading scenarios and focused checklists, which were both characterized as being perspective-based. The only difference between the reading techniques was the active guidance provided by the reading scenarios. We now have replicated the initial study with a controlled experiment using as subjects 43 graduate students in computer science at University of Bari. We did not find evidence that active guidance in reading techniques affects the effectiveness or the efficiency of defect detection. However, inspectors showed a better acceptance of focused checklists than reading scenarios.