Kai Petersen

Guidelines for conducting systematic mapping studies in software engineering: An update

Abstract Context Systematic mapping studies are used to structure a research area, while systematic reviews are focused on gathering and synthesizing evidence. The most recent guidelines for systematic mapping are from 2008. Since that time, many suggestions have been made of how to improve systematic literature reviews (SLRs). There is a need to evaluate how researchers conduct the process of systematic mapping and identify how the guidelines should be updated based on the lessons learned from the existing systematic maps and SLR guidelines. Objective To identify how the systematic mapping process is conducted (including search, study selection, analysis and presentation of data, etc.); to identify improvement potentials in conducting the systematic mapping process and updating the guidelines accordingly. Method We conducted a systematic mapping study of systematic maps, considering some practices of systematic review guidelines as well (in particular in relation to defining the search and to conduct a quality assessment). Results In a large number of studies multiple guidelines are used and combined, which leads to different ways in conducting mapping studies. The reason for combining guidelines was that they differed in the recommendations given. Conclusion The most frequently followed guidelines are not sufficient alone. Hence, there was a need to provide an update of how to conduct systematic mapping studies. New guidelines have been proposed consolidating existing findings.

Context in industrial software engineering research

In order to draw valid conclusions when aggregating evidence it is important to describe the context in which industrial studies were conducted. This paper structures the context for empirical industrial studies and provides a checklist. The aim is to aid researchers in making informed decisions concerning which parts of the context to include in the descriptions. Furthermore, descriptions of industrial studies were surveyed.

A comparison of issues and advantages in agile and incremental development between state of the art and an industrial case

Recent empirical studies have been conducted identifying a number of issues and advantages of incremental and agile methods. However, the majority of studies focused on one model (Extreme Programming) and small projects. To draw more general conclusions we conduct a case study in large-scale development identifying issues and advantages, and compare the results with previous empirical studies on the topic. The principle results are that (1) the case study and literature agree on the benefits while new issues arise when using agile in large-scale and (2) an empirical research framework is needed to make agile studies comparable.

The effect of moving from a plan-driven to an incremental software development approach with agile practices

So far, only few in-depth studies focused on the direct comparison of process models in general, and between plan-driven and incremental/agile approaches in particular. That is, it is not made explicit what the effect is of moving from one model to another model. Furthermore, there is limited evidence on advantages and issues encountered in agile software development, this is specifically true in the context of large-scale development. The objective of the paper is to investigate how the perception of bottlenecks, unnecessary work, and rework (from hereon referred to as issues) changes when migrating from a plan-driven to an incremental software development approach with agile practices (flexible product backlog, face-to-face interaction, and frequent integration), and how commonly perceived these practices are across different systems and development roles. The context in which the objective should be achieved is large-scale development with a market-driven focus. The selection of the context was based on the observation in related work that mostly small software development projects were investigated and that the investigation was focused on one agile model (eXtreme programming). A case study was conducted at a development site of Ericsson AB, located in Sweden in the end of 2007. In total 33 interviews were conducted in order to investigate the perceived change when migrating from plan-driven to incremental and agile software development, the interviews being the primary source of evidence. For triangulation purposes measurements collected by Ericsson were considered, the measurements relating to unnecessary work (amount of discarded requirements) and rework (data on testing efficiency and maintenance effort). Triangulation in this context means that the measurements were used to confirm the perceived changes with an additional data source. In total 64 issues were identified, 24 being of general nature and the remaining 40 being local and therefore unique to individual’s opinions or a specific system. The most common ones were documented and analyzed in detail. The commonality refers to how many persons in different roles and across the systems studied have mentioned the issues for each of the process models. The majority of the most common issues relates to plan-driven development. We also identified common issues remaining for agile after the migration, which were related to testing lead-time, test coverage, software release, and coordination overhead. Improvements were identified as many issues commonly raised for the plan-driven approach were not raised anymore for the incremental and agile approach. It is concluded that the recent introduction (start in 2005 with the study being conducted in the end of 2007) of incremental and agile practices brings added values in comparison to the plan-driven approach, which is evident from the absence of critical issues that are encountered in plan-driven development.

Measuring and predicting software productivity

ContextSoftware productivity measurement is essential in order to control and improve the performance of software development. For example, by identifying role models (e.g. projects, individuals, tasks) when comparing productivity data. The prediction is of relevance to determine whether corrective actions are needed, and to discover which alternative improvement action would yield the best results. ObjectiveIn this study we identify studies for software productivity prediction and measurement. Based on the identified studies we first create a classification scheme and map the studies into the scheme (systematic map). Thereafter, a detailed analysis and synthesis of the studies is conducted. MethodAs a research method for systematically identifying and aggregating the evidence of productivity measurement and prediction approaches systematic mapping and systematic review have been used. ResultsIn total 38 studies have been identified, resulting in a classification scheme for empirical research on software productivity. The mapping allowed to identify the rigor of the evidence with respect to the different productivity approaches. In the detailed analysis the results were tabulated and synthesized to provide recommendations to practitioners. ConclusionRisks with simple ratio-based measurement approaches were shown. In response to the problems data envelopment analysis seems to be a strong approach to capture multivariate productivity measures, and allows to identify reference projects to which inefficient projects should be compared. Regarding simulation no general prediction model can be identified. Simulation and statistical process control are promising methods for software productivity prediction. Overall, further evidence is needed to make stronger claims and recommendations. In particular, the discussion of validity threats should become standard, and models need to be compared with each other.

The Effect of Moving from a Plan-Driven to an Incremental and Agile Development Approach : An Industrial Case Study

So far, only few in-depth studies focused on the direct comparison of process models in general, and between plan-driven and incremental/agile approaches in particular. That is, it is not made explicit what the effect is of moving from one model to another model. Furthermore, there is limited evidence on advantages and issues encountered in agile software development, this is specifically true in the context of large-scale development. The objective of the paper is to investigate how the perception of bottlenecks, unnecessary work, and rework (from hereon referred to as issues) changes when migrating from a plan-driven to an incremental software development approach with agile practices (flexible product backlog, face-to-face interaction, and frequent integration), and how commonly perceived these practices are across different systems and development roles. The context in which the objective should be achieved is large-scale development with a market-driven focus. The selection of the context was based on the observation in related work that mostly small software development projects were investigated and that the investigation was focused on one agile model (eXtreme programming). A case study was conducted at a development site of Ericsson AB, located in Sweden in the end of 2007. In total 33 interviews were conducted in order to investigate the perceived change when migrating from plan-driven to incremental and agile software development, the interviews being the primary source of evidence. For triangulation purposes measurements collected by Ericsson were considered, the measurements relating to unnecessary work (amount of discarded requirements) and rework (data on testing efficiency and maintenance effort). Triangulation in this context means that the measurements were used to confirm the perceived changes with an additional data source. In total 64 issues were identified, 24 being of general nature and the remaining 40 being local and therefore unique to individual’s opinions or a specific system. The most common ones were documented and analyzed in detail. The commonality refers to how many persons in different roles and across the systems studied have mentioned the issues for each of the process models. The majority of the most common issues relates to plan-driven development. We also identified common issues remaining for agile after the migration, which were related to testing lead-time, test coverage, software release, and coordination overhead. Improvements were identified as many issues commonly raised for the plan-driven approach were not raised anymore for the incremental and agile approach. It is concluded that the recent introduction (start in 2005 with the study being conducted in the end of 2007) of incremental and agile practices brings added values in comparison to the plan-driven approach, which is evident from the absence of critical issues that are encountered in plan-driven development.

Software process improvement through the Lean Measurement (SPI-LEAM) method

Software process improvement methods help to continuously refine and adjust the software process to improve its performance (e.g., in terms of lead-time, quality of the software product, reduction of change requests, and so forth). Lean software development propagates two important principles that help process improvement, namely identification of waste in the process and considering interactions between the individual parts of the software process from an end-to-end perspective. A large shift of thinking about the own way of working is often required to adopt lean. One of the potential main sources of failure is to try to make a too large shift about the ways of working at once. Therefore, the change to lean has to be done in a continuous and incremental way. In response to this we propose a novel approach to bring together the quality improvement paradigm and lean software development practices, the approach being called Software Process Improvement through the Lean Measurement (SPI-LEAM) Method. The method allows to assess the performance of the development process and take continuous actions to arrive at a more lean software process over time. The method is under implementation in industry and an initial evaluation of the method has been performed.

Benefits and limitations of automated software testing: systematic literature review and practitioner survey

There is a documented gap between academic and practitioner views on software testing. This paper tries to close the gap by investigating both views regarding the benefits and limits of test automation. The academic views are studied with a systematic literature review while the practitioners views are assessed with a survey, where we received responses from 115 software professionals. The results of the systematic literature review show that the source of evidence regarding benefits and limitations is quite shallow as only 25 papers provide the evidence. Furthermore, it was found that benefits often originated from stronger sources of evidence (experiments and case studies), while limitations often originated from experience reports. We believe that this is caused by publication bias of positive results. The survey showed that benefits of test automation were related to test reusability, repeatability, test coverage and effort saved in test executions. The limitations were high initial invests in automation setup, tool selection and training. Additionally, 45% of the respondents agreed that available tools in the market offer a poor fit for their needs. Finally, it was found that 80% of the practitioners disagreed with the vision that automated testing would fully replace manual testing.

Waste and Lead Time Reduction in a Software Product Customization Process with Value Stream Maps

Custom-developed adaptations of software products are increasingly important to meet different and changing customer needs and heterogeneous system environments. Efficient software customization processes with short lead times are thus a priority for companies to stay competitive. The purpose of this case study is to identify waste-related problems in a software product customization process by using value stream maps (VSM). The study was conducted at the telecom company Ericsson AB; the empirical data was collected using document analysis, extraction of phase times from a requirements tracking tool and interviews. It was used to construct a value stream map that showed the present state of the process. The map was then used in interviews with key stakeholders where they identified waste and proposed measures to avoid them. These solution proposals were subsequently used to construct a future value stream map showing a reduced lead time. Static validation showed that the VSM methodology was useful and that the identified solutions were of value. Our results have applicability for software engineering process improvement in general and for customization processes in particular.

Considering rigor and relevance when evaluating test driven development: A systematic review

Context: Test driven development (TDD) has been extensively researched and compared to traditional approaches (test last development, TLD). Existing literature reviews show varying results for TDD. Objective: This study investigates how the conclusions of existing literature reviews change when taking two study quality dimension into account, namely rigor and relevance. Method: In this study a systematic literature review has been conducted and the results of the identified primary studies have been analyzed with respect to rigor and relevance scores using the assessment rubric proposed by Ivarsson and Gorschek 2011. Rigor and relevance are rated on a scale, which is explained in this paper. Four categories of studies were defined based on high/low rigor and relevance. Results: We found that studies in the four categories come to different conclusions. In particular, studies with a high rigor and relevance scores show clear results for improvement in external quality, which seem to come with a loss of productivity. At the same time high rigor and relevance studies only investigate a small set of variables. Other categories contain many studies showing no difference, hence biasing the results negatively for the overall set of primary studies. Given the classification differences to previous literature reviews could be highlighted. Conclusion: Strong indications are obtained that external quality is positively influenced, which has to be further substantiated by industry experiments and longitudinal case studies. Future studies in the high rigor and relevance category would contribute largely by focusing on a wider set of outcome variables (e.g. internal code quality). We also conclude that considering rigor and relevance in TDD evaluation is important given the differences in results between categories and in comparison to previous reviews.