Wilhelm Meding

A framework for developing measurement systems and its industrial evaluation

As in every engineering discipline, metrics play an important role in software development, with the difference that almost all software projects need the customization of metrics used. In other engineering disciplines, the notion of a measurement system (i.e. a tool used to collect, calculate, and report quantitative data) is well known and defined, whereas it is not as widely used in software engineering. In this paper we present a framework for developing custom measurement systems and its industrial evaluation in a software development unit within Ericsson. The results include the framework for designing measurement systems and its evaluation in real life projects at the company. The results show that with the help of ISO/IEC standards, measurement systems can be effectively used in software industry and that the presented framework improves the way of working with metrics. This paper contributes with the presentation of how automation of metrics collection and processing can be successfully introduced into a large organization and shows the benefits of it: increased efficiency of metrics collection, increased adoption of metrics in the organization, independence from individuals and standardized nomenclature for metrics in the organization.

Developing measurement systems: an industrial case study

The process of measuring in software engineering has already been standardized in the ISO/IEC 15939 standard, where activities related to identifying, creating, and evaluating of measures are described. In the process of measuring software entities, however, an organization usually needs to create custom measurement systems, which are intended to collect, analyze, and present data for a specific purpose. In this paper, we present a proven industrial process for developing measurement systems including the artifacts and deliverables important for a successful deployment of measurement systems in industry. The process has been elicited during a case study at Ericsson and is used in the organization for over 3 years when the paper was written. The process is supported by a framework that simplifies the implementation of the measurement systems and shortens the time from the initial idea to a working measurement system by the factor of 5 compared with using a standard development process not tailored for measurement systems. Copyright

Monitoring bottlenecks in agile and lean software development projects - a method and its industrial use

In the face of growing competition software projects have to deliver software products faster and with better quality - thus leaving little room for unnecessary activities or non-optimal capacity. To achieve the desired high speed of the projects and the optimal capacity,bottlenecks existing in the projects have to be monitored and effectively removed. The objective of this research is to show experiences from a mature software development organization working according to Lean and Agile software development principles. By conducting a formal case study at Ericsson we were able to elicit and automate measures required to monitor bottlenecks in software development workflow, evaluated in one of the projects. The project developed software for one of the telecom products and consisted of over 80 developers. The results of the case study include a measurement system with a number of measures/indicators which can indicate existence of bottlenecks in the flow of work in the project and a number of good practices helping other organizations to start monitoring bottlenecks in an effective way - in particular what to focus on when designing such a measurement system.

Release Readiness Indicator for Mature Agile and Lean Software Development Projects

Large companies like Ericsson increasingly often adopt the principles of Agile and Lean software development and develop large software products in iterative manner – in order to quickly respond to customer needs. In this paper we present the main indicator which is sufficient for a mature software development organization in order to predict the time in weeks to release the product. In our research project we collaborated closely with a large Agile+Lean software development project at Ericsson in Sweden. This large and mature software development project and organization has found this main indicator – release readiness – to be so important that it was used as a key performance indicator and is used in controlling the development of the product and improving organizational performance. The indicator was developed and validated in an action research project at one of the units of Ericsson AB in Sweden in one of its largest projects.

Predicting weekly defect inflow in large software projects based on project planning and test status

Defects discovered during the testing phase in software projects need to be removed before the software is shipped to the customers. The removal of defects can constitute a significant amount of effort in a project and project managers are faced with a decision whether to continue development or shift some resources to cope with defect removal. The goal of this research is to improve the practice of project management by providing a method for predicting the number of defects reported into the defect database in the project. In this paper we present a method for predicting the number of defects reported into the defect database in a large software project on a weekly basis. The method is based on using project progress data, in particular the information about the test progress, to predict defect inflow in the next three coming weeks. The results show that the prediction accuracy of our models is up to 72% (mean magnitude of relative error for predictions of 1 week in advance is 28%) when used in ongoing large software projects. The method is intended to support project managers in more accurate adjusting resources in the project, since they are notified in advance about the potentially large effort needed to correct defects.

A method for forecasting defect backlog in large streamline software development projects and its industrial evaluation

Context: Predicting a number of defects to be resolved in large software projects (defect backlog) usually requires complex statistical methods and thus is hard to use on a daily basis by practitioners in industry. Making predictions in simpler and more robust way is often required by practitioners in software engineering industry. Objective: The objective of this paper is to present a simple and reliable method for forecasting the level of defect backlog in large, lean-based software development projects. Method: The new method was created as part of an action research project conducted at Ericsson. In order to create the method we have evaluated multivariate linear regression, expert estimations and analogy-based predictions w.r.t. their accuracy and ease-of-use in industry. We have also evaluated the new method in a life project at one of the units of Ericsson during a period of 21weeks (from the beginning of the project until the release of the product). Results: The method for forecasting the level of defect backlog uses an indicator of the trend (an arrow) as a basis to forecast the level of defect backlog. Forecasts are based on moving average which combined with the current level of defect backlog was found to be the best prediction method (Mean Magnitude of Relative Error of 16%) for the level of future defect backlog. Conclusion: We have found that ease-of-use and accuracy are the main aspects for practitioners who use predictions in their work. In this paper it is concluded that using the simple moving average provides a sufficiently-good accuracy (much appreciated by practitioners involved in the study). We also conclude that using the indicator (forecasting the trend) instead of the absolute number of defects in the backlog increases the confidence in our method compared to our previous attempts (regression, analogy-based, and expert estimates).

Selecting software reliability growth models and improving their predictive accuracy using historical projects data

8 software reliability growth models are evaluated on 11 large projects.Logistic and Gompertz models have the best fit and asymptote predictions.Using growth rate from earlier projects improves asymptote prediction accuracy.Trend analysis allows choosing the best shape of the model at 50% of project time. During software development two important decisions organizations have to make are: how to allocate testing resources optimally and when the software is ready for release. SRGMs (software reliability growth models) provide empirical basis for evaluating and predicting reliability of software systems. When using SRGMs for the purpose of optimizing testing resource allocation, the models ability to accurately predict the expected defect inflow profile is useful. For assessing release readiness, the asymptote accuracy is the most important attribute. Although more than hundred models for software reliability have been proposed and evaluated over time, there exists no clear guide on which models should be used for a given software development process or for a given industrial domain.Using defect inflow profiles from large software projects from Ericsson, Volvo Car Corporation and Saab, we evaluate commonly used SRGMs for their ability to provide empirical basis for making these decisions. We also demonstrate that using defect intensity growth rate from earlier projects increases the accuracy of the predictions. Our results show that Logistic and Gompertz models are the most accurate models; we further observe that classifying a given project based on its expected shape of defect inflow help to select the most appropriate model.

Using Models to Develop Measurement Systems: A Method and Its Industrial Use

Making the measurement processes work in large software development organizations requires collecting right metrics and collecting them automatically. Collecting the right metrics requires development custom measurement systems which fulfill the actual needs of the company. Effective communication between stakeholders (persons who have the information needs) and the designers of measurement systems are cornerstones in identifying the right metrics and the right amount of them. In this paper we describe a method for developing measurement systems based on models which make this communication more effective. The method supports the designers of measurement systems and managers, for whom the measurement systems are created, in developing more effective measurement systems based on MS Excel. The method comprises of platform independent modeling, platform specific modeling and automated code generation. This method has been used in one of action research projects at Ericsson. We present the results of the evaluation of this method at Ericsson by the end of this paper.

Measuring and Visualizing Code Stability -- A Case Study at Three Companies

Monitoring performance of software development organizations can be achieved from a number of perspectives - e.g. using such tools as Balanced Scorecards or corporate dashboards. In this paper we present results from a study on using code stability indicators as a tool for product stability and organizational performance, conducted at three different software development companies - Ericsson AB, Saab AB Electronic Defense Systems (Saab) and Volvo Group Trucks Technology (Volvo Group). The results show that visualizing the source code changes using heat maps and linking these visualizations to defect inflow profiles provide indicators of how stable the product under development is and whether quality assurance efforts should be directed to specific parts of the product. Observing the indicator and making decisions based on its visualization leads to shorter feedback loops between development and test, thus resulting in lower development costs, shorter lead time and increased quality. The industrial case study in the paper shows that the indicator and its visualization can show whether the modifications of software products are focused on parts of the code base or are spread widely throughout the product.

Identifying Implicit Architectural Dependencies Using Measures of Source Code Change Waves

The principles of Agile software development are increasingly used in large software development projects, e.g. using Scrum of Scrums or combining Agile and Lean development methods. When large software products are developed by self-organized, usually feature-oriented teams, there is a risk that architectural dependencies between software components become uncontrolled. In particular there is a risk that the prescriptive architecture models in form of diagrams are outdated and implicit architectural dependencies may become more frequent than the explicit ones. In this paper we present a method for automated discovery of potential dependencies between software components based on analyzing revision history of software repositories. The result of this method is a map of implicit dependencies which is used by architects in decisions on the evolution of the architecture. The software architects can assess the validity of the dependencies and can prevent unwanted component couplings and design erosion hence minimizing the risk of post-release quality problems. Our method was evaluated in a case study at one large product at Saab Electronic Defense Systems (Saab EDS) and one large software product at Ericsson AB.