Stein Grimstad

A framework for the analysis of software cost estimation accuracy

Many software companies track and analyze project performance by measuring cost estimation accuracy. A high estimation error is frequently interpreted as poor estimation skills. This is not necessarily a correct interpretation. High estimation error can also be a result of other factors, such as high estimation complexity and insufficient cost control of the project. Through a real-life example we illustrate how the lack of proper estimation error analysis technique can bias analyses of cost estimation accuracy and lead to wrong conclusions. Further, we examine a selection of cost estimation studies, and show that they frequently do not take the necessary actions to ensure meaningful interpretations of estimation error data. Motivated by these results, we propose a general framework that, we believe, will improve analyses of software cost estimation error.

The Impact of Irrelevant and Misleading Information on Software Development Effort Estimates: A Randomized Controlled Field Experiment

Studies in laboratory settings report that software development effort estimates can be strongly affected by effort-irrelevant and misleading information. To increase our knowledge about the importance of these effects in field settings, we paid 46 outsourcing companies from various countries to estimate the required effort of the same five software development projects. The companies were allocated randomly to either the original requirement specification or a manipulated version of the original requirement specification. The manipulations were as follows: 1) reduced length of requirement specification with no change of content, 2) information about the low effort spent on the development of the old system to be replaced, 3) information about the clients unrealistic expectations about low cost, and 4) a restriction of a short development period with start up a few months ahead. We found that the effect sizes in the field settings were much smaller than those found for similar manipulations in laboratory settings. Our findings suggest that we should be careful about generalizing to field settings the effect sizes found in laboratory settings. While laboratory settings can be useful to demonstrate the existence of an effect and better understand it, field studies may be needed to study the size and importance of these effects.

Software Development Estimation Biases: The Role of Interdependence

Software development effort estimates are frequently too low, which may lead to poor project plans and project failures. One reason for this bias seems to be that the effort estimates produced by software developers are affected by information that has no relevance for the actual use of effort. We attempted to acquire a better understanding of the underlying mechanisms and the robustness of this type of estimation bias. For this purpose, we hired 374 software developers working in outsourcing companies to participate in a set of three experiments. The experiments examined the connection between estimation bias and developer dimensions: self-construal (how one sees oneself), thinking style, nationality, experience, skill, education, sex, and organizational role. We found that estimation bias was present along most of the studied dimensions. The most interesting finding may be that the estimation bias increased significantly with higher levels of interdependence, i.e., with stronger emphasis on connectedness, social context, and relationships. We propose that this connection may be enabled by an activation of ones self-construal when engaging in effort estimation, and a connection between a more interdependent self-construal and increased search for indirect messages, lower ability to ignore irrelevant context, and a stronger emphasis on socially desirable responses.

Software Development Effort Estimation — Demystifying and Improving Expert Estimation

The main determinant of many types of software-related investments is the amount of development effort required. The ability of software clients to make investment decisions based on cost estimates is consequently strongly tied to the software providers’ ability to estimate the effort accurately. Similarly, the ability of project managers to plan a project and ensure efficient development frequently depends on accurate effort estimates. The importance of accurate effort estimates is illustrated by the findings of a 2007 survey of more than 1,000 IT professionals. The survey reports that two out of the three-most-important causes of IT project failure were related to poor resource estimation, that is, inaccurate effort estimates.

The Impact of Irrelevant Information on Estimates of Software Development Effort

Software professionals typically estimate software development effort based on a requirement specification. Parts of this specification frequently contain information that is irrelevant to the estimation of the actual effort involved in the development of software. We hypothesize that effort-irrelevant information sometimes has a strong impact on effort estimates. To test this hypothesis, we conducted two controlled experiments with software professionals. In each of the experiments, the software professionals received specifications describing the same requirements. However, we gave one group of the software professionals a version of the requirement specification where we had included additional, effort-irrelevant, information. In both experiments we observed that the estimates of most likely effort increased when the estimates were based on requirement specifications that contained the information irrelevant to development effort. The results suggest that when estimation-irrelevant information is included as input to expert judgment-based estimation processes, the estimators find it difficult to distinguish between the estimation-relevant and the estimation-irrelevant information. A possible consequence of our findings is that estimation-irrelevant information should be removed from the requirement specification prior to the use of it as input to estimation work.

A preliminary study of sequence effects in judgment-based software development work-effort estimation

Software development effort estimates are often inaccurate. This study investigates to what degree and why the sequence in which we estimate software work affect the effort estimates. The results may be used to improve judgment-based software development effort estimation processes. Two controlled experiments were conducted. In the first experiment, software professionals were randomly allocated to the groups SMALL and LARGE. First, those in group SMALL estimated the most likely effort required to complete a small software development task and those in group LARGE the effort of a larger task. Then both groups estimated the effort of the same medium-sized task. The first estimate had a large impact on the subsequent. The second experiment aimed at a better understanding of the nature of sequence effects in effort estimation. This experiment suggests that it is the experience and knowledge activated in the previous task that matter, not the estimated value itself. In conclusion, more awareness of the importance of the estimation sequence may lead to more realistic effort estimates. In particular, it may be useful to avoid estimation of simple and small tasks just before larger and more complex tasks in situations where over-optimism is frequent.

Understanding of estimation accuracy in software development projects

Over the past decades large investments in software engineering research and development have been made by academia and the software industry. These efforts have produced considerably insight in the complex domain of software development, and have paid off in the shape of the improved tools, languages, methodologies and techniques. However, a recent review of estimation survey documents that less progress has been made in the area of estimation performance. This is a major concern for the software industry, as lack of estimation performance often causes budget overruns, delays, lost contracts or poor quality software. Because of these rather dramatic consequences, there is a high demand for more research on the topic of effort estimation in software development projects. That demand motivated this PhD. The thesis is written at the Estimation Group at Simula Research Laboratory in Oslo, Norway. The work is supervised by professor Magne Jorgensen, and is part of the SPIKE (Software Process Improvement based on Knowledge and Experience) project which is funded by the Norwegian Research Council