Kjetil Moløkken-Østvold

A comparison of software project overruns - flexible versus sequential development models

Flexible software development models, e.g., evolutionary and incremental models, have become increasingly popular. Advocates claim that among the benefits of using these models is reduced overruns, which is one of the main challenges of software project management. This paper describes an in-depth survey of software development projects. The results support the claim that projects which employ a flexible development model experience less effort overruns than do those which employ a sequential model. The reason for the difference is not obvious. We found, for example, no variation in project size, estimation process, or delivered proportion of planned functionality between projects applying different types of development model. When the managers were asked to provide reasons for software overruns and/or estimation accuracy, the largest difference was that more of flexible projects than sequential projects cited good requirement specifications-and good collaboration/communication with clients as contributing to accurate estimates.

Using planning poker for combining expert estimates in software projects

When producing estimates in software projects, expert opinions are frequently combined. However, it is poorly understood whether, when, and how to combine expert estimates. In order to study the effects of a combination technique called planning poker, the technique was introduced in a software project for half of the tasks. The tasks estimated with planning poker provided: (1) group consensus estimates that were less optimistic than the statistical combination (mean) of individual estimates for the same tasks, and (2) group consensus estimates that were more accurate than the statistical combination of individual estimates for the same tasks. For tasks in the same project, individual experts who estimated a set of control tasks achieved estimation accuracy similar to that achieved by estimators who estimated tasks using planning poker. Moreover, for both planning poker and the control group, measures of the median estimation bias indicated that both groups had unbiased estimates, because the typical estimated task was perfectly on target. A code analysis revealed that for tasks estimated with planning poker, more effort was expended due to the complexity of the changes to be made, possibly caused by the information provided in group discussions.

Group Processes in Software Effort Estimation

The effort required to complete software projects is often estimated, completely or partially, using the judgment of experts, whose assessment may be biased. In general, such bias as there is seems to be towards estimates that are overly optimistic. The degree of bias varies from expert to expert, and seems to depend on both conscious and unconscious processes. One possible approach to reduce this bias towards over-optimism is to combine the judgments of several experts. This paper describes an experiment in which experts with different backgrounds combined their estimates in group discussion. First, 20 software professionals were asked to provide individual estimates of the effort required for a software development project. Subsequently, they formed five estimation groups, each consisting of four experts. Each of these groups agreed on a project effort estimate via the pooling of knowledge in discussion. We found that the groups submitted less optimistic estimates than the individuals. Interestingly, the group discussion-based estimates were closer to the effort expended on the actual project than the average of the individual expert estimates were, i.e., the group discussions led to better estimates than a mechanical averaging of the individual estimates. The groups’ ability to identify a greater number of the activities required by the project is among the possible explanations for this reduction of bias.

The Relationship between Customer Collaboration and Software Project Overruns

Most agile projects rely heavily on good collaboration with the customer in order to achieve project goals and avoid overruns. However, the role of the customer in software projects is not fully understood. Often, successful projects are attributed to developer competence, while unsuccessful projects are attributed to customer incompetence. A study was conducted on eighteen of the latest projects of a software contractor. Quantitative project data was collected, and project managers interviewed, on several issues related to estimates, key project properties, and project outcome. It was found that in projects where collaboration was facilitated by daily communication between the contractor and the customer, they experienced a lesser magnitude of effort overruns. In addition, employing a contract that facilitates risk-sharing may also have a positive impact.

Combining Estimates with Planning Poker--An Empirical Study

Combination of expert opinion is frequently used to produce estimates in software projects. However, if, when and how to combine expert estimates, is poorly understood. In order to study the effects of a combination technique called planning poker, the technique was introduced in a software project for half of the tasks. The tasks estimated with planning poker provided: 1) group consensus estimates that were less optimistic than the mechanical combination of individual estimates for the same tasks, and 2) group consensus estimates that were more accurate than the mechanical combination of individual estimates for the same tasks. The set of control tasks in the same project, estimated by individual experts, achieved similar estimation accuracy as the planning poker tasks. However, for both planning poker and the control group, measures of the median estimation bias indicated that both groups had unbiased estimates, as the typical estimated task was perfectly on target.

Increasing Software Effort Estimation Accuracy Using Experience Data, Estimation Models and Checklists

It is frequently suggested that using experience data, estimation models and checklists can increase software effort estimation accuracy. However, there has been limited empirical research on the subject. We conducted a study of eighteen of the latest projects of a software contractor. Quantitative and qualitative data was collected on several issues related to estimates, key project properties, and project outcome. It was found that in projects where experience data was utilized in the estimation process, they experienced a lesser magnitude of effort overruns. The use of a checklist also appeared to increase estimation accuracy. However, the utilization of an estimation model did not appear to have a substantial impact.

Ethical Concerns when Increasing Realism in Controlled Experiments with Industrial Participants

The emerging interest in realistic controlled experiments in computer science has created a need to examine related research ethics. Increased realism and scale in experimental studies pose new challenges that have not been debated to a sufficient extent. Specifically, there can be conflicts between the ethical principles of scientific value and informed consent. This paper provides an account of related previous work in computer science research ethics. To illustrate, two large-scale software engineering experiments with industrial participants are described. Challenges and solutions in these experiments are discussed in the light of current ethical guidelines. Interviews and debriefing sessions with industrial participants from these, and other, experiments are also provided. These reveal that there will not necessarily be ethical problems with increased realism, provided that the researchers respect the principles of informed consent, benefice and confidentiality.