Dietmar Pfahl

Reporting guidelines for controlled experiments in software engineering

One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) it is difficult to locate relevant information and (2) important information is often missing. Reporting guidelines are expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results. The objective of this paper is to survey the most prominent published proposals for reporting guidelines, and to derive a unified standard that which can serve as a starting point for further discussion. We provide detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical studies, i.e., controlled experiments. Before the guidelines can be evaluated, feedback from the research community is required. For this purpose, we propose to adapt guideline development processes from other disciplines.

Reporting experiments in software engineering

Background: One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) It is difficult to locate relevant information and (2) important information is often missing. Objective: A checklist for reporting results from controlled experiments is expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results. Method: The checklist for reporting is based on (1) a survey of the most prominent published proposals for reporting guidelines in software engineering and (2) an iterative development incorporating feedback from members of the research community. Result: This paper presents a unification of a set of guidelines for reporting experiments in software engineering. Limitation: The checklist has not been evaluated broadly, yet. Conclusion: The resulting checklist provides detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical studies, i.e., experiments (controlled experiments and quasi-experiments).

How do scientists develop and use scientific software

New knowledge in science and engineering relies increasingly on results produced by scientific software. Therefore, knowing how scientists develop and use software in their research is critical to assessing the necessity for improving current development practices and to making decisions about the future allocation of resources. To that end, this paper presents the results of a survey conducted online in October-December 2008 which received almost 2000 responses. Our main conclusions are that (1) the knowledge required to develop and use scientific software is primarily acquired from peers and through self-study, rather than from formal education and training; (2) the number of scientists using supercomputers is small compared to the number using desktop or intermediate computers; (3) most scientists rely primarily on software with a large user base; (4) while many scientists believe that software testing is important, a smaller number believe they have sufficient understanding about testing concepts; and (5) that there is a tendency for scientists to rank standard software engineering concepts higher if they work in large software development projects and teams, but that there is no uniform trend of association between rank of importance of software engineering concepts and project/team size.

Trade-off analysis for requirements selection

Evaluation, prioritization and selection of candidate requirements are of tremendous importance and impact for subsequent software development. Eort, time as well as quality constraints have to be taken into account. Typically, dieren t stakeholders have conicting priorities and the requirements of all these stakeholders have to be balanced in an appropriate way to ensure maximum value of the nal set of requirements. Tradeo analysis is needed to proactively explore the impact of certain decisions in terms of all the criteria and constraints. The proposed method called Quantitative WinWin uses an evolutionary approach to provide support for requirements negotiations. The novelty of the presented idea is four-fold. Firstly, it iteratively uses the Analytical Hierarchy Process (AHP) for a stepwise analysis with the aim to balance the stakeholders’ preferences related to dieren t classes of requirements. Secondly, requirements selection is based on predicting and rebalancing its impact on eort, time and quality. Both prediction and rebalancing uses the simulation model prototype GENSIM. Thirdly, alternative solution sets oered for decision-making are developed incrementally based on thresholds for the degree of importance of requirements and heuristics to nd a best t to constraints. Finally, trade-o analysis is used to determine non-dominated extensions of the maximum value that is achievable under resource and quality constraints. As a main result, quantitative WinWin proposes a small number of possible sets of requirements from which the actual decision-maker can nally select the most appropriate solution.

Using simulation to analyse the impact of software requirement volatility on project performance

During the last decade, software process simulation has been used to address a variety of management issues and questions. These include: understanding; training and learning; planning; control and operational management; strategic management; process improvement and technology adoption. This paper presents a simulation model that was developed to demonstrate the impact of unstable software requirements on project performance, and to analyse how much money should be invested in stabilising software requirements in order to achieve optimal cost effectiveness. The paper reports on all steps of model building, describes the structure of the final simulation model, and presents the most interesting simulation results of an industrial application

Reflections on 10 years of software process simulation modeling: a systematic review

Software process simulation modeling (SPSM) has become an increasinglyactive research area since its introduction in the late 1980s. Particularlyduring the last ten years the related research community and the number ofpublications have been growing. The objective of this research is to provide insightsabout the evolution of SPSM research during the last 10 years. A systematicliterature review was proposed with two subsequent stages to achieve thisgoal. This paper presents the preliminary results of the first stage of the reviewthat is exclusively focusing on a core set of publication sources. More than 200relevant publications were analyzed in order to find answers to the researchquestions, including the purposes and scopes of SPSM, application domains,and predominant research issues. From the analysis the following conclusionscould be drawn: (1) Categories for classifying software process simulationmodels as suggested by the authors of a landmark publication in 1999 should beadjusted and refined to better capture the diversity of published models. (2) Researchimproving the efficiency of SPSM is gaining importance. (3) Hybridprocess simulation models have attracted interest as a possibility to more realisticallycapture complex real-world software processes.

A CBT module with integrated simulation component for software project management education and training

Abstract Due to increasing demand for software project managers in industry, efforts are needed to develop the management-related knowledge and skills of the current and future software workforce. In particular, university education needs to provide to their computer science and software engineering (SE) students not only technology-related skills but, in addition, a basic understanding of typical phenomena occurring in industrial (and academic) software projects. The objective of this paper is to present concepts of a computer-based training (CBT) module for student education in software project management. The single-learner CBT module can be run using standard web-browsers (e.g. Netscape). The simulation component of the CBT module is implemented using the system dynamics (SD) simulation modelling method. The paper presents the design of the simulation model and the training scenario offered by the existing CBT module prototype. Possibilities for empirical validation of the effectiveness of the CBT module in university education are described, results of a first controlled experiment are presented and discussed, and future extensions of the CBT module towards collaborative learning environments are suggested.

Evaluating the learning effectiveness of using simulations in software project management education: results from a twice replicated experiment

Abstract The increasing demand for software project managers in industry requires strategies for the development of management-related knowledge and skills of the current and future software workforce. Although several educational approaches help to develop the necessary skills in a university setting, few empirical studies are currently available to characterise and compare their effects. This paper presents the results of a twice replicated experiment that evaluates the learning effectiveness of using a process simulation model for educating computer science students in software project management. While the experimental group applied a System Dynamics simulation model, the control group used the well-known COCOMO model as a predictive tool for project planning. The results of each empirical study indicate that students using the simulation model gain a better understanding about typical behaviour patterns of software development projects. The combination of the results from the initial experiment and the two replications with meta-analysis techniques corroborates this finding. Additional analysis shows that the observed effect can mainly be attributed to the use of the simulation model in combination with a web-based role-play scenario. This finding is strongly supported by information gathered from the debriefing questionnaires of subjects in the experimental group. They consistently rated the simulation-based role-play scenario as a very useful approach for learning about issues in software project management.

Software process dynamics and agility

Process Content.- Extending Microsoft Team Foundation Server Architecture to Support Collaborative Product Patterns.- The REMIS Approach for Rationale-Driven Process Model Evolution.- On the Measurement of Agility in Software Process.- Coping with the Cone of Uncertainty: An Empirical Study of the SAIV Process Model.- Effects of Architecture and Technical Development Process on Micro-process.- Process Tools and Metrics.- Comparative Experiences with Electronic Process Guide Generator Tools.- Jasmine: A PSP Supporting Tool.- A Tool to Create Process-Agents for OEC-SPM from Historical Project Data.- Process Management.- Safety Critical Software Process Improvement by Multi-objective Optimization Algorithms.- Representing Process Variation with a Process Family.- An Algebraic Approach for Managing Inconsistencies in Software Processes,.- Process Representation, Analysis and Modeling.- Cost Estimation and Analysis for Government Contract Pricing in China.- A Multilateral Negotiation Method for Software Process Modeling.- Distributed Global Development Parametric Cost Modeling.- Process Mining Framework for Software Processes.- Focused Identification of Process Model Changes.- An Approach for Decentralized Process Modeling.- Experience Report.- A Survey of Software Development with Open Source Components in Chinese Software Industry.- Empirical Study on Benchmarking Software Development Tasks.- An Empirical Study on Establishing Quantitative Management Model for Testing Process.- Simulation Modeling.- DynaReP: A Discrete Event Simulation Model for Re-planning of Software Releases.- The Economic Impact of Software Process Variations.- Deriving a Valid Process Simulation from Real World Experiences.- Project Delay Variability Simulation in Software Product Line Development.- Modeling Risk-Benefit Assumptions in Technology Substitution.- Evaluating the Impact of the QuARS Requirements Analysis Tool Using Simulation.- A Framework for Adopting Software Process Simulation in CMMI Organizations.- Achieving Software Project Success: A Semi-quantitative Approach.

Software process change

Keynotes.- A Value-Based Software Process Framework.- Exploring the Business Process-Software Process Relationship.- Assessing 3-D Integrated Software Development Processes: A New Benchmark.- Ubiquitous Process Engineering: Applying Software Process Technology to Other Domains.- Process Tailoring and Decision-Support.- Dependencies Between Data Decisions.- Tailor the Value-Based Software Quality Achievement Process to Project Business Cases.- Optimizing Process Decision in COTS-Based Development Via Risk Based Prioritization.- Process Tools and Metrics.- Project Replayer - An Investigation Tool to Revisit Processes of Past Projects.- Software Process Measurement in the Real World: Dealing with Operating Constraints.- Evaluation of Project Quality: A DEA-Based Approach.- Process Management.- A Pattern-Based Solution to Bridge the Gap Between Theory and Practice in Using Process Models.- On Mobility of Software Processes.- Software Process Fusion: Uniting Pair Programming and Solo Programming Processes.- Towards an Approach for Security Risk Analysis in COTS Based Development.- COCOMO-U: An Extension of COCOMO II for Cost Estimation with Uncertainty.- A Product Line Enhanced Unified Process.- Process Representation, Analysis and Modeling.- Automatic Fault Tree Derivation from Little-JIL Process Definitions.- Workflows and Cooperative Processes.- Spiral Lifecycle Increment Modeling for New Hybrid Processes.- Definition and Analysis of Election Processes.- The Design of a Flexible Software Process Language.- Building Business Process Description and Reasoning Meta-model M bp in A-Prolog.- A Process-Agent Construction Method for Software Process Modeling in SoftPM.- Applying Little-JIL to Describe Process-Agent Knowledge in SoftPM.- Process Simulation Modeling.- Reusable Model Structures and Behaviors for Software Processes.- Organization-Theoretic Perspective for Simulation Modeling of Agile Software Processes.- Semi-quantitative Simulation Modeling of Software Engineering Process.- Process Simulation Applications.- Analysis of Software-Intensive System Acquisition Using Hybrid Software Process Simulation.- Simulation-Based Stability Analysis for Software Release Plans.- Exploring the Impact of Task Allocation Strategies for Global Software Development Using Simulation.- Users and Developers: An Agent-Based Simulation of Open Source Software Evolution.- Simulating the Structural Evolution of Software.- Experience Report.- An Empirical Study on SW Metrics for Embedded System.- Process-Family-Points.- Automated Recognition of Low-Level Process: A Pilot Validation Study of Zorro for Test-Driven Development.- Process Evolution Supported by Rationale: An Empirical Investigation of Process Changes.- Implementing Process Change in a Software Organization - An Experience Based Study.- Practical Experiences of Cost/Schedule Measure Through Earned Value Management and Statistical Process Control.