Oscar Dieste

Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review

This paper reports a systematic review of empirical studies concerning the effectiveness of elicitation techniques, and the subsequent aggregation of empirical evidence gathered from those studies. The most significant results of the aggregation process are as follows: (I) interviews, preferentially structured, appear to be one of the most effective elicitation techniques; (2) many techniques often cited in the literature, like card sorting, ranking or thinking aloud, tend to be less effective than interviews; (3) analyst experience does not appear to be a relevant factor; and (4) the studies conducted have not found the use of intermediate representations during elicitation to have significant positive effects. It should be noted that, as a general rule, the studies from which these results were aggregated have not been replicated, and therefore the above claims cannot be said to be absolutely certain. However, they can be used by researchers as pieces of knowledge to be further investigated and by practitioners in development projects, always taking into account that they are preliminary findings

Developing search strategies for detecting relevant experiments

Our goal is to analyze the optimality of search strategies for use in systematic reviews of software engineering experiments. Studies retrieval is an important problem in any evidence-based discipline. This question has not been examined for evidence-based software engineering as yet. We have run several searches exercising different terms denoting experiments to evaluate their recall and precision. Based on our evaluation, we propose using a high recall strategy when there are plenty of resources or the results need to be exhaustive. For any other case, we propose optimal, or even acceptable, search strategies. As a secondary goal, we have analysed trends and weaknesses in terminology used in articles reporting software engineering experiments. We have found that it is impossible for a search strategy to retrieve 100% of the experiments of interest (as happens in other experimental disciplines), because of the shortage of reporting standards in the community.

Systematic review and aggregation of empirical studies on elicitation techniques

We have located the results of empirical studies on elicitation techniques and aggregated these results to gather empirically grounded evidence. Our chosen surveying methodology was systematic review, whereas we used an adaptation of comparative analysis for aggregation because meta-analysis techniques could not be applied. The review identified 564 publications from the SCOPUS, IEEEXPLORE, and ACM DL databases, as well as Google. We selected and extracted data from 26 of those publications. The selected publications contain 30 empirical studies. These studies were designed to test 43 elicitation techniques and 50 different response variables. We got 100 separate results from the experiments. The aggregation generated 17 pieces of knowledge about the interviewing, laddering, sorting, and protocol analysis elicitation techniques. We provide a set of guidelines based on the gathered pieces of knowledge.

A systematic mapping study on empirical evaluation of software requirements specifications techniques

This paper describes an empirical mapping study, which was designed to identify what aspects of Software Requirement Specifications (SRS) are empirically evaluated, in which context, and by using which research method. On the basis of 46 identified and categorized primary studies, we found that understandability is the most commonly evaluated aspect of SRS, experiments are the most commonly used research method, and the academic environment is where most empirical evaluation takes place.

Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews

Information retrieval is an important problem in any evidence-based discipline. Although evidence- based software engineering (EBSE) is not immune to this fact, this question has not been examined at length. The goal of this paper is to analyse the optimality of search strategies for use in systematic reviews. We tried out 29 search strategies using different terms and combinations of terms. We evaluated their sensitivity and precision with a view to finding an optimum strategy. From this study of search strategies we were able to analyse trends and weaknesses in terminology use in articles reporting experiments.

Requirements Elicitation in Data Mining for Business Intelligence Projects

There are data mining methodologies for business intelligence (DM-BI) projects that highlight the importance of planning an ordered, documented, consistent and traceable requirement’s elicitation throughout the entire project. However, the classical software engineering approach is not completely suitable for DM-BI projects because it neglects the requirements specification aspects of projects. This article focuses on identifying concepts for understand DM-BI project domain from DM-BI field experience, including how requirements can be educed by a proposed DM-BI project requirements elicitation process and how they can be documented by a template set.

Usability evaluation of multi-device/platform user interfaces generated by model-driven engineering

Nowadays several Computer-Aided Software Engineering environments exploit Model-Driven Engineering (MDE) techniques in order to generate a single user interface for a given computing platform or multi-platform user interfaces for several computing platforms simultaneously. Therefore, there is a need to assess the usability of those generated user interfaces, either taken in isolation or compared to each other. This paper describes an MDE approach that generates multi-platform graphical user interfaces (e.g., desktop, web) that will be subject to an exploratory controlled experiment. The usability of user interfaces generated for the two mentioned platforms and used on multiple display devices (i.e., standard size, large, and small screens) has been examined in terms of satisfaction, effectiveness and efficiency. An experiment with a factorial design for repeated measures was conducted for 31 participants, i.e., postgraduate students and professors selected by convenience sampling. The data were collected with the help of questionnaires and forms and were analyzed using parametric and non-parametric tests such as ANOVA with repeated measures and Friedmans test, respectively. Efficiency was significantly better in large screens than in small ones as well as in the desktop platform rather than in the web platform, with a confidence level of 95%. The experiment also suggests that satisfaction tends to be better in standard size screens than in small ones. The results suggest that the tested MDE approach should incorporate enhancements in its multi-device/platform user interface generation process in order to improve its generated usability.

Replication types: towards a shared taxonomy

Context: The software engineering community is becoming more aware of the need for experimental replications. In spite of the importance of this topic, there is still much inconsistency in the terminology used to describe replications. Objective: Understand the perspectives of empirical researchers about various terms used to characterize replications and propose a consistent taxonomy of terms. Method: A survey followed by plenary discussion during the 2013 International Software Engineering Research Network meeting. Results: We propose a taxonomy which consolidates the disparate terminology. This taxonomy had a high level of agreement among workshop attendees. Conclusion: Consistent terminology is important for any field to progress. This work is the first step in that direction. Additional study and discussion is still necessary.

Software industry experiments: a systematic literature review

Background: There is no specialized survey of experiments conducted in the software industry. Goal: Identify the major features of software industry experiments, such as time distribution, independent and dependent variables, subject types, design types and challenges. Method: Systematic literature review, taking the form of a scoping study. Results: We have identified 10 experiments and five quasi-experiments up to July 2012. Most were run as of 2003. The main features of these studies are that they test technologies related to quality and management and analyse outcomes related to effectiveness and effort. Most experiments have a factorial design. The major challenges faced by experimenters are to minimize the cost of running the experiment for the company and to schedule the experiment so as not to interfere with production processes. Conclusion: Companies appear to be disinclined to run experiments because they are not perceived to have direct benefits. We believe that researchers staging a field experiment in a company should adopt a business-aligned stance and plan an experiment that clearly benefits managers and professionals.

Topic selection in industry experiments

This paper shares our experience with initial negotiation and topic elicitation process for conducting industry experiments in six software development organizations in Finland. The process involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings. Fitness criteria developed by researchers were applied to the list of generated topics to decide on a common topic. The challenges we faced include diversity of proposed topics, communication gaps, skepticism about research methods, initial disconnect between research and industry needs, and lack of prior work relationship. Lessons learned include having enough time to establish trust with partners, importance of leveraging the benefits of training and skill development that are inherent in the experimental approach, uniquely positioning the experimental approach within the landscape of other validation approaches more familiar to industrial partners, and introducing the fitness criteria early in the process.