Ivan do Carmo Machado

A systematic mapping study of software product lines testing

ContextIn software development, Testing is an important mechanism both to identify defects and assure that completed products work as specified. This is a common practice in single-system development, and continues to hold in Software Product Lines (SPL). Even though extensive research has been done in the SPL Testing field, it is necessary to assess the current state of research and practice, in order to provide practitioners with evidence that enable fostering its further development. ObjectiveThis paper focuses on Testing in SPL and has the following goals: investigate state-of-the-art testing practices, synthesize available evidence, and identify gaps between required techniques and existing approaches, available in the literature. MethodA systematic mapping study was conducted with a set of nine research questions, in which 120 studies, dated from 1993 to 2009, were evaluated. ResultsAlthough several aspects regarding testing have been covered by single-system development approaches, many cannot be directly applied in the SPL context due to specific issues. In addition, particular aspects regarding SPL are not covered by the existing SPL approaches, and when the aspects are covered, the literature just gives brief overviews. This scenario indicates that additional investigation, empirical and practical, should be performed. ConclusionThe results can help to understand the needs in SPL Testing, by identifying points that still require additional investigation, since important aspects regarding particular points of software product lines have not been addressed yet.

On the Reliability of Mapping Studies in Software Engineering

Background: Systematic literature reviews and systematic mapping studies are becoming increasingly common in software engineering, and hence it becomes even more important to better understand the reliability of such studies. Objective: This paper presents a study of two systematic mapping studies to evaluate the reliability of mapping studies and point out some challenges related to this type of study in software engineering. Method: The research is based on an in-depth case study of two published mapping studies on software product line testing. Results: We found that despite the fact that the two studies are addressing the same topic, there are quite a number of differences when it comes to papers included and in terms of classification of the papers included in the two mapping studies. Conclusions: From this we conclude that although mapping studies are important, their reliability cannot simply be taken for granted. Based on the findings we also provide four conjectures that further research has to address to make secondary studies (systematic mapping studies and systematic literature reviews) even more valuable to both researchers and practitioners. (Less)

On strategies for testing software product lines: A systematic literature review

Context: Testing plays an important role in the quality assurance process for software product line engineering. There are many opportunities for economies of scope and scale in the testing activities, but techniques that can take advantage of these opportunities are still needed. Objective: The objective of this study is to identify testing strategies that have the potential to achieve these economies, and to provide a synthesis of available research on SPL testing strategies, to be applied towards reaching higher defect detection rates and reduced quality assurance effort. Method: We performed a literature review of two hundred seventy-six studies published from the year 1998 up to the 1st semester of 2013. We used several filters to focus the review on the most relevant studies and we give detailed analyses of the core set of studies. Results: The analysis of the reported strategies comprised two fundamental aspects for software product line testing: the selection of products for testing, and the actual test of products. Our findings indicate that the literature offers a large number of techniques to cope with such aspects. However, there is a lack of reports on realistic industrial experiences, which limits the inferences that can be drawn. Conclusion: This study showed a number of leveraged strategies that can support both the selection of products, and the actual testing of products. Future research should also benefit from the problems and advantages identified in this study.

Strategies for testing products in software product lines

The software product line engineering strategy enables the achievement of significant improvements in quality through reuse of carefully crafted software assets across multiple products. However, high levels of quality in the software product line assets, which are used to create products, must be accompanied by effective and efficient test strategies for the products in the software product line. The goal of this study is to understand which strategies for testing products in software product lines have been reported in the literature, enabling discussions on the significant issues, and also pointing out further research directions. A systematic literature review was carried out that identified two hundred seventy-three papers, published from the years 1998 and early in 2012. From such a set of papers, a systematic selection resulted in forty-one relevant papers. The analysis of the reported strategies comprised two important aspects: the selection of products for testing, and the actual test of products. The findings showed a range of strategies, dealing with both aspects, but few empirical evaluations of their effectiveness have been performed, which limits the inferences that can be drawn.

Towards metamodel support for variability and traceability in software product lines

In Software Product Lines (SPL), where a greater variety of products are derived from a common platform and constantly changed and evolved, it is important to manage the SPL variability and the traceability among its artifacts. This paper presents a metamodel which aims to coordinate SPL activities, by managing different SPL phases and their responsibles, and to maintain the traceability and variability among different artifacts. The metamodel was built for a SPL project in a private company working in the medical information management domain, which includes four products encompassing 102 different modules and 840 features. The metamodel is divided into five sub-models: project and risk management, scoping, requirements and testing. It is represented in the UML notation. Organizations using this metamodel as basis for their approaches, can easily understand the relationships between the SPL assets, communicate to the stakeholders, and facilitate the evolution and maintenance of the SPL. The metamodel can also be adapted to the single system development context.

Challenges and opportunities for software change request repositories: a systematic mapping study

Software maintenance starts as soon as the first artifacts are delivered and is essential for the success of the software. However, keeping maintenance activities and their related artifacts on track comes at a high cost. In this respect, change request (CR) repositories are fundamental in software maintenance. They facilitate the management of CRs and are also the central point to coordinate activities and communication among stakeholders. However, the benefits of CR repositories do not come without issues, and commonly occurring ones should be dealt with, such as the following: duplicate CRs, the large number of CRs to assign, or poorly described CRs. Such issues have led researchers to an increased interest in investigating CR repositories, by considering different aspects of software development and CR management. In this paper, we performed a systematic mapping study to characterize this research field. We analyzed 142 studies, which we classified in two ways. First, we classified the studies into different topics and grouped them into two dimensions: challenges and opportunities. Second, the challenge topics were classified in accordance with an existing taxonomy for information retrieval models. In addition, we investigated tools and services for CR management, to understand whether and how they addressed the topics identified. Copyright

A Regression Testing Approach for Software Product Lines Architectures

In the Software Product Lines (SPL) context, where products are derived from a common platform, the reference architecture can be considered the main asset. In order to maintain its correctness and reliability after modifications, a regression testing approach based on architecture specification and code was developed. It aims to reduce the testing effort, by reusing test cases, execution results, as well as, selecting and prioritizing an effective set of test cases. Taking advantage of SPL architectures similarities, this approach can be applied among product architectures and between the reference and product architecture. This study also presents an evaluation performed in order to calibrate and improve the proposed approach.

Risk management in software product lines: an industrial case study

Software Product Lines (SPL) adoption can affect several aspects of an organization and it involves significant investment and risk. This way, SPL risk management is a crucial activity of SPL adoption. This study aims to identify SPL risks during the scoping and requirement disciplines to provide information to better understand risk management in SPL. In order to achieve the previous stated goal, a case study research was applied in an industrial project in the medical information management domain. Using the captured risks, a classification scheme was built and risk mitigation strategies were identified. We spent five months, totaling 79 hours, performing risk management (RM) in the scoping discipline and twelve months, totaling 148 hours, performing RM on the requirements discipline. We identified 32 risks during the scoping discipline and 20 risks during the requirements discipline, 14 risks occurred in both disciplines. Some identified risks are not particular to SPL development, however, they have their impact increased due to the SPL characteristic. All the study results and lessons learned are useful for all project managers and researchers who are considering the introduction of SPL risk management in industry or academia.

Testing Software Product Lines

Two studies of testing practices for software product lines identify gaps between required techniques and existing approaches in the available literature. This Web extra offers extra details for the main article (specifically, the bibliography for the two studies described).

Analysis of Non-functional Properties in Software Product Lines: A Systematic Review

Software Product Lines (SPL) approach has been widely developed in academia and successfully applied in industry. Based on the selection of features, stakeholders can efficiently derive tailor-made programs satisfying different requirements. While SPL was very successful at building products based on identified features, achievements and preservation of many nonfunctional properties (NFPs) remain challenging. A knowledge how to deal with NFPs is still not fully obtained. In this paper, we present a systematic literature review of NFPs analysis for SPL products, focusing on runtime NFPs. The goal of the paper is twofold: (i) to present an holistic overview of SPL approaches that have been reported regarding the analysis of runtime NFPs, and (ii) to categorize NFPs treated in the scientific literature regarding development of SPLs. We analyzed 36 research papers, and identified that system performance attributes are typically the most considered. The results also aid future research studies in NFPs analysis by providing an unbiased view of the body of empirical evidence and by guiding future research directions.