Paola R. G. Accioly

A Model-Driven Approach to Specifying and Monitoring Controlled Experiments in Software Engineering

This paper presents a process-oriented model-driven approach that supports the conduction of controlled experiments in software engineering. The approach consists of: (i) a domain specific language (DSL) for process specification and statistical design of controlled experiments; (ii) model-driven transformations that allow workflow models generations specific to each experiment participant and according to the experiment statistical design; and (iii) a workflow execution environment that allows the monitoring of participant activities in the experiment, besides gathering participants feedback from the experiment. The paper also presents the results of an exploratory study that analyzes the feasibility of the approach and the expressivity of the DSLs in the modeling of a non-trivial software engineering experiment.

Evaluating scenario-based SPL requirements approaches: the case for modularity, stability and expressiveness

AbstractSoftware product lines (SPL) provide support for productivity gains through systematic reuse. Among the various quality attributes supporting these goals, modularity, stability and expressiveness of feature specifications, their composition and configuration knowledge emerge as strategic values in modern software development paradigms. This paper presents a metric-based evaluation aiming at assessing how well the chosen qualities are supported by scenario-based SPL requirements approaches. The selected approaches for this study span from type of notation (textual or graphical based), style to support variability (annotation or composition based), and specification expressiveness. They are compared using the metrics developed in a set of releases from an exemplar case study. Our major findings indicate that composition-based approaches have greater potential to support modularity and stability, and that quantification mechanisms simplify and increase expressiveness of configuration knowledge and composition specifications.

Comparing Two Black-Box Testing Strategies for Software Product Lines

SPL testing has been considered a challenging task,mainly due to the diversity of products that might be generatedfrom an SPL. To deal with this problem, some techniques for derivingproduct specific functional test cases have been proposed.However, this research area still lacks empirical studies showingthe benefits of using such techniques. This paper presents a studythat empirically compares two different black-box manual testdesign techniques: a generic technique that we have observed inan industrial test execution environment and a product specifictechnique whose functional test cases could be derived using anySPL technique that considers variations in functional tests. Weevaluated their impact from the point of view of the test execution process, achieving results that indicate that executing product specific test cases is faster and generates fewer errors.

Assessing Semistructured Merge in Version Control Systems: A Replicated Experiment

Context: To reduce the integration effort arising from conflicting changes resulting from collaborative software development tasks, unstructured merge tools try to automatically solve part of the conflicts via textual similarity, whereas structured and semistructured merge tools try to go further by exploiting the syntactic structure of the involved artifacts. Objective: In this study, aiming at increasing the existing body of evidence and assessing results for systems developed under an alternative version control paradigm, we replicate an experiment conducted by Apel et al. to compare the unstructured and semistructured approach with respect to the occurrence of conflicts reported by both approaches. Method: We used both semistructured and unstructured merge in a sample 2.5 times bigger than the original study regarding the number of projects and 18 times bigger regarding the number of merge scenarios, and we compared the occurrence of conflicts. Results: Similar to the original study, we observed that semistructured merge reduces the number of conflicts in 55% of the scenarios of the new sample. However, the observed average conflict reduction of 62% in these scenarios is far superior than what has been observed before. We also bring new evidence that the use of semistructured merge can reduce the occurrence of conflicting merge scenarios by half. Conclusions: Our findings reinforce the benefits of exploiting the syntactic structure of the artifacts involved in code integration. Besides, the reductions observed in the number and size of conflicts suggest that the use of semistructured merge, when compared to the unstructured approach, might decrease integration effort without compromising correctness.

Evaluating and improving semistructured merge

While unstructured merge tools rely only on textual analysis to detect and resolve conflicts, semistructured merge tools go further by partially exploiting the syntactic structure and semantics of the involved artifacts. Previous studies compare these merge approaches with respect to the number of reported conflicts, showing, for most projects and merge situations, reduction in favor of semistructured merge. However, these studies do not investigate whether this reduction actually leads to integration effort reduction (productivity) without negative impact on the correctness of the merging process (quality). To analyze that, and better understand how merge tools could be improved, in this paper we reproduce more than 30,000 merges from 50 open source projects, identifying conflicts incorrectly reported by one approach but not by the other (false positives), and conflicts correctly reported by one approach but missed by the other (false negatives). Our results and complementary analysis indicate that, in the studied sample, the number of false positives is significantly reduced when using semistructured merge. We also find evidence that its false positives are easier to analyze and resolve than those reported by unstructured merge. However, we find no evidence that semistructured merge leads to fewer false negatives, and we argue that they are harder to detect and resolve than unstructured merge false negatives. Driven by these findings, we implement an improved semistructured merge tool that further combines both approaches to reduce the false positives and false negatives of semistructured merge. We find evidence that the improved tool, when compared to unstructured merge in our sample, reduces the number of reported conflicts by half, has no additional false positives, has at least 8% fewer false negatives, and is not prohibitively slower.

Empirical assessment of two approaches for specifying software product line use case scenarios

Modularity benefits, including the independent maintenance and comprehension of individual modules, have been widely advocated. However, empirical assessments to investigate those benefits have mostly focused on source code, and thus, the relevance of modularity to earlier artifacts is still not so clear (such as requirements and design models). In this paper, we use a multimethod technique, including designed experiments, to empirically evaluate the benefits of modularity in the context of two approaches for specifying product line use case scenarios: PLUSS and MSVCM. The first uses an annotative approach for specifying variability, whereas the second relies on aspect-oriented constructs for separating common and variant scenario specifications. After evaluating these approaches through the specifications of several systems, we find out that MSVCM reduces feature scattering and improves scenario cohesion. These results suggest that evolving a product line specification using MSVCM requires only localized changes. On the other hand, the results of six experiments reveal that MSVCM requires more time to derive the product line specifications and, contrasting with the modularity results, reduces the time to evolve a product line specification only when the subjects have been well trained and are used to the task of evolving product line specifications.

Understanding semi-structured merge conflict characteristics in open-source Java projects

Empirical studies show that merge conflicts frequently occur, impairing developers’ productivity, since merging conflicting contributions might be a demanding and tedious task. However, the structure of changes that lead to conflicts has not been studied yet. Understanding the underlying structure of conflicts, and the involved syntactic language elements might shed light on how to better avoid merge conflicts. To this end, in this paper we derive a catalog of conflict patterns expressed in terms of the structure of code changes that lead to merge conflicts. We focus on conflicts reported by a semistructured merge tool that exploits knowledge about the underlying syntax of the artifacts. This way, we avoid analyzing a large number of spurious conflicts often reported by typical line based merge tools. To assess the occurrence of such patterns in different systems, we conduct an empirical study reproducing 70,047 merges from 123 GitHub Java projects. Our results show that most semistructured merge conflicts in our sample happen because developers independently edit the same or consecutive lines of the same method. However, the probability of creating a merge conflict is approximately the same when editing methods, class fields, and modifier lists. Furthermore, we noticed that most part of conflicting merge scenarios, and merge conflicts, involve more than two developers. Also, that copying and pasting pieces of code, or even entire files, across different repositories is a common practice and cause of conflicts. Finally, we discuss how our results reveal the need for new research studies and suggest potential improvements to tools supporting collaborative software development.

Should we replace our merge tools

While unstructured merge tools try to automatically resolve merge conflicts via textual similarity, semistructured merge tools try to go further by partially exploiting the syntactic structure and semantics of the involved artefacts. Previous studies compare these merge approaches with respect to the number of reported conflicts, showing, for most projects and merge situations, a reduction in favor of semistructured merge. However, these studies do not investigate whether this reduction actually leads to integration effort reduction (Productivity) without negative impact on the correctness of the merging process (Quality). To analyze this, and to better understand how these tools could be improved, we propose empirical studies to identify spurious conflicts reported by one approach but not by the other, and interference reported as conflict by one approach but missed by the other.

Analyzing conflict predictors in open-source Java projects

In collaborative development environments integration conflicts occur frequently. To alleviate this problem, different awareness tools have been proposed to alert developers about potential conflicts before they become too complex. However, there is not much empirical evidence supporting the strategies used by these tools. Learning about what types of changes most likely lead to conflicts might help to derive more appropriate requirements for early conflict detection, and suggest improvements to existing conflict detection tools. To bring such evidence, in this paper we analyze the effectiveness of two types of code changes as conflict predictors. Namely, editions to the same method, and editions to directly dependent methods. We conduct an empirical study analyzing part of the development history of 45 Java projects from GitHub and Travis CI, including 5,647 merge scenarios, to compute the precision and recall for the conflict predictors aforementioned. Our results indicate that the predictors combined have a precision of 57.99% and a recall of 82.67%. Moreover, we conduct a manual analysis which provides insights about strategies that could further increase the precision and the recall.