Ronnie E. S. Santos

Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study

Context. A recent mapping study intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: the empirical studies reporting actually replications and a second group of studies that were concerned with concepts, classifications, guidelines, and other themes about replication. Objective. The purpose of this research is to analyze and discuss the content of the second set of studies about replications published between 1996 and 2013. Method. This is a mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by a non-systematic search that collected some articles published in 2013. Results. We analyzed 36 papers reporting empirical and non-empirical studies about replications in SE published in the last 17 years. These papers explored different topics related to concepts and classifications, presented guidelines, and discussed theoretical issues that are relevant for our understanding of replication in our field.

Motivation to Perform Systematic Reviews and their Impact on Software Engineering Practice

Context: Systematic literature reviews (SLRs) are a particular type of secondary study used as the main research method in evidence based research and practice. The starting point of a SLR should be a problem from the practice and the results should, somehow, have an impact on improving practice. Objective: To investigate the motivation of software engineering researchers to conduct a SLR and how the results of their reviews are potentially impacting the software engineering practice. Method: We conducted a cross-sectional survey with authors of 120 SLRs published between 2004 and 2010 identified by three tertiary studies previously published. Results: Forty-four authors of SLR participate in the survey. The motivation behind the vast majority of the SLRs was to gather knowledge about a particular field of study. However, only six participants affirmed that their reviews actually had a direct impact in industrial practice. Conclusions: The use of SLR has increased over the years but they were mostly focused on academic problems and had very little impact on industrial practice. Therefore, the full potential benefits of evidence-based software engineering are not being achieved.

Preliminary Findings about the Nature of Work in Software Engineering: An Exploratory Survey

Context: Work Design refers to the different ways in which a given work or task can be designed and performed. The study of work design is important because every decision related to how the work is performed can affect the outcomes of individuals and the effectiveness of teamwork. Goal: To investigate work design characteristics of software engineering work and identify areas for future research. Method: We performed a survey with professional software engineers working in different commercial software companies in Brazil. We sent invitations to just over 150 professionals and received 77 valid answers from participants working in 35 distinct software companies. We measured 21 work design constructs, as well as job burnout, role conflict, role ambiguity, and two constructs related to job rotation. Results: Comparisons of our sample with other sample consisting of a diversity of professions showed that software engineering work has distinct characteristics, but some results require further investigation. Conclusion: We identified relevant characteristics of software engineering work and areas for further research. In particular, longitudinal studies are needed to address the temporal variations impossible to identify in cross sectional studies.

Investigations about replication of empirical studies in software engineering

ContextTwo recent mapping studies which were intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994 and 2012) and a second group of studies that are concerned with definitions, classifications, processes, guidelines, and other research topics or themes about replication work in empirical software engineering research (published between 1996 and 2012). ObjectiveIn this current article, our goal is to analyze and discuss the contents of the second set of studies about replications to increase our understanding of the current state of the work on replication in empirical software engineering research. MethodWe applied the systematic literature review method to build a systematic mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by manual and automatic search procedures that collected articles published in 2013. ResultsWe analyzed 37 papers reporting studies about replication published in the last 17years. These papers explore different topics related to concepts and classifications, presented guidelines, and discuss theoretical issues that are relevant for our understanding of replication in our field. We also investigated how these 37 papers have been cited in the 135 replication papers published between 1994 and 2012. ConclusionsReplication in SE still lacks a set of standardized concepts and terminology, which has a negative impact on the replication work in our field. To improve this situation, it is important that the SE research community engage on an effort to create and evaluate taxonomy, frameworks, guidelines, and methodologies to fully support the development of replications.

Replication of Empirical Studies in Software Engineering: An Update of a Systematic Mapping Study

Context: Current empirical research highlight the need for replications of empirical studies because replications plays an important role in the construction of scientific knowledge. Objective: Considering the importance of replications in the consolidation of the knowledge produced in the software engineering research, this study aims to update and extend the results produced in a previous mapping study seeking to discuss the current state of the replication work of empirical studies performed in software engineering research between 2011 and 2012. Method: We applied the systematic review method to search and select published papers, to extract, and synthesize data from reported replications. Results: This study analyzed more than 7,000 articles, from which 39 articles that published replications between 2011 and 2012 were selected. Data extracted from these studies were used to update the information about the replications work in software engineering. Conclusion: The number of replications increased significantly in the period, when compared to the previous mapping study. In particular, the percentage of external replications also increased, with respect to internal ones. However, several other limitations identified in the previous mapping studies are still observed in this new set of replications.

Benefits and limitations of job rotation in software organizations: a systematic literature review

Context. Job Rotation is an organizational practice whereby individuals are regularly moved among jobs or projects in the same organization. Goal: To identify and discuss evidence about job rotation, in order to understand the use, the benefits, and the limitations of this practice in software organizations. Method: A systematic literature review protocol was used to identify and select empirical studies previously published in the software engineering literature, and then coding techniques were used to analyse and synthesize their findings. Results: This review identified 18 empirical papers presenting evidence of 17 distinct studies about job rotation in software engineering. These studies revealed that in software organizations job rotation has been used to enhance communication, organizational understanding, knowledge exchange, and task variety. However, its use also requires extra effort and sometimes complex planning. Conclusion: The research about job rotation in software engineering is restricted, with only one study focusing on this topic and 16 presenting non-intentional evidence about the theme. Our review synthesized evidence that could inform research and practice. However, due to the specific nature of software development tasks and jobs, empirical evidence is still needed to guide the effective application of job rotation in practice.

The use of systematic reviews in evidence based software engineering: a systematic mapping study

Context. A decade ago, Kitchenham, Dybå and Jørgensen argued that software engineering could benefit from an evidence-based research approach similar that that used in medicine, introducing the basis for Evidence Based Software Engineering (EBSE). Objective. Our main goal is to understand the evolution of the use of systematic reviews as the main research method in EBSE, as proposed by Kitchenham et al., by investigating primary and tertiary studies that explore any aspect, theory, or concept around the use of systematic reviews in software engineering. Method. A systematic mapping study protocol was used to find and selected studies about EBSE and systematic reviews in SE, published between 2004 and 2013. Results. We selected 52 unique papers classified as non-empirical studies (12), empirical studies (31), and tertiary studies (9). Conclusion. SLR has become an important component of software engineering research with nearly 200 unique reviews catalogued by the tertiary studies. Most important limitations are related to the industrial relevance and application of the results of reviews and the poor use of synthesis method to aggregate evidence

Member checking in software engineering research: lessons learned from an industrial case study

Context. Member checking can be defined as a research phase performed during a qualitative research in which the researcher compares her interpretations and understanding obtained from the data analysis with the view-points of participants to increase accuracy and consistency of results. This is an important step for any qualitative research. However, considering a sample of 66 case studies developed and published in the context of software engineering, only 10 studies briefly described the use of this technique. Method. In this article, we present a set of lessons learned obtained from planning and performing member checking to validate the results of an industrial case study performed in a large software company. Results. Member checking was effective to validate the findings obtained from the qualitative case study and was also useful to reveal important information not observed in the data analysis process. It has also shown to be effective to observe divergences among different groups of participants. Conclusion. We described how the member checking can be performed, and discussed seven lessons learned in this process. We expect that our experience can be useful to software engineering researchers while performing this research phase in case studies.

Computer games are serious business and so is their quality: particularities of software testing in game development from the perspective of practitioners

Context. Over the last several decades, computer games started to have a significant impact on society. However, although a computer game is a type of software, the process to conceptualize, produce and deliver a game could involve unusual features. In software testing, for instance, studies demonstrated the hesitance of professionals to use automated testing techniques with games, due to the constant changes in requirements and design, and pointed out the need for creating testing tools that take into account the flexibility required for the game development process. Goal. This study aims to improve the current body of knowledge regarding these theme and point out the existing particularities observed in software testing considering the development of a computer game. Method. A mixed-method approach based on a case study and a survey was applied to collect quantitative and qualitative data from practitioners regarding the particularities of software testing in game development. Results. We analyzed over 70 messages posted on three well-established network of question-and-answer communities and received answers of 38 practitioners, and identified important aspects to be observed in the process of planning, performing and reporting tests games. Conclusion. Considering computer games, software testing must focus not only on the common aspects of a general software, but also, track and investigate issues that could be related to game balance, game physics and entertainment related-aspects to guarantee the quality of computer games and a successful testing process.

Would you like to motivate software testers?: ask them how

Context. Considering the importance of software testing to the development of high quality and reliable software systems, this paper aims to investigate how can work-related factors influence the motivation of software testers. Method. We applied a questionnaire that was developed using a previous theory of motivation and satisfaction of software engineers to conduct a survey-based study to explore and understand how professional software testers perceive and value work-related factors that could influence their motivation at work. Results. With a sample of 80 software testers we observed that software testers are strongly motivated by variety of work, creative tasks, recognition for their work, and activities that allow them to acquire new knowledge, but in general the social impact of this activity has low influence on their motivation. Conclusion. This study discusses the difference of opinions among software testers, regarding work-related factors that could impact their motivation, which can be relevant for managers and leaders in software engineering practice.