Andreas Jedlitschka

Reporting guidelines for controlled experiments in software engineering

One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) it is difficult to locate relevant information and (2) important information is often missing. Reporting guidelines are expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results. The objective of this paper is to survey the most prominent published proposals for reporting guidelines, and to derive a unified standard that which can serve as a starting point for further discussion. We provide detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical studies, i.e., controlled experiments. Before the guidelines can be evaluated, feedback from the research community is required. For this purpose, we propose to adapt guideline development processes from other disciplines.

Reporting experiments in software engineering

Background: One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) It is difficult to locate relevant information and (2) important information is often missing. Objective: A checklist for reporting results from controlled experiments is expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results. Method: The checklist for reporting is based on (1) a survey of the most prominent published proposals for reporting guidelines in software engineering and (2) an iterative development incorporating feedback from members of the research community. Result: This paper presents a unification of a set of guidelines for reporting experiments in software engineering. Limitation: The checklist has not been evaluated broadly, yet. Conclusion: The resulting checklist provides detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical studies, i.e., experiments (controlled experiments and quasi-experiments).

Relevant Information Sources for Successful Technology Transfer: A Survey Using Inspections as an Example

A systematic approach to decision making in software engineering is required, for instance, if an organization aims at achieving CMMI level three. Rational decision making regarding the selection and introduction of SE technologies requires adequate information about their suitability for the intended organizational context. Research is often unable to provide such information, and this could be one reason why promising techniques are sometimes not adopted in practice. From a research point of view, successful technology transfer requires knowing which information decision makers in industry need, and where they actually look for it. With this knowledge, empirical research can strive to produce the needed information in order to increase the likelihood of successful technology adoption. To address these questions, we conducted an online survey among German software industry decision makers. To focus the survey, we used inspections as an exemplary technology. We invited 9653 companies to participate, from which we received 92 fully completed questionnaires. Our main findings are that information regarding the impact of technologies on product quality, cost, and development time, as well as on technology cost-benefit ratio is considered most important among decision makers. The preferred sources of information are colleagues, textbooks, and industry workshops.Testing is an important activity to ensure software quality. Big organizations can have several development teams with their products being tested by overloaded test teams. In such situations, test team managers must be able to properly plan their schedules and resources. Also, estimates for the required test execution effort can be an additional criterion for test selection, since effort may be restrictive in practice. Nevertheless, this information is usually not available for test cases never executed before. This paper proposes an estimation model for test execution effort based on the test specifications. For that, we define and validate a measure of size and execution complexity of test cases. This measure is obtained from test specifications written in a controlled natural language. We evaluated the model through an empirical study on the mobile application domain, which results suggested an accuracy improvement when compared with estimations based only on historical test productivity.

An Approach to and Evaluations of Assisted Living Systems Using Ambient Intelligence for Emergency Monitoring and Prevention

Ambient Assisted Living (AAL) is currently one of the important research and development areas, where software engineering aspects play a significant role. The goal of AAL solutions is to apply ambient intelligence technologies to enable people with specific needs to continue to live in their preferred environments. This paper presents an approach and several evaluations for emergency monitoring applications. Experiments in a laboratory setting were performed to evaluate the accuracy of recognizing Activities of Daily Living (ADL). The results show that it is possible to detect ADLs with an accuracy of 92% on average. Hence, we conclude that it is possible to support elderly people in staying longer in their homes by autonomously detecting emergencies on the basis of ADL recognition.

Towards evidence in software engineering

The aggregation of studies is of growing interest for the empirical software engineering community, since the numbers of studies steadily grow. We discuss challenges with the aggregation of studies into a common body of knowledge, based on a quantitative and qualitative evaluation of experience from the Experimental Software Engineering Network, ESERNET. Challenges are that the number of studies available is usually low, and the studies that exist are often too scattered and diverse to allow systematic aggregation as a means for generating evidence. ESERNET therefore attempted to coordinate studies and thus create research synergies to achieve a sufficiently large number of comparable studies to allow for aggregation; however, the coordination approach of ESERNET proved to be insufficient. Based on some lessons learned from ESERNET, a four-step procedure for evolving Empirical Software Engineering towards the generation of evidence is proposed. This consists of (1) developing a methodology for aggregating different kinds of empirical results, (2) establishing guidelines for performing, analyzing, and reporting studies as well as for aggregating the results for every kind of empirical study, (3) extract evidence, that is, apply the methodology to different areas of software engineering, and (4) package the extracted evidence into guidelines for practice.

Feedback control of arm movements using Neuro-Muscular Electrical Stimulation (NMES) combined with a lockable, passive exoskeleton for gravity compensation.

Within the European project MUNDUS, an assistive framework was developed for the support of arm and hand functions during daily life activities in severely impaired people. This contribution aims at designing a feedback control system for Neuro-Muscular Electrical Stimulation (NMES) to enable reaching functions in people with no residual voluntary control of the arm and shoulder due to high level spinal cord injury. NMES is applied to the deltoids and the biceps muscles and integrated with a three degrees of freedom (DoFs) passive exoskeleton, which partially compensates gravitational forces and allows to lock each DOF. The user is able to choose the target hand position and to trigger actions using an eyetracker system. The target position is selected by using the eyetracker and determined by a marker-based tracking system using Microsoft Kinect. A central controller, i.e., a finite state machine, issues a sequence of basic movement commands to the real-time arm controller. The NMES control algorithm sequentially controls each joint angle while locking the other DoFs. Daily activities, such as drinking, brushing hair, pushing an alarm button, etc., can be supported by the system. The robust and easily tunable control approach was evaluated with five healthy subjects during a drinking task. Subjects were asked to remain passive and to allow NMES to induce the movements. In all of them, the controller was able to perform the task, and a mean hand positioning error of less than five centimeters was achieved. The average total time duration for moving the hand from a rest position to a drinking cup, for moving the cup to the mouth and back, and for finally returning the arm to the rest position was 71 s.

Adapting PROFES for use in an agile process: an industry experience report

Background: Agile methods are starting to get established not only in new business organizations, but also in organizations dealing with innovation and early product development in more traditional branches like automotive industry. Customers of those organizations demand a specified quality of the delivered products. Objective: Adapt the PROFES Improvement Methodology for use in an industrial, agile process context, to ensure more predictable product quality. Method: An explorative case study at BMW Car IT, which included several structured interviews with stakeholders such as customers and developers. Result: Adapted PROFES methodology with regard to agility and initial product-process dependencies, which partially confirm some of the original PROFES findings. Conclusion: The cost-value ratio of applying PROFES as an improvement methodology in an agile environment has to be carefully considered.

Empirical Software Engineering Experts on the Use of Students and Professionals in Experiments

Abstract[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.

Experience-based model-driven improvement management with combined data sources from industry and academia

Experience-based improvement using various modeling techniques is an important issue in software engineering. Many approaches have been proposed and applied in both industry and academia, e.g., case studies, pilot projects, controlled experiments, assessments, expert opinion polls, experience bases, goal-oriented measurement, process modeling, statistical modeling, data mining, and simulation. Although these approaches can be combined and organized according to the principles of the quality improvement paradigm (QIP) and the associated experience factory (EF) concepts, there are serious problems with: a) effective and efficient integration of the various approaches; and, b) the exchange of experience and data between industry and academia. In particular, the second problem strongly limits opportunities for joint research efforts and cross-organizational synergy. Based upon lessons learned from large-scale European joint research initiatives involving both industry and academia, this paper proposes the vision of an integrated software process improvement framework that facilitates solutions to the problems mentioned above.

How Can Quality Awareness Support Rapid Software Development? – A Research Preview

Context and Motivation: Rapid software development (RSD) refers to the organizational capability to develop, release, and learn from software in rapid cycles without compromising its quality. To achieve RSD, it is essential to understand and manage software quality along the software lifecycle. Question/Problem: Despite the numerous information sources related to product quality, there is a lack of mechanisms for supporting continuous quality management throughout the whole RSD process. Principal ideas/Results: We propose Q-Rapids, a data-driven, quality-aware RSD framework in which quality and functional requirements are managed together. Quality requirements are incrementally elicited and refined based on data gathered at both development time and runtime. Project, development, and runtime data is aggregated into quality-related indicators to support decision makers in steering future development cycles. Contributions: Q-Rapids aims to increase software quality through continuous data gathering and analysis, as well as continuous management of quality requirements.