Amela Karahasanovic

A survey of controlled experiments in software engineering

The classical method for identifying cause-effect relationships is to conduct controlled experiments. This paper reports upon the present state of how controlled experiments in software engineering are conducted and the extent to which relevant information is reported. Among the 5,453 scientific articles published in 12 leading software engineering journals and conferences in the decade from 1993 to 2002, 103 articles (1.9 percent) reported controlled experiments in which individuals or teams performed one or more software engineering tasks. This survey quantitatively characterizes the topics of the experiments and their subjects (number of subjects, students versus professionals, recruitment, and rewards for participation), tasks (type of task, duration, and type and size of application) and environments (location, development tools). Furthermore, the survey reports on how internal and external validity is addressed and the extent to which experiments are replicated. The gathered data reflects the relevance of software engineering experiments to industrial practice and the scientific maturity of software engineering research.

Conducting realistic experiments in software engineering

An important goal of most empirical software engineering research is the transfer of research results to industrial applications. Two important obstacles for this transfer are the lack of control of variables of case studies, i.e., the lack of explanatory power, and the lack of realism of controlled experiments. While it may be difficult to increase the explanatory power of case studies, there is a large potential for increasing the realism of controlled software engineering experiments. To convince industry about the validity and applicability of the experimental results, the tasks, subjects and the environments of the experiments should be as realistic as practically possible. Such experiments are, however, more expensive than experiments involving students, small tasks and pen-and-paper environments. Consequently, a change towards more realistic experiments requires a change in the amount of resources spent on software engineering experiments. This paper argues that software engineering researchers should apply for resources enabling expensive and realistic software engineering experiments similar to how other researchers apply for resources for expensive software and hardware that are necessary for their research. The paper describes experiences from recent experiments that varied in size from involving one software professional for 5 days to 130 software professionals, from 9 consultancy companies, for one day each.

Co-creation and user-generated content-elderly people's user requirements

There is an increasing demand on citizens to participate in social network websites and to create and share their own user-generated content (UGC), such as photographs, videos, and blogs. So far, little is known about how elderly people respond to these new trends and master the techniques required. This paper reports on three studies that investigated elderly peoples user requirements related to consumption, sharing and co-creation of UGC in new media. The first study, conducted in Norway, identifies patterns of Internet usage, age differences, and participation in online communities and the consumption, sharing and co-creation of UGC on a macro level. The second study, conducted in Belgium, investigated the social requirements of elderly people on a group level. The third study, also conducted in Belgium, investigated user and context requirements on an individual level. The results of the first study show that the elderly rarely participate in online communities and share audio-visual UGC. However, they embrace some aspects of the new media and more often express themselves politically. The results of the second study show that the elderly are very motivated to contribute with UGC, given the right circumstances. The results of the third study show that it is important for elderly people that they be able to use the new technologies easily and identifies their worries about using them.

Challenges and Recommendations When Increasing the Realism of Controlled Software Engineering Experiments

An important goal of most empirical software engineering experiments is the transfer of the research results to industrial applications. To convince industry about the validity and applicability of the results of controlled software engineering experiments, the tasks, subjects and the environments should be as realistic as practically possible. Such experiments are, however, more demanding and expensive than experiments involving students, small tasks and pen-and-paper environments. This chapter describes challenges of increasing the realism of controlled experiments and lessons learned from the experiments that have been conducted at Simula Research Laboratory.

Comprehension strategies and difficulties in maintaining object-oriented systems: An explorative study

Program comprehension is a major time-consuming activity in software maintenance. Understanding the underlying mechanisms of program comprehension is therefore necessary for improving software maintenance. It has been argued that acquiring knowledge of how a program works before modifying it (the systematic strategy) is unrealistic in larger programs. The goal of the experiment presented in this paper is to explore this claim. The experiment examines strategies for program comprehension and cognitive difficulties of developers who maintain an unfamiliar object-oriented system. The subjects were 38 students in their third or fourth year of study in computer science. They used a professional Java tool to perform several maintenance tasks on a medium-size Java application system in a 6-h long experiment. The results showed that the subjects who applied the systematic strategy were more likely to produce correct solutions. Two major groups of difficulties were related to the comprehension of the application structure, namely to the understanding of GUI implementation and OO comprehension and programming. Acquisition of strategic knowledge might improve program comprehension in software maintenance.

Inferring Skill from Tests of Programming Performance: Combining Time and Quality

The skills of software developers are important to the success of software projects. Also, when studying the general effect of a tool or method, it is important to control for individual differences in skill. However, the way skill is assessed is often ad hoc, or based on unvalidated methods. According to established test theory, validated tests of skill should infer skill levels from well-defined performance measures on multiple, small, representative tasks. In this respect, we show how time and quality, which are often analyzed separately, can be combined as task performance and subsequently be aggregated as an approximation of skill. Our results show significant positive correlations between our proposed measures of skill and other variables, such as seniority, lines of code written, and self-evaluated expertise. The method for combining time and quality is a promising first step to measuring programming skill in both industry and research settings.

Ensuring Trust, Privacy, and Etiquette in Web 2.0 Applications

An analysis of three user studies of Web 2.0 applications reveals the most important requirements related to ethical issues. The development of features that support these requirements should be tailored to the type of application and specific community needs.

Visualizing impacts of database schema changes - A controlled experiment

Research in schema evolution has been driven by the need for more effective software development and maintenance. Finding impacts of schema changes on the applications and presenting them in an appropriate way are particularly challenging. We have developed a tool that finds impacts of schema changes on applications in object-oriented systems. This tool displays components (packages, classes, interfaces, methods and fields) of a database application system as a graph. Components potentially affected by a change are indicated by changing the shape of the boxes representing those components. Two versions of the tool are available. One version identifies affected parts of applications at the granularity of packages, classes, and interfaces, whereas the other version identifies affected parts at the finer granularity of fields and methods. This paper presents the design and results of a controlled student experiment testing these two granularity levels with respect to productivity and user satisfaction. There are indications that identifying impacts at the finer granularity can reduce the time needed to conduct schema changes and reduce the number of errors. Our results also show that the subjects of the experiment appreciated the idea of visualizing the impacts of schema changes.

Epilogue – service innovation actor engagement: an integrative model

Purpose While (customer) engagement has been proposed as a volitional concept, our structuration theory/S-D logic-informed analyses of actors’ (e.g. employees’) engagement in service innovation reveal engagement as a boundedly volitional theoretical entity, which arises from actors’ structural and agency-based characteristics and constraints. In line with this observation, the purpose of this paper is to develop a conceptual model of actor (i.e. customer, firm, employee) engagement with service innovation. Design/methodology/approach Based on the observed gap, the authors propose an integrative S-D logic/structuration theoretical model that outlines three particular service innovation actors’ (i.e. customers’, the firm’s and employees’) engagement, which comprises institution-driven (i.e. fixed) and agency-driven (i.e. variable) engagement facets. In addition, the authors integrate the key expected characteristics of positively (vs negatively) valenced service innovation engagement for each of these actor groups in the analyses. Findings The authors develop a 12-cell matrix (conceptual model) that outlines particular service innovation actors’ institution-driven and agency-driven engagement facets and outline their expected impact on actors’ ensuing positively and negatively valenced engagement. Research limitations/implications The authors discuss key theoretical implications arising from the analyses. Originality/value Outlining service innovation actors’ structure- and agency-driven engagement facets, the authors’ model can be used to explain or predict customers’, the firm’s or employees’ service innovation engagement-based activities.

Design Feedback From Users Through an Online Social Platform: Benefits and Limitations

Online social platforms, such as blogs, discussion forums and social networking sites, are increasingly explored as venues for user-centred evaluations; in particular, for design feedback from users. We present a multi-case study providing needed knowledge on such evaluations. Our findings are based on analyses of the design feedback and post-factum data collections with development team representatives and users. The development team representatives reported as key benefits that the evaluations provided insight into users’ needs and competencies, input into ongoing design discussions and support for idea generation in the development team, but found the lack of direct contact and control with the users to be an important limitation. The users appreciated the opportunity to contribute to the design process, but the majority reported not to build on each other’s contributions. Involving a relatively large number of users was found to be beneficial for generating constructive design suggestions. Practical implications and future research challenges are suggested.