Carlos Jensen

Code coverage for suite evaluation by developers

One of the key challenges of developers testing code is determining a test suites quality -- its ability to find faults. The most common approach is to use code coverage as a measure for test suite quality, and diminishing returns in coverage or high absolute coverage as a stopping rule. In testing research, suite quality is often evaluated by a suites ability to kill mutants (artificially seeded potential faults). Determining which criteria best predict mutation kills is critical to practical estimation of test suite quality. Previous work has only used small sets of programs, and usually compares multiple suites for a single program. Practitioners, however, seldom compare suites --- they evaluate one suite. Using suites (both manual and automatically generated) from a large set of real-world open-source projects shows that evaluation results differ from those for suite-comparison: statement (not block, branch, or path) coverage predicts mutation kills best.

The life and times of files and information: a study of desktop provenance

In the field of Human-Computer Interaction, provenance refers to the history and genealogy of a document or file. Provenance helps us to understand the evolution and relationships of files; how and when different versions of a document were created, or how different documents in a collection build on each other through copy-paste events. Though methods for tracking provenance and the subsequent use of this meta-data have been proposed and developed into tools, there have been no studies documenting the types and frequency of provenance events in typical computer use. This is knowledge essential for the design of efficient query methods and information displays. We conducted a longitudinal study of knowledge workers at Intel Corporation tracking provenance events in their computer use. We also interviewed knowledge workers to determine the effectiveness of provenance cues for document recall. Our data shows that provenance relationships are common, and provenance cues aid recall.

Beyond pretty pictures: Examining the benefits of code visualization for Open Source newcomers

Joining an Open Source project is not easy. Newcomers often experience a steep learning curve dealing with technical complexity, lack of domain knowledge, and the amount of project information available for starters. This paper looks at the information needs of newcomers and the potential benefits of information visualization in supporting newcomers through a controlled experiment. Our results show that current OSS environments and development tools are lacking in support for the information needs of newcomers, and that existing visualization tools and techniques can help. We also discuss the potential problems and pitfalls associated with the inappropriate use of code visualization tools.

Participatory design with older adults: an analysis of creativity in the design of mobile healthcare applications

Researchers often use participatory design -- involving endusers in technology ideation -- as this is found to lead to more useful and relevant products. Researchers have sought to involve older adults in the design of emerging technologies like smartphones, with which older adults often have little experience. Therefore, their effectiveness as co-designers could be questioned. We examine whether older adults can create novel design ideas, and whether critiquing existing applications prior to ideation helps or hinders creativity. Panelists from industry and academia evaluated design ideas generated by focus groups of older adults. Out of five groups, the most creative design idea came from one with no smartphone experience or critique exposure. We found that while only some designs scored high on the novelty dimension of creativity, participants were enthusiastic about participating and adapted quickly. We found evidence that critiquing existing applications prior to ideation did more harm than good, potentially due to design fixation. We recommend continuing to involve older adults in the technology design ideation phase.

Tracking website data-collection and privacy practices with the iWatch web crawler

In this paper we introduce the iWatch web crawler, a tool designed to catalogue and analyze online data practices and the use of privacy related indicators and technologies. Our goal in developing iWatch was to make possible a new type of analysis of trends, the impact of legislation on practices, and geographic and social differences online. In this paper we present preliminary findings from two sets of data collected 15 months apart and analyzed with this tool. Our combined samples included more than 240,000 pages from over 24,000 domains and 47 different countries. In addition to providing useful and needed data on the state of online data practices, we show that iWatch is a promising approach to the study of the web ecosystem.

Mutations: How Close are they to Real Faults?

Mutation analysis is often used to compare the effectiveness of different test suites or testing techniques. One of the main assumptions underlying this technique is the Competent Programmer Hypothesis, which proposes that programs are very close to a correct version, or that the difference between current and correct code for each fault is very small. Researchers have assumed on the basis of the Competent Programmer Hypothesis that the faults produced by mutation analysis are similar to real faults. While there exists some evidence that supports this assumption, these studies are based on analysis of a limited and potentially non-representative set of programs and are hence not conclusive. In this paper, we separately investigate the characteristics of bug-fixes and other changes in a very large set of randomly selected projects using four different programming languages. Our analysis suggests that a typical fault involves about three to four tokens, and is seldom equivalent to any traditional mutation operator. We also find the most frequently occurring syntactical patterns, and identify the factors that affect the real bug-fix change distribution. Our analysis suggests that different languages have different distributions, which in turn suggests that operators optimal in one language may not be optimal for others. Moreover, our results suggest that mutation analysis stands in need of better empirical support of the connection between mutant detection and detection of actual program faults in a larger body of real programs.

What health topics older adults want to track: a participatory design study

Older adults are increasingly savvy consumers of smartphone-based health solutions and information. These technologies may enable older adults to age-in-place more successfully. However, many app creators fail to do needs assessments of their end-users. To rectify this issue, we involved older adults (aged 65+) in the beginning stages of designing a mobile health and wellness application. We conducted a participatory design study, where 5 groups of older adults created 5 designs. Four groups identified at least 1 health metric not currently offered in either the iPhone app store or the Google Play store. At the end of the sessions we administered a questionnaire to determine what health topics participants would like to track via smartphone or tablet. The designs included 13 health topics that were not on the questionnaire. Seventeen of eighteen participants expressed interest in tracking health metrics using a smartphone/tablet despite having little experience with these devices. This shows that older adults have unique ideas that are not being considered by current technology designers. We conclude with recommendations for future development, and propose continuing to involve to older adults in participatory design.

On the limits of mutation reduction strategies

Although mutation analysis is considered the best way to evaluate the effectiveness of a test suite, hefty computational cost often limits its use. To address this problem, various mutation reduction strategies have been proposed, all seeking to reduce the number of mutants while maintaining the representativeness of an exhaustive mutation analysis. While research has focused on the reduction achieved, the effectiveness of these strategies in selecting representative mutants, and the limits in doing so have not been investigated, either theoretically or empirically. We investigate the practical limits to the effectiveness of mutation reduction strategies, and provide a simple theoretical framework for thinking about the absolute limits. Our results show that the limit in improvement of effectiveness over random sampling for real-world open source programs is a mean of only 13.078%. Interestingly, there is no limit to the improvement that can be made by addition of new mutation operators. Given that this is the maximum that can be achieved with perfect advance knowledge of mutation kills, what can be practically achieved may be much worse. We conclude that more effort should be focused on enhancing mutations than removing operators in the name of selective mutation for questionable benefit.

Understanding how and why open source contributors use diagrams in the development of Ubuntu

Some of the most interesting differences between Open Source Software (OSS) development and commercial co-located software development lie in the communication and collaboration practices of these two groups of developers. One interesting practice is that of diagramming. Though well studied and important in many aspects of co-located software development (including communication and collaboration among developers), its role in OSS development has not been thoroughly studied. In this paper, we report our investigation on how and why Ubuntu contributors use diagrams in their work. Our study shows that diagrams are not actively used in many scenarios where they commonly would in co-located software development efforts. We describe differences in the use and practices of diagramming, their possible reasons, and present design considerations for potential systems aimed at better supporting diagram use in OSS development.

Coping with duplicate bug reports in free/open source software projects

Free/Open Source Software (FOSS) communities often use open bug reporting to allow users to participate by reporting bugs. This practice can lead to more duplicate reports, as users can be less rigorous about researching existing bug reports. This paper examines how FOSS projects deal with duplicate bug reports. We examined 12 FOSS projects: 4 small, 4 medium and 4 large, where size was determined by number of code contributors. First, we found that contrary to what has been reported from studies of individual large projects like Mozilla and Eclipse, duplicate bug reports are a problem for FOSS projects, especially medium-sized, which struggle with a large number of submissions without the resources of large projects. Second, we found that the focus of a project does not affect the number of duplicate bug reports. Our findings indicate a need for additional scaffolding and training for bug reporters.