Thomas D. LaToza

Developers ask reachability questions

A reachability question is a search across feasible paths through a program for target statements matching search criteria. In three separate studies, we found that reachability questions are common and often time consuming to answer. In the first study, we observed 13 developers in the lab and found that half of the bugs developers inserted were associated with reachability questions. In the second study, 460 professional software developers reported asking questions that may be answered using reachability questions more than 9 times a day, and 82% rated one or more as at least somewhat hard to answer. In the third study, we observed 17 developers in the field and found that 9 of the 10 longest activities were associated with reachability questions. These findings suggest that answering reachability questions is an important source of difficulty understanding large, complex codebases.

Hard-to-answer questions about code

To build new tools and programming languages that make it easier for professional software developers to create, debug, and understand code, it is helpful to better understand the questions that developers ask during coding activities. We surveyed professional software developers and asked them to list hard-to-answer questions that they had recently asked about code. 179 respondents reported 371 questions. We then clustered these questions into 21 categories and 94 distinct questions. The most frequently reported categories dealt with intent and rationale -- what does this code do, what is it intended to do, and why was it done this way? Many questions described very specific situations -- e.g., what does the code do when an error occurs, how to refactor without breaking callers, or the implications of a specific change on security. These questions revealed opportunities for both existing research tools to help developers and for developing new languages and tools that make answering these questions easier.

A practical guide to controlled experiments of software engineering tools with human participants

Empirical studies, often in the form of controlled experiments, have been widely adopted in software engineering research as a way to evaluate the merits of new software engineering tools. However, controlled experiments involving human participants actually using new tools are still rare, and when they are conducted, some have serious validity concerns. Recent research has also shown that many software engineering researchers view this form of tool evaluation as too risky and too difficult to conduct, as they might ultimately lead to inconclusive or negative results. In this paper, we aim both to help researchers minimize the risks of this form of tool evaluation, and to increase their quality, by offering practical methodological guidance on designing and running controlled experiments with developers. Our guidance fills gaps in the empirical literature by explaining, from a practical perspective, options in the recruitment and selection of human participants, informed consent, experimental procedures, demographic measurements, group assignment, training, the selecting and design of tasks, the measurement of common outcome variables such as success and time on task, and study debriefing. Throughout, we situate this guidance in the results of a new systematic review of the tool evaluations that were published in over 1,700 software engineering papers published from 2001 to 2011.

Microtask programming: building software with a crowd

Microtask crowdsourcing organizes complex work into workflows, decomposing large tasks into small, relatively independent microtasks. Applied to software development, this model might increase participation in open source software development by lowering the barriers to contribu-tion and dramatically decrease time to market by increasing the parallelism in development work. To explore this idea, we have developed an approach to decomposing programming work into microtasks. Work is coordinated through tracking changes to a graph of artifacts, generating appropriate microtasks and propagating change notifications to artifacts with dependencies. We have implemented our approach in CrowdCode, a cloud IDE for crowd development. To evaluate the feasibility of microtask programming, we performed a small study and found that a small crowd of 12 workers was able to successfully write 480 lines of code and 61 unit tests in 14.25 person-hours of time.

Crowdsourcing in Software Engineering: Models, Motivations, and Challenges

Almost surreptitiously, crowdsourcing has entered software engineering practice. In-house development, contracting, and outsourcing still dominate, but many development projects use crowdsourcing-for example, to squash bugs, test software, or gather alternative UI designs. Although the overall impact has been mundane so far, crowdsourcing could lead to fundamental, disruptive changes in how software is developed. Various crowdsourcing models have been applied to software development. Such changes offer exciting opportunities, but several challenges must be met for crowdsourcing software development to reach its potential.

Supporting informal design with interactive whiteboards

Whiteboards serve an important role in supporting informal design, providing a fluid and flexible medium for collaborative design. Interactive whiteboards offer the potential for enhanced support for manipulating content, managing sketches, and distributed work, but little is known about how this support affects the practice of informal design. To understand the opportunities and challenges, we first conducted a literature review, identifying 14 behaviors that occur during informal design. We then designed an interactive whiteboard system to support all of these behaviors and deployed the system to three groups of designers. Through usage logs and interviews, we examined the effects of interactivity on whiteboard use across a wide spectrum of design behaviors, identifying ways in which interactive whiteboards support the practices used in physical whiteboards and where they enable designers to work more effectively.

Borrowing from the crowd: a study of recombination in software design competitions

One form of crowdsourcing is the competition, which poses an open call for competing solutions. Commercial systems such as TopCoder have begun to explore the application of competitions to software development, but have important limitations diminishing the potential benefits drawn from the crowd. In particular, they employ a model of independent work that ignores the opportunity for designs to arise from the ideas of multiple designers. In this paper, we examine the potential for software design competitions to incorporate recombination, in which competing designers are given the designs of others and encouraged to use them to revise their own designs. To explore this, we conducted two software design competitions in which participants were asked to produce both an initial and a revised design, drawing on lessons learned from the crowd. We found that, in both competitions, all participants borrowed ideas and most improved the quality of their designs. Our findings demonstrate the potential benefits of recombination in software design and suggest several ways in which software design competitions can be improved.

Questions about object structure during coding activities

Recent tools have been designed to help developers understand the potential runtime structure of objects in a system at compile time. Such tools let developers interactively explore diagrams of object structure. But do developers ask questions about object structure? If so, when? We conducted a small pilot study of developers working on coding tasks designed to require thinking about relationships between objects. Developers did indeed ask a number of questions about various types of relationships such as containment, ownership, object identities and aliasing. Finally, some of our results revealed usability challenges tools should address to more effectively answer these questions.

Developer refinement of runtime architectural structure

Runtime architecture is important because it helps reason about quality attributes such as performance and security. We conducted an on-site field study to help us understand how developers understand object relationships, and what tool features a developer might need to convey their mental model of object relationships. The subject of our study was an experienced, professional programmer. We asked the developer questions to check if he understood which tiers objects belonged to, and whether certain objects were conceptually part of other objects. We also took an initial runtime architecture and refined it to convey his intent. The developers mental model seemed to agree with the one supported by the current, batch-oriented approach for extracting architectures. This indicates that an interactive tool that allows making an object conceptually part of another, and abstracting away a low-level object would be useful to iteratively refine an initial object model.

CodeExchange: Supporting Reformulation of Internet-Scale Code Queries in Context (T)

Programming today regularly involves searching for source code online, whether through a general search engine such as Google or a specialized code search engine such as SearchCode, Ohloh, or GitHub. Searching typically is an iterative process, with develop-ers adjusting the keywords they use based on the results of the previous query. However, searching in this manner is not ideal, because just using keywords places limits on what developers can express as well as the overall interaction that is required. Based on the observation that the results from one query create a con-text in which a next is formulated, we present CodeExchange, a new code search engine that we developed to explicitly leverage this context to support fluid, expressive reformulation of queries. We motivate the need for CodeExchange, highlight its key design decisions and overall architecture, and evaluate its use in both a field deployment and a laboratory study.