Paige Rodeghero

Improving automated source code summarization via an eye-tracking study of programmers

Source Code Summarization is an emerging technology for automatically generating brief descriptions of code. Current summarization techniques work by selecting a subset of the statements and keywords from the code, and then including information from those statements and keywords in the summary. The quality of the summary depends heavily on the process of selecting the subset: a high-quality selection would contain the same statements and keywords that a programmer would choose. Unfortunately, little evidence exists about the statements and keywords that programmers view as important when they summarize source code. In this paper, we present an eye-tracking study of 10 professional Java programmers in which the programmers read Java methods and wrote English summaries of those methods. We apply the findings to build a novel summarization tool. Then, we evaluate this tool and provide evidence to support the development of source code summarization systems.

An Empirical Study on the Patterns of Eye Movement during Summarization Tasks

Eye movement patterns are the order in which keywords or sections of keywords are read. These patterns are an important component of how programmers read source code. One strategy for determining how programmers perform summarization tasks is through eye tracking studies. These studies examine where people focus their attention while viewing text or images. In this study, we expand on eye tracking analysis to determine the eye movement patterns of programmers. We begin the study with a qualitative exploration of the eye movement patterns used by 10 professional programmers from an earlier study. We then use what we learned qualitatively to perform a quantitative analysis of those patterns. We found that all ten of the programmers followed nearly identical eye movement patterns. These patterns were analogous to eye movement patterns of reading natural language.

Detecting user story information in developer-client conversations to generate extractive summaries

User stories are descriptions of functionality that a software user needs. They play an important role in determining which software requirements and bug fixes should be handled and in what order. Developers elicit user stories through meetings with customers. But user story elicitation is complex, and involves many passes to accommodate shifting and unclear customer needs. The result is that developers must take detailed notes during meetings or risk missing important information. Ideally, developers would be freed of the need to take notes themselves, and instead speak naturally with their customers. This paper is a step towards that ideal. We present a technique for automatically extracting information relevant to user stories from recorded conversations between customers and developers. We perform a qualitative study to demonstrate that user story information exists in these conversations in a sufficient quantity to extract automatically. From this, we found that roughly 10.2% of these conversations contained user story information. Then, we test our technique in a quantitative study to determine the degree to which our technique can extract user story information. In our experiment, our process obtained about 70.8% precision and 18.3% recall on the information.

An empirical study on how expert knowledge affects bug reports

Bug reports are crucial software artifacts for both software maintenance researchers and practitioners. A typical use of bug reports by researchers is to evaluate automated software maintenance tools: a large repository of reports is used as input for a tool, and metrics are calculated from the tools output. But this process is quite different from practitioners, who distinguish between reports written by experts, such as programmers, and reports written by non‐experts, such as users. Practitioners recognize that the content of a bug report depends on its authors expert knowledge. In this paper, we present an empirical study of the textual difference between bug reports written by experts and non‐experts. We find that a significant difference exists and that this difference has a significant impact on the results from a state‐of‐the‐art feature location tool. Through an additional study, we also found no evidence that these encountered differences were caused by the increased usage of terms from the source code in the expert bug reports. Our recommendation is that researchers evaluate maintenance tools using different sets of bug reports for experts and non‐experts. Copyright

A comparison of program comprehension strategies by blind and sighted programmers

Programmers who are blind use a screen reader to speak source code one word at a time, as though the code were text. This process of reading is in stark contrast to sighted programmers, who skim source code rapidly with their eyes. At present, it is not known whether the difference in these processes has effects on the program comprehension gained from reading code. These effects are important because they could reduce both the usefulness of accessibility tools and the generalizability of software engineering studies to persons with low vision. In this paper, we present an empirical study comparing the program comprehension of blind and sighted programmers. We found that both blind and sighted programmers prioritize reading method signatures over other areas of code. Both groups obtained an equal and high degree of comprehension, despite the different reading processes.

API usage in descriptions of source code functionality

In this paper, we present a study exploring the use of API keywords within method summaries. We conducted a web-based study where we asked participants to rank Java method summaries based on five levels of detail, from low level to high level. We found that programmers widely use API in both high and low level summaries. Specifically, we found that 76.78% of higher level summaries contain Java API keywords. Additionally, we found that 93.75% of lower level summaries also contain them. This also shows that, in general, as the detail level decreases, the number of API keywords within the summary increases. It is our hope that this line of research will spark a discussion about API usage outside of source code. It is possible that method summaries are not the only form of documentation that API usage plays an important role. We believe these may be important results that could lead to an improvement for API usability design.