Jooyong Yi

Codeflaws: a programming competition benchmark for evaluating automated program repair tools

Several automated program repair techniques have been proposed to reduce the time and effort spent in bug-fixing. While these repair tools are designed to be generic such that they could address many software faults, different repair tools may fix certain types of faults more effectively than other tools. Therefore, it is important to compare more objectively the effectiveness of different repair tools on various fault types. However, existing benchmarks on automated program repairs do not allow thorough investigation of the relationship between fault types and the effectiveness of repair tools. We present Codeflaws, a set of 3902 defects from 7436 programs automatically classified across 39 defect classes (we refer to different types of fault as defect classes derived from the syntactic differences between a buggy program and a patched program).

What do software engineers care about? gaps between research and practice

It is a cliche to say that there is a gap between research and practice. As the interest and importance in the practical impact of research has been growing, the gap between research and practice is expected to be narrowing. However, our study reveals that there still seems to be a wide gap. We survey so ware engineers about what they care about when developing so ware. We then compare our survey results with the research topics of the papers published in ICSE/FSE recently. We found the following discrepancy: while so ware engineers care more about so ware development productivity than the quality of so ware, papers on research areas closely related to so ware productivity--such as so ware development process management and so ware development techniques--are significantly less published than papers on so ware verification and validation that account for more than half of publications. We also found that so ware engineers are in great need for techniques for accurate effort estimation, and they are not necessarily knowledgable about techniques they can use to meet their needs.

A feasibility study of using automated program repair for introductory programming assignments

Despite the fact an intelligent tutoring system for programming (ITSP) education has long attracted interest, its widespread use has been hindered by the difficulty of generating personalized feedback automatically. Meanwhile, automated program repair (APR) is an emerging new technology that automatically fixes software bugs, and it has been shown that APR can fix the bugs of large real-world software. In this paper, we study the feasibility of marrying intelligent programming tutoring and APR. We perform our feasibility study with four state-of-the-art APR tools (GenProg, AE, Angelix, and Prophet), and 661 programs written by the students taking an introductory programming course. We found that when APR tools are used out of the box, only about 30% of the programs in our dataset are repaired. This low repair rate is largely due to the student programs often being significantly incorrect - in contrast, professional software for which APR was successfully applied typically fails only a small portion of tests. To bridge this gap, we adopt in APR a new repair policy akin to the hint generation policy employed in the existing ITSP. This new repair policy admits partial repairs that address part of failing tests, which results in 84% improvement of repair rate. We also performed a user study with 263 novice students and 37 graders, and identified an understudied problem; while novice students do not seem to know how to effectively make use of generated repairs as hints, the graders do seem to gain benefits from repairs.

A new architecture and implementation strategy for non-invasive software measurement systems

Despite that non-invasive software measurement tools have proven their usefulness in software production, their adoption in software industry is still limited. Reasons for the limited distributions have been studied and analysed recently. In this paper, we propose a new architecture for non-invasive software measurement systems that address the problems of the existing systems. The outcome of our early experimentation is quite promising and gives us the desired additional confidence on its successful distribution.

Understanding the impact of pair programming on the minds of developers

Software is mostly, if not entirely, a knowledge artifact. Software best practices are often thought to work because they induce more productive behaviour in software developers. In this paper we deployed a new generation tool, portable multichannel EEG, to obtain direct physical insight into the mental processes of working software developers engaged in their standard activities. We have demonstrated the feasibility of this approach and obtained a glimpse of its potential power to distinguish physical brain activity of developers working with different methodologies.

Precooked developer dashboards: what to show and how to use

Designing an effective and useful dashboard is expensive and it would be important to determine if it is possible to elaborate a generic useful and effective dashboard, usable in a variety of circumstances. To determine if it is possible to develop such dashboard and, if so, its structure we interviewed 67 software engineers from 44 different companies. Their answers made us confident in the possibility of building such dashboard.

A correlation study between automated program repair and test-suite metrics

Automated program repair has attracted attention due to its potential to reduce debugging cost. Prior works show the feasibility of automated repair, and the research focus is gradually shifting towards the quality of generated patches. One promising direction is to control the quality of generated patches by controlling the quality of test-suites used. In this paper, 1we investigate the question: Can traditional test-suite metrics used in software testing be used for automated program repair?. We empirically investigate the effectiveness of test-suite metrics (statement / branch coverage and mutation score) in controlling the reliability of repairs (the likelihood that repairs cause regressions). We conduct the largest-scale experiments to date with real-world software, and perform the first correlation study between test-suite metrics and the reliability of generated repairs. Our results show that by increasing test-suite metrics, the reliability of repairs tend to increase. Particularly, such trend is most strongly observed in statement coverage. This implies that traditional test-suite metrics used in software testing can also be used to improve the reliability of repairs in program repair.

Toward a Better Understanding of How to Develop Software Under Stress - Drafting the Lines for Future Research.

The software is often produced under significant time constraints. Our idea is to understand the effects of various software development practices on the performance of developers working in stressful environments, and identify the best operating conditions for software developed under stressful conditions collecting data through questionnaires, non-invasive software measurement tools that can collect measurable data about software engineers and the software they develop, without intervening their activities, and biophysical sensors and then try to recreated also in different processes or key development practices such conditions.

Towards Non-invasive Software Measurement System: Architecture and Implementation

Despite that non-invasive software measurement tools have proven their usefulness in software production, their adoption in software industry is still limited. Reasons for the limited distributions have been studied and analyzed in works like (Coman et al, Proceedings of 476 the 31st International Conference on Software Engineering (ICSE 2009), Vancouver 89–99, 2009) [1]. In this paper, we propose a new architecture for non-invasive software measurement systems that address the problems of the existing systems. The outcome of our early experimentation is quite promising and gives us the desired additional confidence on its successful distribution.