Chung Man Tang

Accuracy Graphs of Spectrum-Based Fault Localization Formulas

The effectiveness of spectrum-based fault localization techniques primarily relies on the accuracy of their fault localization formulas. Theoretical studies prove the relative accuracy orders of selected formulas under certain assumptions, forming a graph of their theoretical accuracy relations. However, it is unclear whether in such a graph the relative positions of these formulas may change when some assumptions are relaxed. On the other hand, empirical studies can measure the actual accuracy of any formula in controlled settings that more closely approximate practical scenarios but in less general contexts. In this paper, we propose an empirical framework of accuracy graphs and their construction that reveal the relative accuracy of formulas. Our work not only evaluates the association between certain assumptions and the theoretical relations among formulas, but also expands our knowledge to reveal new potential accuracy relationships of other formulas which have not been discovered by theoretical analysis. Using our proposed framework, we identified a list of formula pairs in which a formula is consistently statistically more accurate than or similar in accuracy to another, enlightening directions for further theoretical analysis.

DFL: Dual-Service Fault Localization

In engineering a service, software developers often construct and deploy a newer (forthcoming) version of the service to replace the current version. A forthcoming version is often placed online for users to consume and report feedback. In the case of observed failures, the forthcoming version should be debugged and further evolved. In this paper, we propose the model of dual-service fault localization (DFL) to aid this evolution process. Many prior research studies on spectrum-based fault localization (SBFL) consider each version separately. The DFL model correlates the dynamic execution spectra of the current and the forthcoming versions of the same service placed for live test of the forthcoming version, and dynamically generates an adaptive fault localization formula to estimate the code regions in the forthcoming service responsible for the observed failures. We report an experiment in which we initialized the DFL model into six instances, each using an ensemble technique dynamically composed from 11 existing SBFL formulas, and applied the model to four benchmarks. The results show that DFL is feasible and multiple instances are statistically more effective than, if not as effective as, the best of these individual SBFL formulas on each benchmark.

An Exploratory Study on Instructors’ Agreement on the Correctness of Computer Program Outputs

Many universities have developed Automated Program Assessment Systems to automate the tasks of assessing students’ computer programs so as to enhance students’ learning and relieve instructors’ workload. These systems typically evaluate the correctness of a program by comparing its actual outputs with the instructor’s pre-defined expected outputs. However, an actual output may still be correct even if it deviates from the expected output. One challenge in building such a system is to devise an automated mechanism for determining program output correctness that matches the instructor’s own judgment. This is difficult if instructors have different individual judgments. This paper reports an exploratory empirical study which evaluates instructors’ agreement on the correctness of students’ program outputs. Our study demonstrates reasonably good overall agreement between the instructors and reveals the categories of program output variants for which they are more likely to agree or disagree.

Automatic Assessment via Intelligent Analysis of Students' Program Output Patterns.

Automatic assessment of computer programming exercises offers a number of benefits to both learners and educators, including timely and customised feedback, as well as saving of human effort in grading. However, due to the high variety of programs submitted by students, exact matching between the expected output and different output variants is undesirable and how to do the matching properly is a challenging and practical problem. Existing approaches to address this problem adopt various inexact matching strategies, but typically they are unscalable, incapable of expressing a diversity of program outputs, or require high level of expertise. In this paper, we propose Hierarchical Program Output Structure (HiPOS), which provides higher expressiveness and flexibility, to model the program output. Based on HiPOS, we design different levels of matching rules in the matching rule hierarchy to determine the admissible program output variants in a flexible and scalable manner. We conducted experiments and compare our approach of automatic assessment to human judgement. The results show that our proposed method achieved an accuracy of 0.8467 in determining the admissible program output variants.

Theoretical, Weak and Strong Accuracy Graphs of Spectrum-Based Fault Localization Formulas

Driven by the need to know which spectrum-based fault localization techniques are more effective in locating faults, many studies have sought to compare the accuracy of different formulas used in these techniques, resulting in findings of both theoretical and empirical accuracy relations of these formulas. Theoretical accuracy relations are independent of the specific programs and other settings involved, but limited by underlying assumptions and manual work in proofs. An accuracy graph can be constructed to holistically represent the proved relations. On the other hand, empirical studies are free of specific theoretical assumptions and can be highly automatable and scalable. A recent study has developed a systematic methodology based on statistical tests to reveal consistent and statistically sound empirical accuracy relations. That work has demonstrated the merits of empirical accuracy graphs in revealing relations that can be hard to prove. In this paper, we propose to use a stronger criterion for comparing formulas, describe an exploratory experiment to construct accuracy graphs based on the criterion, and report interesting relations found from the resulting accuracy graphs.