Birgit Hofer

On the empirical evaluation of fault localization techniques for spreadsheets

Spreadsheets are by far the most prominent example of end-user programs of ample size and substantial structural complexity. In addition, spreadsheets are usually not tested very rigorously and thus comprise faults. Locating faults is a hard task due to the size and the structure, which is usually not directly visible to the user, i.e., the functions are hidden behind the cells and only the computed values are presented. Hence, there is a strong need for debugging support. In this paper, we adapt three program-debugging approaches that have been designed for more traditional procedural or object-oriented programming languages. These techniques are Spectrum-based Fault Localization, Spectrum-Enhanced Dynamic Slicing, and Constraint-based Debugging. Beside the theoretical foundations, we present a more sophisticated empirical evaluation including a comparison of these approaches. The empirical evaluation shows that Sfl (Spectrum-based Fault Localization) and Sendys (Spectrum ENhanced Dynamic Slicing) are the most promising techniques.

Avoiding, Finding and Fixing Spreadsheet Errors - A Survey of Automated Approaches for Spreadsheet QA

Spreadsheet programs can be found everywhere in organizations and they are used for a variety of purposes, including financial calculations, planning, data aggregation and decision making tasks. A number of research surveys have however shown that such programs are particularly prone to errors. Some reasons for the error-proneness of spreadsheets are that spreadsheets are developed by end users and that standard software quality assurance processes are mostly not applied. Correspondingly, during the last two decades, researchers have proposed a number of techniques and automated tools aimed at supporting the end user in the development of error-free spreadsheets. In this paper, we provide a review of the research literature and develop a classification of automated spreadsheet quality assurance (QA) approaches, which range from spreadsheet visualization, static analysis and quality reports, over testing and support to model-based spreadsheet development. Based on this review, we outline possible opportunities for future work in the area of automated spreadsheet QA.

On the empirical evaluation of similarity coefficients for spreadsheets fault localization

Spreadsheets are by far the most prominent example of end-user programs of ample size and substantial structural complexity. They are usually not thoroughly tested so they often contain faults. Debugging spreadsheets is a hard task due to the size and structure, which is usually not directly visible to the user, i.e., the functions are hidden and only the computed values are presented. A way to locate faulty cells in spreadsheets is by adapting software debugging approaches for traditional procedural or object-oriented programming languages. One of such approaches is spectrum-based fault localization (Sfl). In this paper, we study the impact of different similarity coefficients on the accuracy of Sfl applied to the spreadsheet domain. Our empirical evaluation shows that three of the 42 studied coefficients (Ochiai, Jaccard and Sorensen-Dice) require less effort by the user while inspecting the diagnostic report, and can also be used interchangeably without a loss of accuracy. In addition, we illustrate the influence of the number of correct and incorrect output cells on the diagnostic report.

Spectrum enhanced dynamic slicing for better fault localization

Debugging consumes a considerable amount of time in software engineering, but it is rarely automated. In this paper, we focus on improving existing fault localization techniques. Spectrum-based fault localization (SFL) and slicing-hitting-set-computation (SHSC) are two techniques based on program execution traces. Both techniques come with small computational overhead and aid programmers to faster identify possible locations of faults. However, they have disadvantages: SHSC results in an undesirable high ranking of statements which are executed in many test cases, such as constructors. SFL operates on block level. Therefore, it cannot provide fine-grained results. We combine SHSC with SFL in order to eliminate these disadvantages. Our objective is to improve the ranking of faulty statements so that they allow for better fault localization than when using the previously mentioned methods separately. We show empirically that the resulting approach reduces the number of statements a programmer needs to check manually. In particular, we gain improvements of about 50% percent for SHSC and 25 % for SFL.

GUI savvy end-to-end testing with smart monkeys

In this article we report on the development of a graphical user interface-savvy test monkey and its successful application to the Windows calculator. Our novel test monkey allows for a pragmatic approach in providing an abstract model of the GUI relevant behavior of the application under test and relies on a readily available GUI automation tool. Besides of outlining the employed test oracles we explain our novel decision-based state machine model, the associated language and the random test algorithm. Moreover we outline the pragmatic model creation concept and report on its concrete application in an end-to-end test setting with a Windows Vista front-end. Notably in this specific scenario, our novel monkey was able to identify a misbehavior in a well-established application and provided valuable insight for reproducing the detected fault.

The Right Choice Matters! SMT Solving Substantially Improves Model-Based Debugging of Spreadsheets

Spreadsheets are undoubtedly the most prominent example of end-user programs vastly used in practice. Spreadsheets are often complex and comprise several hundreds or even thousands of formulas making fault localization a very hard and painstaking task. Although there has been work based on model-based reasoning for fault localization in spreadsheets, its applicability in practice is still limited mainly due to two reasons. First, most of the available constraint solvers, used for computing diagnoses, do not sufficiently support Real numbers. Second, the runtime especially for larger spreadsheets is too long preventing truly interactive debugging. In order to eliminate limitations regarding data type support and runtime performance, we suggest the use of SMT solvers for spreadsheet debugging. Besides lying out the foundations of SMT solving for diagnoses, we introduce its application to spreadsheet debugging. The empirical evaluation shows that the SMT solver Z3 requires six times less time for computing diagnoses compared to public available constraint solvers.

Why Does my Spreadsheet Compute Wrong Values

Spreadsheets are by far the most used programs that are written by end-users. They often build the basis for decisions in companies and governmental organizations and therefore they have a high impact on our daily life. Ensuring correctness of spreadsheets is thus an important task. But what happens after detecting a faulty behavior? This question has not been sufficiently answered. Therefore, we focus on fault localization techniques for spreadsheets. In this paper, we introduce a novel dependency-based approach for model-based fault localization in spreadsheets. This approach improves diagnostic accuracy while keeping computation times short, thus making the automated fault localization more appropriate for practical applications. The presented approach allows for an acceptable fault localization time of less than a second, and reduces the number of computed root cause candidates by 15 % on average, when compared with another dependency-based approach.

AI for the win: improving spectrum-based fault localization

A considerable amount of time in software engineering is spent in debugging. In practice, mainly debugging tools which allow for executing a program step-by-step and setting break points are used. This debugging method is however very time consuming and cumbersome. There is a need for tools which undertake the task of narrowing down the most likely fault locations. These tools must complete this task with as little user interaction as possible and the results computed must be beneficial so that such tools appeal to programmers. In order to come up with such tools, we present three variants of the well-known spectrum-based fault localization technique that are enhanced by using methods from Artificial Intelligence. Each of the three combined approaches outperforms the underlying basic method concerning diagnostic accuracy. Hence, the presented approaches support the hypothesis that combining techniques from different areas is beneficial. In addition to the introduction of these techniques, we perform an empirical evaluation, discuss open challenges of debugging and outline possible solutions.

Combining Models for Improved Fault Localization in Spreadsheets

Spreadsheets are the most prominent example of end-user programing, but they unfortunately are often erroneous, and thus, they compute wrong values. Localizing the true cause of such an observed misbehavior can be cumbersome and frustrating especially for large spreadsheets. Therefore, supporting techniques and tools for fault localization are highly required. Model-based software debugging (MBSD) is a well-known technique for fault localization in software written in imperative and object-oriented programing languages like C, C++, and Java. In this paper, we explain how to use MBSD for fault localization in spreadsheets and compare three types of models for MBSD, namely the value-based model (VBM), the dependency based model (DBM), and an improved version of the DBM. Whereas the VBM computes the lowest number of diagnoses, both DBMs convince by their low computational complexity. Hence, a combination of these two types of models is desired, and we present a solution that combines value-based and DBM in this paper. Moreover, we discuss a detailed evaluation of the models and the combined approach, which indicates that the combined approach computes the same number of diagnoses like the VBMs while requiring less computation time. Hence, the proposed approach is more appropriate to be used in tools for fault localization in spreadsheets.

Combining slicing and constraint solving for better debugging: the CONBAS approach

Although slices provide a good basis for analyzing programs during debugging, they lack in their capabilities providing precise information regarding the most likely root causes of faults. Hence, a lot of work is left to the programmer during fault localization. In this paper, we present an approach that combines an advanced dynamic slicing method with constraint solving in order to reduce the number of delivered fault candidates. The approach is called Constraints Based Slicing (Conbas). The idea behind Conbas is to convert an execution trace of a failing test case into its constraint representation and to check if it is possible to find values for all variables in the execution trace so that there is no contradiction with the test case. For doing so, we make use of the correctness and incorrectness assumptions behind a diagnosis, the given failing test case. Beside the theoretical foundations and the algorithm, we present empirical results and discuss future research. The obtained empirical results indicate an improvement of about 28% for the single fault and 50% for the double-fault case compared to dynamic slicing approaches.