Razieh Nokhbeh Zaeem

Automated Generation of Oracles for Testing User-Interaction Features of Mobile Apps

As the use of mobile devices becomes increasingly ubiquitous, the need for systematically testing applications (apps) that run on these devices grows more and more. However, testing mobile apps is particularly expensive and tedious, often requiring substantial manual effort. While researchers have made much progress in automated testing of mobile apps during recent years, a key problem that remains largely untracked is the classic oracle problem, i.e., to determine the correctness of test executions. This paper presents a novel approach to automatically generate test cases, that include test oracles, for mobile apps. The foundation for our approach is a comprehensive study that we conducted of real defects in mobile apps. Our key insight, from this study, is that there is a class of features that we term user-interaction features, which is implicated in a significant fraction of bugs and for which oracles can be constructed - in an application agnostic manner -- based on our common understanding of how apps behave. We present an extensible framework that supports such domain specific, yet application agnostic, test oracles, and allows generation of test sequences that leverage these oracles. Our tool embodies our approach for generating test cases that include oracles. Experimental results using 6 Android apps show the effectiveness of our tool in finding potentially serious bugs, while generating compact test suites for user-interaction features.

Improving the effectiveness of spectra-based fault localization using specifications

Fault localization i.e., locating faulty lines of code, is a key step in removing bugs and often requires substantial manual effort. Recent years have seen many automated localization techniques, specifically using the programs passing and failing test runs, i.e., test spectra. However, the effectiveness of these approaches is sensitive to factors such as the type and number of faults, and the quality of the test-suite. This paper presents a novel technique that applies spectra-based localization in synergy with specification-based analysis to more accurately locate faults. Our insight is that unsatisfiability analysis of violated specifications, enabled by SAT technology, could be used to (1) compute unsatisfiable cores that contain likely faulty statements and (2) generate tests that help spectra-based localization. Our technique is iterative and driven by a feedback loop that enables more precise fault localization. SAT-TAR is a framework that embodies our technique for Java programs, including those with multiple faults. An experimental evaluation using a suite of widely-studied data structure programs, including the ANTLR and JTopas parser applications, shows that our technique localizes faults more accurately than state-of-the-art approaches.

Contract-based data structure repair using alloy

Contracts and specifications have long been used in object-oriented design, programming and testing to enhance reliability before software deployment. However, the use of specifications in deployed software is commonly limited to runtime checking where assertions form a basis for detecting incorrect program states to terminate the erroneous executions. This paper presents a contract-based approach for data structure repair, which allows repairing erroneous executions in deployed software by repairing erroneous states. The key novelty is the support for rich behavioral specifications, such as those that relate pre-states with post-states of the method to accurately specify expected behavior and hence to enable precise repair. The approach is based on the view of a specification as a nondeterministic implementation, which may permit a high degree of non-determinism. The key insight is to use any correct state mutations by an otherwise erroneous execution to prune the non-determinism in the specification, thereby transmuting the specification to an implementation that does not incur a prohibitively high performance penalty. While invariants, pre-conditions and post-conditions could be provided in different modeling languages, we leverage the Alloy tool-set, specifically the Alloy language and the Alloy Analyzer for systematically repairing erroneous states. Four different algorithms are presented and implemented in our data structure repair framework. Experiments using complex specifications show the approach holds much promise in increasing software reliability.

Test input generation using dynamic programming

Constraint-based input generation is an effective technique for testing programs, such as compilers and web browsers, which have complex inputs. However, efficient generation of such inputs remains a challenging problem. We present a novel input generation technique that takes constraints written as recursive predicates in the underlying programming language and uses dynamic programming to solve the constraints efficiently. Our key insight is to leverage the recursive structure of desired inputs and partition the problem of generating an input into several sub-problems of generating smaller inputs that exhibit the same structure, and then to use dynamic programming -- a well-known problem solving methodology designed to exploit common sub-problems -- to combine them. A lazy initialization strategy and symbolic execution optimize our basic technique. Our technique provides not only bounded exhaustive input generation but also enables random input generation. We show the correctness of our technique. Furthermore, we present an experimental evaluation, which shows that our technique can provide over an order of magnitude performance improvement for input generation compared to Korat (an efficient solver for structural constraints) and Pex (a state-of-the-art tool for symbolic execution). Finally, we use our technique to effectively find bugs in production versions of Google Chrome and Apple Safari web browsers.

History-Aware data structure repair using SAT

Data structure repair corrects erroneous executions in deployed programs while they execute, eliminating costly downtime. Recent techniques show how to leverage specifications and a SAT solver to enforce specification conformance at runtime. While this powerful methodology increases the reliability of deployed programs, scalability remains a key technical challenge--satisfying a specification often results in the exploration of a huge state space. We present a novel technique, called history-aware contract-based repair for more efficient data structure repair using SAT. Our insight is two-fold: (1) the dynamic program trace of field writes and reads provides useful guidance to repair incorrect state mutations by a faulty program; and (2) we show how to execute SAT using unsatisfiable cores it generates, in an efficient iterative approach on successive problems with increasing state spaces, in order to utilize the history of previous runs as captured in the unsatisfiable core. We implement this approach in a new tool, called Cobbler, that repairs Java programs. Experimental results on two large applications and a library implementation of a linked list show that Cobbler significantly outperforms previous techniques for specification-based repair using SAT, and finds and repairs a previously undetected bug.

Automated Test Generation and Mutation Testing for Alloy

We present two novel approaches for automated testing of models written in Alloy – a well-known declarative, first-order language that is supported by a fully automatic SAT-based analysis engine. The first approach introduces automated test generation for Alloy and is embodied by three techniques that create test suites in the traditional spirit of black-box, white-box, and mutation-based testing. The second approach introduces mutation testing for Alloy and defines how to create mutants of Alloy models, compute mutation testing results, and check for equivalent mutants using SAT. The two approaches build on the theoretical foundation defined previously by our AUnit framework, which introduced the idea of unit testing for Alloy in the spirit of unit testing for imperative languages. While test generation and mutation testing are heavily studied problems with many solutions in the context of imperative languages, the key novelty of our work is to introduce and address these problems for the declarative programming paradigm, specifically for the Alloy language. Experimental results using several Alloy subjects, including those with real faults, demonstrate the efficacy of our framework.

Towards a test automation framework for alloy

Writing declarative models of software designs and analyzing them to detect defects is an effective methodology for developing more dependable software systems. However, writing such models correctly can be challenging for practitioners who may not be proficient in declarative programming, and their models themselves may be buggy. We introduce the foundations of a novel test automation framework, AUnit, which we envision for testing declarative models written in Alloy -- a first-order, relational language that is supported by its SAT-based analyzer. We take inspiration from the success of the family of xUnit frameworks that are used widely in practice for test automation, albeit for imperative or object-oriented programs. The key novelty of our work is to define a basis for unit testing for Alloy, specifically, to define the concepts of test case and coverage, and coverage criteria for declarative models. We reduce the problems of declarative test execution and coverage computation to evaluation without requiring SAT solving. Our vision is to blend how developers write unit tests in commonly used programming languages with how Alloy users formulate their models in Alloy, thereby facilitating the development and testing of Alloy models for both new Alloy users as well as experts. We illustrate our ideas using a small but complex Alloy model. While we focus on Alloy, our ideas generalize to other declarative languages (such as Z, B, ASM).

Repair Abstractions for More Efficient Data Structure Repair

Despite the substantial advances in techniques for finding and removing bugs, code is often deployed with (unknown or known) bugs, which pose a fundamental problem for software reliability. A promising approach to address this problem is data structure repair—a runtime approach designed to perform repair actions, i.e., mutations of erroneous data structures to repair (certain) errors in program state, to allow the program to recover from those errors and continue to execute. While data structure repair holds much promise, current techniques for repair do not scale to real applications.

PrivacyCheck: Automatic Summarization of Privacy Policies Using Data Mining

Prior research shows that only a tiny percentage of users actually read the online privacy policies they implicitly agree to while using a website. Prior research also suggests that users ignore privacy policies because these policies are lengthy and, on average, require 2 years of college education to comprehend. We propose a novel technique that tackles this problem by automatically extracting summaries of online privacy policies. We use data mining models to analyze the text of privacy policies and answer 10 basic questions concerning the privacy and security of user data, what information is gathered from them, and how this information is used. In order to train the data mining models, we thoroughly study privacy policies of 400 companies (considering 10% of all listings on NYSE, Nasdaq, and AMEX stock markets) across industries. Our free Chrome browser extension, PrivacyCheck, utilizes the data mining models to summarize any HTML page that contains a privacy policy. PrivacyCheck stands out from currently available counterparts because it is readily applicable on any online privacy policy. Cross-validation results show that PrivacyCheck summaries are accurate 40% to 73% of the time. Over 400 independent Chrome users are currently using PrivacyCheck.

Introducing specification-based data structure repair using alloy

While several different techniques utilize specifications to check correctness of programs before they are deployed, the use of specifications in deployed software is more limited, largely taking the form of runtime checking where assertions form a basis for detecting erroneous program states and terminating erroneous executions in failures. Recent approaches [1] proposed constraint-based repair where data structure constraints are used to repair erroneous states. However, data structure constraints are too weak a form of specification for error recovery in general. We have developed a specification-based approach for data structure repair, which allows repairing erroneous executions in deployed software by repairing erroneous states. The key novelty is our support for rich behavioral specifications, such as those that relate pre-states with post-states to accurately specify expected behavior and hence to enable precise repair.