Eitan Farchi

Automatic query wefinement using lexical affinities with maximal information gain

This work describes an automatic query refinement technique, which focuses on improving precision of the top ranked documents. The terms used for refinement are lexical affinities (LAs), pairs of closely related words which contain exactly one of the original query terms. Adding these terms to the query is equivalent to re-ranking search results, thus, precision is improved while recall is preserved. We describe a novel method that selects the most informative LAs for refinement, namely, those LAs that best separate relevant documents from irrelevant documents in the set of results. The information gain of candidate LAs is determined using unsupervised estimation that is based on the scoring function of the search engine. This method is thus fully automatic and its quality depends on the quality of the scoring function. Experiments we conducted with TREC data clearly show a significant improvement in the precision of the top ranked documents.

Using binary decision diagrams for combinatorial test design

Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. Various combinatorial testing tools exist, implementing different approaches to finding a set of tests that satisfies t-wise coverage. However, little consideration has been given to the process of defining the test space for CTD, which is usually a manual, labor-intensive, and error-prone effort. Potential errors include missing parameters and their values, wrong identification of parameters and of valid value combinations, and errors in the definition of restrictions that cause them not to capture the intended combinations. From our experience, lack of support for the test space definition process is one of the main obstacles in applying CTD to a wide range of testing domains.n In this work, we present a Cartesian product based methodology and technology that assist in defining a complete and consistent test space for CTD. We then show how using binary decision diagrams (BDDs) to represent and build the test space dramatically increases the scalability of our approach, making it applicable to large and complex real-life test design tasks, for which explicit representation of the test space is infeasible.n Finally, we show how BDDs can be used also to solve the CTD problem itself. We present a new and highly effective BDD-based approach for solving CTD, which finds a set of tests that satisfies t-wise coverage by subset selection. Our approach supports also advanced requirements such as requirements on the distribution of values in the selected tests. We apply our algorithm to real-life testing problems of varying complexity, and show its superior performance.

Helping the Tester Get It Right: Towards Supporting Agile Combinatorial Test Design

Combinatorial test design CTD is an effective test planning technique that reveals faulty feature interaction in a given system. CTD takes a systematic approach to formally model the system to be tested, and propose test cases ensuring coverage of given conditions or interactions between parameters. In this position paper we propose a framework for supporting agile CTD, a human-centered methodology, which takes into account the human testers possible mistakes and supports revision and refinement. In this approach a combinatorial model of the system and test plans are constructed in an incremental and iterative way, providing the tester with the ability to refine and validate the constructions. We propose a formal framework which can be used as a theoretical foundation for the development of agile CTD support tools, and describe a use case of an envisioned tool.

Cross-Entropy-Based Replay of Concurrent Programs

Replay is an important technique in program analysis, allowing to reproduce bugs, to track changes, and to repeat executions for better understanding of the results. Unfortunately, since re-executing a concurrent program does not necessarily produce the same ordering of events, replay of such programs becomes a difficult task. The most common approach to replay of concurrent programs is based on analyzing the logical dependencies among concurrent events and requires a complete recording of the execution we are trying to replay as well as a complete control over the programs scheduler. In realistic settings, we usually have only a partial recording of the execution and only partial control over the scheduling decisions, thus such an analysis is often impossible. In this paper, we present an approach for replay in the presence of partial information and partial control. Our approach is based on a novel application of the cross-entropy method, and it does not require any logical analysis of dependencies among concurrent events. Roughly speaking, given a partial recording R of an execution, we define a performance function on executions, which reaches its maximum on R (or any other execution that coincides with R on the recorded events). Then, the program is executed many times in iterations, on each iteration adjusting the probabilistic scheduling decisions so that the performance function is maximized. Our method is also applicable to debugging of concurrent programs, in which the program is changed before it replayed in order to increase the information from its execution. We implemented our replay method on concurrent Java programs and we show that it consistently achieves a close replay in presence of incomplete information and incomplete control, as well as when the program is changed before it is replayed.

Formal verification of concurrent software: two case studies

Software model checking came to the focus of research recently, with sharp growth in the number of safety-critical applications and in the complexity of software. In model checking of software we meet some obstacles which do not exist in model checking of hardware: the state explosion problem is more acute, the model often consists of many processes that run concurrently, and there also can be a requirement for correct behavior in the presence of failures. Also, the programs are written in high-level programming languages, which causes two problems: the model-checker has to understand a programming language,and the state space of the program can be infinite (for example, the range of values for integer values is infinite in theory). In this paper, we present two case studies of real projects at IBM, which were formally modeled and verified using ExpliSAT model checker, as well a testing tool ConTest. The second case also involves modeling limited Byzantine (malicious) failures of processes. We discuss the special structure of the control flow graph of these programs and perform probabilistic analysis of the number of random executions needed in order to execute all control flow paths with high probability. We also compare the performance of ConTest and ExpliSAT on these case studies.

Using Projections to Debug Large Combinatorial Models

Combinatorial test design (CTD) is an effective test planning technique that reveals faults resulting from parameters interactions in a system. The test space is manually modeled by a set of parameters, their respective values, and restrictions on the value combinations - referred to as a CTD model. Each possible combination of values in the cross product of the parameters, that is not excluded by restrictions, represents a valid test. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is usually a user input. In many real-life testing problems, the relationships between the different test parameters are complex. Thus, precisely capturing them by restrictions in the CTD model might be a very challenging and time consuming task. Since the test space is of exponential size in the number of parameters, it is impossible to exhaustively review all potential tests. In this paper, we present technology that supports the modeling process by enabling repeated reviews of projections of the test space on a subset of the parameters, while indicating how the value combinations under review are affected by the restrictions. In addition, we generate explanations as to why the restrictions exclude specific value combinations of the subsets of parameters under review. These explanations can be used to identify modeling mistakes, as well as to increase the understanding of the test space. Furthermore, we identify specific excluded combinations that may require special attention, and list them for review together with their corresponding exclusion explanation. To enable the review of subsets of the exponential test space, indicate their status, and identify excluded combinations for review, we use a compact representation of the test space that is based on Binary Decision Diagrams. For the generation of explanations we use satisfiability solvers. We evaluate the proposed technology on real-life CTD models and demonstrate its effectiveness.

Distributed desk checking

Desk checking is known to be an effective reading technique for early detection of sequential program errors. This paper discusses how to extend desk checking for concurrent and distributed programs. In addition to exponential possible schedules, concurrent and distributed programs have execution states that include more than one process. The new distributed desk‐checking technique supports the selection of schedules and execution states to be reviewed. The cross‐product functional coverage technique assists in the selection process. Schedule selection guidelines that facilitate early detection and coverage are introduced. It is demonstrated that code inspection can be applied effectively to large industrial applications using the selection mechanism introduced by this technique. Industrial pilots show that distributed desk checking is an effective early error‐detection review technique. Copyright

Exploring the Role of Logic and Formal Methods in Information Systems Education

This position paper contributes to the ongoing debate on the role played by logic and formal methods courses in the computing curricula. We report on an exploratory empirical study investigating the perceptions of Information Systems students on the benefits of a completed course on logic and formal specification. Participants indicated that the course had fostered their analytical thinking abilities and provided them with tools to handle abstraction and decomposition. This provides a starting point for a discourse on the benefits of formal methods courses for IS practitioners.

Choosing among alternative pasts

The primary difficulty with testing concurrent programs is their non‐determinism, where two executions with the same input can yield different results due to a changed thread schedule (also known as interleaving). This problem is aggravated by the fact that most thread schedulers are almost deterministic, and generate the same interleavings over and over for a given testing environment. The traditional approach to testing concurrent programs is to identify and examine the race conditions. A different solution involves noise‐making, which generates different interleavings at runtime, for example, using embedded sleep statements. This paper proposes a totally different technique for generating a rich set of interleavings. In this approach, operations on shared variables are tracked. Every time a shared variable is read, the value to be read is selected from the set of values that were held by this variable during the program execution. The algorithm identifies those values that the variable could hold in some interleaving consistent with the past observed events. Within this subset, the value choice can be random, biased‐random, based on coverage, etc. The problem of identifying read values that are consistent with the past observations is far from simple, since past decisions on value selection affect future ones. Our solution is computationally intensive and, therefore, impractical as is. However, insights gained from this solution lead to new heuristics for noise‐making. Copyright