Shin Hong

Testing concurrent programs to achieve high synchronization coverage

The effectiveness of software testing is often assessed by measuring coverage of some aspect of the software, such as its code. There is much research aimed at increasing code coverage of sequential software. However, there has been little research on increasing coverage for concurrent software. This paper presents a new technique that aims to achieve high coverage of concurrent programs by generating thread schedules to cover uncovered coverage requirements. Our technique first estimates synchronization-pair coverage requirements, and then generates thread schedules that are likely to cover uncovered coverage requirements. This paper also presents a description of a prototype tool that we implemented in Java, and the results of a set of studies we performed using the tool on a several open-source programs. The results show that, for our subject programs, our technique achieves higher coverage faster than random testing techniques; the estimation-based heuristic contributes substantially to the effectiveness of our technique.

Detecting Concurrency Errors in Client-Side Java Script Web Applications

As web technologies have evolved, the complexity of dynamic web applications has increased significantly and web applications suffer concurrency errors due to unexpected orders of interactions among web browsers, users, the network, and so forth. In this paper, we present WAVE (Web Applications Virtual Environment), a testing framework to detect concurrency errors in client-side web applications written in JavaScript. WAVE generates various sequences of operations as test cases for a web application and executes a sequence of operations by dynamically controlling interactions of a target web application with the execution environment. We demonstrate that WAVE is effective and efficient for detecting concurrency errors through experiments on eight examples and five non-trivial real-world web applications.

Understanding user understanding: determining correctness of generated program invariants

Recently, work has begun on automating the generation of test oracles, which are necessary to fully automate the testing process. One approach to such automation involves dynamic invariant generation which extracts invariants from program executions. To use such invariants as test oracles, however, it is necessary to distinguish correct from incorrect invariants, a process that currently requires human intervention. In this work we examine this process. In particular, we examine the ability of 30 users, across two empirical studies, to classify invariants generated from three Java programs. Our results indicate that users struggle to classify generated invariants: on average, they misclassify 9.1% to 31.7% of correct invariants and 26.1%-58.6% of incorrect invariants. These results contradict prior studies that suggest that classification by users is easy, and indicate that further work needs to be done to bridge the gap between the effectiveness of dynamic invariant generation in theory, and the ability of users to apply it in practice. Along these lines, we suggest several areas for future work.

The Impact of Concurrent Coverage Metrics on Testing Effectiveness

When testing multithreaded programs, the number of possible thread interactions makes exploring all interactions infeasible in practice. In response, researchers have developed concurrent coverage metrics for multithreaded programs. These metrics allow them to estimate how well they have exercised concurrent program behavior, just as branch and statement coverage metrics do for sequential program testing. However, unlike sequential coverage metrics, the effectiveness of concurrent coverage metrics in testing remains largely unexamined. In this paper, we explore the relationship between concurrent coverage and fault detection effectiveness by studying the application of eight concurrent coverage metrics in testing nine concurrent programs. Our results show that existing concurrent coverage metrics are often moderate to strong predictors of concurrent testing effectiveness, and are generally reasonable targets for test suite generation. Nevertheless, their relative effectiveness as predictors and test generation targets varies across programs, and thus additional work is needed in this area.

Mutation-Based Fault Localization for Real-World Multilingual Programs (T)

Programmers maintain and evolve their software in a variety of programming languages to take advantage of various control/data abstractions and legacy libraries. The programming language ecosystem has diversified over the last few decades, and non-trivial programs are likely to be written in more than a single language. Unfortunately, language interfaces such as Java Native Interface and Python/C are difficult to use correctly and the scope of fault localization goes beyond language boundaries, which makes debugging multilingual bugs challenging. To overcome the aforementioned limitations, we propose a mutation-based fault localization technique for real-world multilingual programs. To improve the accuracy of locating multilingual bugs, we have developed and applied new mutation operators as well as conventional mutation operators. The results of the empirical evaluation for six non-trivial real-world multilingual bugs are promising in that the proposed technique identifies the buggy statements as the most suspicious statements for all six bugs.

Are concurrency coverage metrics effective for testing: a comprehensive empirical investigation

Testing multithreaded programs is inherently challenging, as programs can exhibit numerous thread interactions. To help engineers test these programs cost‐effectively, researchers have proposed concurrency coverage metrics. These metrics are intended to be used as predictors for testing effectiveness and provide targets for test generation. The effectiveness of these metrics, however, remains largely unexamined. In this work, we explore the impact of concurrency coverage metrics on testing effectiveness and examine the relationship between coverage, fault detection, and test suite size. We study eight existing concurrency coverage metrics and six new metrics formed by combining complementary metrics. Our results indicate that the metrics are moderate to strong predictors of testing effectiveness and effective at providing test generation targets. Nevertheless, metric effectiveness varies across programs, and even combinations of complementary metrics do not consistently provide effective testing. These results highlight the need for additional work on concurrency coverage metrics. Copyright

A survey of race bug detection techniques for multithreaded programmes

As multithreaded programmes become popular to fully utilize multicore CPUs, many race bug detection techniques have been developed to find concurrency errors in multithreaded programmes effectively. Because these techniques have different views on target programme execution and detect race bugs of various types, it is difficult to characterize, compare and improve race bug detection techniques. This paper presents a formal execution model, which can uniformly represent various views of race bug detection techniques on target programme execution. Then, this paper classifies 43 race bug detection techniques according to their target race bugs. We classify race bugs on whether or not a bug violates operation block specification and/or data association specification. This survey provides researchers with a clear top‐down view of various race bug detection techniques. In addition, the concrete examples of various race bugs in this survey can help field engineers avoid race bugs in their multithreaded programmes. Copyright

Effective pattern-driven concurrency bug detection for operating systems

Highlights? We identified characteristics of concurrency bugs in Linux kernel by reviewing ChangeLog bug reports. ? We developed a pattern-driven COncurrency Bug dETector (COBET) that utilizes composite bug patterns with semantic conditions to detect complex bugs. ? TWe defined four concurrency bug patterns and detected ten new bugs in Linux kernel. As multi-core hardware has become more popular, concurrent programming is being more widely adopted in software. In particular, operating systems such as Linux utilize multi-threaded techniques heavily to enhance performance. However, current analysis techniques and tools for validating concurrent programs often fail to detect concurrency bugs in operating systems (OSes) due to the complex characteristics of OSes. To detect concurrency bugs in OSes in a practical manner, we have developed the COncurrency Bug dETector (COBET) framework based on composite bug patterns augmented with semantic conditions. The effectiveness, efficiency, and applicability of COBET were demonstrated by detecting 10 new bugs in file systems, device drivers, and network modules of Linux 2.6.30.4 as confirmed by the Linux maintainers.

MUSEUM: Debugging real-world multilingual programs using mutation analysis

Context: The programming language ecosystem has diversified over the last few decades. Non-trivial programs are likely to be written in more than a single language to take advantage of various control/data abstractions and legacy libraries. Objective: Debugging multilingual bugs is challenging because language interfaces are difficult to use correctly and the scope of fault localization goes beyond language boundaries. To locate the causes of real-world multilingual bugs, this article proposes a mutation-based fault localization technique (MUSEUM). Method: MUSEUM modifies a buggy program systematically with our new mutation operators as well as conventional mutation operators, observes the dynamic behavioral changes in a test suite, and reports suspicious statements. To reduce the analysis cost, MUSEUM selects a subset of mutated programs and test cases. Results: Our empirical evaluation shows that MUSEUM is (i) effective: it identifies the buggy statements as the most suspicious statements for both resolved and unresolved non-trivial bugs in real-world multilingual programming projects; and (ii) efficient: it locates the buggy statements in modest amount of time using multiple machines in parallel. Also, by applying selective mutation analysis (i.e., selecting subsets of mutants and test cases to use), MUSEUM achieves significant speedup with marginal accuracy loss compared to the full mutation analysis. Conclusion: It is concluded that MUSEUM locates real-world multilingual bugs accurately. This result shows that mutation analysis can provide an effective, efficient, and language semantics agnostic analysis on multilingual code. Our light-weight analysis approach would play important roles as programmers write and debug large and complex programs in diverse programming languages.

Model-based Kernel Testing for Concurrency Bugs through Counter Example Replay

Despite the growing need for customized operating system kernels for embedded devices, kernel development continues to suffer from high development and testing costs for several reasons, including the high complexity of the kernel code, the infeasibility of unit testing, exponential numbers of concurrent behaviors, and a lack of proper tool support. To alleviate these difficulties, this study proposes the MOdel-based KERnel Testing (MOKERT) framework, which supports detection of concurrency bugs in the kernel by combining both model checking techniques and testing methods. The MOKERT framework was applied to the file systems of the Linux 2.6 kernel and found a data race bug in the proc file system.