Changjiang Jia

A Subsumption Hierarchy of Test Case Prioritization for Composite Services

Many composite workflow services utilize non-imperative XML technologies such as WSDL, XPath, XML schema, and XML messages. Regression testing should assure the services against regression faults that appear in both the workflows and these artifacts. In this paper, we propose a refinement-oriented level-exploration strategy and a multilevel coverage model that captures progressively the coverage of different types of artifacts by the test cases. We show that by using them, the test case prioritization techniques initialized on top of existing greedy-based test case prioritization strategy form a subsumption hierarchy such that a technique can produce more test suite permutations than a technique that subsumes it. Our experimental study of a model instance shows that a technique generally achieves a higher fault detection rate than a subsumed technique, which validates that the proposed hierarchy and model have the potential to improve the cost-effectiveness of test case prioritization techniques.

5W+1H pattern

Novices often find it difficult to kick off a mapping study or literature review.A 5W+1H pattern is proposed to ease the difficulty of conducting literature surveys.The 5W+1H pattern helps formulate initial research questions and structure reports.A case study shows the applicability of the 5W+1H pattern and the lessons learned.A three-year mapping study on cloud software testing research is presented. A common type of study used by researchers to map out the landscape of a research topic is known as mapping study. Such a study typically begins with an exploratory search on the possible ideas of the research topic, which is often done in an unsystematic manner. Hence, the activity of formulating research questions in mapping studies is ill-defined, rendering it difficult for researchers who are new to the topic. There is a need to guide them kicking off a mapping study of an unfamiliar domain. This paper proposes a 5W+1H pattern to help investigators systematically examine a generic set of dimensions in a mapping study toward the formulation of research questions before identifying, reading, and analyzing sufficient articles of the topic. We have validated the feasibility of our proposal by conducting a case study of a mapping study on cloud software testing, that is, software testing for and on cloud computing platforms. The case study reveals that the 5W+1H pattern can lead investigators to define a set of systematic, generic, and complementary research questions, enabling them to kick off and expedite the mapping study process in a well-defined manner. We also share our experiences and lessons learned from our case study on the use of the 5W+1H pattern in mapping studies.

ASN: A Dynamic Barrier-Based Approach to Confirmation of Deadlocks from Warnings for Large-Scale Multithreaded Programs

Many large-scale multithreaded programs incur deadlock bugs. Existing deadlock warning detection techniques only report warning scenarios, which may or may not be real deadlocks. Each warning should be further verified on whether it may manifest into a real deadlock. For this purpose, a number of active randomized testing schedulers have been developed to trigger them, and yet pervious experiments show that their deadlock confirmation probability can be low. This paper presents ASN, a novel barrier-based randomized scheduler that triggers real deadlocks with high probabilities. We exploit the insights that in a confirmation run, the threads involved in a real deadlock should properly acquire one or more sets of locks prior to deadlocking. ASN automatically identifies three interesting sets of such positions. It guides the threads participating in a given warning to stay at these position sets in turn. When all the threads are staying at the last position set, ASN checks whether any deadlock that matches with the given warning has been triggered. We have evaluated ASN on 15 deadlock bugs in a suite of real-world multithreaded programs. The results show that ASN either confirms more deadlocks from the benchmark suite or triggers the same deadlocks with significantly higher probabilities than existing schedulers.

Test Pair Selection for Test Case Prioritization in Regression Testing for WS-BPEL Programs

Many web services not only communicate through XML-based messages, but also may dynamically modify their behaviors by applying different interpretations on XML messages through updating the associated XML Schemas or XML-based interface specifications. Such artifacts are usually complex, allowing XML-based messages conforming to these specifications structurally complex. Testing should cost-effectively cover all scenarios. Test case prioritization is a dimension of regression testing that assures a program from unintended modifications by reordering the test cases within a test suite. However, many existing test case prioritization techniques for regression testing treat test cases of different complexity generically. In this paper, the authors exploit the insights on the structural similarity of XML-based artifacts between test cases in both static and dynamic dimensions, and propose a family of test case prioritization techniques that selects pairs of test case without replacement in turn. To the best of their knowledge, it is the first test case prioritization proposal that selects test case pairs for prioritization. The authors validate their techniques by a suite of benchmarks. The empirical results show that when incorporating all dimensions, some members of our technique family can be more effective than conventional coverage-based techniques.

Prioritizing Structurally Complex Test Pairs for Validating WS-BPEL Evolutions

Many web services represent their artifacts in the semi-structural format. Such artifacts may or may not be structurally complex. Many existing test case prioritization techniques however treat test cases of different complexity generically. In this paper, we exploit the insights on the structural similarity of XML-based artifacts between test cases, and propose a family of test case prioritization techniques that iteratively selects test case pairs without replacement. The validation experiment shows that these techniques can be more cost-effective than the studied existing techniques in exposing faults.

Using the 5W+1H Model in Reporting Systematic Literature Review: A Case Study on Software Testing for Cloud Computing

This paper documents a case study of using the 5W+1H model for reporting systematic literature review on software testing for cloud computing. To our knowledge, this is the first systematic literature review that applies the 5W+1H model, which is widely used in the journalism domain, to report the full picture of the research area in both software engineering and services computing. Existing guidelines on systematic literature review heavily rely on the researcher to pose the right research questions, and the review results are tightly focused on these research questions. For researchers new to a field, defining the right research questions that are effective in revealing the critical issues in the field can be challenging. Our case study demonstrates that the 5W+1H model provides an easy aid for the researcher to get over such initial challenges. As the researcher becomes more familiar with the field, he/she may then refine the research questions by adding more topic-specific contexts. In this way, the 5W+1H model serves to provide an exploratory framework to shape a systematic literature review. Applying to software testing for cloud computing, we are able to synthesize a comprehensive picture of recent researches on the field, including publication pattern, article citation immediacy, research topic diversity, research ideas for addressing testing challenges at different cloud service architectural layers. Based on the case study, we summarize the lessons learned on using the 5W+1H model in reporting systematic literature review.

ASP: Abstraction Subspace Partitioning for Detection of Atomicity Violations with an Empirical Study

Dynamic concurrency bug detectors predict and then examine suspicious instances of atomicity violations from executions of multithreaded programs. Only few predicted instances are real bugs. Prioritizing such instances can make the examinations cost-effective, but is there any design factor exhibiting significant influence? This work presents the first controlled experiment that studies two design factors, abstraction level and subspace, in partitioning such instances through 35 resultant partition-based techniques on 10 benchmarks with known vulnerability-related bugs. The empirical analysis reveals significant findings. First, partition-based prioritization can significantly improve the fault detection rate. Second, coarse-grained techniques are more effective than fine-grained ones, and using some one-dimensional subspaces is more effective than using other dimensional subspaces. Third, eight previously unknown techniques can be more effective than the technique modeled after a state-of-the-art dynamic detector.

Architecturing Dynamic Data Race Detection as a Cloud-Based Service

A web-based service consists of layers of programs (components) in the technology stack. Analyzing program executions of these components separately allows service vendors to acquire insights into specific program behaviors or problems in these components, thereby pinpointing areas of improvement in their offering services. Many existing approaches for testing as a service take an orchestration approach that splits components under test and the analysis services into a set of distributed modules communicating through message-based approaches. In this paper, we present the first work in providing dynamic analysis as a service using a virtual machine (VM)-based approach on dynamic data race detection. Such a detection needs to track a huge number of events performed by each thread of a program execution of a service component, making such an analysis unsuitable to use message passing to transit huge numbers of events individually. In our model, we instruct VMs to perform holistic dynamic race detections on service components and only transfer the detection results to our service selection component. With such result data as the guidance, the service selection component accordingly selects VM instances to fulfill subsequent analysis requests. The experimental results show that our model is feasible.

A Study on the Efficiency Aspect of Data Race Detection: A Compiler Optimization Level Perspective

Dynamically detecting data races in multithreaded programs incurs significant slowdown and memory overheads. Many existing techniques have been put forward to improve the performance slowdown through different dimensions such as sampling, detection precision, and data structures to track the happened-before relations among events in execution traces. Compiling the program source code with different compiler optimization options, such as reducing the object code size as the selected optimization objective, may produce different versions of the object code. Does optimizing the object code with a standard optimization option help improve the performance of the precise online race detection? To study this question and a family of related questions, this paper reports a pilot study based on four benchmarks from the PARSEC 3.0 suite compiled with six GCC compiler optimization options. We observe from the empirical data that in terms of performance slowdown, the standard optimization options behave comparably to the optimization options for speed and code size, but behave quite different from the baseline option. Moreover, in terms of memory cost, the standard optimization options incur similar memory costs as the baseline option and the option for speed, and consume less memory than the option for code size.

Which compiler optimization options should I use for detecting data races in multithreaded programs

Different compiler optimization options may produce different versions of object code. To the best of our knowledge, existing studies on concurrency bug detection in the public literature have not reported the effects of different compiler optimization options on detection effectiveness. This paper reports a preliminary but the first study in the exploratory nature to investigate this aspect. The study examines the happened-before based predictive data race detection scenarios on four benchmarks from the PARSEC 3.0 suite compiled under six different GNU GCC optimization options. We observe from the data set that the same race detection technique may produce different sets of races or different detection probabilities under different optimization scenarios. Based on the observations, we formulate two hypotheses for future investigations.