Sreedevi Sampath

Developing a Single Model and Test Prioritization Strategies for Event-Driven Software

Event-Driven Software (EDS) can change state based on incoming events; common examples are GUI and Web applications. These EDSs pose a challenge to testing because there are a large number of possible event sequences that users can invoke through a user interface. While valuable contributions have been made for testing these two subclasses of EDS, such efforts have been disjoint. This work provides the first single model that is generic enough to study GUI and Web applications together. In this paper, we use the model to define generic prioritization criteria that are applicable to both GUI and Web applications. Our ultimate goal is to evolve the model and use it to develop a unified theory of how all EDS should be tested. An empirical study reveals that the GUI and Web-based applications, when recast using the new model, show similar behavior. For example, a criterion that gives priority to all pairs of event interactions did well for GUI and Web applications; another criterion that gives priority to the smallest number of parameter value settings did poorly for both. These results reinforce our belief that these two subclasses of applications should be modeled and studied together.

Automated replay and failure detection for web applications

User-session-based testing of web applications gathers user sessions to create and continually update test suites based on real user input in the field. To support this approach during maintenance and beta testing phases, we have built an automated framework for testing web-based software that focuses on scalability and evolving the test suite automatically as the applications operational profile changes. This paper reports on the automation of the replay and oracle components for web applications, which pose issues beyond those in the equivalent testing steps for traditional, stand-alone applications. Concurrency, nondeterminism, dependence on persistent state and previous user sessions, a complex application infrastructure, and a large number of output formats necessitate developing different replay and oracle comparator operators, which have tradeoffs in fault detection effectiveness, precision of analysis, and efficiency. We have designed, implemented, and evaluated a set of automated replay techniques and oracle comparators for user-session-based testing of web applications. This paper describes the issues, algorithms, heuristics, and an experimental case study with user sessions for two web applications. From our results, we conclude that testers performing user-session-based testing should consider their expectations for program coverage and fault detection when choosing a replay and oracle technique.

Applying Concept Analysis to User-Session-Based Testing of Web Applications

The continuous use of the Web for daily operations by businesses, consumers, and the government has created a great demand for reliable Web applications. One promising approach to testing the functionality of Web applications leverages the user-session data collected by Web servers. User-session-based testing automatically generates test cases based on real user profiles. The key contribution of this paper is the application of concept analysis for clustering user sessions and a set of heuristics for test case selection. Existing incremental concept analysis algorithms are exploited to avoid collecting and maintaining large user-session data sets and to thus provide scalability. We have completely automated the process from user session collection and test suite reduction through test case replay. Our incremental test suite update algorithm, coupled with our experimental study, indicates that concept analysis provides a promising means for incrementally updating reduced test suites in response to newly captured user sessions with little loss in fault detection capability and program coverage.

Prioritizing User-Session-Based Test Cases for Web Applications Testing

Web applications have rapidly become a critical part of business for many organizations. However, increased usage of Web applications has not been reciprocated with corresponding increases in reliability. Unique characteristics, such as quick turnaround time, coupled with growing popularity motivate the need for efficient and effective Web application testing strategies. In this paper, we propose several new test suite prioritization strategies for Web applications and examine whether these strategies can improve the rate of fault detection for three Web applications and their preexisting test suites. We prioritize test suites by test lengths, frequency of appearance of request sequences, and systematic coverage of parameter-values and their interactions. Experimental results show that the proposed prioritization criteria often improve the rate of fault detection of the test suites when compared to random ordering of test cases. In general, the best prioritization metrics either (1) consider frequency of appearance of sequences of requests or (2) systematically cover combinations of parameter-values as early as possible.

A scalable approach to user-session based testing of Web applications through concept analysis

The continuous use of the Web for daily operations by businesses, consumers, and government has created a great demand for reliable Web applications. One promising approach to testing the functionality of Web applications leverages user-session data collected by Web servers. This approach automatically generates test cases based on real user profiles. The key contribution of This work is the application of concept analysis for clustering user sessions for test suite reduction. Existing incremental concept analysis algorithms can be exploited to avoid collecting large user-session data sets and thus provide scalability. We have completely automated the process from user session collection and reduction through replay. Our incremental test suite update algorithm coupled with our experimental study indicate that concept analysis provides a promising means for incrementally updating reduced test suites in response to newly captured user sessions with some loss in fault detection capability and practically no coverage loss.

An empirical comparison of test suite reduction techniques for user-session-based testing of Web applications

Automated cost-effective test strategies are needed to provide reliable, secure, and usable Web applications. As a software maintainer updates an application, test cases must accurately reflect usage to expose faults that users are most likely to encounter. User-session-based testing is an automated approach to enhancing an initial test suite with real user data, enabling additional testing during maintenance as well as adding test data that represents usage as operational profiles evolve. Test suite reduction techniques are critical to the cost effectiveness of user-session-based testing because a key issue is the cost of collecting, analyzing, and replaying the large number of test cases generated from user-session data. We performed an empirical study comparing the test suite size, program coverage, fault detection capability, and costs of three requirements-based reduction techniques and three variations of concept analysis reduction applied to two Web applications. The statistical analysis of our results indicates that concept analysis-based reduction is a cost-effective alternative to requirements-based approaches.

A combinatorial approach to building navigation graphs for dynamic web applications

Modeling the navigation structure of a dynamic web application is a challenging task because of the presence of dynamic pages. In particular, there are two problems to be dealt with: (1) the page explosion problem, i.e., the number of dynamic pages may be huge or even infinite; and (2) the request generation problem, i.e., many dynamic pages may not be reached unless appropriate user requests are supplied. As a user request typically consists of multiple parameter values, the request generation problem can be further divided into two problems: (1) How to select appropriate values for individual parameters? (2) How to effectively combine individual parameter values to generate requests? This paper presents a combinatorial approach to building a navigation graph. The novelty of our approach is two-fold. First, we use an abstraction scheme to control the page explosion problem. In this scheme, pages that are likely to have the same navigation behavior are grouped together, and are represented as a single node in a navigation graph. Grouping pages reduces and bounds the size of a navigation graph for practical applications. Second, assuming that values of individual parameters are supplied by using other techniques or generated manually by the user, we combine parameter values in a way that achieves a well-defined combinatorial coverage called pairwise coverage. Using pairwise coverage can significantly reduce the number of requests that have to be submitted while still achieving effective coverage of the navigation structure. We report a prototype tool called Tansuo, and apply the tool to five open source web applications. Our empirical results indicate that Tansuo can efficiently generate web navigation graphs for these applications.

Composing a framework to automate testing of operational Web-based software

Low reliability in Web-based applications can result in detrimental effects for business, government, and consumers as they become increasingly dependent on the Internet for routine operations. A short time to market, large user community, demand for continuous availability, and frequent updates motivate automated, cost-effective testing strategies. To investigate the practical tradeoffs of different automated strategies for key components of the Web-based software testing process, we have designed a framework for Web-based software testing that focuses on scalability and evolving the test suite automatically as the applications operational profile changes. We have developed an initial prototype that not only demonstrates how existing tools can be used together but provides insight into the cost effectiveness of the overall approach. This paper describes the testing framework, discusses the issues in building and reusing tools in an integrated manner, and presents a case study that exemplifies the usability, costs, and scalability of the approach.

An Interaction-Based Test Sequence Generation Approach for Testing Web Applications

Web applications often use dynamic pages that interact with each other by accessing shared objects, e.g., session objects. Interactions between dynamic pages need to be carefully tested, as they may give rise to subtle faults that cannot be detected by testing individual pages in isolation. Since it is impractical to test all possible interactions, a trade-off must be made between test coverage (in terms of number of interactions covered in the tests) and test effort. In this paper, we present a test sequence generation approach to cover all pairwise interactions, i.e., interactions between any two pages. Intuitively, if a page P could reach another page Ppsila, there must exist a test sequence in which both P and Ppsila are visited in the given order. We report a test sequence generation algorithm and two case studies in which test sequences are generated to achieve pairwise interaction coverage for two Web applications. The empirical results indicate that our approach achieves good code coverage and is effective for detecting interaction faults in the subject applications.

Test suite prioritization by cost-based combinatorial interaction coverage

Test suite prioritization techniques modify the order in which tests within a test suite run. The goal is to order tests such that they detect faults as early as possible in the test execution cycle. Prioritization by combinatorial interaction coverage is a recent criterion that has been useful for prioritizing test suites for GUI and web applications. While studies show that this prioritization criterion can be valuable, previous studies compute the interaction coverage without considering the cost of individual tests. This paper proposes a new cost-based combinatorial interaction coverage metric, an algorithm to compute the new metric, and an empirical study with three subject web applications. Two of our studies show that prioritization by the new metric improves the rate at which faults are detected in relation to cost. A third study reveals an interesting result that the success of the cost-based metric is influenced by the distribution of t-tuples in the selected test cases.