Willa K. Ehrlich

Applying design of experiments to software testing: experience report

Recently, a class of experimental designs has been devised that guarantee input domain coverage up to all combinations of k test factors taken t at a time. With such designs, all pair-wise combinations (or triplets or quadruplets, etc.) are selected at least once. To evaluate their applicability to software testing, we analyzed the extent to which software coverage (i.e., code execution) achieved by these designs for r = 1 ,...,k is representative of that achieved by exhaustively testing all factor combinations. The block coverage obtained for tS2 was comparable with that achieved by exhaustively testing all factor combinations but higher-order values of t were required for path coverage. Implications of these results for software testing are discussed. field, and that there is a total of 151 characters that can be entered across all 20 screen fields, then there are 95’5’~4.3x102g8 different inputs or test cases corresponding to the different ways a user can populate the screen. Clearly, exhaustive input testing is not feasible and hence, a software tester needs to efficiently generate an effective set of test cases as a means of verifying the correct operation of the software.

End to End Performance Modeling of Web Server Architectures

Web server performance in a distributed Object-Oriented (OO) environment is a complex interplay between a variety of factors (e.g., hardware platform, threading model, object scope model, server operating system, network bandwidth, disk file size, caching). In this paper, we present a model-based approach to Web Server performance evaluation in terms of an end-to-end queueing model implemented in a simulation tool. We have applied this model to Active Server Page (ASP) and Common Object Model (COM) technology in Microsofts Internet Information Server and to the Java Server Page (JSP) and JavaBean technology in both IIS and Netscape Enterprise Server (NES). Our results indicate that for the ASP Script Engine, performance predictions from the simulation model matched the performance observed in a test environment. However, for the JSP Script Engine, the model predicted higher throughput than laboratory test results at high load. This result suggests that Web Server performance can be severely limited by a software bottleneck that causes requests to be serialized. This may cause a request to wait for some resource (i.e., a lock) as opposed to consuming CPU or memory. Implications of these results for Web Server performance in general are discussed

Software reliability assessment using accelerated testing methods

The use of operational profiles and usage-based testing has received considerable attention recently in the software engineering literature. Testing under the actual operational profile can, however, be expensive, time consuming or even infeasible in situations where the performance of a system is dominated by infrequent but highly critical events. We consider a real application that deals with telecommunications network restoration after network failure caused by cuts in fibre optic cables. We use this application to demonstrate the usefulness of traditional accelerated testing methods to test and estimate software reliability. These methods, which have been extensively used in hardware reliability, have an important role to play in software reliability assessment as well.