James A. Whittaker

A Markov chain model for statistical software testing

Statistical testing of software establishes a basis for statistical inference about a software systems expected field quality. This paper describes a method for statistical testing based on a Markov chain model of software usage. The significance of the Markov chain is twofold. First, it allows test input sequences to be generated from multiple probability distributions, making it more general than many existing techniques. Analytical results associated with Markov chains facilitate informative analysis of the sequences before they are generated, indicating how the test is likely to unfold. Second, the test input sequences generated from the chain and applied to the software are themselves a stochastic model and are used to create a second Markov chain to encapsulate the history of the test, including any observed failure information. The influence of the failures is assessed through analytical computations on this chain. We also derive a stopping criterion for the testing process based on a comparison of the sequence generating properties of the two chains. >

What is software testing? And why is it so hard?

The author sheds some light on why testing todays software products is so challenging, and he identifies several solid approaches that all testers should be able to thoughtfully apply. The effecti...The author sheds some light on why testing todays software products is so challenging, and he identifies several solid approaches that all testers should be able to thoughtfully apply. The effective tester has a rich toolkit of fundamental testing techniques, understands how the product will be used in its operating environment, and has a nose for where subtle bugs might lurk in the product and a bag of tricks for flushing them out. The methods described can help testers provide a sensible answer to the question of what they really mean when they say they have finished testing a software system.

Markov analysis of software specifications

A procedure for modeling software usage with the finite state, discrete parameter Markov chain is described. It involves rigorous analysis of the specification before design and coding begin. Many benefits emerge from this process, including the ability to synthesize a macro level usage distribution from a micro level understanding of how the software will be used. This usage distribution becomes the basis for a statistical test of the software, which is fundamental to the Cleanroom development process. Some analytical results known for Markov chains that have meaningful implications and interpretations for the software development process are described.

Toward a more reliable theory of software reliability

The notions of time and the operational profile incorporated into software reliability are incomplete. Reliability should be redefined as a function of application complexity, test effectiveness, and operating environment. We do not yet have a reliability equation that application complexity, test effectiveness, test suite diversity, and a fuller definition of the operational profile. We challenge the software reliability community to consider these ideas in future models. The reward for successfully doing so likely will be the widespread adoption of software reliability prediction by the majority of software publishers.

Stochastic software testing

This paper presents a method for test case selection that allows a formal approach to testing software. The two main ideas are (1) that testers create stochastic models of software behavior instead of crafting individual test cases and (2) that specific test cases are generated from the stochastic models and applied to the software under test. This paper describes a method for creating a stochastic model in the context of a solved example. We concentrate on Markov models and show how non‐Markovian behavior can be embedded in such models without violating the Markov property.

Building software securely from the ground up

0 7 4 0 7 4 5 9 / 0 2 /

50 years of software: key principles for quality

1 7 . 0 0

Software security vulnerability testing in hostile environments

Software quality is no better today than it was decades ago. In some cases, its worse. A look at the past may help us change the future for the better. Although a complete catalog of the past 50 years would be impossible in this short article, we give a decade-by-decade synopsis of software development theory and practice, focusing particularly on attitudes and trends that have shaped current software development methods. Perhaps by examining past trends-successes and failures-we can uncover clues as to what avenues to explore in improving future software systems.

Why secure applications are difficult to write

Traditional Black box software testing can be effective at exposing some classes of software failures. Security class failures, however, do not tend to manifest readily using these techniques. The problem is that many security failures occur in stressed environments, which appear in the field, but are often neglected during testing because of the difficulty to simulate these conditions. Software can only be considered secure if it behaves securely under all operating environments. Hostile environment testing must thus be a part of any overall testing strategy. This paper describes this necessity and a black box approach for creating such environments in order to expose security vulnerabilities.

A constraint-based approach to the representation of software usage models

The author categorizes application security concerns into four specific issues: input streams, output production, internal data, and algorithms and computation. The first two concerns - input and output - are related to the environment in which applications execute. The last two - data and algorithms - are related to an applications awareness of its own internal secrets. Those secrets could be the data an application stores or the algorithm it uses to perform its work. All four issues relate to awareness: secure software must always be aware of what is going on, both inside its perimeter and out to respond effectively to malicious threats.