Sten F. Andler

Combination Testing Strategies: A Survey ⁄

Combination strategies are test case selection methods that identify test cases by combining values of the different test object input parameters based on some combinatorial strategy. This survey presents 16 different combination strategies, covering more than 40 papers that focus on one or several combination strategies. This collection represents most of the existing work performed on combination strategies. This survey describes the basic algorithms used by the combination strategies. Some properties of combination strategies, including coverage criteria and theoretical bounds on the size of test suites, are also included in this description. This survey paper also includes a subsumption hierarchy that attempts to relate the various coverage criteria associated with the identified combination strategies. Copyright

DeeDS towards a distributed and active real-time database system

DeeDS combines active database functionality with critical timing constraints and integrated system monitoring. Since the reactive database mechanisms, or rule management system, must meet critical deadlines, we must employ methods that make triggering of rules and execution of actions predictable. We will focus on the scheduling issues associated with dynamic scheduling of workloads where the triggered transactions have hard, firm or soft deadlines, and how transient overloads may be resolved by substituting transactions by computationally cheaper ones. The rationale for a loosely coupled general purpose event monitoring facility, that works in tight connection with the scheduler, is presented. For performance and predictability, the scheduler and event monitor are executing on a separate CPU from the rest of the system. Real-time database accesses in DeeDS are made predictable and efficient by employing methods such as main memory resident data, full replication, eventual consistency, and prevention of global deadlocks.

An evaluation of combination strategies for test case selection

This paper presents results from a comparative evaluation of five combination strategies. Combination strategies are test case selection methods that combine “interesting” values of the input parameters of a test subject to form test cases. This research comparatively evaluated five combination strategies; the All Combination strategy (AC), the Each Choice strategy (EC), the Base Choice strategy (BC), Orthogonal Arrays (OA) and the algorithm from the Automatic Efficient Test Generator (AETG). AC satisfies n-wise coverage, EC and BC satisfy 1-wise coverage, and OA and AETG satisfy pair-wise coverage. The All Combinations strategy was used as a “gold standard” strategy; it subsumes the others but is usually too expensive for practical use. The others were used in an experiment that used five programs seeded with 128 faults. The combination strategies were evaluated with respect to the number of test cases, the number of faults found, failure size, and number of decisions covered. The strategy that requires the least number of tests, Each Choice, found the smallest number of faults. Although the Base Choice strategy requires fewer test cases than Orthogonal Arrays and AETG, it found as many faults. Analysis also shows some properties of the combination strategies that appear significant. The two most important results are that the Each Choice strategy is unpredictable in terms of which faults will be revealed, possibly indicating that faults are found by chance, and that the Base Choice and the pair-wise combination strategies to some extent target different types of faults.

Mutation-based testing criteria for timeliness

Temporal correctness is crucial to the dependability of real-time systems. Few methods exist to test for temporal correctness and most existing methods are ad-hoc. A problem with testing real-time applications is the dependency on the execution time and execution order of individual tasks. Thus, the response times for the tasks may be non-deterministic with respect to inputs. Conventional test coverage criteria ignore task interleaving and tinting and, thus do not help determine which execution orders need to be exercised to test for temporal correctness. This paper presents test criteria based on mutation to test timeliness. We also show how previously proposed methods in specification based testing, can be applied to testing real-time systems

Self-stabilization and eventual consistency in replicated real-time databases

Distributed databases generally have strict consistency requirements on data replicas, meaning that they are not allowed to diverge outside of transaction boundaries. For certain applications, this is too pessimistic, and it is often better to trade off consistency for higher availability, performance, or predictability. In this paper, we describe a replication protocol for a distributed database which is eventually consistent; it allows replicas to diverge if the system can be guaranteed to eventually converge to a consistent state. We also compare eventual consistency to self-stabilization, and outline how self-stabilization theory may help in proving properties of eventually consistent systems.

Continuous consistency management in distributed real-time databases with multiple writers of replicated data

We introduce a continuous convergence protocol for handling locally committed and possibly conflicting updates to replicated data. The protocol supports local consistency and predictability while allowing replicas to deterministically diverge and converge as updates are committed and replicated. We discuss how applications may exploit the protocol characteristics and describe an implementation where conflicting updates are detected, qualified by a partial update order, and resolved using application-specific forward conflict resolution.

Bounds on test effort for event-triggered real-time systems

The test effort required for full test coverage is much higher in event-triggered than in time-triggered real-time systems. Thus, it is hard to attain sufficient confidence in the correctness of event-triggered real-time applications by testing. We present a general upper bound on the test effort of constrained event-triggered real-time systems, assuming multiple resources (a refinement of previous results). The emphasis is on system-level testing of application timeliness, assuming that sufficient confidence in its functional correctness has been attained. The covered fault types are mainly incorrect assumptions about temporal attributes. An analysis of our approach shows that designated pre-emption points are required. A key factor in this approach is the ability to reduce the test effort while maintaining full test coverage.

Virtual Full Replication by Adaptive Segmentation

We propose virtual full replication by adaptive segmentation (ViFuR-A), and evaluate its ability to maintain scalability in a replicated real-time database. With full replication and eventual consistency, transaction timeliness becomes independent of network delays for all transactions. However, full replication does not scale well, since all updates must be replicated to all nodes, also when data is needed only at a subset of the nodes. With virtual full replication that adapts to actual data needs, resource usage can be bounded and the database can be made scalable. We propose a scheme for adaptive segmentation that detects new data needs and adapts replication. The scheme includes an architecture, a scalable protocol and a replicated directory service that together maintains scalability. We show that adaptive segmentation bounds the required storage at a significantly lower level compared to static segmentation, for a typical workload where the data needs change repeatedly. Adaptation time can be kept constant for the workload when there are sufficient resources. Also, the storage is constant with an increasing amount of nodes and linear with an increasing rate of change to data needs.

SMARTracIn : a concept for spoof resistant tracking of vessels and detection of adverse intentions

The aim of maritime surveillance systems is to detect threats early enough to take appropriate actions. We present the results of a study on maritime domain awareness performed during the fall of 2008. We analyze an identified capability gap of worldwide surveillance in the maritime domain, and report from a user workshop addressing the identified gap. We describe a SMARTracIn concept system that integrates information from surveillance systems with background knowledge on normal conditions to help users detect and visualize anomalies in vessel traffic. Land-based systems that cover the coastal waters as well as airborne, space-borne and ships covering open sea are considered. Sensor data are combined with intelligence information from ship reporting systems and databases. We describe how information fusion, anomaly detection and semantic technology can be used to help users achieve more detailed maritime domain awareness. Human operators are a vital part of this system and should be active components in the fusion process. We focus on the problem of detecting anomalous behavior in ocean-going traffic, and a room and door segmentation concept to achieve this. This requires the ability to identify vessels that enter into areas covered by sensors as well as the use of information management systems that allow us to quickly find all relevant information.

An empirical comparison of Bayesian and credal networks for dependable high-level information fusion

Bayesian networks are often proposed as a method for high-level information fusion. However, a Bayesian network relies on strong assumptions about the underlying probabilities. In many cases it is not realistic to require such precise probability assessments. We show that there exists a significant set of problems where credal networks outperform Bayesian networks, thus enabling more dependable decision making for this type of problems. A credal network is a graphical probabilistic method that utilizes sets of probability distributions, e.g., interval probabilities, for representation of belief. Such a representation allows one to properly express epistemic uncertainty, i.e., uncertainty that can be reduced if more information becomes available. Since reducing uncertainty has been proposed as one of the main goals of information fusion, the ability to represent epistemic uncertainty becomes an important aspect in all fusion applications.