Benedikt Eberhardinger

Towards Testing Self-organizing, Adaptive Systems

The characteristics of self-adaptive, self-organizing systems lead to a significant higher flexibility and robustness against a changing environment. This flexibility makes it hard to test these systems adequately. To assure their quality, however, it is inevitable to do so. We introduce a new approach for systematically testing these self-* systems based on a feedback control-oriented system architecture called Corridor Enforcing Infrastructure (CEI). Thus, it is possible to examine particular situations, where the system is forced to reorganize or adapt to new situations. This is where the self-* mechanisms come into play and can be tested separately.

Runtime Model-Based Safety Analysis of Self-Organizing Systems with S#

Self-organizing systems present a challenge for model-based safety analysis techniques: At design time, the potential system configurations are unknown, making it necessary to postpone the safety analyses to runtime. At runtime, however, model checking based safety analysis techniques are often too time-consuming because of the large state spaces that have to be analyzed. Based on the S# frameworks support for runtime model adaptation, we modularize runtime safety analyses by splitting them into two parts, modeling and analyzing the self-organizing and non-self-organizing parts separately. With some additional heuristics, the resulting state space reduction facilitates the use of model checking based safety analysis techniques to analyze the safety of self-organizing systems. We outline this approach on a self-organizing production cell, assessing the self-organizations impact on the overall safety of the system.

Model-driven synthesis of monitoring infrastructure for reliable adaptive multi-agent systems

Knowledge about the current state of the system serves at least two purposes: it is the basis for decisions to act and adapt to ensure reliable operation and it can be used to verify the correctness of the system at runtime. Both purposes require that current information is available at runtime that can be evaluated. Thus, the system designers have to create a complex monitoring infrastructure that suits the purposes of the system. We propose a combination of proven techniques that can be used as the basis for such a monitoring infrastructure. We combine it with a model-driven approach that allows a model transformation of information contained in the requirements and design documents to implementations of observers and controllers that allow adaptation at runtime based on current information as well as runtime verification. The approach can be easily integrated into an iterative-incremental software engineering process and is illustrated with two complex case studies.

A Research Overview and Evaluation of Performance Metrics for Self-Organization Algorithms

Self-organization (SO) algorithms are supposed to restructure and reconfigure the system at run-time in order to empower it to fulfill its requirements under uncertain environmental conditions. For this purpose, information about the state of the environment and the system is used in feedback loops to establish a flexible, powerful system. Consequently, the performance of the SO algorithms has a significant effect on the overall performance of the system. Indeed, it is hard to design high-performing SO algorithms, because the environmental conditions the system has to operate in are partially unpredictable at design time. A crucial aid for the development of SO algorithms are tools that enable the evaluation of the algorithms performance at design time. These tools could also be used to select the best-fitting algorithm and parametrization for a specific application, among others. We show how existing performance metrics can be applied to SO algorithms by evaluating different partition-based algorithms. Based on these results, we discuss the advantages and limitations of the existing metrics and deduce requirements for performance metrics for SO algorithms.

An Approach for Isolated Testing of Self-Organization Algorithms

We provide a systematic approach for testing self-organization (SO) algorithms. The main challenges for such a testing domain are the strongly ramified state space, the possible error masking, the interleaving of mechanisms, and the oracle problem resulting from the main characteristics of SO algorithms: their inherent non-deterministic behavior on the one hand, and their dynamic environment on the other. A key to success for our SO algorithm testing framework is automation, since it is rarely possible to cope with the ramified state space manually. The test automation is based on a model-based testing approach where probabilistic environment profiles are used to derive test cases that are performed and evaluated on isolated SO algorithms. Besides isolation, we are able to achieve representative test results with respect to a specific application. For illustration purposes, we apply the concepts of our framework to partitioning-based SO algorithms and provide an evaluation in the context of an existing smart-grid application.

Back-to-Back Testing of Self-organization Mechanisms

When developing SO mechanisms, mapping requirements to actual designs and implementations demands a lot of expertise. Among other things, it is important to define the right degree of freedom for the system that allows for self-organization. Back-to-back testing supports this hard engineering task by an adequate testing method helping to reveal failures in this design and implementation procedure. Within this paper we propose a model-based approach for back-to-back testing. The approach is built on top of the S# framework and integrated into the Visual Studio development environment, enabling the creation of executable test models with comprehensive tooling support for model debugging. By applying the concepts to a self-organizing production cell, we show how it is used to fully automatically reveal faults of a SO mechanism.

Risk-Based Interoperability Testing Using Reinforcement Learning

Risk-based test strategies enable the tester to harmonize the number of specified test cases with imposed time and cost constraints. However, the risk assessment itself often requires a considerable effort of cost and time, since it is rarely automated. Especially for complex tasks such as testing the interoperability of different components it is expensive to manually assess the criticality of possible faults. We present a method that operationalizes the risk assessment for interoperability testing. This method uses behavior models of the system under test and reinforcement learning techniques to break down the criticality of given failure situations to the relevance of single system actions for being tested. Based on this risk assessment, a desired number of test cases is generated which covers as much relevance as possible. Risk models and test cases have been generated for a mobile payment system within an industrial case study.

PosoMAS: An Extensible, Modular SE Process for Open Self-organising Systems

This paper introduces PosoMAS, the Process for open, self-organising Multi-Agent Systems. The process is composed of a number of practices, reusable and customisable building parts, and integrated into the lifecycle of the Open Unified Process to yield an iterative, incremental software engineering process tailored to open self-organising systems. The individual practices are introduced and their interplay described. We evaluate PosoMAS in two case studies and provide a qualitative comparison with existing AOSE processes.

Testing Self-Organizing, Adaptive Systems

The characteristics of self-organizing, adaptive systems (SOAS) lead to a significantly higher flexibility and robustness against an ever-changing environment. This flexibility makes it hard to test these systems adequately, which is, however, inevitable in order to assure their quality. The PhD thesis faces the following key challenges for testing SOAS: state space explosion, interleaved feedback loops, and error masking. In the context of a self-organizing energy grid, we present the first results of the thesis and give an overview of the research plan.

Test suite reduction for self-organizing systems: a mutation-based approach

We study regression testing and test suite reduction for self-organizing (SO) systems. The complex environments of SO systems typically require large test suites. The physical distribution of their components and their history-dependent behavior, however, make test execution very expensive. Consequently, an efficient test suite reduction mechanism is needed. The fundamental characteristic of SO systems is their ability to reconfigure themselves. We thus in- vestigate a mutation-based approach concentrating on reconfigura- tions, more specifically the communication between the distributed components in reconfigurations. Due to distribution, we argue for an explicit consideration of higher-order mutants and find a short- cut that makes the number of test cases to execute before reduction feasible. For the reduction task, we evaluate the applicability of two existing clustering techniques, Affinity Propagation and Dissimilar- ity-based Sparse Subset Selection. It turns out that these techniques are able to drastically reduce the original test suite while retaining a good mutation score. We discuss the approach by means of a test suite for a self-organizing production cell as a running example.