Stefan Rudolph

Reinforcement Learning for Coverage Optimization Through PTZ Camera Alignment in Highly Dynamic Environments

In recent years, Smart Camera Networks are a field of intensive research, because of the versatile application areas of such systems. This work contributes to realize such applications by focusing on the problem of optimizing the coverage in a pan tilt zoom (PTZ) camera network during runtime. We propose to approach this problem with Reinforcement Learning (RL) techniques. Therefore, we first introduce our underlying model and its RL context. Then, we present the fairly new RL algorithm Distributed W-Learning, which is specialized for Multi Agent Systems. We compared the algorithm against current non-learning, state-of-the-art algorithms. The paper demonstrates the potential benefit of applying RL to coverage problems. Especially the performance of PTZ cameras in highly dynamic environments can be increased significantly.

A Mutual Influence Detection Algorithm for Systems with Local Performance Measurement

Self-adapting and self-organizing systems with local neighbor ship relations can face implicit mutual influences that may prevent an optimal behavior of the system. In this work, we present a novel algorithm that detects these influences in systems with local performance measures. We demonstrate the benefit of this algorithm by applying it to a smart camera network scenario, where neighbored cameras share parts of their fields of view. Furthermore, we discuss the gain of performance in learning camera control strategies.

A Cognitive-Inspired Model for Self-Organizing Networks

In this work we propose a computational scheme inspired by the workings of human cognition. We embed some fundamental aspects of the human cognitive system into this scheme in order to obtain a minimization of computational resources and the evolution of a dynamic knowledge network over time, and apply it to computer networks. Such algorithm is capable of generating suitable strategies to explore huge graphs like the Internet that are too large and too dynamic to be ever perfectly known. The developed algorithm equips each node with a local information about possible hubs which are present in its environment. Such information can be used by a node to change its connections whenever its fitness is not satisfying some given requirements. Eventually, we compare our algorithm with a randomized approach within an ecological scenario for the ICT domain, where a network of nodes carries a certain set of objects, and each node retrieves a subset at a certain time, constrained with limited resources in terms of energy and bandwidth. We show that a cognitive-inspired approach improves the overall networks topology better than a randomized algorithm.

Design and Evaluation of an Extended Learning Classifier-Based StarCraft Micro AI

Due to the manifold challenges that arise when developing an artificial intelligence that can compete with human players, the popular realtime-strategy game Starcraft: Broodwar (BW) has received attention from the computational intelligence research community. It is an ideal testbed for methods for self-adaption at runtime designed to work in complex technical systems. In this work, we utilize the broadlys-used Extended Classifier System (XCS) as a basis to develop different models of BW micro AIs: the Defender, the Attacker, the Explorer and the Strategist. We evaluate theses AIs with a focus on their adaptive and co-evolutionary behaviors. To this end, we stage and analyze the outcomes of a tournament among the proposed AIs and we also test them against a non-adaptive player to provide a proper baseline for comparison and learning evolution. Of the proposed AIs, we found the Explorer to be the best performing design, but, also that the Strategist shows an interesting behavioral evolution.

Towards Autonomous Self-Tests at Runtime

Self-adaptive and self-organising (SASO) systems are one promising approach to counter the raising interconnectedness and complexity in technical systems [1]. In particular, decisions about parametrisation, behaviour, and even structure are moved into the responsibility of the systems themselves: from design-time to runtime. This means that hardly all conditions a system may face can be foreseen during development. Consequently, a full test coverage at design-time is seldom possible as well. We argue that such a transfer of decisions to runtime also impacts the approach to test the resulting systems. If conditions, interaction partners, and resulting behaviour occur only at runtime, testing these aspects has to occur at runtime as well. Besides standard functionality tests (e.g. watchdogs), the distributed and component-oriented nature of self-organising systems can be re-used for highly autonomous and adaptive test mechanisms by establishing and dissolving relationships of test-tester pairs at runtime. The basic idea is to augment more static self-tests (i.e. integrated in hardware, or internal software routines) with dynamic tests - ranging from availability tests at the lowest level to comprising tests at the highest level (i.e. verifying that a component has not been taken over by an attacker). Since we want to avoid single points of failure and maintain scalability in large-scale self-organised systems, we propose a fully self-organised approach, the autonomous self-tests. In this article, we sketch a concept to runtime testing in terms of self-tests, briefly summarise the state-of-the-art, and illustrate the idea by means of examples from the smart camera and intrusion detection domains.

Self-learning Smart Cameras - Harnessing the Generalization Capability of XCS.

In this paper, we show how an evolutionary rule-based machine learning technique can be applied to tackle the task of self-configuration of smart camera networks. More precisely, the Extended Classifier System (XCS) is utilized to learn a configuration strategy for the pan, tilt, and zoom of smart cameras. Thereby, we extend our previous approach, which is based on Q-Learning, by harnessing the generalization capability of Learning Classifier Systems (LCS), i.e. avoiding to separately approximate the quality of each possible (re-)configuration (action) in reaction to a certain situation (state). Instead, situations in which the same reconfiguration is adequate are grouped to one single rule. We demonstrate that our XCS-based approach outperforms the Q-learning method on the basis of empirical evaluations on scenarios of different severity.

Modeling Epidemic Risk Perception in Networks with Community Structure

We study the influence of global, local and community-level risk perception on the extinction probability of a disease in several models of social networks. In particular, we study the infection progression as a susceptible-infected-susceptible (SIS) model on several modular networks, formed by a certain number of random and scale-free communities. We find that in the scale-free networks the progression is faster than in random ones with the same average connectivity degree. For what concerns the role of perception, we find that the knowledge of the infection level in one’s own neighborhood is the most effective property in stopping the spreading of a disease, but at the same time the more expensive one in terms of the quantity of required information, thus the cost/effectiveness optimum is a tradeoff between several parameters.