Sofie Haesaert

Multi-agent discrete-time graphical games and reinforcement learning solutions

This paper introduces a new class of multi-agent discrete-time dynamic games, known in the literature as dynamic graphical games. For that reason a local performance index is defined for each agent that depends only on the local information available to each agent. Nash equilibrium policies and best-response policies are given in terms of the solutions to the discrete-time coupled Hamilton-Jacobi equations. Since in these games the interactions between the agents are prescribed by a communication graph structure we have to introduce a new notion of Nash equilibrium. It is proved that this notion holds if all agents are in Nash equilibrium and the graph is strongly connected. A novel reinforcement learning value iteration algorithm is given to solve the dynamic graphical games in an online manner along with its proof of convergence. The policies of the agents form a Nash equilibrium when all the agents in the neighborhood update their policies, and a best response outcome when the agents in the neighborhood are kept constant. The paper brings together discrete Hamiltonian mechanics, distributed multi-agent control, optimal control theory, and game theory to formulate and solve these multi-agent dynamic graphical games. A simulation example shows the effectiveness of the proposed approach in a leader-synchronization case along with optimality guarantees.

Multi-agent discrete-time graphical games: interactive Nash equilibrium and value iteration solution

This paper introduces a new class of multi-agent discrete-time dynamical games known as dynamic graphical games, where the interactions between agents are prescribed by a communication graph structure. The graphical game results from multi-agent dynamical systems, where pinning control is used to make all the agents synchronize to the state of a command generator or target agent. The relation of dynamic graphical games and standard multi-player games is shown. A new notion of Interactive Nash equilibrium is introduced which holds if the agents are all in Nash equilibrium and the graph is strongly connected. The paper brings together discrete Hamiltonian mechanics, distributed multi-agent control, optimal control theory, and game theory to formulate and solve these multi-agent graphical games. The relationships between the discrete-time Hamilton Jacobi equation and discrete-time Bellman equation are used to formulate a discrete-time Hamilton Jacobi Bellman equation for dynamic graphical games. Proofs of Nash, stability, and convergence are given. A reinforcement learning value iteration algorithm is given to solve the dynamic graphical games.

Data-driven property verification of grey-box systems by bayesian experiment design

A measurement-based statistical verification approach is developed for systems with partly unknown dynamics. These grey-box systems are subject to identification experiments which, new in this contribution, enable accepting or rejecting system properties expressed in a linear-time logic. We employ a Bayesian framework for the computation of a confidence level on the properties and for the design of optimal experiments. Applied to dynamical systems, this work enables data-driven verification of partly-known system dynamics with controllable non-determinism (inputs) and noisy output observations. A numerical case study concerning the safety of a dynamical system is used to elucidate this data-driven and model-based verification technique.

Data-driven and model-based verification: A Bayesian identification approach

This work develops a measurement-driven and model-based formal verification approach, applicable to systems with partly unknown dynamics. We provide a principled method, grounded on reachability analysis and on Bayesian inference, to compute the confidence that a physical system driven by external inputs and accessed under noisy measurements verifies a temporal logic property. A case study is discussed, where we investigate the bounded- and unbounded-time safety of a partly unknown linear time invariant system.

Correct-by-design output feedback of LTI systems

Current state-of-the-art correct-by-design controllers are designed for full-state measurable systems. This work extends the applicability of correct-by-design controllers to partially observable linear, time-invariant (LTI) models. Towards the certification of the synthesised controllers, approximate simulation relations are leveraged to attain a quantification for the accuracy of introduced approximations. Additionally, the robustness of the approach allows the extension to models with the presence of probabilistic disturbances on state transitions and on output measurements. In a case study from smart buildings we evaluate the new output-based correct-by-design controller on a physical system with limited sensor information.

Approximate and Reinforcement Learning techniques to solve non-convex Economic Dispatch problems

Economic Dispatch is one of the power systems management tools. It is used to allocate an amount of power generation to the generating units to meet the active load demands. The Economic Dispatch problem is a large-scale nonlinear constrained optimization problem. In this paper, two novel techniques are developed to solve the non-convex Economic Dispatch problem. Firstly, a novel approximation of the non-convex generation cost function is developed to solve non-convex Economic Dispatch problem with the transmission losses. This approximation enables the use of gradient and Newton techniques to solve the non-convex Economic Dispatch problem. Secondly, Q-Learning with eligibility traces technique is adopted to solve the non-convex Economic Dispatch problem with valve point loading effects, multiple fuel options, and power transmission losses. The eligibility traces are used to speed up the Q-Learning process. This technique showed superior results compared to other heuristic techniques.

Data-driven and model-based verification via Bayesian identification and reachability analysis

This work develops a measurement-driven and model-based formal verification approach, applicable to dynamical systems with partly unknown dynamics. We provide a new principled method, grounded on Bayesian inference and on reachability analysis respectively, to compute the confidence that a physical system driven by external inputs and accessed under noisy measurements verifies a given property expressed as a temporal logic formula. A case study discusses the bounded- and unbounded-time safety verification of a partly unknown system, encompassed within a class of linear, time-invariant dynamical models with inputs and output measurements.

Verification of General Markov Decision Processes by Approximate Similarity Relations and Policy Refinement

In this work we introduce new approximate similarity relations that are shown to be key for policy (or control) synthesis over general Markov decision processes. The models of interest are discrete-time Markov decision processes, endowed with uncountably-infinite state spaces and metric output (or observation) spaces. The new relations, underpinned by the use of metrics, allow in particular for a useful trade-off between deviations over probability distributions on states, and distances between model outputs. We show that the new probabilistic similarity relations can be effectively employed over general Markov decision processes for verification purposes, and specifically for control refinement from abstract models.

Experiment design for formal verification via stochastic optimal control

A measurement-based statistical verification approach is developed for systems with partly unknown dynamics. Grey-box systems, which are specified as a model class, are subject to identification experiments that enable accepting or rejecting system properties expressed as formulae in a linear-time logic with a given confidence. We employ a Bayesian framework for the computation of the confidence level and for the design of experiments to increase the confidence. The experiment design is formulated as a stochastic optimal control problem, which solvable via dynamic programming. Applied to linear control systems, this work enables efficient data-driven verification of partly-known dynamics with controllable non-determinism (inputs) and noisy output observations. A numerical case study concerning the safety of a dynamical system is used to elucidate this approach.

Automated experiment design for data-efficient verification of parametric Markov decision processes

We present a new method for statistical verification of quantitative properties over a partially unknown system with actions, utilising a parameterised model (in this work, a parametric Markov decision process) and data collected from experiments performed on the underlying system. We obtain the confidence that the underlying system satisfies a given property, and show that the method uses data efficiently and thus is robust to the amount of data available. These characteristics are achieved by firstly exploiting parameter synthesis to establish a feasible set of parameters for which the underlying system will satisfy the property; secondly, by actively synthesising experiments to increase amount of information in the collected data that is relevant to the property; and finally propagating this information over the model parameters, obtaining a confidence that reflects our belief whether or not the system parameters lie in the feasible set, thereby solving the verification problem.