Jan Skach

Infinite time horizon active fault diagnosis based on approximate dynamic programming

The paper deals with designing an approximate active fault detector for stochastic linear Markovian switching systems over an infinite time horizon. The problem is formulated as a functional optimization problem that can be solved using approximate dynamic programming. First, the Generalized Pseudo Bayes (GPB) algorithm is employed to solve the state estimation problem. Then the original formulation is restated by introducing a hyper-state that comprises a finite dimensional statistics obtained from the GPB algorithm. Since the hyper-state is of a higher dimension, a nonparametric local approximation of the Bellman function is used together with the value iteration algorithm to design the approximate active fault detector. The performance of the designed approximate active fault detector is demonstrated through a numerical example.

Approximation methods for optimal active fault detection

In this paper the problem of active fault detection for a non-linear discrete-time stochastic system over an infinite time horizon is considered. The faults in the system are modeled by switching between fault-free and finitely many faulty models. The optimal active fault detector is derived by reformulating the original problem and solving corresponding Bellmans functional equation. A suboptimal solution is based on application of the value iteration or the policy iteration algorithms and a representation of the continuous hyper-state space by a uniform grid. A comparison of both algorithms for solving Bellmans functional equation is illustrated in a numerical example.

Actor-Critic Off-Policy Learning for Optimal Control of Multiple-Model Discrete-Time Systems

In this paper, motivated by human neurocognitive experiments, a model-free off-policy reinforcement learning algorithm is developed to solve the optimal tracking control of multiple-model linear discrete-time systems. First, an adaptive self-organizing map neural network is used to determine the system behavior from measured data and to assign a responsibility signal to each of system possible behaviors. A new model is added if a sudden change of system behavior is detected from the measured data and the behavior has not been previously detected. A value function is represented by partially weighted value functions. Then, the off-policy iteration algorithm is generalized to multiple-model learning to find a solution without any knowledge about the system dynamics or reference trajectory dynamics. The off-policy approach helps to increase data efficiency and speed of tuning since a stream of experiences obtained from executing a behavior policy is reused to update several value functions corresponding to different learning policies sequentially. Two numerical examples serve as a demonstration of the off-policy algorithm performance.

Approximate active fault detection and control

This paper deals with approximate active fault detection and control for nonlinear discrete-time stochastic systems over an infinite time horizon. Multiple model framework is used to represent fault-free and finitely many faulty models. An imperfect state information problem is reformulated using a hyper-state and dynamic programming is applied to solve the problem numerically. The proposed active fault detector and controller is illustrated in a numerical example of an air handling unit.

Optimal active fault diagnosis by temporal-difference learning

In this paper, a novel solution to the active fault diagnosis problem for stochastic linear Markovian switching systems on the infinite-time horizon is proposed. The imperfect state information problem of designing an active fault detector that minimizes a general detection cost criterion is reformulated as the perfect state information problem using sufficient statistics. The reformulation decreases theoretical complexity and enables to find a suboptimal solution by dynamic programming. However, classical approaches are computationally complex or fail to identify the most representative states of the system. This paper combines the active fault detection, state estimation, and reinforcement learning. In the proposed algorithm, temporal difference learning is used to train the active fault detector based on input-output data from the system simulation. The designed detector can be then used online. A numerical example is presented to verify the proposed algorithm.

Temporal-difference Q-learning in active fault diagnosis

The paper deals with a novel design of an approximate active fault detector for discrete-time stochastic linear Markovian switching systems on the infinite-time horizon. The problem is formulated as an optimization problem with the aim to minimize a general discounted detection cost criterion. The proposed solution is inspired by approximate dynamic programming and reinforcement learning. The active fault detector is trained by a temporal-difference Q-learning algorithm with a linear parametric Q-function approximation adjusted to fit the true Q-function. The main advantage is that this approach is computationally less expensive than a temporal-difference learning with a value function.

Active fault detection: A comparison of probabilistic methods

The paper deals with probabilistic methods for designing the active fault detectors that improve the quality of detection using an auxiliary input signal. Two probabilistic methods that assume a similar stochastic model of a monitored system are considered and compared with a special attention to various difficulties associated with active fault detector designs. The active fault detector design based on a general detection cost function is compared with the model sequence selection error minimization design in terms of assumptions and theoretical properties. Practical aspects of both methods are also considered and demonstrated through a numerical example.

Nonlinear analysis of position estimate in global navigation satellite systems

The paper presents a nonlinear analysis of position estimation based on a global navigation satellite system. A classical problem formulation and iterative solution that results in the weighted least squares estimate of the receiver state are assumed. The analysis employs the Taylors theorem to express the nonlinear measurement model using the first order Taylor polynomial at the state estimate and the Lagrange form of the remainder. A sensitivity analysis of the Jacobian matrix pseudoinverse is performed, and an upper bound on the size of the Lagrange remainder is derived using eigenstructure of the Hessian matrix. The results obtained show that both the sensitivity of the pseudoinverse and the size of the quadratic term are not significant, and thus the linear approximation commonly used to derive stochastic properties of the state estimate is reasonable. Although this result has been experimentally confirmed by numerous successful applications, this analysis can serve as a more rigorous basis when the design procedures for a safety critical system have to be satisfied.