William M. McEneaney

Risk-Sensitive Control on an Infinite Time Horizon

Stochastic control problems on an infinite time horizon with exponential cost criteria are considered. The Donsker--Varadhan large deviation rate is used as a criterion to be optimized. The optimum rate is characterized as the value of an associated stochastic differential game, with an ergodic (expected average cost per unit time) cost criterion. If we take a small-noise limit, a deterministic differential game with average cost per unit time cost criterion is obtained. This differential game is related to robust control of nonlinear systems.

A Max-Plus-Based Algorithm for a Hamilton--Jacobi--Bellman Equation of Nonlinear Filtering

The Hamilton--Jacobi--Bellman (HJB) equation associated with the {robust/\hinfty} filter (as well as the Mortensen filter) is considered. These filters employ a model where the disturbances have finite power. The HJB equation for the filter information state is a first-order equation with a term that is quadratic in the gradient. Yet the solution operator is linear in the max-plus algebra. This property is exploited by the development of a numerical algorithm where the effect of the solution operator on a set of basis functions is computed off-line. The precomputed solutions are stored as vectors of coefficients of the basis functions. These coefficients are then used directly in the real-time computations.

A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs

In previous works of the author and others, max-plus methods have been explored for the solution of first-order, nonlinear Hamilton-Jacobi-Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. These methods exploit the max-plus linearity of the associated semigroups. In particular, although the problems are nonlinear, the semigroups are linear in the max-plus sense. These methods have been used successfully to compute solutions. Although they provide certain computational-speed advantages, they still generally suffer from the curse of dimensionality. Here we consider HJB PDEs in which the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to the solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We obtain a numerical method not subject to the curse of dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with a kernel obtained from the originating semigroup. One considers repeated application of the dual-space semigroup to obtain the solution.

Risk-Sensitive and Robust Escape Criteria

The problem of controlling a noisy process so as to prevent it from leaving a prescribed set has a number of interesting applications. In this paper, new approaches to this problem are considered. First, a risk-sensitive criterion for a stochastic diffusion process model is examined, and it is shown that the value is a classical solution of a related PDE. The qualitative properties of this criterion are favorably contrasted with those of existing criteria in the risk-averse limit. It is proved that in the risk-averse limit the value of the risk-sensitive criterion converges to a viscosity solution of a first-order PDE. It is then demonstrated that the value function of a deterministic differential game is also a viscosity solution to the PDE. This game gives a robust control formulation of the escape time problem and is analogous to H

Adversarial Reasoning: Computational Approaches to Reading the Opponent's Mind

^{\infty}

Stochastic game approach to air operations

control. In particular, the opposing player attempts to push the process out of the prescribed set and suffers an L2 cost for his efforts. Lower bounds on the escape time as a function of this cost are obtained.

Curse-of-complexity attenuation in the curse-of-dimensionality-free method for HJB PDEs

ion lementation FIGURE 32.3 The structure of modeling elements. CRC-C6781 CH032.tex 5/6/2007 15: 2 Page 4 32-4 Handbook of Real-Time and Embedded Systems 32.2.1 The Data Model: Algebras We believe, like many others, that data types (typing) are a very helpful concept in a structured modeling of application domains and software structures (for details, see Ref. 11). From a mathematical point of view, a data model consists of a heterogeneous algebra. Such algebras are given by families of carrier sets and families of functions (including predicates and thus relations). More technically, we assume a set TYPE of types (sometimes also called sorts). Given types we consider a set FUNCT of function symbols with a predefined functionality (TYPE∗ stands for the set of finite sequences over the set of types TYPE) fct : FUNCT → (TYPE∗ × TYPE) The function fct associates with every function symbol in FUNCT its domain types and its range types. Both the sets TYPE and FUNCT provide only names for sets and functions. The pair (TYPE, FUNCT) together with the type assignment fct is often called the signature of the algebra. The signature is the static (also called syntactic) part of a data model. Every algebra of signature (TYPE, FUNCT) provides a carrier set (a set of data elements) for every type and a function of the requested functionality for every function symbol. For each type T ∈ TYPE, we denote by CAR(T) its carrier set. There are many ways to describe data models such as algebraic specifications, E/R diagrams (see Ref. 27), or class diagrams. 32.2.2 Syntactic Interfaces of Systems and Their Components A system and also a system component is an active information processing unit that encapsulates a state and communicates asynchronously with its environment through its interface syntactically characterized by a set of input and output channels. This communication takes place within a global (discrete) time frame. In this section we introduce the notion of a syntactic interface of systems and system components. The syntactic interface models by which communication lines, which we call channels, the system or a system component is connected to the environment and which messages are communicated over the channels. We distinguish between input and output channels. The channels and their messages determine the events of interactions that are possible for a system or a system component. In the following sections we introduce several views such as state machines, semantic interfaces, and architectures that all fit into the syntactic interface view. As we will see, each system can be used as a component in a larger system and each component of a system is a system by itself. As a result, there is no difference between the notion of a system and that of a system component. 32.2.2.1 Typed Channels In this section we introduce the concept of a typed channel. A typed channel is a directed communication line over which messages of its specific type are communicated. A typed channel c is a pair c = (e, T) consisting of an identifier e, called the channel identifier, and the type T, called the channel type. Let CID be a set of identifiers for channels. For a set C ⊆ CID × TYPE of typed channels we denote by SET(C) ⊆ CID its set of channel identifiers: SET(C) = {e ∈ CID : ∃ T ∈ TYPE : (e, T) ∈ C} A set C ⊆ CID × TYPE of typed channels is called a typed channel set, if every channel identifier e ∈ SET(C) has a unique type in C (in other words, we do not allow in typed channel sets the same channel identifier to occur twice with different types). Formally, we assume for a typed channel set: (c, T1) ∈ C ∧ (c, T2) ∈ C ⇒ T1 = T2 By TypeC(c) we denote for c ∈ C with c = (e, T) the type T. CRC-C6781 CH032.tex 5/6/2007 15: 2 Page 5 Modular Hierarchies of Models for Embedded Systems 32-5 A typed channel set C1 is called a subtype of a typed channel set C2 if the following formula holds: (c, T1) ∈ C1 ⇒ ∃ T2 ∈ TYPE : (c, T2) ∈ C2 ∧ CAR(T1) ⊆ CAR(T2) Then SET(C1) ⊆ SET(C2) holds. We then write

A new fundamental solution for differential Riccati equations arising in control

A command and control (C/sup 2/) problem for military air operations is addressed. Specifically, we consider C/sup 2/ problems for air vehicles against ground-based targets and defensive systems. The problem is viewed as a stochastic game. We restrict our attention to the C/sup 2/ level where the problem may consist of a few unmanned combat air vehicles (UCAVs) or aircraft (or possibly teams of vehicles), less than say, a half-dozen enemy surface-to-air missile air defense units (SAMs), a few enemy assets (viewed as targets from our standpoint), and some enemy decoys (assumed to mimic SAM radar signatures). At this low level, some targets are mapped out and possible SAM sites that are unavoidably part of the situation are known. One may then employ a discrete stochastic game problem formulation to determine which of these SAMs should optimally be engaged (if any), and by what series of air vehicle operations. We provide analysis, numerical implementation, and simulation for full state-feedback and measurement feedback control within this C/sup 2/ context. Sensitivity to parameter uncertainty is discussed. Some insight into the structure of optimal and near-optimal strategies for C/sup 2/ is obtained. The analysis is extended to the case of observations which may be affected by adversarial inputs. A heuristic based on risk-sensitive control is applied, and it is found that this produces improved results over more standard approaches.

Uniqueness for Viscosity Solutions of Nonstationary Hamilton--Jacobi--Bellman Equations Under Some A Priori Conditions (with Applications)

Recently, a curse-of-dimensionality-free method was developed for solution of Hamilton-Jacobi-Bellman partial differential equations (HJB PDEs) for nonlinear control problems, using semiconvex duality and max-plus analysis. The curse-of-dimensionality-free method may be applied to HJB PDEs where the Hamiltonian is given as (or well-approximated by) a pointwise maximum of quadratic forms. Such HJB PDEs also arise in certain switched linear systems. The method constructs the correct solution of an HJB PDE from a max-plus linear combination of quadratics. The method completely avoids the curse-of-dimensionality, and is subject to cubic computational growth as a function of space dimension. However, it is subject to a curse-of-complexity. In particular, the number of quadratics in the approximation grows exponentially with the number of iterations. Efficacy of such a method depends on the pruning of quadratics to keep the complexity growth at a reasonable level. Here we apply a pruning algorithm based on semidefinite programming. Computational speeds are exceptional, with an example HJB PDE in six-dimensional Euclidean space solved to the indicated quality in approximately 30 minutes on a typical desktop machine.

Curse of dimensionality reduction in max-plus based approximation methods: Theoretical estimates and improved pruning algorithms

The matrix differential Riccati equation (DRE) is ubiquitous in control and systems theory. The presence of the quadratic term implies that a simple linear-systems fundamental solution does not exist. Of course it is well-known that the Bernoulli substitution may be applied to obtain a linear system of doubled size. Here, however, tools from max-plus analysis and semiconvex duality are brought to bear on the DRE. We consider the DRE as a finite-dimensional solution to a deterministic linear/quadratic control problem. Taking the semiconvex dual of the associated semigroup, one obtains the solution operator as a max-plus integral operator with quadratic kernel. The kernel is equivalently represented as a matrix. Using the semigroup property of the dual operator, one obtains a matrix operation whereby the kernel matrix propagates as a semigroup. The propagation forward is through some simple matrix operations. This time-indexed family of matrices forms a new fundamental solution for the DRE. Solution for any initial condition is obtained by a few matrix operations on the fundamental solution and the initial condition. In analogy with standard-algebra linear systems, the fundamental solution can be viewed as an exponential form over a certain idempotent semiring. This fundamental solution has a particularly nice control interpretation, and might lead to improved DRE solution speeds.