Francesca P. Carli

A Maximum Entropy Solution of the Covariance Extension Problem for Reciprocal Processes

Stationary reciprocal processes defined on a finite interval of the integer line can be seen as a special class of Markov random fields restricted to one dimension. Nonstationary reciprocal processes have been extensively studied in the past especially by Jamison et al. The specialization of the nonstationary theory to the stationary case, however, does not seem to have been pursued in sufficient depth in the literature. Stationary reciprocal processes (and reciprocal stochastic models) are potentially useful for describing signals which naturally live in a finite region of the time (or space) line. Estimation or identification of these models starting from observed data seems still to be an open problem which can lead to many interesting applications in signal and image processing. In this paper, we discuss a class of reciprocal processes which is the acausal analog of auto-regressive (AR) processes, familiar in control and signal processing. We show that maximum likelihood identification of these processes leads to a covariance extension problem for block-circulant covariance matrices. This generalizes the famous covariance band extension problem for stationary processes on the integer line. As in the usual stationary setting on the integer line, the covariance extension problem turns out to be a basic conceptual and practical step in solving the identification problem. We show that the maximum entropy principle leads to a complete solution of the problem.

On the Covariance Completion Problem Under a Circulant Structure

Covariance matrices with a circulant structure arise in the context of discrete-time periodic processes and their significance stems also partly from the fact that they can be diagonalized via a Fourier transformation. This note deals with the problem of completion of partially specified circulant covariance matrices. The particular completion that has maximal determinant (i.e., the so-called maximum entropy completion) was considered in Carli where it was shown that if a single band is unspecified and to be completed, the algebraic restriction that enforces the circulant structure is automatically satisfied and that the inverse of the maximizer has a band of zero values that corresponds to the unspecified band in the data, i.e., it has the Dempster property. The purpose of the present note is to develop an independent proof of this result which in fact extends naturally to any number of missing bands as well as arbitrary missing elements. More specifically, we show that this general fact is a direct consequence of the invariance of the determinant under the group of transformations that leave circulant matrices invariant. A description of the complete set of all positive extensions of partially specified circulant matrices is also given and certain connections between such sets and the factorization of certain polynomials in many variables, facilitated by the circulant structure, is highlighted.

Modelling and Simulation of Images by Reciprocal Processes

In this paper we propose an approach to image modelling and simulation, by a simple class of linear constant parameter stochastic models known as stationary rciprocal processes. These processes can be seen as a special class of Gibbs-Markov random fields. Stationary reciprocal processes admit constant parameter descriptor type representations of a certain kind which can be seen as a natural non-causal extension of the linear state space models used in time series analysis. Estimation and identification of these models starting from observed data is solvable by fast and efficient numerical techniques. In particular it turns out that the statistical estimation and simulation of stationary reciprocal models leads to an extension problem for symmetric positive-definite block-circulant matrices which can be solved by Fourier methods.

Maximum entropy properties of discrete-time first-order stable spline kernel

The first order stable spline (SS-1) kernel (also known as the tuned-correlated (TC) kernel) is used extensively in regularized system identification, where the impulse response is modeled as a zero-mean Gaussian process whose covariance function is given by well designed and tuned kernels. In this paper, we discuss the maximum entropy properties of this kernel. In particular, we formulate the exact maximum entropy problem solved by the SS-1 kernel without Gaussian and uniform sampling assumptions. Under general sampling assumption, we also derive the special structure of the SS-1 kernel (e.g. its tridiagonal inverse and factorization have closed form expression), also giving to it a maximum entropy covariance completion interpretation.

Efficient algorithms for large scale linear system identification using stable spline estimators

Abstract A new nonparametric approach for system identification has been recently proposed where, in place of postulating parametric classes of impulse responses, the estimation process starts from an infinite-dimensional space. In particular, the impulse response is seen as the realization of a zero-mean Gaussian process. Its covariance, the so called stable spline kernel, encodes information on system stability and depends on few hyperparameters estimated from data via marginal likelihood optimization. This approach has been proved to compare much favorably with classical parametric methods but, in data rich situations, a possible drawback may be represented by its computational complexity which scales with the cube of the number of available samples. In this work we design a new computational strategy which may reduce significantly the computational load required by the stable spline estimator, thus extending its practical applicability also to large-scale scenarios.

Maximum Entropy Kernels for System Identification

Bayesian nonparametric approaches have been recently introduced in system identification scenario where the impulse response is modeled as the realization of a zero-mean Gaussian process whose covariance (kernel) has to be estimated from data. In this scheme, quality of the estimates crucially depends on the parametrization of the covariance of the Gaussian process. A family of kernels that have been shown to be particularly effective in the system identification framework is the family of Diagonal/Correlated (DC) kernels. Maximum entropy properties of a related family of kernels, the Tuned/Correlated (TC) kernels, have been recently pointed out in the literature. In this technical note, we show that maximum entropy properties indeed extend to the whole family of DC kernels. The maximum entropy interpretation can be exploited in conjunction with results on matrix completion problems in the graphical models literature to shed light on the structure of the DC kernel. In particular, we prove that the DC kernel admits a closed-form factorization, inverse, and determinant. These results can be exploited both to improve the numerical stability and to reduce the computational complexity associated with the computation of the DC estimator.

On the maximum entropy property of the first-order stable spline kernel and its implications

A new nonparametric approach for system identification has been recently proposed where the impulse response is seen as the realization of a zero-mean Gaussian process whose covariance, the so-called stable spline kernel, guarantees that the impulse response is almost surely stable. Maximum entropy properties of the stable spline kernel have been pointed out in the literature. In this paper we provide an independent proof that relies on the theory of matrix extension problems in the graphical model literature and leads to a closed form expression for the inverse of the first order stable spline kernel as well as to a new factorization in the form UWU with U upper triangular and W diagonal. Interestingly, all first-order stable spline kernels share the same factor U and W admits a closed form representation in terms of the kernel hyperparameter, making the factorization computationally inexpensive. Maximum likelihood properties of the stable spline kernel are also highlighted. These results can be applied both to improve the stability and to reduce the computational complexity associated with the computation of stable spline estimators.

On the estimation of hyperparameters for Bayesian system identification with exponentially decaying kernels

A Bayesian formulation of system identification problems has become popular recently; this is mainly due to the introduction of a family of prior descriptions (kernels) which encode structural properties of dynamical systems such as stability. The simplest instance of this kernel prescribes that the impulse response coefficients are independent random variables with exponentially decaying variance. Selecting the most suitable kernel within this class, which involves tuning the rate at which variance decay, is an important step. This paper studies the properties of the so-called “marginal likelihood” approach providing an interpretation in terms of Mean Squared Error properties of the resulting estimators.

Sparse multiple kernels for impulse response estimation with majorization minimization algorithms

This contribution aims to enrich the recently introduced kernel-based regularization method for linear system identification. Instead of a single kernel, we use multiple kernels, which can be instances of any existing kernels for the impulse response estimation of linear systems. We also introduce a new class of kernels constructed based on output error (OE) model estimates. In this way, a more flexible and richer representation of the kernel is obtained. Due to this representation the associated hyper-parameter estimation problem has two good features. First, it is a difference of convex functions programming (DCP) problem. While it is still nonconvex, it can be transformed into a sequence of convex optimization problems with majorization minimization (MM) algorithms and a local minima can thus be found iteratively. Second, it leads to sparse hyper-parameters and thus sparse multiple kernels. This feature shows the kernel-based regularization method with multiple kernels has the potential to tackle various problems of finding sparse solutions in linear system identification.

On the geometry of message passing algorithms for Gaussian reciprocal processes

Reciprocal processes are acausal generalizations of Markov processes introduced by Bernstein in 1932. In the literature, a significant amount of attention has been focused on developing dynamical models for reciprocal processes. Recently, probabilistic graphical models for reciprocal processes have been provided. This opens the way to the application of efficient inference algorithms in the machine learning literature to solve the smoothing problem for reciprocal processes. Such algorithms are known to converge if the underlying graph is a tree. This is not the case for a reciprocal process, whose associated graphical model is a single loop network. The contribution of this paper is twofold. First, we introduce belief propagation for Gaussian reciprocal processes. Second, we establish a link between convergence analysis of belief propagation for Gaussian reciprocal processes and stability theory for differentially positive systems.