Ivan Markovsky

Overview of total least-squares methods

We review the development and extensions of the classical total least-squares method and describe algorithms for its generalization to weighted and structured approximation problems. In the generic case, the classical total least-squares problem has a unique solution, which is given in analytic form in terms of the singular value decomposition of the data matrix. The weighted and structured total least-squares problems have no such analytic solution and are currently solved numerically by local optimization methods. We explain how special structure of the weight matrix and the data matrix can be exploited for efficient cost function and first derivative computation. This allows to obtain computationally efficient solution methods. The total least-squares family of methods has a wide range of applications in system theory, signal processing, and computer algebra. We describe the applications for deconvolution, linear prediction, and errors-in-variables system identification.

Survey paper: Structured low-rank approximation and its applications

Fitting data by a bounded complexity linear model is equivalent to low-rank approximation of a matrix constructed from the data. The data matrix being Hankel structured is equivalent to the existence of a linear time-invariant system that fits the data and the rank constraint is related to a bound on the model complexity. In the special case of fitting by a static model, the data matrix and its low-rank approximation are unstructured. We outline applications in system theory (approximate realization, model reduction, output error, and errors-in-variables identification), signal processing (harmonic retrieval, sum-of-damped exponentials, and finite impulse response modeling), and computer algebra (approximate common divisor). Algorithms based on heuristics and local optimization methods are presented. Generalizations of the low-rank approximation problem result from different approximation criteria (e.g., weighted norm) and constraints on the data matrix (e.g., nonnegativity). Related problems are rank minimization and structured pseudospectra.

Supervisory hybrid systems

Supervisory hybrid systems are systems generating a mixture of continuous-valued and discrete-valued signals. This systems paradigm is particularly useful in modeling applications where high-level decision making is used to supervise process behavior. Hybrid system methodologies are also applicable to switched systems where the system switches between various setpoints or operational modes to extend its effective operating range. Hybrid systems, therefore, embrace a diverse set of applications. There has been considerable activity in the area of hybrid systems theory, and this article provides an introduction to some of the basic concepts and trends in this emergent field. The term hybrid refers to a mixing of two fundamentally different types of objects or methods. The paper deals with supervisory hybrid systems. Supervisory hybrid systems are systems that combine discrete event and continuous-valued dynamics. The article is organized as follows. We first provide an example of a hybrid system, to be used throughout as a pedagogical tool illustrating various concepts in hybrid systems theory. We then discuss modeling frameworks for hybrid systems, paying specific attention to the hybrid automaton. Not only may the system have a hybrid character, but the specifications on desired system behaviors may also be hybrid. The article also discusses specification logics that express system requirements on both the discrete and continuous states of the system. The article continues with a survey of current methods and concepts used to verify or validate desired system behaviors, and concludes with a survey of current methods for hybrid control system synthesis.

Exact and Approximate Modeling of Linear Systems: A Behavioral Approach

Exact and Approximate Modeling of Linear Systems: A Behavioral Approach elegantly introduces the behavioral approach to mathematical modeling, an approach that requires models to be viewed as sets of possible outcomes rather than to be a priori bound to particular representations. The authors discuss exact and approximate fitting of data by linear, bilinear, and quadratic static models and linear dynamic models, a formulation that enables readers to select the most suitable representation for a particular purpose. This book presents exact subspace-type and approximate optimization-based identification methods, as well as representation-free problem formulations, an overview of solution approaches, and software implementation. Readers will find an exposition of a wide variety of modeling problems starting from observed data. The presented theory leads to algorithms that are implemented in C language and in MATLAB.

A note on persistency of excitation

We prove that if a component of the response signal of a controllable linear time-invariant system is persistently exciting of sufficiently high order, then the windows of the signal span the full system behavior. This is then applied to obtain conditions under which the state trajectory of a state representation spans the whole state space.

Application of structured total least squares for system identification and model reduction

The following identification problem is considered: minimize the /spl lscr//sub 2/ norm of the difference between a given time series and an approximating one under the constraint that the approximating time series is a trajectory of a linear time invariant system of a fixed complexity. The complexity is measured by the input dimension and the maximum lag. The problem is known as the global total least squares and alternatively can be viewed as maximum likelihood identification in the errors-in-variables setup. Multiple time series and latent variables can be considered in the same setting. The identification problem is related to the structured total least squares problem. The paper presents an efficient software package that implements the theory in practice. The proposed method and software are tested on data sets from the database for the identification of systems DAISY.

Low Rank Approximation

Data Approximation by Low-complexity Models details the theory, algorithms, and applications of structured low-rank approximation. Efficient local optimization methods and effective suboptimal convex relaxations for Toeplitz, Hankel, and Sylvester structured problems are presented. Much of the text is devoted to describing the applications of the theory including: system and control theory; signal processing; computer algebra for approximate factorization and common divisor computation; computer vision for image deblurring and segmentation; machine learning for information retrieval and clustering; bioinformatics for microarray data analysis; chemometrics for multivariate calibration; and psychometrics for factor analysis. Software implementation of the methods is given, making the theory directly applicable in practice. All numerical examples are included in demonstration files giving hands-on experience and exercises and MATLAB examples assist in the assimilation of the theory.

Data-driven simulation and control

Classical linear time-invariant system simulation methods are based on a transfer function, impulse response, or input/state/output representation. We present a method for computing the response of a system to a given input and initial conditions directly from a trajectory of the system, without explicitly identifying the system from the data. Similar to the classical approach for simulation, the classical approach for control is model-based: first a model representation is derived from given data of the plant and then a control law is synthesised using the model and the control specifications. We present an approach for computing a linear quadratic tracking control signal that circumvents the identification step. The results are derived assuming exact data and the simulated response or control input is constructed off-line.

Consistent least squares fitting of ellipsoids

Summary.A parameter estimation problem for ellipsoid fitting in the presence of measurement errors is considered. The ordinary least squares estimator is inconsistent, and due to the nonlinearity of the model, the orthogonal regression estimator is inconsistent as well, i.e., these estimators do not converge to the true value of the parameters, as the sample size tends to infinity. A consistent estimator is proposed, based on a proper correction of the ordinary least squares estimator. The correction is explicitly given in terms of the true value of the noise variance.

Software for weighted structured low-rank approximation

A software package is presented that computes locally optimal solutions to low-rank approximation problems with the following features: *mosaic Hankel structure constraint on the approximating matrix, *weighted 2-norm approximation criterion, *fixed elements in the approximating matrix, *missing elements in the data matrix, and *linear constraints on an approximating matrixs left kernel basis. It implements a variable projection type algorithm and allows the user to choose standard local optimization methods for the solution of the parameter optimization problem. For an mxn data matrix, with n>m, the computational complexity of the cost function and derivative evaluation is O(m^2n). The package is suitable for applications with n@?m. In statistical estimation and data modeling-the main application areas of the package-n@?m corresponds to modeling of large amount of data by a low-complexity model. Performance results on benchmark system identification problems from the database DAISY and approximate common divisor problems are presented.