Tyrus Berry

Time-Scale Separation from Diffusion-Mapped Delay Coordinates

It has long been known that the method of time-delay embedding can be used to reconstruct nonlinear dynamics from time series data. A less-appreciated fact is that the induced geometry of time-delay coordinates increasingly biases the reconstruction toward the stable directions as delays are added. This bias can be exploited, using the diffusion maps approach to dimension reduction, to extract dynamics on desired time scales from high-dimensional observed data. We demonstrate the technique on a wide range of examples, including data generated by a model of meandering spiral waves and video recordings of a liquid-crystal experiment.

Adaptive ensemble Kalman filtering of non-linear systems

A necessary ingredient of an ensemble Kalman filter (EnKF) is covariance inflation, used to control filter divergence and compensate for model error. There is an on-going search for inflation tunings that can be learned adaptively. Early in the development of Kalman filtering, Mehra (1970, 1972) enabled adaptivity in the context of linear dynamics with white noise model errors by showing how to estimate the model error and observation covariances. We propose an adaptive scheme, based on lifting Mehras idea to the non-linear case, that recovers the model error and observation noise covariances in simple cases, and in more complicated cases, results in a natural additive inflation that improves state estimation. It can be incorporated into non-linear filters such as the extended Kalman filter (EKF), the EnKF and their localised versions. We test the adaptive EnKF on a 40-dimensional Lorenz96 model and show the significant improvements in state estimation that are possible. We also discuss the extent to which such an adaptive filter can compensate for model error, and demonstrate the use of localisation to reduce ensemble sizes for large problems.

Nonparametric forecasting of low-dimensional dynamical systems

This paper presents a nonparametric modeling approach for forecasting stochastic dynamical systems on low-dimensional manifolds. The key idea is to represent the discrete shift maps on a smooth basis which can be obtained by the diffusion maps algorithm. In the limit of large data, this approach converges to a Galerkin projection of the semigroup solution to the underlying dynamics on a basis adapted to the invariant measure. This approach allows one to quantify uncertainties (in fact, evolve the probability distribution) for nontrivial dynamical systems with equation-free modeling. We verify our approach on various examples, ranging from an inhomogeneous anisotropic stochastic differential equation on a torus, the chaotic Lorenz three-dimensional model, and the Niño-3.4 data set which is used as a proxy of the El Niño Southern Oscillation.

Linear theory for filtering nonlinear multiscale systems with model error.

In this paper, we study filtering of multiscale dynamical systems with model error arising from limitations in resolving the smaller scale processes. In particular, the analysis assumes the availability of continuous-time noisy observations of all components of the slow variables. Mathematically, this paper presents new results on higher order asymptotic expansion of the first two moments of a conditional measure. In particular, we are interested in the application of filtering multiscale problems in which the conditional distribution is defined over the slow variables, given noisy observation of the slow variables alone. From the mathematical analysis, we learn that for a continuous time linear model with Gaussian noise, there exists a unique choice of parameters in a linear reduced model for the slow variables which gives the optimal filtering when only the slow variables are observed. Moreover, these parameters simultaneously give the optimal equilibrium statistical estimates of the underlying system, and as a consequence they can be estimated offline from the equilibrium statistics of the true signal. By examining a nonlinear test model, we show that the linear theory extends in this non-Gaussian, nonlinear configuration as long as we know the optimal stochastic parametrization and the correct observation model. However, when the stochastic parametrization model is inappropriate, parameters chosen for good filter performance may give poor equilibrium statistical estimates and vice versa; this finding is based on analytical and numerical results on our nonlinear test model and the two-layer Lorenz-96 model. Finally, even when the correct stochastic ansatz is given, it is imperative to estimate the parameters simultaneously and to account for the nonlinear feedback of the stochastic parameters into the reduced filter estimates. In numerical experiments on the two-layer Lorenz-96 model, we find that the parameters estimated online, as part of a filtering procedure, simultaneously produce accurate filtering and equilibrium statistical prediction. In contrast, an offline estimation technique based on a linear regression, which fits the parameters to a training dataset without using the filter, yields filter estimates which are worse than the observations or even divergent when the slow variables are not fully observed. This finding does not imply that all offline methods are inherently inferior to the online method for nonlinear estimation problems, it only suggests that an ideal estimation technique should estimate all parameters simultaneously whether it is online or offline.

Forecasting turbulent modes with nonparametric diffusion models: Learning from noisy data

Abstract In this paper, we apply a recently developed nonparametric modeling approach, the “diffusion forecast”, to predict the time-evolution of Fourier modes of turbulent dynamical systems. While the diffusion forecasting method assumes the availability of a noise-free training data set observing the full state space of the dynamics, in real applications we often have only partial observations which are corrupted by noise. To alleviate these practical issues, following the theory of embedology, the diffusion model is built using the delay-embedding coordinates of the data. We show that this delay embedding biases the geometry of the data in a way which extracts the most stable component of the dynamics and reduces the influence of independent additive observation noise. The resulting diffusion forecast model approximates the semigroup solutions of the generator of the underlying dynamics in the limit of large data and when the observation noise vanishes. As in any standard forecasting problem, the forecasting skill depends crucially on the accuracy of the initial conditions. We introduce a novel Bayesian method for filtering the discrete-time noisy observations which works with the diffusion forecast to determine the forecast initial densities. Numerically, we compare this nonparametric approach with standard stochastic parametric models on a wide-range of well-studied turbulent modes, including the Lorenz-96 model in weakly chaotic to fully turbulent regimes and the barotropic modes of a quasi-geostrophic model with baroclinic instabilities. We show that when the only available data is the low-dimensional set of noisy modes that are being modeled, the diffusion forecast is indeed competitive to the perfect model.

Nonparametric Uncertainty Quantification for Stochastic Gradient Flows

This paper presents a nonparametric statistical modeling method for quantifying uncertainty in stochastic gradient systems with isotropic diffusion. The central idea is to apply the diffusion maps algorithm to a training data set to produce a stochastic matrix whose generator is a discrete approximation to the backward Kolmogorov operator of the underlying dynamics. The eigenvectors of this stochastic matrix, which we will refer to as the diffusion coordinates, are discrete approximations to the eigenfunctions of the Kolmogorov operator and form an orthonormal basis for functions defined on the data set. Using this basis, we consider the projection of three uncertainty quantification (UQ) problems (prediction, filtering, and response) into the diffusion coordinates. In these coordinates, the nonlinear prediction and response problems reduce to solving systems of infinite-dimensional linear ordinary differential equations. Similarly, the continuous-time nonlinear filtering problem reduces to solving a syst...

Semiparametric modeling

Semiparametric forecasting and filtering are introduced as a method of addressing model errors arising from unresolved physical phenomena. While traditional parametric models are able to learn high-dimensional systems from small data sets, their rigid parametric structure makes them vulnerable to model error. On the other hand, nonparametric models have a very flexible structure, but they suffer from the curse-of-dimensionality and are not practical for high-dimensional systems. The semiparametric approach loosens the structure of a parametric model by fitting a data-driven nonparametric model for the parameters. Given a parametric dynamical model and a noisy data set of historical observations, an adaptive Kalman filter is used to extract a time-series of the parameter values. A nonparametric forecasting model for the parameters is built by projecting the discrete shift map onto a data-driven basis of smooth functions. Existing techniques for filtering and forecasting algorithms extend naturally to the semiparametric model which can effectively compensate for model error, with forecasting skill approaching that of the perfect model. Semiparametric forecasting and filtering are a generalization of statistical semiparametric models to time-dependent distributions evolving under dynamical systems.

Semiparametric forecasting and filtering: correcting low-dimensional model error in parametric models

Semiparametric forecasting and filtering are introduced as a method of addressing model errors arising from unresolved physical phenomena. While traditional parametric models are able to learn high-dimensional systems from small data sets, their rigid parametric structure makes them vulnerable to model error. On the other hand, nonparametric models have a very flexible structure, but they suffer from the curse-of-dimensionality and are not practical for high-dimensional systems. The semiparametric approach loosens the structure of a parametric model by fitting a data-driven nonparametric model for the parameters. Given a parametric dynamical model and a noisy data set of historical observations, an adaptive Kalman filter is used to extract a time-series of the parameter values. A nonparametric forecasting model for the parameters is built by projecting the discrete shift map onto a data-driven basis of smooth functions. Existing techniques for filtering and forecasting algorithms extend naturally to the semiparametric model which can effectively compensate for model error, with forecasting skill approaching that of the perfect model. Semiparametric forecasting and filtering are a generalization of statistical semiparametric models to time-dependent distributions evolving under dynamical systems.

Correcting Biased Observation Model Error in Data Assimilation

AbstractWhile the formulation of most data assimilation schemes assumes an unbiased observation model error, in real applications model error with nontrivial biases is unavoidable. A practical example is errors in the radiative transfer model (which is used to assimilate satellite measurements) in the presence of clouds. Together with the dynamical model error, the result is that many (in fact 99%) of the cloudy observed measurements are not being used although they may contain useful information. This paper presents a novel nonparametric Bayesian scheme that is able to learn the observation model error distribution and correct the bias in incoming observations. This scheme can be used in tandem with any data assimilation forecasting system. The proposed model error estimator uses nonparametric likelihood functions constructed with data-driven basis functions based on the theory of kernel embeddings of conditional distributions developed in the machine learning community. Numerically, positive results are...

Density estimation on manifolds with boundary

Density estimation is a crucial component of many machine learning methods, and manifold learning in particular, where geometry is to be constructed from data alone. A significant practical limitation of the current density estimation literature is that methods have not been developed for manifolds with boundary, except in simple cases of linear manifolds where the location of the boundary is assumed to be known. We overcome this limitation by developing a density estimation method for manifolds with boundary that does not require any prior knowledge of the location of the boundary. To accomplish this we introduce statistics that provably estimate the distance and direction of the boundary, which allows us to apply a cut-and-normalize boundary correction. By combining multiple cut-and-normalize estimators we introduce a consistent kernel density estimator that has uniform bias, at interior and boundary points, on manifolds with boundary.