Kevin Judd

On selecting models for nonlinear time series

Abstract Constructing models from time series with nontrivial dynamics involves the problem of how to choose the best model from within a class of models, or to choose between competing classes. This paper discusses a method of building nonlinear models of possibly chaotic systems from data, while maintaining good robustness against noise. The models that are built are close to the simplest possible according to a description length criterion. The method will deliver a linear model if that has shorter description length than a nonlinear model. We show how our models can be used for prediction, smoothing and interpolation in the usual way. We also show how to apply the results to identification of chaos by detecting the presence of homoclinic orbits directly from time series.

Dynamics from multivariate time series

Abstract Multivariate time series data are common in experimental and industrial systems. If the generating system has nonlinear dynamics, we may be able to construct a model that reproduces the dynamics and can be used for control and other purposes. In principle, multivariate time series are not necessary for recovering dynamics: according to the embedding theorem, only one time series should be needed. However, for real data, there may be large gains in using all of the measurements. In this paper we examine the issues of how to use multiple data streams most effectively for modeling and prediction. For example, perhaps the data are redundant in that only a subset of the data streams is useful. And how should we embed the data, if indeed embedding is required at all? We show how these questions can be answered, and describe some numerical experiments which show that using multivariate time series can significantly improve predictability. We also demonstrate a somewhat surprising synchronization between different reconstructions.

Embedding as a modeling problem

Abstract Standard approaches to time-delay embedding will often fail to provide an embedding that is useful for many common applications. This happens in particular when there are multiple timescales in the dynamics. We present a modified procedure, non-uniform embedding, which overcomes such problems in many cases. For more complex nonlinear dynamics we introduce variable embedding, where, in a suitable sense, the embedding changes with the state of the system. We show how to implement these procedures by combining embedding and modeling into a single procedure with a single optimization goal.

An improved estimator of dimension and some comments on providing confidence intervals

Abstract An estimator of dimension is described that has many advantages over the conventional implementation of the Grassberger-Procaccia method. It can in some circumstances provide a confidence interval for the dimension estimate. For high dimension sets (greater than about 3.5) the dimension estimate has a bias which depends on the structure of the set. If one has suitable knowledge of the structure of the set it is possible to make a bias adjustment. Some numerical calculations are presented which indicate the largest error in the dimension estimate that can be expected. The problem of providing confidence intervals for dimension estimates in general is not solved completely, but we do take a step away from a meaningless number.

Dangers of geometric filtering

Abstract We argue that there are unrecognised dangers in the popular and powerful geometric filtering method. Carelessly applied, geometric filtering can produce apparent structure in data which is pure noise, or can cause severe distortions in clean deterministic data. The explanations for these effects are straightforward and the dangers are easily avoided by taking simple precautions in the filtering process.

Estimating dimension from small samples

Abstract Details of a new algorithm to estimate the dimension of a measure from point samples are described. The new algorithm emphasizes that dimension need not be the same at all scales, that is, there need not be a “scaling region.” The algorithm is verified with Gaussian measures and is found to be reliable and have less demanding data requirements than conventional estimators. Using certain convergence properties of Gaussian measures, it is shown that good estimates can be made from very small samples.

Time Step Sensitivity of Nonlinear Atmospheric Models: Numerical Convergence, Truncation Error Growth, and Ensemble Design

Computational models based on discrete dynamical equations are a successful way of approaching the problem of predicting or forecasting the future evolution of dynamical systems. For linear and mildly nonlinear models, the solutions of the numerical algorithms on which they are based converge to the analytic solutions of the underlying differential equations for small time steps and grid sizes. In this paper, the authors investigate the time step sensitivity of three nonlinear atmospheric models of different levels of complexity: the Lorenz equations, a quasigeostrophic (QG) model, and a global weather prediction system (NOGAPS). It is illustrated here how, for chaotic systems, numerical convergence cannot be guaranteed forever. The time of decoupling of solutions for different time steps follows a logarithmic rule (as a function of time step) similar for the three models. In regimes that are not fully chaotic, the Lorenz equations are used to illustrate how different time steps may lead to different model climates and even different regimes. A simple model of truncation error growth in chaotic systems is proposed. This model decomposes the error onto its stable and unstable components and reproduces well the short- and medium-term behavior of the QG model truncation error growth, with an initial period of slow growth (a plateau) before the exponential growth phase. Experiments with NOGAPS suggest that truncation error can be a substantial component of total forecast error of the model. Ensemble simulations with NOGAPS show that using different time steps may be a simple and natural way of introducing an important component of model error in ensemble design.

Pulse propagation networks: a neural network model that uses temporal coding by action potentials

Abstract In this paper we study a model of a neural network that is fundamentally different from currently popular models. In this model we consider every action potential in the network, rather than average firing rates; this enables us to consider temporal coding by action potentials. This kind of model is not new, but we believe our results on computational ability to be new. We introduce a specific model, which we call a pulse propagation network (PPN), and consider this model from the point of view of information processing, as a dynamical system and as a computing machine. We show, in particular, that as a computing machine it can operate with real numbers and consequently it is of a class more powerful than a conventional Turing machine. In the process of this analysis, we develop a framework of concepts and techniques useful to understand and analyze these PPN.

Indistinguishable states I.: perfect model scenario

An accurate forecast of a nonlinear system will require an accurate estimation of the initial state. It is shown that even under the ideal conditions of a perfect model and infinite past observations of a deterministic nonlinear system, uncertainty in the observations makes exact state estimation is impossible. Consistent with the noisy observations there is a set of states indistinguishable from the true state. This implies that an accurate forecast must be based on a probability density on the indistinguishable states. This paper shows that this density can be calculated by first calculating a maximum likelihood estimate of the state, and then an ensemble estimate of the density of states that are indistinguishable from the maximum likelihood state. A new method for calculating the maximum likelihood estimate of the true state is presented which allows practical ensemble forecasting even when the recurrence time of the system is long. In a subsequent paper the theory and practice described in this paper are extended to an imperfect model scenario.

Correlation dimension: a pivotal statistic for non-constrained realizations of composite hypotheses in surrogate data analysis

Abstract Currently surrogate data analysis can be used to determine if data is consistent with various linear systems, or something else (a nonlinear system). In this paper we propose an extension of these methods in an attempt to make more specific classifications within the class of nonlinear systems. In the method of surrogate data one estimates the probability distribution of values of a test statistic for a set of experimental data under the assumption that the data is consistent with a given hypothesis. If the probability distribution of the test statistic is different for different dynamical systems consistent with the hypothesis, one must ensure that the surrogate generation technique generates surrogate data that are a good approximation to the data. This is often achieved with a careful choice of surrogate generation method and for noise driven linear surrogates such methods are commonly used. This paper argues that, in many cases (particularly for nonlinear hypotheses), it is easier to select a test statistic for which the probability distribution of test statistic values is the same for all systems consistent with the hypothesis. For most linear hypotheses one can use a reliable estimator of a dynamic invariant of the underlying class of processes. For more complex, nonlinear hypothesis it requires suitable restatement (or cautious statement) of the hypothesis. Using such statistics one can build nonlinear models of the data and apply the methods of surrogate data to determine if the data is consistent with a simulation from a broad class of models. These ideas are illustrated with estimates of probability distribution functions for correlation dimension estimates of experimental and artificial data, and linear and nonlinear hypotheses.