Hailiang Du

Laplace’s Demon and the Adventures of His Apprentices

The sensitive dependence on initial conditions (SDIC) associated with nonlinear models imposes limitations on the models’ predictive power. We draw attention to an additional limitation than has been underappreciated, namely, structural model error (SME). A model has SME if the model dynamics differ from the dynamics in the target system. If a nonlinear model has only the slightest SME, then its ability to generate decision-relevant predictions is compromised. Given a perfect model, we can take the effects of SDIC into account by substituting probabilistic predictions for point predictions. This route is foreclosed in the case of SME, which puts us in a worse epistemic situation than SDIC.

Pseudo-Orbit Data Assimilation. Part I: The Perfect Model Scenario

State estimation lies at the heart of many meteorological tasks. Pseudo-orbit-based data assimilation provides an attractive alternative approach to data assimilation in nonlinear systems such as weather forecasting models. In the perfect model scenario, noisy observations prevent a precise estimate of the current state. In this setting, ensemble Kalman filter approaches are hampered by their foundational assumptions of dynamical linearity, while variational approaches may fail in practice owing to local minima in their cost function. The pseudo-orbit data assimilation approach improves state estimation by enhancing the balance between the information derived from the dynamic equations and that derived from the observations. The potential use of this approach for numerical weather prediction is explored in the perfect model scenario within two deterministic chaotic systems: the two-dimensional Ikeda map and 18-dimensional Lorenz96 flow. Empirical results demonstrate improved performance over that of the two most common traditional approaches of data assimilation (ensemble Kalman filter and four-dimensional variational assimilation).

Towards improving the framework for probabilistic forecast evaluation

The evaluation of forecast performance plays a central role both in the interpretation and use of forecast systems and in their development. Different evaluation measures (scores) are available, often quantifying different characteristics of forecast performance. The properties of several proper scores for probabilistic forecast evaluation are contrasted and then used to interpret decadal probability hindcasts of global mean temperature. The Continuous Ranked Probability Score (CRPS), Proper Linear (PL) score, and IJ Good’s logarithmic score (also referred to as Ignorance) are compared; although information from all three may be useful, the logarithmic score has an immediate interpretation and is not insensitive to forecast busts. Neither CRPS nor PL is local; this is shown to produce counter intuitive evaluations by CRPS. Benchmark forecasts from empirical models like Dynamic Climatology place the scores in context. Comparing scores for forecast systems based on physical models (in this case HadCM3, from the CMIP5 decadal archive) against such benchmarks is more informative than internal comparison systems based on similar physical simulation models with each other. It is shown that a forecast system based on HadCM3 out performs Dynamic Climatology in decadal global mean temperature hindcasts; Dynamic Climatology previously outperformed a forecast system based upon HadGEM2 and reasons for these results are suggested. Forecasts of aggregate data (5-year means of global mean temperature) are, of course, narrower than forecasts of annual averages due to the suppression of variance; while the average “distance” between the forecasts and a target may be expected to decrease, little if any discernible improvement in probabilistic skill is achieved.

Pseudo-Orbit Data Assimilation. Part II: Assimilation with Imperfect Models

Data assimilation and state estimation for nonlinear models is a challenging task mathematically. Performing this task in real time, as in operational weather forecasting, is even more challenging as the models are imperfect: the mathematical system that generated the observations (if such a thing exists) is not a member of the available model class (i.e., the set of mathematical structures admitted as potential models). To the extent that traditional approaches address structural model error at all, most fail to produce consistent treatments. This results in questionable estimates both of the model state and of its uncertainty. A promising alternative approach is proposed to produce more consistent estimates of the model state and to estimate the (state dependent) model error simultaneously. This alternative consists of pseudo-orbit data assimilation with a stopping criterion. It is argued to be more efficient and more coherent than one alternative variational approach [a version of weak-constraint fourdimensional variational data assimilation (4DVAR)]. Results that demonstrate the pseudo-orbit data assimilation approach can also outperform an ensemble Kalman filter approach are presented. Both comparisons are made in the context of the 18-dimensional Lorenz96 flow and the two-dimensional Ikeda map. Many challenges remain outsidethe perfect model scenario,both in defining the goalsof data assimilation and in achievinghigh-quality state estimation. The pseudo-orbit data assimilation approach provides a new tool for approaching this open problem.

Model error and ensemble forecasting: a cautionary tale

Many scientific enterprises nowadays involve using computer simulations to model complex phenomena of interest. Many models make probabilistic predictions using the methodology of initial condition ensemble forecasting, sometimes called ICE forecasting. Weather models, climate models, financial market models and hydrological models, among others, all do this sort of thing. We will see (using a simple example) how this a methodology successfully deals with certain kinds of uncertainty. The main aim of this paper is negative, however. We will show that ICE forecasting does not help with a certain other kind of errors, namely with structural model error. We argue that if a model is non-linear and if there is only the slightest model imperfection, then treating model outputs as decision relevant probabilistic forecasts can be seriously misleading. Models of the systems mentioned above are extremely complex. The problem we want to discuss shows up in much simpler models, so the remainder of our discussion will be about a particular simple model-system pair that we begin to introduce in the next section. After that, in Section 3 we introduce ICE forecasting and in Section 4 we demonstrate a problem with it. We relate our discussion of the simple model back to weather and climate models in Section 5 and we respond to some objections in Section 6.

Multi-model cross-pollination in time

The predictive skill of complex models is rarely uniform in model-state space; in weather forecasting models, for example, the skill of the model can be greater in the regions of most interest to a particular operational agency than it is in “remote” regions of the globe. Given a collection of models, a multi-model forecast system using the cross-pollination in time approach can be generalized to take advantage of instances where some models produce forecasts with more information regarding specific components of the model-state than other models, systematically. This generalization is stated and then successfully demonstrated in a moderate (∼40) dimensional nonlinear dynamical system, suggested by Lorenz, using four imperfect models with similar global forecast skill. Applications to weather forecasting and in economic forecasting are discussed. Given that the relative importance of different phenomena in shaping the weather changes in latitude, changes in attitude among forecast centers in terms of the resources assigned to each phenomena are to be expected. The demonstration establishes that cross-pollinating elements of forecast trajectories enriches the collection of simulations upon which the forecast is built, and given the same collection of models can yield a new forecast system with significantly more skill than the original forecast system.