Jochen Bröcker

Scoring Probabilistic Forecasts: The Importance of Being Proper

Questions remain regarding how the skill of operational probabilistic forecasts is most usefully evaluated or compared, even though probability forecasts have been a long-standing aim in meteorological forecasting. This paper explains the importance of employing proper scores when selecting between the various measures of forecast skill. It is demonstrated that only proper scores provide internally consistent evaluations of probability forecasts, justifying the focus on proper scores independent of any attempt to influence the behavior of a forecaster. Another property of scores (i.e., locality) is discussed. Several scores are examined in this light. There is, effectively, only one proper, local score for probability forecasts of a continuous variable. It is also noted that operational needs of weather forecasts suggest that the current concept of a score may be too narrow; a possible generalization is motivated and discussed in the context of propriety and locality.

Increasing the Reliability of Reliability Diagrams

The reliability diagram is a common diagnostic graph used to summarize and evaluate probabilistic forecasts. Its strengths lie in the ease with which it is produced and the transparency of its definition. While visually appealing, major long-noted shortcomings lie in the difficulty of interpreting the graph visually; for the most part, ambiguities arise from variations in the distributions of forecast probabilities and from various binning procedures. A resampling method for assigning consistency bars to the observed frequencies is introduced that allows for immediate visual evaluation as to just how likely the observed relative frequencies are under the assumption that the predicted probabilities are reliable. Further, an alternative presentation of the same information on probability paper eases quantitative evaluation and comparison. Both presentations can easily be employed for any method of binning.

Reliability, sufficiency, and the decomposition of proper scores

Scoring rules are an important tool for evaluating the performance of probabilistic forecasting schemes. In the binary case, scoring rules (which are strictly proper) allow for a decomposition into terms related to the resolution and to the reliability of the forecast. This fact is particularly well known for the Brier Score. In this paper, this result is extended to forecasts for finite–valued targets. Both resolution and reliability are shown to have a positive effect on the score. It is demonstrated that resolution and reliability are directly related to forecast attributes which are desirable on grounds independent of the notion of scores. This finding can be considered an epistemological justification of measuring forecast quality by proper scores. A link is provided to the original work of DeGroot and Fienberg (1982), extending their concepts of sufficiency and refinement. The relation to the conjectured sharpness principle of Gneiting et al. (2005a) is elucidated.

Nonlinear noise reduction

Different methods for removing noise contaminating time series are presented, which all exploit the underlying (deterministic) dynamics. All approaches are embedded in a probabilistic framework for stochastic systems and signals, where the two main tasks, state and orbit estimation, are distinguished. Estimation of the true current state (without noise) is based on previously sampled elements of the time series, only, and corresponds to filtering. With orbit estimation, the entire measured time series is used to determine a less noisy orbit. In this case not only past values but also future samples are used, which, of course, improves performance.

Examining reliability of seasonal to decadal sea surface temperature forecasts: The role of ensemble dispersion

Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.Useful probabilistic climate forecasts on decadal timescales should be reliable (i.e. forecast probabilities match the observed relative frequencies) but this is seldom examined. This paper assesses a necessary condition for reliability, that the ratio of ensemble spread to forecast error being close to one, for seasonal to decadal sea surface temperature retrospective forecasts from the Met Office Decadal Prediction System (DePreSys). Factors which may affect reliability are diagnosed by comparing this spread-error ratio for an initial condition ensemble and two perturbed physics ensembles for initialized and uninitialized predictions. At lead times less than 2 years, the initialized ensembles tend to be under-dispersed, and hence produce overconfident and hence unreliable forecasts. For longer lead times, all three ensembles are predominantly over-dispersed. Such over-dispersion is primarily related to excessive inter-annual variability in the climate model. These findings highlight the need to carefully evaluate simulated variability in seasonal and decadal prediction systems.

On variational data assimilation in continuous time

Variational data assimilation in continuous time is revisited. The central techniques applied in this paper are in part adopted from the theory of optimal nonlinear control. Alternatively, the investigated approach can be considered as a continuous time generalisation of what is known as weakly constrained four dimensional variational assimilation (WC–4DVAR) in the geosciences. The technique allows to assimilate trajectories in the case of partial observations and in the presence of model error. Several mathematical aspects of the approach are studied. Computationally, it amounts to solving a two point boundary value problem. For imperfect models, the trade off between small dynamical error (i.e. the trajectory obeys the model dynamics) and small observational error (i.e. the trajectory closely follows the observations) is investigated. For (nearly) perfect models, this trade off turns out to be (nearly) trivial in some sense, yet allowing for some dynamical error is shown to have positive effects even in this situation. The presented formalism is dynamical in character; no assumptions need to be made about the presence (or absence) of dynamical or observational noise, let alone about their statistics.

Efficient noncausal noise reduction for deterministic time series

We present a simple noncausal noise reduction algorithm for time series that consist of noisy measurements of the state vectors of a deterministic (chaotic) nonlinear system. The underlying dynamical system is assumed to be known and to operate in discrete time. The noise reduction algorithm is an iterative scheme for finding exact deterministic orbits close to the measured noisy orbits. Furthermore, we discuss cases where the solution is not the original orbit but homoclinic to it. (c) 2001 American Institute of Physics.

Sensitivity and out-of-sample error in continuous time data assimilation

Data assimilation refers to the problem of finding trajectories of a prescribed dynamical model in such a way that the output of the model (usually some function of the model states) follows a given time series of observations. Typically though, these two requirements cannot both be met at the same time–tracking the observations is not possible without the trajectory deviating from the proposed model equations, while adherence to the model requires deviations from the observations. Thus, data assimilation faces a trade-off. In this contribution, the sensitivity of the data assimilation with respect to perturbations in the observations is identified as the parameter which controls the trade-off. A relation between the sensitivity and the out-of-sample error is established, which allows the latter to be calculated under operational conditions. A minimum out-of-sample error is proposed as a criterion to set an appropriate sensitivity and to settle the discussed trade-off. Two approaches to data assimilation are considered, namely variational data assimilation and Newtonian nudging, also known as synchronization. Numerical examples demonstrate the feasibility of the approach. Copyright

Some Remarks on the Reliability of Categorical Probability Forecasts

Studies on forecast evaluation often rely on estimating limiting observed frequencies conditioned on specific forecast probabilities (the reliability diagram or calibration function). Obviously, statistical estimates of the calibration function are based on only limited amounts of data and therefore contain residual errors. Although errors and variations of calibration function estimates have been studied previously, either they are often assumed to be small or unimportant, or they are ignored altogether. It is demonstrated how these errors can be described in terms of bias and variance, two concepts well known in the statistics literature. Bias and variance adversely affect estimates of the reliability and sharpness terms of the Brier score, recalibration of forecasts, and the assessment of forecast reliability through reliability diagram plots. Ways to communicate and appreciate these errors are presented. It is argued that these errors can become quite substantial if individual sample points have too large influence on the estimate, which can be avoided by using regularization techniques. As an illustration, it is discussed how to choose an appropriate bin size in the binning and counting method, and an appropriate bandwidth parameter for kernel estimates.

Prediction of Extreme Events

We discuss concepts for the prediction of extreme events based on time series data. We consider both probabilistic forecasts and predictions by precursors. Probabilistic forecasts employ estimates of the probability for the event to follow, whereas precursors are temporal patterns in the data typically preceeding events. Theoretical considerations lead to the construction of schemes that are optimal with respect to several scoring rules. We discuss scenarios for which, in contrast to intuition, events with larger magnitude are better predictable than events with smaller magnitude.