Fadoua Balabdaoui

Using Bayesian Model Averaging to Calibrate Forecast Ensembles

Ensembles used for probabilistic weather forecasting often exhibit a spread-error correlation, but they tend to be underdispersive. This paper proposes a statistical method for postprocessing ensembles based on Bayesian model averaging (BMA), which is a standard method for combining predictive distributions from different sources. The BMA predictive probability density function (PDF) of any quantity of interest is a weighted average of PDFs centered on the individual bias-corrected forecasts, where the weights are equal to posterior probabilities of the models generating the forecasts and reflect the models’ relative contributions to predictive skill over the training period. The BMA weights can be used to assess the usefulness of ensemble members, and this can be used as a basis for selecting ensemble members; this can be useful given the cost of running large ensembles. The BMA PDF can be represented as an unweighted ensemble of any desired size, by simulating from the BMA predictive distribution. The BMA predictive variance can be decomposed into two components, one corresponding to the between-forecast variability, and the second to the within-forecast variability. Predictive PDFs or intervals based solely on the ensemble spread incorporate the first component but not the second. Thus BMA provides a theoretical explanation of the tendency of ensembles to exhibit a spread-error correlation but yet be underdispersive. The method was applied to 48-h forecasts of surface temperature in the Pacific Northwest in January– June 2000 using the University of Washington fifth-generation Pennsylvania State University–NCAR Mesoscale Model (MM5) ensemble. The predictive PDFs were much better calibrated than the raw ensemble, and the BMA forecasts were sharp in that 90% BMA prediction intervals were 66% shorter on average than those produced by sample climatology. As a by-product, BMA yields a deterministic point forecast, and this had root-mean-square errors 7% lower than the best of the ensemble members and 8% lower than the ensemble mean. Similar results were obtained for forecasts of sea level pressure. Simulation experiments show that BMA performs reasonably well when the underlying ensemble is calibrated, or even overdispersed.

A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions

Calibration, the statistical consistency of forecast distributions and the observations, is a central requirement for probabilistic predictions. Calibration of continuous forecasts is typically assessed using the probability integral transform histogram. In this article, we propose significance tests based on scoring rules to assess calibration of continuous predictive distributions. For an ideal normal forecast we derive the first two moments of two commonly used scoring rules: the logarithmic and the continuous ranked probability score. This naturally leads to the construction of two unconditional tests for normal predictions. More generally, we propose a novel score regression approach, where the individual scores are regressed on suitable functions of the predictive variance. This conditional approach is applicable even for certain nonnormal predictions based on the Dawid-Sebastiani score. Two case studies illustrate that the score regression approach has typically more power in detecting miscalibrated forecasts than the other approaches considered, including a recently proposed technique based on conditional exceedance probability curves.

Testing monotonicity via local least concave majorants

We propose a new testing procedure for detecting localized departures from monotonicity of a signal embedded in white noise. In fact, we perform simultaneously several tests that aim at detecting departures from concavity for the integrated signal over various intervals of different sizes and localizations. Each of these local tests relies on estimating the distance between the restriction of the integrated signal to some interval and its least concave majorant. Our test can be easily implemented and is proved to achieve the optimal uniform separation rate simultaneously

A Kiefer - Wolfowitz Theorem for Convex Densities

Kiefer and Wolfowitz (14) showed that if F is a strictly curved concave distribution function (corresponding to a strictly monotone density f ), then the Maximum Likelihood Estimator � Fn, which is, in fact, the least concave majorant of the empirical distribution function Fn ,d if fers from the empirical distribution function in the uniform norm by no more than a con- stant times (n−1 logn)2/3 almost surely. We review their result and give an updated version of their proof. We prove a comparable theorem for the class of distribution functions F with convex decreasing densities f , but with the max- imum likelihood estimator � Fn of F replaced by the least squares estimator � Fn: if X1,...,Xn are sampled from a distribution function F with strictly convex density f , then the least squares estimator � Fn of F and the empirical distribu- tion function Fn differ in the uniform norm by no more than a constant times (n−1 logn) 3/5 almost surely. The proofs rely on bounds on the interpolation error for complete spline interpolation due to Hall (12), Hall and Meyer (13), building on earlier work by Birkhoff and de Boor (4). These results, which are crucial for the developments here, are all nicely summarized and exposited in de Boor (5).

Inference for a two-component mixture of symmetric distributions under log-concavity

In this article, we reconsider the problem of estimating the unknown symmetric density in a two-component location mixture model under the assumption that the symmetric density is log-concave. When consistent estimators for the shift locations and mixing probability are used, we show that the nonparametric log-concave Maximum Likelihood estimator (MLE) of both the mixed density and that of the unknown symmetric component are consistent in the Hellinger distance. In case the estimators for the shift locations and mixing probability are p n-consistent, we establish that these MLE’s converge to the truth at the rate n 2=5 in the L1 distance. To estimate the shift locations and mixing probability, we use the estimators proposed by Hunter et al. (2007). The unknown symmetric density is eciently computed using the R package logcondens.mode.

On asymptotics of the discrete convex LSE of a p.m.f.

In this article, we derive the weak limiting distribution of the least squares estimator (LSE) of a convex probability mass function (pmf) with a finite support. We show that it can be defined via a certain convex projection of a Gaussian vector. Furthermore, samples of any given size from this limit distribution can be generated using an efficient Dykstra-like algorithm.

The distribution of the maximal difference between a Brownian bridge and its concave majorant

We provide a representation of the maximal difference between a standard Brownian bridge and its concave majorant on the unit interval, from which we deduce expressions for the distribution and density functions and moments of this difference. This maximal difference has an application in nonparametric statistics where it arises in testing monotonicity of a density or regression curve.

Least-squares estimation of two-ordered monotone regression curves

In this paper, we consider the problem of finding the least-squares estimators of two isotonic regression curves and under the additional constraint that they are ordered, for example, . Given two sets of n data points y 1, …, y n and z 1, …, z n observed at (the same) design points, the estimates of the true curves are obtained by minimising the weighted least-squares criterion over the class of pairs of vectors (a, b)∈ℝ n ×ℝ n such that a 1≤a 2≤···≤a n , b 1≤b 2≤···≤b n , and a i ≤b i , i=1, …, n. The characterisation of the estimators is established. To compute these estimators, we use an iterative projected subgradient algorithm, where the projection is performed with a ‘generalised’ pool-adjacent-violaters algorithm, a byproduct of this work. Then, we apply the estimation method to real data from mechanical engineering.

Demonstrating Single and Multiple Currents Through the E. coli-SecYEG-Pore: Testing for the Number of Modes of Noisy Observations

We analyze a new dataset from an electrophysiological recording of transmembrane currents through a bacterial membrane channel to demonstrate the existence of single and multiple channel currents. Protein channels mediate transport through biological membranes; knowledge of the channel properties gained from electrophysiological recordings is important for a targeted drug design. We investigate the bacterial membrane protein SecYEG which is of essential importance for the secretory pathway for sorting of newly synthesized proteins to their place of function in the cell. Our results strongly indicate that in the SecYEG pore the different modes of the density of channel currents are approximately equidistant and correspond to different numbers of open channels in the membrane. A current of ≈12 pA under the present experimental conditions turns out to be characteristic of the presence of a single open SecYEG pore, a fact that had not been electrophysiologically characterized so far. Electrophysiological recordings of single protein channels show a substantial amount of background noise. The data at our disposal can be modeled as the independent sum of an error variable and the realization of the ionic current. Thus, we are led to deconvoluting the density of the observations in order to recover the density f of the ionic currents, and then investigating the number of modes of f. To this end we propose an extension of Silverman’s (1981) test for the number of modes to deconvolution kernel density estimation, and develop the relevant theory. The finite sample performance of the test is investigated in a simulation study. Technical details for the proofs in this article are available as supplementary material online.

Calibration tests for multivariate Gaussian forecasts

Forecasts by nature should take the form of probabilistic distributions. Calibration, the statistical consistency of forecast distributions and observations, is a central property of good probabilistic forecasts. Calibration of univariate forecasts has been widely discussed, and significance tests are commonly used to investigate whether a prediction model is miscalibrated. However, calibration tests for multivariate forecasts are rare. In this paper, we propose calibration tests for multivariate Gaussian forecasts based on two types of the Dawid-Sebastiani score (DSS): the multivariate DSS (mDSS) and the individual DSS (iDSS). Analytic results and simulation studies show that the tests have sufficient power to detect miscalibrated forecasts with incorrect mean or incorrect variance. But for forecasts with incorrect correlation coefficients, only the tests based on mDSS are sensitive to miscalibration. As an illustration, we apply the methodology to weekly data on Norovirus disease incidence among males and females in Germany, in 2011-2014. The results further show that tests for multivariate forecasts are useful tools and superior to univariate calibration tests for correlated multivariate forecasts.