Robert L. Winkler

A General Framework for Forecast Verification

Abstract A general framework for forecast verification based on the joint distribution of forecasts and observations is described. For further elaboration of the framework, two factorizations of the joint distribution are investigated: 1) the calibration-refinement factorization, which involves the conditional distributions of observations given forecasts and the marginal distribution of forecasts, and 2) the likelihood-base factorization, which involve the conditional distributions of forecasts given observations and the marginal distribution of observations. The names given to the factorizations reflect the fact that they relate to different attributes of the forecasts and/or observations. Several examples are used to illustrate the interpretation of these factorizations in the context of verification and to describe the relationship between the respective factorizations. Some insight into the potential utility of the framework is provided by demonstrating that basic elements and summary measures of the...

The Assessment of Prior Distributions in Bayesian Analysis

Abstract In the Bayesian framework, quantified judgments about uncertainty are an indispensable input to methods of statistical inference and decision. Ultimately, all components of the formal mathematical models underlying inferential procedures represent quantified judgments. In this study, the focus is on just one component, the prior distribution, and on some of the problems of assessment that arise when a person tries to express prior distributions in quantitative form. The objective is to point toward assessment procedures that can actually be used. One particular type of statistical problem is considered and several techniques of assessment are presented, together with the necessary instruction so that these techniques can be understood and applied. A questionnaire is developed and used in a study in which people actually assess prior distributions. The results indicate that, by and large, it is feasible to question people about subjective prior probability distributions, although this depends on t...

Combining Economic Forecasts

A method for combining forecasts may or may not account for dependence and differing precision among forecasts. In this article we test a variety of such methods in the context of combining forecasts of GNP from four major econometric models. The methods include one in which forecasting errors are jointly normally distributed and several variants of this model as well as some simpler procedures and a Bayesian approach with a prior distribution based on exchangeability of forecasters. The results indicate that a simple average, the normal model with an independence assumption, and the Bayesian model perform better than the other approaches that are studied here.

Interactive Elicitation of Opinion for a Normal Linear Model

Abstract This article describes the mathematical theory underlying an interactive computer program for eliciting the hyperparameters of a subjective conjugate distribution for the multiple linear regression model with the usual normal error structure. Although the methods are heuristic, they are shown to produce hyperparameter estimates satisfying the constraints satisfied by the hyperparameters themselves. An application is given to the problem of predicting the time to fatigue failure of an asphalt-concrete road as a function of several design variables concerning the road.

“Good” Probability Assessors

Abstract Since a meteorologists predictions are subjective, a framework for the evaluation of meteorological probability assessors must be consistent with the theory of subjective probability. Such a framework is described in this paper. First, two standards of “goodness,” one based upon normative considerations and one based upon substantive considerations, are proposed. Specific properties which a meteorologists assessments should possess are identified for each standard. Then, several measures of “goodness,” or scoring rules, which indicate the extent to which such assessments possess certain properties, are described. Finally, several important uses of these scoring rules are considered.

The Combination of Forecasts

Aggregating information by combining forecasts from two or more forecasting methods is an alternative to using just a single method. In this paper we provide extensive empirical results showing that combined forecasts obtained through weighted averages can be quite accurate. Five procedures for estimating weights are investigated, and two appear to be superior to the others. These two procedures provide forecasts that are more accurate overall than forecasts from individual methods. Furthermore, they are superior to forecasts found from a simple unweighted average of the same methods.

Limits for the Precision and Value of Information from Dependent Sources

In many inferential and decision-making situations, information is obtained from a number of information sources, and the separate pieces of information are often not independent. This paper investigates the impact of dependence on the precision and value of information. The results indicate that positive dependence among information sources can have a serious detrimental effect on the precision and value of the information. Differences in precision between the dependent and independent cases can be remarkably large. With dependence, the incremental value of information can decrease very rapidly, and the limiting value of information as more sources are considered can be considerably less than the expected value of perfect information. The results of this paper have implications for the acquisition and use of information in decision-making problems.

Probability Forecasting in Meteorology

Abstract Efforts to quantify the uncertainty in weather forecasts began more than 75 years ago, and many studies and experiments involving objective and subjective probability forecasting have been conducted in meteorology in the intervening period. Moreover, the U.S. National Weather Service (NWS) initiated a nationwide program in 1965 in which precipitation probability forecasts were formulated on an operational basis and routinely disseminated to the general public. In addition, the NWS now prepares objective probability forecasts for many variables, using statistical procedures. Hence probability forecasting in meteorology is unique in that very large sets of probability forecasts that have been subjected to detailed evaluation are available. This article has four objectives: (a) to review the history of probability forecasting in meteorology to acquaint statisticians with this body of literature; (b) to describe recent methodological, experimental, and operational activities in this field; (c) to exa...

Scoring Rules and the Evaluation of Probability Assessors

Abstract The personalistic theory of probability prescribes that personal probability assessments to be used in decision-making situations should correspond with the assessors judgments. A payoff function which depends on the assessors stated probabilities and on the event which actually occurs may be used (1) to keep the assessor honest or (2) to evaluate the assessor. It is shown that with the exception of a logarithmic payoff function, these two uses of payoff functions for assessors are not compatible. This conflict is explained in terms of the differences in the situations facing the assessor and the evaluator (the user of the probabilistic predictions).

Uncertainty in probabilistic risk assessment

Abstract Dealing with uncertainty is an important and difficult aspect of analyses for complex systems. Such systems involve many uncertainties, and assessing probabilities to represent these uncertainties is itself a complex undertaking utilizing a variety of information sources. At a very basic level, uncertainty is uncertainty, and attempting to distinguish between ‘types of uncertainty’ is questionable. At a practical level, on the other hand, a close look at such distinctions suggests that they are driven by important modelling issues related to model structuring, probability assessment, information gathering, and sensitivity analysis. Anything that brings more attention to these issues should improve the state of the art. However, I would prefer to attack the issues directly instead of working indirectly through the notion of ‘types of uncertainty.’