Allan H. Murphy

Scoring rules in probability assessment and evaluation

The purpose of this paper is to briefly discuss some important current questions and problem related to the use of scoring rules (SRs) both in connection with the actual asse%ment of probabilities and with the evaluation of probability forecasts and probability assessors. With regard to the assesstneftt process, we consider both the case in which the assessor’s utility function is linear and the case in which his utility function is nonlinear. Under linear utility, important problems of concern are the sensitivity of SRs to deviations from optimality (with a strictly proper SR, optimality consists of the assessor making his statements correspond to his judgments) and the effect of psychological considerations arising from the use of different SRs. Under nonlinear utility, SRs should be modified to allow for the nonlinearity in such a manner that for a specific utility function. the modified SRs are strictly proper. This introduces the difficult question of the assessment of the assessor’s utility function. With regard to the evaiunrion process (as opposed to the assessment process), we consider the process from an inferential viewpoint and from a decision-theoretic viewpoint. From an inferential viewpoint, attributessuch as validity may be of interest, and in certain circumstances these attributes may be related to SRs. The attributes of interest, of course, depend on the framework within which the evaluation process is undertaken. From a decision-thLwretic viewpoint, SRs may be related to a decision maker’s utilities or expected utilities (under uncertainty about the utilities) if the decision maker uses the assessed probabilities in an actual decision situation. In summary, there are many important qutistions and problems related to SRs, and the need for future research on these probletns seems clear. Such research should lead to ;I greatly improved understanding of the processes of probability assessment and evaluation.

Experiments in the laboratory and the real world

Abstract The age—old question of the generalizability of the results of experiments that are conducted in artificial laboratory settings to more realistic inferential and decision-making situations is considered in this paper. Conservatism in probability revision provides an example of a result that (1) has received wide attention, including attention in terms of implications for real-world decision making, on the basis of experiments conducted in artificial settings and (2) is now apparently thought by many to be highly situational and not at all a ubiquitous phenomenon, in which case its implications for real-world decision making are not as extensive as originally claimed. In this paper conservatism is considered in some detail within the context of the generalizability question. In a more general vein, we discuss some of the difficulties inherent in experimentation in realistic settings, suggest possible procedures for avoiding or at least alleviating such difficulties, and make a plea for more realistic experiments.

THE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON1,, 2

Abstract In this paper, we compare the ranked probability score (RPS) and the probability score (PS) and examine the nature of the sensitivity of the RPS to distance. First, we briefly describe the nature of and the relationship between the frameworks within which the RPS and the PS were formulated. Second, we consider certain properties of the RPS and the PS including their range, their values for categorical and uniform forecasts, and their “proper” nature. Third, we describe the RPS and the PS in a manner that reveals the structure of and the relationship between these scoring rules. Fourth, we considered the RPS with reference to two definitions of distance and examine the nature of the sensitivity of the RPS to distance. The comparison of the RPS and the PS suggests that the RPS rather than the PS should be used to evaluate probability forecats, at least in those situations in which the variable of concern is ordered.

The Use of Credible Intervals in Temperature Forecasting: Some Experimental Results

Since 1965, the National Weather Service (NWS) in the United States has routinely issued precipitation probability forecasts to the general public. Forecasts of maximum and minimum temperature, however, are still expressed in categorical terms. NWS forecasts usually give point forecasts when forecasting temperature (e. g., “the high temperature tomorrow will be 75°F”), and such forecasts do not provide any information about the uncertainty inherent in the forecasts. Point forecasts are sometimes replaced by interval forecasts (e. g., “the high temperature tomorrow will be between 73°F and 77°F”), but such forecasts only provide a very informal representation of the forecaster’s uncertainty. The potential user of the forecast does not know whether the forecaster is almost certain that the high temperature will fall in the forecast interval or whether the forecaster feels, say, that there is only a 50-50 chance that the high temperature will fall in the forecast interval.

On the Generalizability of Experimental Results

The age-old question of the generalizability of the results of experiments that are conducted in artificial laboratory settings to more realistic inferential and decision-making situations is considered in this paper. Conservatism in probability revision provides an example of a result that (1) has received wide attention, including attention in terms of implications for real-world decision making, on the basis of experiments conducted in artificial settings and (2) is now apparently thought by many to be highly situational and not at all a ubiquitous phenomenon, in which case its implications for real-world decision making are not as extensive as originally claimed. In this paper we consider the questions of generalizations from the laboratory to the real world in some detail, both within the context of the experiments regarding conservatism and within a more general context. In addition, we discuss some of the difficulties inherent in experimentation in realistic settings, suggest possible procedures for avoiding or at least alleviating such difficulties, and make a plea for more realistic experiments.

Experimental point and area precipitation probability forecasts for a forecast area with significant local effects

Abstract This paper describes the results of a recent experiment designed to investigate the ability of forecasters to make point and area precipitation probability forecasts. The location of the experiment (Rapid City, South Dakota) was chosen because of the existence of a network of raingages and because of significant local effects that cause variation among points in terms of precipitation occurrence. A similar experiment conducted at a different location in 1972–73 yielded encouraging results (see Winkler and Murphy, 1976), but the lack of variability in precipitation occurrence among points in the forecast area in that experiment made it difficult to answer several important questions of interest. On each forecasting occasion during the Rapid City experiment, the forecasters made an average point probability forecast for the area, individual point probability forecasts for four specific points in the area, an area probability forecast, and an expected areal coverage forecast. The results indicate th...

Subjective Probability Forecasting: Some Real World Experiments

Three experiments in subjective probability forecasting were designed, and these experiments were conducted in four forecast offices of the U.S. National Weather Service. The first experiment involved credible interval temperature forecasts, the second experiment involved point and area precipitation probability forecasts, and the third experiment involved the effect of guidance forecasts on precipitation probability forecasts. In each case, some background material is presented; the design of the experiment is discussed; some preliminary results of the experiment are presented; and some implications of the experiment and the results for probability forecasting in general and probability forecasting in meteorology in particular are discussed.

Engineering Meteorology: Meteorology and Heating Load Requirements

Temperature and other meteorological parameters, including wind velocity and solar radiation, have been examined to determine their individual and combined effects on fuel requirements. Regression methods established a correlation coefficient of 0.89 between standard degree-days and gas consumption for 147 days during the 1951–52 heating season. A temperature-refined and wind-adjusted degree-day yielded a correlalation coefficient of 0.94, a material improvement in estimating the heating load.