Simon J. Mason

Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation

The areas beneath the relative (or receiver) operating characteristics (ROC) and relative operating levels (ROL) curves can be used as summary measures of forecast quality, but statistical significance tests for these areas are conducted infrequently in the atmospheric sciences. A development of signal-detection theory, the ROC curve has been widely applied in the medical and psychology fields where significance tests and relationships to other common statistical methods have been established and described. This valuable literature appears to be largely unknown to the atmospheric sciences where applications of ROC and related techniques are becoming more common. This paper presents a survey of that literature with a focus on the interpretation of the ROC area in the field of forecast verification. We extend these foundations to demonstrate that similar principles can be applied to the interpretation and significance testing of the ROL area. It is shown that the ROC area is equivalent to the Mann–Whitney U-statistic testing the significance of forecast event probabilities for cases where events actually occurred with those where events did not occur. A similar derivation shows that the ROL area is equivalent to the Mann–Whitney U-statistic testing the magnitude of events with respect to whether or not an event has been forecast. Because the Mann–Whitney U-statistic follows a known probability distribution, under certain assumptions it can be used to define the statistical significance of ROC and ROL areas and for comparing the areas of competing forecasts. For large samples the significance of either measure can be accurately assessed using a normal-distribution approximation. Copyright

Malaria early warnings based on seasonal climate forecasts from multi-model ensembles.

The control of epidemic malaria is a priority for the international health community and specific targets for the early detection and effective control of epidemics have been agreed. Interannual climate variability is an important determinant of epidemics in parts of Africa where climate drives both mosquito vector dynamics and parasite development rates. Hence, skilful seasonal climate forecasts may provide early warning of changes of risk in epidemic-prone regions. Here we discuss the development of a system to forecast probabilities of anomalously high and low malaria incidence with dynamically based, seasonal-timescale, multi-model ensemble predictions of climate, using leading global coupled ocean–atmosphere climate models developed in Europe. This forecast system is successfully applied to the prediction of malaria risk in Botswana, where links between malaria and climate variability are well established, adding up to four months lead time over malaria warnings issued with observed precipitation and having a comparably high level of probabilistic prediction skill. In years in which the forecast probability distribution is different from that of climatology, malaria decision-makers can use this information for improved resource allocation.

Conditional Probabilities, Relative Operating Characteristics, and Relative Operating Levels

Abstract The relative operating characteristic (ROC) curve is a highly flexible method for representing the quality of dichotomous, categorical, continuous, and probabilistic forecasts. The method is based on ratios that measure the proportions of events and nonevents for which warnings were provided. These ratios provide estimates of the probabilities that an event will be forewarned and that an incorrect warning will be provided for a nonevent. Some guidelines for interpreting the ROC curve are provided. While the ROC curve is of direct interest to the user, the warning is provided in advance of the outcome and so there is additional value in knowing the probability of an event occurring contingent upon a warning being provided or not provided. An alternative method to the ROC curve is proposed that represents forecast quality when expressed in terms of probabilities of events occurring contingent upon the warnings provided. The ratios used provide estimates of the probability of an event occurring give...

Probabilistic Precipitation Anomalies Associated with ENSO

Extreme phases of the El Nino–Southern Oscillation (ENSO) phenomenon have been blamed for precipitation anomalies in many areas of the world. In some areas the probability of above-normal precipita...

Multimodel Ensembling in Seasonal Climate Forecasting at IRI

The International Research Institute (IRI) for Climate Prediction seasonal forecast system is based largely on the predictions of ensembles of several atmospheric general circulation models (AGCMs) forced by two versions of an SST prediction—one consisting of persisted SST anomalies from the current observations and one of evolving SST anomalies as predicted by a set of dynamical and statistical SST prediction models. Recently, an objective multimodel ensembling procedure has replaced a more laborious and subjective weighting of the predictions of the several AGCMs. Here the skills of the multimodel predictions produced retrospectively over the first 4 years of IRI forecasts are examined and compared with the skills of the more subjectively derived forecasts actually issued. The multimodel ensemble predictions are generally found to be an acceptable replacement, although the precipitation forecasts do benefit from inclusion of empirical forecast tools. Planned pattern-level model output statistics (MOS) c...

A verification framework for interannual-to-decadal predictions experiments

Decadal predictions have a high profile in the climate science community and beyond, yet very little is known about their skill. Nor is there any agreed protocol for estimating their skill. This paper proposes a sound and coordinated framework for verification of decadal hindcast experiments. The framework is illustrated for decadal hindcasts tailored to meet the requirements and specifications of CMIP5 (Coupled Model Intercomparison Project phase 5). The chosen metrics address key questions about the information content in initialized decadal hindcasts. These questions are: (1) Do the initial conditions in the hindcasts lead to more accurate predictions of the climate, compared to un-initialized climate change projections? and (2) Is the prediction model’s ensemble spread an appropriate representation of forecast uncertainty on average? The first question is addressed through deterministic metrics that compare the initialized and uninitialized hindcasts. The second question is addressed through a probabilistic metric applied to the initialized hindcasts and comparing different ways to ascribe forecast uncertainty. Verification is advocated at smoothed regional scales that can illuminate broad areas of predictability, as well as at the grid scale, since many users of the decadal prediction experiments who feed the climate data into applications or decision models will use the data at grid scale, or downscale it to even higher resolution. An overall statement on skill of CMIP5 decadal hindcasts is not the aim of this paper. The results presented are only illustrative of the framework, which would enable such studies. However, broad conclusions that are beginning to emerge from the CMIP5 results include (1) Most predictability at the interannual-to-decadal scale, relative to climatological averages, comes from external forcing, particularly for temperature; (2) though moderate, additional skill is added by the initial conditions over what is imparted by external forcing alone; however, the impact of initialization may result in overall worse predictions in some regions than provided by uninitialized climate change projections; (3) limited hindcast records and the dearth of climate-quality observational data impede our ability to quantify expected skill as well as model biases; and (4) as is common to seasonal-to-interannual model predictions, the spread of the ensemble members is not necessarily a good representation of forecast uncertainty. The authors recommend that this framework be adopted to serve as a starting point to compare prediction quality across prediction systems. The framework can provide a baseline against which future improvements can be quantified. The framework also provides guidance on the use of these model predictions, which differ in fundamental ways from the climate change projections that much of the community has become familiar with, including adjustment of mean and conditional biases, and consideration of how to best approach forecast uncertainty.

Changes in extreme rainfall events in South Africa

Extreme rainfall events can have severe impacts on society, so possible long-term changes in the intensity of extreme events are of concern. Testing for long-term changes in the intensity of extreme events is complicated by data inhomogeneities resulting from site and instrumentation changes. Using rainfall data from stations in South Africa that have not involved site relocations, but which have not been tested for inhomogeneities resulting from changes in instrumentation, a method of testing for changes in the intensity of extreme events is adopted. Significant increases in the intensity of extreme rainfall events between 1931–1960 and 1961–1990 are identified over about 70% of the country. The intensity of the 10-year high rainfall events has increased by over 10% over large areas of the country, except in parts of the north-east, north-west and in the winter rainfall region of the south-west. Percentage increases in the intensity of high rainfall events are largest for the most extreme events. While some inhomogeneities remain in the data used, the observed changes in the intensity of extreme rainfall events over South Africa are thought to be at least partly real.

Evaluation of the IRI'S “Net Assessment” Seasonal Climate Forecasts: 1997–2001

Abstract The International Research Institute for Climate Prediction (IRI) net assessment seasonal temperature and precipitation forecasts are evaluated for the 4-yr period from October–December 1997 to October–December 2001. These probabilistic forecasts represent the human distillation of seasonal climate predictions from various sources. The ranked probability skill score (RPSS) serves as the verification measure. The evaluation is offered as time-averaged spatial maps of the RPSS as well as area-averaged time series. A key element of this evaluation is the examination of the extent to which the consolidation of several predictions, accomplished here subjectively by the forecasters, contributes to or detracts from the forecast skill possible from any individual prediction tool. Overall, the skills of the net assessment forecasts for both temperature and precipitation are positive throughout the 1997–2001 period. The skill may have been enhanced during the peak of the 1997/98 El Nino, particularly for t...

Review of seasonal climate forecasting for agriculture in sub- Saharan Africa

SUMMARY We review the use and value of seasonal climate forecasting for agriculture in sub-Saharan Africa (SSA), with a view to understanding and exploiting opportunities to realize more of its potential benefits. Interaction between the atmosphere and underlying oceans provides the basis for probabilistic forecasts of climate conditions at a seasonal lead-time, including during cropping seasons in parts of SSA. Regional climate outlook forums (RCOF) and national meteorological services (NMS) have been at the forefront of efforts to provide forecast information for agriculture. A survey showed that African NMS often go well beyond the RCOF process to improve seasonal forecast information and disseminate it to the agricultural sector. Evidence from a combination of understanding of how climatic uncertainty impacts agriculture, modelbased ex-ante analyses, subjective expressions of demand or value, and the few well-documented evaluations of actual use and resulting benefit suggests that seasonal forecasts may have considerable potential to improve agricultural management and rural livelihoods. However, constraints related to legitimacy, salience, access, understanding, capacity to respond and data scarcity have so far limited the widespread use and benefit from seasonal prediction among smallholder farmers. Those constraints that reflect inadequate information products, policies or institutional process can potentially be overcome. Additional opportunities to benefit rural communities come from expanding the use of seasonal forecast information for coordinating input and credit supply, food crisis management, trade and agricultural insurance. The surge of activity surrounding seasonal forecasting in SSA following the 1997/98 El Nino has waned in recent years, but emerging initiatives, such as the Global Framework for Climate Services and ClimDev-Africa, are poised to reinvigorate support for seasonal forecast information services for agriculture. We conclude with a discussion of institutional and policy changes that we believe will greatly enhance the benefits of seasonal forecasting to agriculture in SSA.

On Using “Climatology” as a Reference Strategy in the Brier and Ranked Probability Skill Scores

Abstract The Brier and ranked probability skill scores are widely used as skill metrics of probabilistic forecasts of weather and climate. As skill scores, they compare the extent to which a forecast strategy outperforms a (usually simpler) reference forecast strategy. The most widely used reference strategy is that of “climatology,” in which the climatological probability (or probabilities in the case of the ranked probability skill score) of the forecast variable is issued perpetually. The Brier and ranked probability skill scores are often considered harsh standards. It is shown that the scores are harsh because the expected value of these skill scores is less than 0 if nonclimatological forecast probabilities are issued. As a result, negative skill scores can often hide useful information content in the forecasts. An alternative formulation of the skill scores based on a reference strategy in which the outcome is independent of the forecast is equivalent to using randomly assigned probabilities but is...