Fred Collopy

Error Measures for Generalizing About Forecasting Methods: Empirical Comparisons

This study evaluated measures for making comparisons of errors across time series. We analyzed 90 annual and 101 quarterly economic time series. We judged error measures on reliability, construct validity, sensitivity to small changes, protection against outliers, and their relationship to decision making. The results lead us to recommend the Geometric Mean of the Relative Absolute Error (GMRAE) when the task involves calibrating a model for a set of time series. The GMRAE compares the absolute error of a given method to that from the random walk forecast. For selecting the most accurate methods, we recommend the Median RAE (MdRAE) when few series are available and the Median Absolute Percentage Error (MdAPE) otherwise. The Root Mean Square Error (RMSE) is not reliable, and is therefore inappropriate for comparing accuracy across series.

How effective are neural networks at forecasting and prediction? A review and evaluation

Despite increasing applications of artificial neural networks (NNs) to forecasting over the past decade, opinions regarding their contribution are mixed. Evaluating research in this area has been difficult, due to lack of clear criteria. We identified eleven guidelines that could be used in evaluating this literature. Using these, we examined applications of NNs to business forecasting and prediction. We located 48 studies done between 1988 and 1994. For each, we evaluated how effectively the proposed technique was compared with alternatives (effectiveness of validation) and how well the technique was implemented (effectiveness of implementation). We found that eleven of the studies were both effectively validated and implemented. Another eleven studies were effectively validated and produced positive results, even though there were some problems with respect to the quality of their NN implementations. Of these 22 studies, 18 supported the potential of NNs for forecasting and prediction. Copyright

Automatic Identification of Time Series Features for Rule-Based Forecasting

Rule-based forecasting (RBF) is an expert system that uses features of time series to select and weight extrapolation techniques. Thus, it is dependent upon the identification of features of the time series. Judgmental coding of these features is expensive and the reliability of the ratings is modest. We developed and automated heuristics to detect six features that had previously been judgmentally identified in RBF: outliers, level shifts, change in basic trend, unstable recent trend, unusual last observation, and functional form. These heuristics rely on simple statistics such as first differences and regression estimates. In general, there was agreement between automated and judgmental codings for all features other than functional form. Heuristic coding was more sensitive than judgment and consequently, identified more series with a certain feature than judgmental coding. We compared forecast accuracy using automated codings with that using judgmental codings across 122 series. Forecasts were produced for six horizons, resulting in a total of 732 forecasts. Accuracy for 30% of the 122 annual time series was similar to that reported for RBF. For the remaining series, there were as many that did better with automated feature detection as there were that did worse. In other words, the use of automated feature detection heuristics reduced the costs of using RBF without negatively affecting forecast accuracy.

Expert Opinions About Extrapolation and the Mystery of the Overlooked Discontinuities

We report on the opinions of 49 forecasting experts on guidelines for extrapolation methods. They agreed that seasonality, trend, aggregation, and discontinuities were key features to use for selecting extrapolation methods. The strong agreement about the importance of discontinuities was surprising because this topic has been largely ignored in the forecasting literature.

The role and validity of judgment in forecasting

Abstract All forecasting methods involve judgment but forecasting techniques are often dichotomised as judgmental or statistical. Most forecasting research has focused on the development and testing of statistical techniques. However, in practice, human reasoning and judgment play a primary role. Even when statistical methods are used, results are often adjusted in accord with expert judgment (Bunn and Wright 1991). This editorial introduces the papers included in this special issue of the International Journal of Forecasting and places them within a broader research context. The discussion of this context is structured in three sections: judgmental probability forecasting; judgmental time series forecasting; and interaction of judgment and statistical models.

Rule-Based Forecasting: Using Judgment in Time-Series Extrapolation

Rule-Based Forecasting (RBF) is an expert system that uses judgment to develop and apply rules for combining extrapolations. The judgment comes from two sources, forecasting expertise and domain knowledge. Forecasting expertise is based on more than a half century of research. Domain knowledge is obtained in a structured way; one example of domain knowledge is managers’ expectations about trends, which we call “causal forces.” Time series are described in terms of up to 28 conditions, which are used to assign weights to extrapolations. Empirical results on multiple sets of time series show that RBF produces more accurate forecasts than those from traditional extrapolation methods or equal-weights combined extrapolations. RBF is most useful when it is based on good domain knowledge, the domain knowledge is important, the series is well-behaved (such that patterns can be identified), there is a strong trend in the data, and the forecast horizon is long. Under ideal conditions, the error for RBF’s forecasts were one-third less than those for equal-weights combining. When these conditions are absent, RBF will neither improve nor harm forecast accuracy. Some of RBF’s rules can be used with traditional extrapolation procedures. In a series of studies, rules based on causal forces improved the selection of forecasting methods, the structuring of time series, and the assessment of prediction intervals.

An Application of Rule-Based Forecasting to a Situation Lacking Domain Knowledge

Rule-based forecasting (RBF) uses rules to combine forecasts from simple extrapolation methods. Weights for combining the rules use statistical and domain-based features of time series. RBF was originally developed, tested, and validated only on annual data. For the M3-Competition, three major modifications were made to RBF. First, due to the absence of much in the way of domain knowledge, we prepared the forecasts under the assumption that no domain knowledge was available. This removes what we believe is one of RBFs primary advantages. We had to re-calibrate some of the rules relating to causal forces to allow for this lack of domain knowledge. Second, automatic identification procedures were used for six time-series features that had previously been identified using judgment. This was done to reduce cost and improve reliability. Third, we simplified the rule-base by removing one method from the four that were used in the original implementation. Although this resulted in some loss in accuracy, it reduced the number of rules in the rule-base from 99 to 64. This version of RBF still benefits from the use of prior findings on extrapolation, so we expected that it would be substantially more accurate than the random walk and somewhat more accurate than equal weights combining. Because most of the previous work on RBF was done using annual data, we especially expected it to perform well with annual data.

Expert Systems for Forecasting

Expert systems use rules to represent experts’ reasoning in solving problems. The rules are based on knowledge about methods and the problem domain. To acquire knowledge for an expert system, one should rely on a variety of sources, such as textbooks, research papers, interviews, surveys, and protocol analyses. Protocol analyses are especially useful if the area to be modeled is complex or if experts lack an awareness of their processes. Expert systems should be easy to use, incorporate the best available knowledge, and reveal the reasoning behind the recommendations they make. In forecasting, the most promising applications of expert systems are to replace unaided judgment in cases requiring many forecasts, to model complex problems where data on the dependent variable are of poor quality, and to handle semi-structured problems. We found 15 comparisons of forecast validity involving expert systems. As expected, expert systems were more accurate than unaided judgment, six comparisons to one, with one tie. Expert systems were less accurate than judgmental bootstrapping in two comparisons with two ties. There was little evidence with which to compare expert systems and econometric models; expert systems were better in one study and tied in two.

Color, Form, and Motion: Dimensions of a Musical Art of Light

Lumia are an art form that permits visual artists to play images in the way that musicians play with sounds. Though the idea of creating lumia has a long historical tradition, modern graphicallybased computers make it possible to design instruments for creating lumia that are more flexible and easier to play than at any previous time in the history of art. In designing and playing lumia, three principal dimensions require attention: color, form, and motion. By organizing the design of lumia and instruments for creating them along these dimensions, it is possible to learn a great deal from art theory and history, as guidelines have been devised for the effective use of each of these dimensions.

Research Report-Principles for Examining Predictive Validity: The Case of Information Systems Spending Forecasts

Research over two decades has advanced the knowledge of how to assess predictive validity. We believe this has value to information systems IS researchers. To demonstrate, we used a widely cited study of IS spending. In that study, price-adjusted diffusion models were proposed to explain and to forecast aggregate U.S. information systems spending. That study concluded that such models would produce more accurate forecasts than would simple linear trend extrapolation. However, one can argue that the validation procedure provided an advantage to the diffusion models. We reexamined the results using an alternative validation procedure based on three principles extracted from forecasting research: 1 use ex ante out-of-sample performance rather than the fit to the historical data, 2 use well-accepted models as a basis for comparison, and 3 use an adequate sample of forecasts. Validation using this alternative procedure did confirm the importance of the price-adjustment, but simple trend extrapolations were found to be more accurate than the price-adjusted diffusion models.