Monica Adya

How effective are neural networks at forecasting and prediction? A review and evaluation

Despite increasing applications of artificial neural networks (NNs) to forecasting over the past decade, opinions regarding their contribution are mixed. Evaluating research in this area has been difficult, due to lack of clear criteria. We identified eleven guidelines that could be used in evaluating this literature. Using these, we examined applications of NNs to business forecasting and prediction. We located 48 studies done between 1988 and 1994. For each, we evaluated how effectively the proposed technique was compared with alternatives (effectiveness of validation) and how well the technique was implemented (effectiveness of implementation). We found that eleven of the studies were both effectively validated and implemented. Another eleven studies were effectively validated and produced positive results, even though there were some problems with respect to the quality of their NN implementations. Of these 22 studies, 18 supported the potential of NNs for forecasting and prediction. Copyright

Automatic Identification of Time Series Features for Rule-Based Forecasting

Rule-based forecasting (RBF) is an expert system that uses features of time series to select and weight extrapolation techniques. Thus, it is dependent upon the identification of features of the time series. Judgmental coding of these features is expensive and the reliability of the ratings is modest. We developed and automated heuristics to detect six features that had previously been judgmentally identified in RBF: outliers, level shifts, change in basic trend, unstable recent trend, unusual last observation, and functional form. These heuristics rely on simple statistics such as first differences and regression estimates. In general, there was agreement between automated and judgmental codings for all features other than functional form. Heuristic coding was more sensitive than judgment and consequently, identified more series with a certain feature than judgmental coding. We compared forecast accuracy using automated codings with that using judgmental codings across 122 series. Forecasts were produced for six horizons, resulting in a total of 732 forecasts. Accuracy for 30% of the 122 annual time series was similar to that reported for RBF. For the remaining series, there were as many that did better with automated feature detection as there were that did worse. In other words, the use of automated feature detection heuristics reduced the costs of using RBF without negatively affecting forecast accuracy.

Rule-Based Forecasting: Using Judgment in Time-Series Extrapolation

Rule-Based Forecasting (RBF) is an expert system that uses judgment to develop and apply rules for combining extrapolations. The judgment comes from two sources, forecasting expertise and domain knowledge. Forecasting expertise is based on more than a half century of research. Domain knowledge is obtained in a structured way; one example of domain knowledge is managers’ expectations about trends, which we call “causal forces.” Time series are described in terms of up to 28 conditions, which are used to assign weights to extrapolations. Empirical results on multiple sets of time series show that RBF produces more accurate forecasts than those from traditional extrapolation methods or equal-weights combined extrapolations. RBF is most useful when it is based on good domain knowledge, the domain knowledge is important, the series is well-behaved (such that patterns can be identified), there is a strong trend in the data, and the forecast horizon is long. Under ideal conditions, the error for RBF’s forecasts were one-third less than those for equal-weights combining. When these conditions are absent, RBF will neither improve nor harm forecast accuracy. Some of RBF’s rules can be used with traditional extrapolation procedures. In a series of studies, rules based on causal forces improved the selection of forecasting methods, the structuring of time series, and the assessment of prediction intervals.

An Application of Rule-Based Forecasting to a Situation Lacking Domain Knowledge

Rule-based forecasting (RBF) uses rules to combine forecasts from simple extrapolation methods. Weights for combining the rules use statistical and domain-based features of time series. RBF was originally developed, tested, and validated only on annual data. For the M3-Competition, three major modifications were made to RBF. First, due to the absence of much in the way of domain knowledge, we prepared the forecasts under the assumption that no domain knowledge was available. This removes what we believe is one of RBFs primary advantages. We had to re-calibrate some of the rules relating to causal forces to allow for this lack of domain knowledge. Second, automatic identification procedures were used for six time-series features that had previously been identified using judgment. This was done to reduce cost and improve reliability. Third, we simplified the rule-base by removing one method from the four that were used in the original implementation. Although this resulted in some loss in accuracy, it reduced the number of rules in the rule-base from 99 to 64. This version of RBF still benefits from the use of prior findings on extrapolation, so we expected that it would be substantially more accurate than the random walk and somewhat more accurate than equal weights combining. Because most of the previous work on RBF was done using annual data, we especially expected it to perform well with annual data.

Expert Systems for Forecasting

Expert systems use rules to represent experts’ reasoning in solving problems. The rules are based on knowledge about methods and the problem domain. To acquire knowledge for an expert system, one should rely on a variety of sources, such as textbooks, research papers, interviews, surveys, and protocol analyses. Protocol analyses are especially useful if the area to be modeled is complex or if experts lack an awareness of their processes. Expert systems should be easy to use, incorporate the best available knowledge, and reveal the reasoning behind the recommendations they make. In forecasting, the most promising applications of expert systems are to replace unaided judgment in cases requiring many forecasts, to model complex problems where data on the dependent variable are of poor quality, and to handle semi-structured problems. We found 15 comparisons of forecast validity involving expert systems. As expected, expert systems were more accurate than unaided judgment, six comparisons to one, with one tie. Expert systems were less accurate than judgmental bootstrapping in two comparisons with two ties. There was little evidence with which to compare expert systems and econometric models; expert systems were better in one study and tied in two.

Flexible Global Software Development (GSD): Antecedents of Success in Requirements Analysis

Globalization of software development has resulted in a rapid shift away from the traditional collocated, on-site development model, to the offshoring model. Emerging trends indicate an increasing interest in offshoring even in early phases like requirements analysis. Additionally, the flexibility offered by the agile development approach makes it attractive for adaptation in globally distributed software work. A question of significance then is what impacts the success of offshoring earlier phases, like requirements analysis, in a flexible and globally distributed environment? This article incorporates the stance of control theory to posit a research model that examines antecedent factors such as requirements change, facilitation by vendor and client site-coordinators, control, and computer-mediated communication. The impact of these factors on success of requirements analysis projects in a “flexible†global setting is tested using two quasi-experiments involving students from Management Development Institute, India and Marquette University, USA. Results indicate that formal modes of control significantly influence project success during requirements analysis. Further, facilitation by both client and vendor site coordinators positively impacts requirements analysis success.

Project Quality of Off-Shore Virtual Teams Engaged in Software Requirements Analysis: An Exploratory Comparative Study

The off-shore software development companies in countries such as India use a global delivery model in which initial requirement analysis phase of software projects get executed at client locations to leverage frequent and deep interaction between user and developer teams. Subsequent phases such as design, coding and testing are completed at off-shore locations. Emerging trends indicate an increasing interest in off-shoring even requirements analysis phase using computer mediated communication. We conducted an exploratory research study involving students from Management Development Institute (MDI), India and Marquette University (MU), U.S.A. to determine quality of such off-shored requirements analysis projects. Our findings suggest that project quality of teams engaged in pure off-shore mode is comparable to that of teams engaged in collocated mode. However, the effect of controls such as user project monitoring on the quality of off-shored projects needs to be studied further.

Bringing global sourcing into the classroom: experiential learning via software development project

The growing trend in offshore software development has imposed new skills requirements on collaborating global partners. In the U.S. this has translated into skill sets that include communications, project management, business analysis, and team management. In a virtual setting, these skills take on a complex proportion. This paper describes an educational initiative in offshore software development between undergraduate students enrolled in a project management course at Marquette University, USA and graduate business students enrolled in an Information Systems Analysis and Design course at Management Development Institute, India. The course replicated an offshore client/vendor relationship in a virtual setting. For faculty considering such initiatives, this paper describes the setting and factors critical to success of this initiative and cautions against others that can be detrimental to such an effort.

Corrections to rule-based forecasting: findings from a replication

Abstract Rule-Based Forecasting (RBF) is an expert system that combines forecasts from simple extrapolation methods based on features of time series. In this study, we provide corrections to ten of the 99 rules contained in RBF. These corrections were identified during a replication of RBF. Empirical comparisons indicate that the corrections did not lead to a noticeable improvement in accuracy when tested against some of the original data. However, in light of the fact that several studies are extending the work on RBF, it is important to report on these corrections to RBF.

Can the US minimum data set be used for predicting admissions to acute care facilities

This paper is intended to give an overview of Knowledge Discovery in Large Datasets (KDD) and data mining applications in healthcare particularly as related to the Minimum Data Set, a resident assessment tool which is used in US long-term care facilities. The US Health Care Finance Administration, which mandates the use of this tool, has accumulated massive warehouses of MDS data. The pressure in healthcare to increase efficiency and effectiveness while improving patient outcomes requires that we find new ways to harness these vast resources. The intent of this preliminary study design paper is to discuss the development of an approach which utilizes the MDS, in conjunction with KDD and classification algorithms, in an attempt to predict admission from a long-term care facility to an acute care facility. The use of acute care services by long term care residents is a negative outcome, potentially avoidable, and expensive. The value of the MDS warehouse can be realized by the use of the stored data in ways that can improve patient outcomes and avoid the use of expensive acute care services. This study, when completed, will test whether the MDS warehouse can be used to describe patient outcomes and possibly be of predictive value.