Durga Lal Shrestha

2006 Special issue: Machine learning approaches for estimation of prediction interval for the model output

A novel method for estimating prediction uncertainty using machine learning techniques is presented. Uncertainty is expressed in the form of the two quantiles (constituting the prediction interval) of the underlying distribution of prediction errors. The idea is to partition the input space into different zones or clusters having similar model errors using fuzzy c-means clustering. The prediction interval is constructed for each cluster on the basis of empirical distributions of the errors associated with all instances belonging to the cluster under consideration and propagated from each cluster to the examples according to their membership grades in each cluster. Then a regression model is built for in-sample data using computed prediction limits as targets, and finally, this model is applied to estimate the prediction intervals (limits) for out-of-sample data. The method was tested on artificial and real hydrologic data sets using various machine learning techniques. Preliminary results show that the method is superior to other methods estimating the prediction interval. A new method for evaluating performance for estimating prediction interval is proposed as well.

Experiments with AdaBoost.RT, an improved boosting scheme for regression

The application of boosting technique to regression problems has received relatively little attention in contrast to research aimed at classification problems. This letter describes a new boosting algorithm, AdaBoost.RT, for regression problems. Its idea is in filtering out the examples with the relative estimation error that is higher than the preset threshold value, and then following the AdaBoost procedure. Thus, it requires selecting the suboptimal value of the error threshold to demarcate examples as poorly or well predicted. Some experimental results using the M5 model tree as a weak learning machine for several benchmark data sets are reported. The results are compared to other boosting methods, bagging, artificial neural networks, and a single M5 model tree. The preliminary empirical comparisons show higher performance of AdaBoost.RT for most of the considered data sets.

A novel method to estimate model uncertainty using machine learning techniques

A novel method is presented for model uncertainty estimation using machine learning techniques and its application in rainfall runoff modeling. In this method, first, the probability distribution of the model error is estimated separately for different hydrological situations and second, the parameters characterizing this distribution are aggregated and used as output target values for building the training sets for the machine learning model. This latter model, being trained, encapsulates the information about the model error localized for different hydrological conditions in the past and is used to estimate the probability distribution of the model error for the new hydrological model runs. The M5 model tree is used as a machine learning model. The method is tested to estimate uncertainty of a conceptual rainfall runoff model of the Bagmati catchment in Nepal. In this paper the method is extended further to enable it to predict an approximation of the whole error distribution, and also the new results of comparing this method to other uncertainty estimation approaches are reported. It can be concluded that the method generates consistent, interpretable and improved model uncertainty estimates.

ANN-based representation of parametric and residual uncertainty of models

In this paper we investigate the possibility of using ANN for modeling uncertainty of models (with some focus on environmental models). We assume that all uncertainties of the prediction made by such model M are represented by probability distribution function (pdf) of its error, and build regression models of the quantiles of this pdf. The original version of the technique termed UNEEC (published earlier) deals with residual uncertainty of calibrated (trained) deterministic models, and uses fuzzy clustering and soft weighting of local models to deal with the fact that uncertainty of environmental models is different in different regions of the state space. The extended version of the method presented here allows also for explicit handling the uncertainty in parameters of environmental model M. The resulting ANN encapsulates both the results of Monte-Carlo simulations as well as the residual uncertainty. On two data sets it is shown that the presented approach allows for generating consistent predictions of models uncertainty.

Comparing machine learning methods in estimation of model uncertainty

The paper presents a generalization of the framework for assessment of predictive models uncertainty using machine learning techniques. Historical model errors which are mismatch between observed and predicted values are assumed to be indicators of total model uncertainty; it is measured in the form of prediction intervals, and comprises all sources of uncertainty including model structure, model parameters, input and output data. Several machine learning methods are compared. The approach is tested on a conceptual hydrological model set up to predict stream flows of the Brue catchment in the United Kingdom.

ANNs and Other Machine Learning Techniques in Modelling Models' Uncertainty

The paper presents examples of using ANNs and other machine learning (ML) techniques to assess uncertainty of a mathematical (computer-based) model M . Two approaches have been developed to estimate parametric and residual uncertainty, and they were tested on process based hydrological models. One approach emulates computationally expensive Monte Carlo simulations, and the second one uses residuals of a calibrated model M outputs to assess the remaining uncertainty of this model. ML models are trained to approximate the functional relationships between the input (and state) variables of the model M and the uncertainty descriptors. ML model, being trained, encapsulates the information about the model M errors specific for different conditions in the past, and is used to estimate the probability distribution of the model M error for the new model runs. Methods are tested to estimate uncertainty of a conceptual rainfall-runoff model of a catchment in UK.

Estimation of prediction intervals for the model outputs using machine learning

A new method for estimating prediction intervals for a model output using machine learning is presented. In it, first the prediction intervals for in-sample data using clustering techniques to identify the distinguishable regions in input space with similar distributions of model errors are constructed. Then regression model is built for in-sample data using computed prediction intervals as targets, and, finally, this model is applied to estimate the prediction intervals for out-of-sample data. The method was tested on artificial and real hydrologic data sets using various machine learning techniques. Preliminary results show that the method is superior to other methods estimating the prediction intervals. A new method for evaluating performance for estimating prediction intervals is proposed as well.