Conceição Amado

Water distribution systems flow monitoring and anomalous event detection: A practical approach

Methods to detect outliers in network flow measurements that may be due to pipe bursts or unusual consumptions are fundamental to improve water distribution system on-line operation and management, and to ensure reliable historical data for sustainable planning and design of these systems. To detect and classify anomalous events in flow data from district metering areas a four-step methodology was adopted, implemented and tested: i) data acquisition, ii) data validation and normalization, iii) anomalous observation detection, iv) anomalous event detection and characterization. This approach is based on the renewed concept of outlier regions and depends on a reduced number of configuration parameters: the number of past observations, the true positive rate and the false positive rate. Results indicate that this approach is flexible and applicable to the detection of different types of events (e.g., pipe burst, unusual consumption) and to different flow time series (e.g., instantaneous, minimum night flow).

Comparative Study of Three Stochastic Models for Prediction of Pipe Failures in Water Supply Systems

The prediction of pipe failures in urban water systems is a complex process because the available failure records, originating in work orders, are often short and incomplete. To identify a robust and simple model with good failure prediction results using short data history, three existing models were compared in this study: the single-variate Poisson process, the Weibull accelerated lifetime model, and the linear-extended Yule process. This work also presents modifications to these models that enable them to produce more accurate predictions and overcome computational issues for practical software implementation. The three models, together with the improvements where applicable, were applied to water supply system data provided by a Portuguese water utility, and the results were comparatively analysed to assess the accuracy of each model. The Weibull accelerated lifetime model yielded the best results among the three models, accurately predicting failures and detecting pipes with high failure likelihood; however, it is based on Monte Carlo simulations, which can be time-consuming. The linear extended Yule process could also effectively detect pipes with higher failure likelihood; however, it presented a clear tendency to overestimate the number of future failures. The single-variate Poisson process is the simplest of the three models and produced failure prediction results of lower quality.

Robust Bootstrap with Non Random Weights Based on the Influence Function

Abstract The existence of outliers in a sample is an obvious problem which can become worse when the usual bootstrap is applied, because some resamples may have a higher contamination level than the initial sample. Bootstrapping using robust estimators may be a solution to this problem. However, in many instances, this will not be enough because it can lead to several complications, such as: (i) the breakdown point for the whole procedure may be small even when based on an estimator with a high breakdown point; (ii) mathematical difficulties; (iii) very high computation time. In this paper, we suggest a modification of the bootstrap procedure in order to solve these problems which consists of forming each bootstrap sample by resampling with different probabilities so that the potentially more harmful observations have smaller probabilities of selection. The aim is to protect the whole procedure against a given number of arbitrary outliers. As an illustration, we consider point and interval estimation for the correlation coefficient. We use Monte Carlo methods to compare this method with another robust bootstrap procedure, the winsorized bootstrap, recently suggested by Singh [Singh, K. (1998). Breakdown theory for bootstrap quantiles. Ann. Statist. 26:1719–1732].

Stochastic data mining tools for pipe blockage failure prediction

Abstract Failure prediction plays an important role in the management of urban water systems infrastructures. An accurate description of the deterioration of urban drainage systems is essential for optimal investment and rehabilitation planning. In the study presented in this paper, a new method to predict sewer pipe failure based on robust decision trees is proposed. Five other different stochastic failure prediction models – the non-homogeneous Poisson process, the zero-inflated non-homogeneous Poisson process, classical decision tress (CART and Random Forest algorithms), the Weibull accelerated lifetime model and the linear extended Yule process – are also implemented and explored in order to identify models that combine good failure prediction results with robustness. The six models were tested on the asset register and pipe failure register of a large US wastewater utility; only pipe blockage failures were considered in this study. The linear extended Yule process and the zero-inflated non-homogeneous Poisson process presented the overall best results throughout the models’ comparison, showing a good ability to detect pipes with high likelihood of blockage failure. Decision trees based on robust random forests only detected pipes with high likelihood of failure when considering a short-term prediction window; the accuracy of the predictions was one of the best when using the robust decision tree model. The Weibull accelerated lifetime model provided some of the best medium-term predictions but performed less well for shorter prediction windows.

A Comprehensive Approach for Spatial and Temporal Water Demand Profiling to Improve Management in Network Areas

The aim of this paper is to present a comprehensive approach for spatial and temporal demand profiling in water distribution systems. Multiple linear regression models for estimating network design parameters and decision trees for predicting daily demand patterns are presented. Proposed approach is a four-step procedure: data collection, data processing, data characterization, and spatial and temporal demand profiling. Continuous flow measurements and infrastructure and billing data were collected from a large set of water network areas and combined with census data. Main results indicate that family structures (i.e., families with elderly or adolescents), individuals’ mobility (i.e., people employed in the tertiary sector and university graduates) and public consumption (i.e., public spaces’ irrigation) are key-variables to profile water demand. Profiling models are of the utmost importance to describe water demand in areas with no monitoring but with similar socio-demographic characteristics to the ones analyzed, to improve network operation and to support network planning and design in new areas. Obtained models have been tested for new areas, showing good prediction performances.

Analysing the importance of variables for sewer failure prediction

ABSTRACT When defining the variables to predict sewer failure and therefore optimise sewer systems maintenance, it is important to identify the ones that most significantly influence the quality of the predictions or to define the smallest number of variables that is sufficient to obtain accurate predictions. In this study, three different statistical variable selection algorithms are applied for the first time to identify the most important variables for sewer failure prediction: the mutual information indicator, the out-of-bag samples concept, based on the random forest algorithm, and the stepwise search approach. The methods were applied to a real data-set that consists of the categorisation of sewer condition and associated physical characteristics. The mutual information and the stepwise search methods provided good predictions while those obtained using out-of-bag samples based on random forest were somewhat different, justified by the lack of robustness to imbalanced class distributions.

Image reconstruction based on circulant matrices

Abstract We propose a new method for image reconstruction based on circulant matrices. The novelty of this method is the image treatment using a simple and classical algebraic structure, the circulant matrix, which significantly reduces the computational effort, nevertheless providing reliable outputs. We compare the results with well established techniques such as the Principal Component Analysis (PCA) and the Discrete Fourier Transform (DFT), and the recently introduced Randomized Singular Value Decomposition (RSVD). We conclude that the quality is comparable whilst the computational time is considerably reduced.

Experimental repetitions and blockage of large stems at ogee crested spillways with piers

ABSTRACT Large wood is often transported by rivers into reservoirs during heavy rainfall events. When a critical section like a spillway is blocked and discharge capacity reduced, an uncontrolled increase of the reservoir water level may occur. This study aims to statistically analyse the importance of repetitions for the accuracy of experimental campaigns when studying blocking probabilities at ogee crested spillways equipped with piers. Systematic and reliable estimations based on physical models are critical for developing preventive measures against large wood blockage. Two statistical methods have been described and applied to calculate confidence intervals. A minimum number of repetitions for a maximum acceptable error is recommended for blocking probabilities. The minimum number of experimental repetitions has been statistically justified in accordance with a reasonable use of resources for experimental campaigns. In addition, a maximum acceptable level of error is proposed as a common metric of accuracy in large wood studies.

Robust Bootstrap Confidence Intervals: An Application Study for Denoising Images

Blurred images are a common problem in image processing and image deblurring techniques are sensitive to image noise. Some recent proposals use confidence intervals to image deblurring under the usual assumptions of Gaussian noise. However, non-normal noise, and particularly the presence of outliers, severely degrades the performance of the restoration. This results in poor state estimates and invalid inference. In this work, we propose a new image cleaning method that removes noise in blurred images based on robust confidence intervals. We consider that the observation noise distribution can be represented as a member of a contaminated normal neighbourhood and the analysis is based on nonparametric bootstrap confidence intervals. An illustration of this technique is presented. From the results we conclude that, regardless the distribution of the random noise, we obtained a blurry image ready to start the restoration process, without the problem of random noise even though not normal distributed.