Zbigniew Telec

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

In the paper we present some guidelines for the application of nonparametric statistical tests and post-hoc procedures devised to perform multiple comparisons of machine learning algorithms. We emphasize that it is necessary to distinguish between pairwise and multiple comparison tests. We show that the pairwise Wilcoxon test, when employed to multiple comparisons, will lead to overoptimistic conclusions. We carry out intensive normality examination employing ten different tests showing that the output of machine learning algorithms for regression problems does not satisfy normality requirements. We conduct experiments on nonparametric statistical tests and post-hoc procedures designed for multiple 1×N and N ×N comparisons with six different neural regression algorithms over 29 benchmark regression data sets. Our investigation proves the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.

Investigation of bagging ensembles of genetic neural networks and fuzzy systems for real estate appraisal

Artificial neural networks are often used to generate real appraisal models utilized in automated valuation systems. Neural networks are widely recognized as weak learners therefore are often used to create ensemble models which provide better prediction accuracy. In the paper the investigation of bagging ensembles combining genetic neural networks as well as genetic fuzzy systems is presented. The study was conducted with a newly developed system in Matlab to generate and test hybrid and multiple models of computational intelligence using different resampling methods. The results of experiments showed that genetic neural network and fuzzy systems ensembles outperformed a pairwise comparison method used by the experts to estimate the values of residential premises over majority of datasets.

Nonparametric statistical analysis of machine learning algorithms for regression problems

Several experiments aimed to apply recently proposed statistical procedures which are recommended for analysing multiple 1×n and n×n comparisons of machine learning algorithms were conducted. 11 regression algorithms comprising 5 deterministic and 6 neural network ones implemented in the data mining system KEEL were employed. All experiments were performed using 29 benchmark datasets for regression. The investigation proved the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.

Exploration of bagging ensembles comprising genetic fuzzy models to assist with real estate appraisals

The study reported was devoted to investigate to what extent bagging approach could lead to the improvement of the accuracy machine learning regression models. Four algorithms implemented in the KEEL tool, including two evolutionary fuzzy systems, decision trees for regression, and neural network, were used in the experiments. The results showed that some bagging ensembles ensured higher prediction accuracy than single models.

Analysis of bagging ensembles of fuzzy models for premises valuation

The investigation of 16 fuzzy algorithms implemented in data mining system KEEL from the point of view of their usefulness to create bagging ensemble models to assist with real estate appraisal were presented in the paper. All the experiments were conducted with a real-world dataset derived from a cadastral system and registry of real estate transactions. The results showed there were significant differences in accuracy between individual algorithms. The analysis of measures of error diversity revealed that only the highest values of an average pairwise correlation of outputs were a profitable criterion for the selection of ensemble members.

A Multi-agent System to Assist with Real Estate Appraisals Using Bagging Ensembles

The multi-agent system for real estate appraisals MAREA was extended to include aggregating agents, which could create ensemble models applying the bagging approach, was presented in the paper. The major part of the study was devoted to investigate to what extent bagging approach could lead to the improvement of the accuracy machine learning regression models. Four algorithms implemented in the KEEL tool, including linear regression, decision trees for regression, support vector machines, and artificial neural network of MLP type, were used in the experiments. The results showed that bagging ensembles ensured higher prediction accuracy than single models.

Evaluation of Neural Network Ensemble Approach to Predict from a Data Stream

We have recently worked out a method for building reliable predictive models from a data stream of real estate transactions which applies the ensembles of genetic fuzzy systems and neural networks. The method consists in building models over the chunks of a data stream determined by a sliding time window and enlarging gradually an ensemble by models generated in the course of time. The aged models are utilized to compose ensembles and their output is updated with trend functions reflecting the changes of prices in the market. In the paper we present the next series of extensive experiments to evaluate our method with the ensembles of artificial neural networks. We examine the impact of the number of aged models used to compose an ensemble on the accuracy and the influence of the degree of polynomial trend functions employed to modify the results on the performance of neural network ensembles. The experimental results were analysed using statistical approach embracing nonparametric tests followed by post-hoc procedures designed for multiple N×N comparisons.

Application of mixture of experts to construct real estate appraisal models

Several experiments were conducted in order to investigate the usefulness of mixture of experts (ME) approach to an online internet system assisting in real estate appraisal All experiments were performed using 28 real-world datasets composed of data taken from a cadastral system and GIS data derived from a cadastral map The analysis of the results was performed using recently proposed statistical methodology including nonparametric tests followed by post-hoc procedures designed especially for multiple 1×n and n×n comparisons GLM (general linear model) architectures of mixture of experts achieved better results for ME with an adaptive variance parameter for each expert, whereas MLP (multilayer perceptron) architectures - for standard mixtures of experts.

A multi-agent system to assist with property valuation using heterogeneous ensembles of fuzzy models

The multi-agent system for real estate appraisals MAREA was extended to include aggregating agents, which are equipped with heuristic optimization algorithms and can create heterogeneous ensemble models, was presented in the paper. The major part of the study was devoted to investigate the predictive accuracy of heterogeneous ensembles comprising fuzzy models and to compare them with homogenous bagging ensembles. Six optimization heuristics including genetic, tabu search, simulated annealing, minimum average and random algorithms were implemented and applied to obtain the best ensembles for different number of fuzzy models.

Concept of a Multi-Agent System for Assisting in Real Estate Appraisals

The general architecture of a multi-agent system for real estate appraisal (MAREA) is presented in the paper. The appraisal data warehouse is filled with data drawn from source cadastral databases. Data driven appraisal models are created using different machine learning algorithms. Architecture of the MAREA system in the JADE platform was also proposed. Several experiments aimed to assess the usefulness of different machine learning algorithms for the system were conducted using the KEEL tool.