Bogdan Trawiński

Nonparametric statistical analysis for multiple comparison of machine learning regression algorithms

In the paper we present some guidelines for the application of nonparametric statistical tests and post-hoc procedures devised to perform multiple comparisons of machine learning algorithms. We emphasize that it is necessary to distinguish between pairwise and multiple comparison tests. We show that the pairwise Wilcoxon test, when employed to multiple comparisons, will lead to overoptimistic conclusions. We carry out intensive normality examination employing ten different tests showing that the output of machine learning algorithms for regression problems does not satisfy normality requirements. We conduct experiments on nonparametric statistical tests and post-hoc procedures designed for multiple 1×N and N ×N comparisons with six different neural regression algorithms over 29 benchmark regression data sets. Our investigation proves the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.

Investigation of evolutionary optimization methods of TSK fuzzy model for real estate appraisal

Takagi-Sugeno-Kang fuzzy model to assist with real estate appraisals is described and optimized using evolutionary algorithms. Two approaches were compared in the paper. The first one consisted in learning the rule base and the second one in combining learning the rule base and tuning the membership functions in one process. Moreover two model variants with three and five triangular and trapezoidal membership functions describing each input variable were tested. Several TSK fuzzy models comprising different number of input variables were evaluated using the MATLAB. The evolutionary algorithms were based on Pittsburgh approach with the real coded chromosomes of constant length comprising whole rule base or both the rule base and all parameters of all membership functions. The experiments were conducted using training and testing sets prepared on the basis of actual 150 sales transactions made in one of Polish cities and located in a residential section. The results obtained were not decisive and further research in this area is needed.

On employing fuzzy modeling algorithms for the valuation of residential premises

In this paper, we investigate fuzzy modeling techniques for predicting the prices of residential premises, based on some main drivers such as usable area of premises, age of a building, number of rooms in a flat, floor on which a flat is located, number of storeys in a building as well as the distance from the city center. Our proposed modeling techniques rely on two aspects: the first one (called SparseFIS) is a batch off-line modeling method and tries to out-sparse an initial dense rule population by optimizing the rule weights within an iterative optimization procedure subject to constrain the number of important rules; the second one (called FLEXFIS) is a single-pass incremental method which is able to build up fuzzy models in an on-line sample-wise learning context. As such, it is able to adapt former generated prediction models with new data recordings on demand and also to cope with on-line data streams. The final obtained fuzzy models provide some interpretable insight into the relations between the various features and residential prices in form of linguistically readable rules (IF-THEN conditions). Both methods will be compared with a state-of-the-art premise estimation method usually conducted by many experts and exploiting heuristic concepts such as sliding time window, nearest neighbors and averaging. The comparison is based on a two real-world data set including prices for residential premises within the years 1998-2008.

Comparative Analysis of Premises Valuation Models Using KEEL, RapidMiner, and WEKA

The experiments aimed to compare machine learning algorithms to create models for the valuation of residential premises, implemented in popular data mining systems KEEL, RapidMiner and WEKA, were carried out. Six common methods comprising two neural network algorithms, two decision trees for regression, and linear regression and support vector machine were applied to actual data sets derived from the cadastral system and the registry of real estate transactions. A dozen of commonly used performance measures was applied to evaluate models built by respective algorithms. Some differences between models were observed.

Comparison of bagging, boosting and stacking ensembles applied to real estate appraisal

The experiments, aimed to compare three methods to create ensemble models implemented in a popular data mining system called WEKA, were carried out. Six common algorithms comprising two neural network algorithms, two decision trees for regression, linear regression, and support vector machine were used to construct ensemble models. All algorithms were employed to real-world datasets derived from the cadastral system and the registry of real estate transactions. Nonparametric Wilcoxon signed-rank tests to evaluate the differences between ensembles and original models were conducted. The results obtained show there is no single algorithm which produces the best ensembles and it is worth to seek an optimal hybrid multi-model solution.

Investigation of bagging ensembles of genetic neural networks and fuzzy systems for real estate appraisal

Artificial neural networks are often used to generate real appraisal models utilized in automated valuation systems. Neural networks are widely recognized as weak learners therefore are often used to create ensemble models which provide better prediction accuracy. In the paper the investigation of bagging ensembles combining genetic neural networks as well as genetic fuzzy systems is presented. The study was conducted with a newly developed system in Matlab to generate and test hybrid and multiple models of computational intelligence using different resampling methods. The results of experiments showed that genetic neural network and fuzzy systems ensembles outperformed a pairwise comparison method used by the experts to estimate the values of residential premises over majority of datasets.

Nonparametric statistical analysis of machine learning algorithms for regression problems

Several experiments aimed to apply recently proposed statistical procedures which are recommended for analysing multiple 1×n and n×n comparisons of machine learning algorithms were conducted. 11 regression algorithms comprising 5 deterministic and 6 neural network ones implemented in the data mining system KEEL were employed. All experiments were performed using 29 benchmark datasets for regression. The investigation proved the usefulness and strength of multiple comparison statistical procedures to analyse and select machine learning algorithms.

Exploration of bagging ensembles comprising genetic fuzzy models to assist with real estate appraisals

The study reported was devoted to investigate to what extent bagging approach could lead to the improvement of the accuracy machine learning regression models. Four algorithms implemented in the KEEL tool, including two evolutionary fuzzy systems, decision trees for regression, and neural network, were used in the experiments. The results showed that some bagging ensembles ensured higher prediction accuracy than single models.

Analysis of bagging ensembles of fuzzy models for premises valuation

The investigation of 16 fuzzy algorithms implemented in data mining system KEEL from the point of view of their usefulness to create bagging ensemble models to assist with real estate appraisal were presented in the paper. All the experiments were conducted with a real-world dataset derived from a cadastral system and registry of real estate transactions. The results showed there were significant differences in accuracy between individual algorithms. The analysis of measures of error diversity revealed that only the highest values of an average pairwise correlation of outputs were a profitable criterion for the selection of ensemble members.

Selection of Heterogeneous Fuzzy Model Ensembles Using Self-adaptive Genetic Algorithms

The problem of model selection to compose a heterogeneous bagging ensemble was addressed in the paper. To solve the problem, three self-adapting genetic algorithms were proposed with different control parameters of mutation, crossover, and selection adjusted during the execution. The algorithms were applied to create heterogeneous ensembles comprising regression fuzzy models to aid in real estate appraisals. The results of experiments revealed that the self-adaptive algorithms converged faster than the classic genetic algorithms. The heterogeneous ensembles created by self-adapting methods showed a very good predictive accuracy when compared with the homogeneous ensembles obtained in earlier research.