Marko Robnik-Šikonja

Theoretical and Empirical Analysis of ReliefF and RReliefF

Relief algorithms are general and successful attribute estimators. They are able to detect conditional dependencies between attributes and provide a unified view on the attribute estimation in regression and classification. In addition, their quality estimates have a natural interpretation. While they have commonly been viewed as feature subset selection methods that are applied in prepossessing step before a model is learned, they have actually been used successfully in a variety of settings, e.g., to select splits or to guide constructive induction in the building phase of decision or regression tree learning, as the attribute weighting method and also in the inductive logic programming.A broad spectrum of successful uses calls for especially careful investigation of various features Relief algorithms have. In this paper we theoretically and empirically investigate and discuss how and why they work, their theoretical and practical properties, their parameters, what kind of dependencies they detect, how do they scale up to large number of examples and features, how to sample data for them, how robust are they regarding the noise, how irrelevant and redundant attributes influence their output and how different metrics influences them.

Overcoming the Myopia of Inductive Learning Algorithms with RELIEFF

Current inductive machine learning algorithms typically use greedy search with limited lookahead. This prevents them to detect significant conditional dependencies between the attributes that describe training objects. Instead of myopic impurity functions and lookahead, we propose to use RELIEFF, an extension of RELIEF developed by Kira and Rendell [10, 11], for heuristic guidance of inductive learning algorithms. We have reimplemented Assistant, a system for top down induction of decision trees, using RELIEFF as an estimator of attributes at each selection step. The algorithm is tested on several artificial and several real world problems and the results are compared with some other well known machine learning algorithms. Excellent results on artificial data sets and two real world problems show the advantage of the presented approach to inductive learning.

Explaining Classifications For Individual Instances

We present a method for explaining predictions for individual instances. The presented approach is general and can be used with all classification models that output probabilities. It is based on the decomposition of a models predictions on individual contributions of each attribute. Our method works for the so-called black box models such as support vector machines, neural networks, and nearest neighbor algorithms, as well as for ensemble methods such as boosting and random forests. We demonstrate that the generated explanations closely follow the learned models and present a visualization technique that shows the utility of our approach and enables the comparison of different prediction methods.

Prognostic factors in the prediction of chronic wound healing by electrical stimulation

The aim of the study is to determine the effects of wound, patient and treatment attributes on the wound healing rate and to propose a system for wound healing rate prediction. Predicting the wound healing rate from the initial wound, patient and treatment data collected in a database of 300 chronic wounds is not possible. After considering weekly follow-ups, it was determined that the best prognostic factors are weekly follow-ups of the wound healing process, which alone were found to predict accurately the wound healing rate after a minimum follow-up period of four weeks (at least five measurements of wound area). After combining the follow-ups with wound, patient and treatment attributes, the minimum follow-up period was reduced to two weeks (at least three measurements of wound area). After a follow-up period of two weeks, it was possible to predict the wound healing rate of an independent test set of chronic wounds with a relative squared error of 0.347, and after three weeks, with a relative squared error of 0.181 (using regression trees with linear equations in its leaves). Regression trees with a relative squared error close to 0 produce better prediction than with an error closer to 1. Results show that the type of treatment is just one of many prognostic factors. Arranged in order of decreasing prediction capability, prognostic factors are: wound size, patients age, elapsed time from wound appearance to the beginning of the treatment, width-to-length ratio, location and type of treatment. The data collected support former findings that the biphasic- and direct-current stimulation contributes to faster healing of chronic wounds. The model of wound healing dynamics aids the prediction of chronic wound healing rate, and hence helps with the formulation of appropriate treatment decisions.

ClowdFlows: Online workflows for distributed big data mining

Abstract The paper presents a platform for distributed computing, developed using the latest software technologies and computing paradigms to enable big data mining. The platform, called ClowdFlows, is implemented as a cloud-based web application with a graphical user interface which supports the construction and execution of data mining workflows, including web services used as workflow components. As a web application, the ClowdFlows platform poses no software requirements and can be used from any modern browser, including mobile devices. The constructed workflows can be declared either as private or public, which enables sharing the developed solutions, data and results on the web and in scientific publications. The server-side software of ClowdFlows can be multiplied and distributed to any number of computing nodes. From a developer’s perspective the platform is easy to extend and supports distributed development with packages. The paper focuses on big data processing in the batch and real-time processing mode. Big data analytics is provided through several algorithms, including novel ensemble techniques, implemented using the map-reduce paradigm and a special stream mining module for continuous parallel workflow execution. The batch mode and real-time processing mode are demonstrated with practical use cases. Performance analysis shows the benefit of using all available data for learning in distributed mode compared to using only subsets of data in non-distributed mode. The ability of ClowdFlows to handle big data sets and its nearly perfect linear speedup is demonstrated.

Evaluation of prediction reliability in regression using the transduction principle

In machine learning community there are many efforts to improve overall reliability of predictors measured as an error on the testing set. But in contrast, very little research has been done concerning prediction reliability of a single answer. This article describes an algorithm that can be used for evaluation of prediction reliability in regression. The basic idea of the algorithm is based on construction of transductive predictors. Using them, the algorithm makes inference from the differences between initial and transductive predictions to the error on a single new case. The implementation of the algorithm with regression tress managed to significantly reduce the relative mean squared error on the majority of the tested domains.

Comprehensible evaluation of prognostic factors and prediction of wound healing

We analyzed the data of a controlled clinical study of the chronic wound healing acceleration as a result of electrical stimulation. The study involved a conventional conservative treatment, sham treatment, biphasic pulsed current, and direct current electrical stimulation. Data was collected over 10 years and suffices for an analysis with machine learning methods. So far, only a limited number of studies have investigated the wound and patient attributes which affect the chronic wound healing. There is none to our knowledge to include treatment attributes. The aims of our study are to determine effects of the wound, patient and treatment attributes on the wound healing process and to propose a system for prediction of the wound healing rate. First we analyzed which wound and patient attributes play a predominant role in the wound healing process and investigated a possibility to predict the wound healing rate at the beginning of the treatment based on the initial wound, patient and treatment attributes. Later we tried to enhance the wound healing rate prediction accuracy by predicting it after a few weeks of the wound healing follow-up. Using the attribute estimation algorithms ReliefF and RReliefF we obtained a ranking of the prognostic factors which was comprehensible to experts. We used regression and classification trees to build models for prediction of the wound healing rate. The obtained results are encouraging and may form a basis for an expert system for the chronic wound healing rate prediction. If the wound healing rate is known, then the provided information can help to formulate the appropriate treatment decisions and orient resources towards individuals with poor prognosis.

Evaluation of ordinal attributes at value level

We propose a novel context sensitive algorithm for evaluation of ordinal attributes which exploits the information hidden in ordering of attributes’ and class’ values and provides a separate score for each value of the attribute. Similar to feature selection algorithm ReliefF, the proposed algorithm exploits the contextual information via selection of nearest instances. The ordEval algorithm outputs probabilistic factors corresponding to the effect an increase/decrease of attribute’s value has on the class value. While the ordEval algorithm is general and can be used for analysis of any survey with graded answers, we show its utility on an important marketing problem of customer (dis)satisfaction. We develop a visualization technique and show how we can use it to detect and confirm several findings from marketing theory.

Experiments with cost-sensitive feature evaluation

Many machine learning tasks contain feature evaluation as one of its important components. This work is concerned with attribute estimation in the problems where class distribution is unbalanced or the misclassification costs are unequal. We test some common attribute evaluation heuristics and propose their cost-sensitive adaptations. The new measures are tested on problems which can reveal their strengths and weaknesses.

A Decade of Euroleague Basketball: an Analysis of Trends and Recent Rule Change Effects

Abstract The International Basketball Federation (FIBA) recently introduced major rule changes that came into effect with the 2010/11 season. Most notably, moving the three-point arc and changing the shot-clock. The purpose of this study was to investigate and quantify how these changes affect the game performance of top-level European basketball players. In order to better understand these changes, we also investigated past seasons and showed the presence of several trends, even in the absence of significant rule changes. A large set of game statistics for 10 seasons and 2198 Euroleague basketball games in which top European clubs competed was analyzed. Results show that the effects of the rule changes are contrary to trends in recent years