Erik Štrumbelj

Explanation and reliability of prediction models: the case of breast cancer recurrence

In this paper, we describe the first practical application of two methods, which bridge the gap between the non-expert user and machine learning models. The first is a method for explaining classifiers’ predictions, which provides the user with additional information about the decision-making process of a classifier. The second is a reliability estimation methodology for regression predictions, which helps the users to decide to what extent to trust a particular prediction. Both methods are successfully applied to a novel breast cancer recurrence prediction data set and the results are evaluated by expert oncologists.

Explaining prediction models and individual predictions with feature contributions

We present a sensitivity analysis-based method for explaining prediction models that can be applied to any type of classification or regression model. Its advantage over existing general methods is that all subsets of input features are perturbed, so interactions and redundancies between features are taken into account. Furthermore, when explaining an additive model, the method is equivalent to commonly used additive model-specific methods. We illustrate the method’s usefulness with examples from artificial and real-world data sets and an empirical analysis of running times. Results from a controlled experiment with 122 participants suggest that the method’s explanations improved the participants’ understanding of the model.

Basketball Shot Types and Shot Success in Different Levels of Competitive Basketball

The purpose of our research was to investigate the relative frequencies of different types of basketball shots (above head, hook shot, layup, dunk, tip-in), some details about their technical execution (one-legged, two-legged, drive, cut, …), and shot success in different levels of basketball competitions. We analysed video footage and categorized 5024 basketball shots from 40 basketball games and 5 different levels of competitive basketball (National Basketball Association (NBA), Euroleague, Slovenian 1st Division, and two Youth basketball competitions). Statistical analysis with hierarchical multinomial logistic regression models reveals that there are substantial differences between competitions. However, most differences decrease or disappear entirely after we adjust for differences in situations that arise in different competitions (shot location, player type, and attacks in transition). Differences after adjustment are mostly between the Senior and Youth competitions: more shots executed jumping or standing on one leg, more uncategorised shot types, and more dribbling or cutting to the basket in the Youth competitions, which can all be attributed to lesser technical and physical ability of developing basketball players. The two discernible differences within the Senior competitions are that, in the NBA, dunks are more frequent and hook shots are less frequent compared to European basketball, which can be attributed to better athleticism of NBA players. The effect situational variables have on shot types and shot success are found to be very similar for all competitions.

A general method for visualizing and explaining black-box regression models

We propose a method for explaining regression models and their predictions for individual instances. The method successfully reveals how individual features influence the model and can be used with any type of regression model in a uniform way. We used different types of models and data sets to demonstrate that the method is a useful tool for explaining, comparing, and identifying errors in regression models.

A Decade of Euroleague Basketball: an Analysis of Trends and Recent Rule Change Effects

Abstract The International Basketball Federation (FIBA) recently introduced major rule changes that came into effect with the 2010/11 season. Most notably, moving the three-point arc and changing the shot-clock. The purpose of this study was to investigate and quantify how these changes affect the game performance of top-level European basketball players. In order to better understand these changes, we also investigated past seasons and showed the presence of several trends, even in the absence of significant rule changes. A large set of game statistics for 10 seasons and 2198 Euroleague basketball games in which top European clubs competed was analyzed. Results show that the effects of the rule changes are contrary to trends in recent years

A Comment on the Bias of Probabilities Derived From Betting Odds and Their Use in Measuring Outcome Uncertainty

Probabilities from bookmaker odds are often used in measures of short-run outcome uncertainty. We analyzed the most commonly used methods for deriving probability forecasts from odds and found that basic normalization (BN) produces biased probabilities. Furthermore, differences between probabilities produced with BN, regression models, or Shin probabilities are large enough to lead to contradictory conclusions when used to measure outcome uncertainty. We also provide evidence against the reported bias of bookmakers favoring better supported teams and show how past evidence of such a bias is possibly only due to a misinterpretation of the results.

Towards a Model Independent Method for Explaining Classification for Individual Instances

Recently, a method for explaining the models decision for an instance was introduced by Robnik-Sikonja and Kononenko. It is a rare example of a model-independent explanation method. In this paper we make a step towards formalization of the model-independent explanation methods by defining the criteria and a testing environment for such methods. We extensively test the aforementioned method and its variations. The results confirm some of the qualities of the original method as well as expose several of its shortcomings. We propose a new method, based on attribute interactions, that overcomes the shortcomings of the original method and serves as a theoretical framework for further work.

Efficiently explaining decisions of probabilistic RBF classification networks

For many important practical applications model transparency is an important requirement. A probabilistic radial basis function (PRBF) network is an effective non-linear classifier, but similarly to most other neural network models it is not straightforward to obtain explanations for its decisions. Recently two general methods for explaining of a models decisions for individual instances have been introduced which are based on the decomposition of a models prediction into contributions of each attribute. By exploiting the marginalization property of the Gaussian distribution, we show that PRBF is especially suitable for these explanation techniques. By explaining the PRBFs decisions for new unlabeled cases we demonstrate resulting methods and accompany presentation with visualization technique that works both for single instances as well as for the attributes and their values, thus providing a valuable tool for inspection of the otherwise opaque models.

A Bayesian hierarchical latent trait model for estimating rater bias and reliability in large-scale performance assessment

We propose a novel approach to modelling rater effects in scoring-based assessment. The approach is based on a Bayesian hierarchical model and simulations from the posterior distribution. We apply it to large-scale essay assessment data over a period of 5 years. Empirical results suggest that the model provides a good fit for both the total scores and when applied to individual rubrics. We estimate the median impact of rater effects on the final grade to be ± 2 points on a 50 point scale, while 10% of essays would receive a score at least ± 5 different from their actual quality. Most of the impact is due to rater unreliability, not rater bias.

A Bayesian approach to forecasting daily air-pollutant levels

Forecasting air-pollutant levels is an important issue, due to their adverse effects on public health, and often a legislative necessity. The advantage of Bayesian methods is their ability to provide density predictions which can easily be transformed into ordinal or binary predictions given a set of thresholds. We develop a Bayesian approach to forecasting PM