How Much Can We See? A Note on Quantifying Explainability of Machine Learning Models
HH OW M UCH C AN W E S EE ? A N OTE ON Q UANTIFYING E XPLAINABILITY OF M ACHINE L EARNING M ODELS
A P
REPRINT
Gero Szepannek
Stralsund University of Applied SciencesZur Schwedenschanze 1518435 StralsundGermanyDecember 17, 2019 A BSTRACT
One of the most popular approaches to understanding feature effects of modern black box machinelearning models are partial dependence plots (PDP). These plots are easy to understand but only ableto visualize low order dependencies. The paper is about the question ”How much can we see?” : Aframework is developed to quantify the explainability of arbitrary machine learning models, i.e. upto what degree the visualization as given by a PDP is able to explain the predictions of the model.The result allows for a judgement whether an attempt to explain a black box model is sufficient ornot.
In the recent past a considerable number of auto machine learning frameworks such as
H2O , auto-sklearn (Feureret al, 2015) or mlr3 (Bischl et al, 2016) have been developed and made publicly available and thus simplify creationof complex machine learning models. On the other hand, advances in hardware technology allow these models to getmore and more complex with huge numbers of parameters such as deep learning models (cf. e.g. LeCun et al, 2015).Properly parameterized modern ML algorithms are often of superior predictive accuracy.The popularity of modern ML algorithms is based on the fact that they are very flexible with regard to to detectionof complex nonlinear high dimensional multivariate dependencies without the need for an explicit specification of thetype of the functional relationship of the dependence. As a consequence the resulting models are often called to be ofblack box nature which has led to an increasing need of tools for their interpretation.Depending on the context, there are different requirements to explainability (cf. e.g. Biecek, 2018; Szepannek andAschenbruck, 2019) given by different targets of explanation such as explanations of predictions for individual obser-vations (Ribeiro et al, 2016; Štrumbelj and Kononenko, 2014; Lundberg and Lee, 2017; Staniak and Biecek, 2018),importance of features (Breiman, 2001; Casalicchio et al, 2018) and feature effects (Friedman, 2001; Apley, 2016;Goldstein et al, 2015).This paper concentrates on the latter: feature effects do investigate the dependency of the predictions by a model onone (or several) predictors. Molnar et al (2019) work out that superior performance comes along with the ability tomodel nonlinear high oder dependencies which are naturally hard to understand for humans. As a remedy, criteria aredeveloped in order to quantify the interpretability of a model and in consequence allow for multi-objective optimizationof the model selection process with respect to both: predictive performance and interpretability.The approach in this paper is somewhat different: Starting with any model (which is often the best one in terms ofpredcitive accuracy) one of the most popular approaches to understanding of feature effects are partial dependenceplots (PDP) which are introduced in Section 2. Partial dependence plots are easy to understand but only able tovisualize low order dependencies. The question that is asked in this paper is ”How much can we see?” : In Section 3 aframework is developed to quantify the explainability of a model, i.e. up to what degree the visualization as given by a r X i v : . [ s t a t . M L ] D ec ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT −3 −2 −1 0 1 2 − − PDP ( X ) x P D P and P r ed i c t i on s −10 −5 0 5 10 − − PDP vs. Predictions
Prediction P D P ( X ) Figure 1: PDP for variable X (left) and match of partial dependence function P D ( X ) and predicted values ˆ f ( x ) .a PDP is able to explain a model. This allows us to judge whether an attempt to explain the predictions of a model issufficient or not. In Section 4 the approach is demonstrated on two examples uing both artificial as well as real-worlddata and finally, a summary and an outlook are given in Section 5. Partial depencence plots (PDP Friedman, 2001) are a model-agnostic approach in order to understand feature effectsand are applicable to arbitrary models, here denoted by ˆ f ( x ) . The vector of predictor variables x = ( x s , x c ) is furthersubdivided into two subsets: x s and x c . The partial dependence function is given by P D s ( X ) = P D s ( X s ) = (cid:90) ˆ f ( X s , X c ) dP ( X c ) , (1)i.e. it computes the average prediction given the variable subset X s takes the values x s . In practise, the partialdependence curve is estimated by (cid:100) P D s ( x ) = (cid:100) P D s ( x s ) = 1 n n (cid:88) i =1 ˆ f ( x s , x ic ) . (2)Note that for X s = X the partial dependence function P D s ( x ) corresponds to ˆ f ( x ) and in the extreme, for thevariable subset s = ∅ , i.e. X c = X , this will end up in: P D ∅ ( X ) = P D ∅ = (cid:90) ˆ f ( X ) dP ( X ) , (3)which is independent of x and corresponds to the constant average prediction of the model estimated by: (cid:100) P D ∅ ( x ) = (cid:100) P D ∅ = 1 n n (cid:88) i =1 ˆ f ( x i ) . (4) In the rest of the paper a measure is defined in order to quantify up to what degree this visualization as given by a PDPis able to explain a model. As an introductory example consider simulated data of two independent random variables X , X ∼ N (0 , and a dependent variable Y according to the data generating process:2ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT −10 −5 0 5 10 − − PDP vs. Predictions
Prediction P D P ( X ) −10 −5 0 5 10 − − PDP vs. Predictions
Prediction P D P ( X , X ) Figure 2: Match of partial dependence function
P D ( X ) and predicted values for the first example (left). The plot onthe right illustrates, that for a 2D-PDP using all input variables X s = X a perfect match is obtained. Y = aX + bX + (cid:15), (5)with a = 5 , b = 3 and a standard normally distributed error term (note that the error term could also be omitted, here). Y depends linearly on X and X . Afterwards a default random forest model (using both variables X and X andthe R package randomForest , Liaw and Wiener, 2002) is computed. Figure 1 (left) shows the corresponding partialdependence plot for variable X together with the predictions for all observations. It can be recognized that – of course– the PDP does not exactly match the predictions. In Figure 1 (right) the x-axis is changed: here, the predictions ofthe model ˆ f ( x i ) (x-axis) are plotted against their corresponding values of the partial dependence function P D X ( x i ) (y-axis). The better the PDP would represent the model the closer the points should be to the diagonal.A first step towards defining explainability consists in answering the question: How close is what I see to the truepredictions of the model?
For this reason, a starting point for further analysis is given by computing the differencesbetween the partial dependence function
P D s ( X s ) and the model’s predictions. A natural approach to quantifiyingthese differences is given by computing the expected squared difference: ASE ( P D s ) = (cid:90) (cid:16) ˆ f ( X ) − (cid:100) P D s ( X ) (cid:17) dP ( X s ) , (6)which can be empirically estimated by: (cid:100) ASE ( P D s ) = 1 n n (cid:88) i =1 (cid:16) ˆ f ( x i ) − (cid:100) P D s ( x i ) (cid:17) . (7)Remarkably the ASE does not calculate the error between model’s predictions and the obervations but between thepartial dependence function and the model’s predictions here. Further, in order to benchmark the
ASE ( P D s ) of apartial dependence function it can be compared to the ASE ( P D ∅ ) of the naive constant average prediction P D ∅ : ASE ( P D ∅ ) = (cid:90) (cid:16) ˆ f ( x ) − P D ∅ (cid:17) dP ( X s ) , (8)and its empirical estimate: (cid:100) ASE ( P D ∅ ) = 1 n n (cid:88) i =1 (cid:16) ˆ f ( x i ) − (cid:100) P D ∅ (cid:17) . (9)3ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT
10 20 30
PDP
LSTAT P r ed i c t i on
10 20 30 40 50
PDP vs. Predictions
Prediction P D P ( L S T A T ) Figure 3: Most explainable PDP for a random forest model on the boston housing data (left) as well as match ofpreditions and PDP (right).Finally one can relate both
ASE ( P D s ) and ASE ( P D ∅ ) and define explainability Υ of any black box model ˆ f ( X ) by a partial dependence function P D s ( X ) by the ratio Υ( P D s ) = 1 − ASE ( P D s ) ASE ( P D ∅ ) (10)similar to the common R goodness of fit statistic: Υ close to 1 means that a model is well represented by a PDP andthe smaller it is the less of the model’s predictions are explained. Starting again with the introductory example from the previous Section. From data generation the choice of a > b results in a higher variation of Y with regard to X . Accordingly, it can be expected that P D ( X ) is closer tothe model’s predictions than the P D ( X ) (cf. Figure 2, left) and thus has a higher explainability. Computing bothexplainabilities confirms this: Υ( P D X ) = 0 . > .
356 = Υ(
P D X ) . For a two dimensional PDP the partialdependence function corresponds to the true predictions resulting in an explainability of 1, i.e. the model is perfectlyexplained by the partial dependence curve (Figure 2, right).Variable Υ Variable Υ lstat 0.512 age 0.018rm 0.410 b 0.012lon 0.085 chas 0.004nox 0.056 zn 0.002ptratio 0.056 lat 0.001indus 0.046 rad -0.002tax 0.030 dis -0.004crim 0.025Table 1: Explainability of 1D PDPs for a random forest model of the Boston housing data based on different variables.As a nd example the popular Boston housing real world data set (Dua and Graff, 2017) is used which has also beendone by other authors (cf. e.g. Greenwell, 2017) in order to illustrate partial dependence plots. Again, a defaultrandom forest model has been built as in the example before. Figure 3 (left) shows the PDP for variable LSTAT . Thecorresponding explainabilities identify these PDPs to be the two most useful ones (cf. Table 4).4ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT
10 20 30
PD(x) lstat r m
10 20 30
PD(x) − f(x) lstat r m −19.4−9.709.719.4
10 20 30 40 50
PDP vs. Predictions
Prediction P D P Figure 4: Two dimensional PDP for the variables variables
LSTAT and RM of a random forest on the Boston housingdata (left) and the corresponding match of PDP and preditions(right).Nonetheless, from the explainabilities of all single variable’s partial dependence functions it is also obvious thatconsidering single PDPs alone is not sufficient to understand the behaviour of the model in this case. Taking a closerlook at the partial dependence function vs. the predicted values on the data set (Figure 3, left) shows further thate.g. for large values of the variable LSTAT the partial dependence function (dotted red line) appears to systematicallyoverestimate the predictions for this example.Taking into account for the explainability Υ provides us with the information of how strong the true predictions deviatefrom what we do see in the partial dependence plot. Comparison of Figure 3 (right) and Figure 4, (right) illustratesthat the two dimensional PDP of the two most explainable variables LSTAT and RM is much more explainable in thiscase ( Υ = 0 . ). A coloured scatterplot can be used in order to visualize the two dimensional PDP together withthe distribution of the observations in both variables (Figure 4, left) as well as the gap between the partial dependencefunction and the predicted values by the model (Figure 4, center) where in both plots blue represents low (/negative)and red represents high (/positive) values.Generally, partial dependence functions are not resticted to 1D or 2D thus one can analyze how much the partialdependence function gets closer to the model if we include additional variables. Table 2 shows the results of a forwardvariable selection using Υ as selection criterion for stepwise inclusion of variables in X s :Step Variable Ups Step Variable Ups1 lstat 0.512 9 tax 0.9862 rm 0.759 10 age 0.9933 lon 0.805 11 b 0.9974 nox 0.847 12 lat 0.9995 crim 0.894 13 rad 0.9996 dis 0.931 14 chas 1.0007 ptratio 0.958 15 zn 1.0008 indus 0.974Table 2: Results of forward variable selection for X s according to maximise explainability Υ for the Boston housingdata.It can be seen that with as few as six variables an explainability of Υ > . is obtained. Nonetheless, there isstill need for collection of experiences what level of Υ could be considered as a sufficient explanation of a model.Furthermore, although it is principally possible to compute partial dependence for vectors X s of any dimension itsvisualization is restricted to dim ( X s ) ≤ . As an attempt to consider more than two variables at once, a scatterplotmatrix of two-dimensional PDPs can be computed. Figure 5 shows such a scatterplot matrix for the first four variablesof the Boston housing data according to Υ -based variable selection: It can be easily recognized from the plot that ahigh number of rooms rm as well as a low percentage of habitants with lower status lstat are most important forprediction of high house prices as well as a cooccurence of both. But this visualization of course still fails to visualize5ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT lstat lstat r m −71.3 −71.2 −71.1 −71.0 −70.9 −70.8 lstat r m lstat r m rm −71.3 −71.2 −71.1 −71.0 −70.9 −70.8 lstat r m lstat r m lon − . − . − . − . − . − . lstat r m nox Figure 5: Scatterplot matrix of 2D PDPs for the first four variables according to Υ -based variable selection.nonlinear dependencies of higher order which are potentially also identified by the model ˆ f ( x ) and it remains an opentopic for future resarch activities to develop methodologies to understand high order interactions of variables withinmodels, e.g. based on the works of Britton (2019) and Gosiewska and Biecek (2019). In contrast, some authors suggestrestricting to interpretable models (Rudin, 2019) which often (but not always, cf. e.g. Bücker et al, 2019) trades offwith predictive power. In general the benefit of using a rather complex models should be analyzed for each situationseparately. Molnar et al (2019) suggest a strategy for simultaneously optimizing a compromise between accuracy andinterpretability. Partial dependence plots as one of the most common tools to explain feature effects of black box machine learningmodels are investigated with regard to the extent that they are able to explain a model’s predictions.Using differences between the predictions of the model and their corresponding values of a partial dependence functiona framework has been developed to quantify how well a PDP is able to explain the underlying model. The result interms of the measure of explainability Υ allows to assess whether the explanation of a black box model may besufficient or not.As a graphical approach to assess explainability the match between the partial dependence function and the model’soutput as a scatterplot of the data in the ( ˆ f ( x ) , P D ( x )) plane is proposed. For two-dimensional PDPs the differencesbetween both functions can be visually localized in a coloured scatterplot of ( P D ( x ) − ˆ f ( x )) .6ow Much Can We See? A Note on Quantifying Explainability of Machine Learning Models A PREPRINT
Two simple examples have been presented in order to illustrate the concept of explainability. It can be seen thatlooking at PDPs is not necessarily sufficient to understand a model’s behaviour. As an open issue it has to be notedthat although PDP visualizations are restricted to dimensions lower or equal than two the models in general use morethan two variables. Of course, analysts are able to look at several PDPs at the same time, e.g. using scatterplotmatrices of partial dependence plots but up to our knowledge no literature is available howfar humans are able tocombine information of more than two PDPs in order to get a clearer picture of a model’s behaviour which remains asa topic of future research.
References
Apley D (2016) Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. URL https://arxiv.org/abs/1612.08468
Biecek P (2018) DALEX: Explainers for complex predictive models in R. Journal of Machine Learning Research19(84):1–5Bischl B, Lang M, Kotthoff L, Schiffner J, Richter J, Studerus E, Casalicchio G, Jones ZM (2016) mlr: Machinelearning in R. Journal of Machine Learning Research 17(170):1–5Breiman L (2001) Random forests. Machine Learning 45(1):5–32Britton M (2019) VINE: Visualizing Statistical Interactions in Black Box Models. URL https://arxiv.org/abs/1904.00561
Bücker M, Szepannek G, Biecek P, Gosiewska A, Staniak M (2019) Transparency of machine learning models incredit scoring. In: Credit Scoring and Credit Control XVI), URL
Casalicchio G, Molnar C, Bisch B (2018) Visualizing the feature importance for black box models. In: Joint EuropeanConference on Machine Learning and Knowledge Discovery in Databases, pp 655–670Dua D, Graff C (2017) UCI machine learning repository. URL http://archive.ics.uci.edu/ml
Feurer M, Klein A, Eggensperger K, Springenberg J, Blum M, Hutter F (2015) Efficient and robust automated machinelearning. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in Neural InformationProcessing Systems 28, pp 2962–2970Friedman J (2001) Greedy function approximation: A gradient boosting machine. Annals of Statistics 29:1189–1232Goldstein A, Kapelner A, Bleich J, Pitkin E (2015) Peeking inside the black box: Visualizing statistical learningwith plots of individual conditional expectation. Journal of Computational and Graphical Statistics 24(1):44–65,DOI 10.1080/10618600.2014.907095Gosiewska A, Biecek P (2019) iBreakDown: Uncertainty of Model Explanations for Non-additive Predictive Models.URL https://arxiv.org/abs/1903.11420
Greenwell B (2017) pdp: An R Package for Constructing Partial Dependence Plots. The R Journal 9(1):421–436,DOI 10.32614/RJ-2017-016LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444Liaw A, Wiener M (2002) Classification and regression by randomforest. R News 2(3):18–22Lundberg SM, Lee SI (2017) A unified approach to interpreting model predictions. In: Advances in Neural InformationProcessing Systems, pp 4765–4774Molnar C, Casalicchio G, Bischl B (2019) Quantifiying Interpretability of Arbitrary Machine Learning ModelsThrough Functional Decomposition. URL https://arxiv.org/abs/1904.03867
Ribeiro MT, Singh S, Guestrin C (2016) "Why should I trust you?": Explaining the predictions of any classifier. In:Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp1135–1144, DOI 10.1145/2939672.2939778Rudin C (2019) Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use InterpretableModels Instead. URL https://arxiv.org/abs/1811.10154
Staniak M, Biecek P (2018) Explanations of Model Predictions with live and breakDown Packages. The R Journal10(2):395–409, DOI 10.32614/RJ-2018-072, URL https://doi.org/10.32614/RJ-2018-072https://doi.org/10.32614/RJ-2018-072