Ada-SISE: Adaptive Semantic Input Sampling for Efficient Explanation of Convolutional Neural Networks
Mahesh Sudhakar, Sam Sattarzadeh, Konstantinos N. Plataniotis, Jongseong Jang, Yeonjeong Jeong, Hyunwoo Kim
AADA-SISE: ADAPTIVE SEMANTIC INPUT SAMPLING FOR EFFICIENT EXPLANATION OFCONVOLUTIONAL NEURAL NETWORKS
Mahesh Sudhakar (cid:63) , Sam Sattarzadeh (cid:63) , Konstantinos N. Plataniotis (cid:63) ,Jongseong Jang † , Yeonjeong Jeong † , Hyunwoo Kim † (cid:63) Department of Electrical & Computer Engineering, University of Toronto † Fundamental Research Lab, LG AI Research
ABSTRACT
Explainable AI (XAI) is an active research area to in-terpret a neural network’s decision by ensuring transparencyand trust in the task-specified learned models. Recently,perturbation-based model analysis has shown better inter-pretation, but backpropagation techniques are still prevailingbecause of their computational efficiency. In this work, wecombine both approaches as a hybrid visual explanation al-gorithm and propose an efficient interpretation method forconvolutional neural networks. Our method adaptively se-lects the most critical features that mainly contribute towardsa prediction to probe the model by finding the activated fea-tures. Experimental results show that the proposed methodcan reduce the execution time up to 30% while enhancingcompetitive interpretability without compromising the qual-ity of explanation generated.
Index Terms — CNNs, Deep Learning, Explainable AI,Interpretable ML, Neural Network Interpretability.
1. INTRODUCTION
Over the recent past years, access to a lot of digital data,the advances in computing facilities, and the facile accessto many readily available pre-trained models have fueled thegrowth in deep learning. Although such models produce highaccuracy in object recognition, the interpretability [1] of theirdecisions is also essential to convince the stakeholders or lo-cate any potential bias in the underlying data. With AI cur-rently being employed in various fields such as in healthcare,consumer retails, and banking, it is high time to develop “Re-sponsible AI” [2] for society. To ensure the uniformity of thetraining data’s distribution, lately, there is an increase in mod-ern open-source toolkits [3, 4] that acts as a common frame-work to evaluate a model’s fairness.Explainable AI (XAI) has recently been offering many al-gorithms to interpret a model’s behavior. Based on their us-age at the training process’s timeline, XAI approaches can bebroadly classified into ad-hoc and post-hoc methods. In termsof their explanation ability to interpret a single instance or thewhole decision process, XAI can be classified into local and global . They can also be categorized into model-agnostic and model-specific methods, based on the requirement to specifythe model’s architecture.In this work, we study such a recent post-hoc , local , and model-specific XAI algorithm - Semantic Input Sampling forExplanation (SISE) [5] developed for image classificationtasks. Building on this method, we propose a way to improveits run-time while retaining its overall performance withoutcompromising the visual explanation’s quality. Our approachintroduces a novel way to adaptively select the most impor-tant feature information to be considered for the subsequentsteps of the algorithm’s operation. This modification acts asa smart filtering procedure that mutates the existing methodinto an automated, unified solution by eliminating the needfor an end-user to tune the hyper-parameters. To demon-strate this claim, we evaluate our approach with the originalalgorithm’s performance in terms of the visual explanationquality, overall benchmark analysis, and execution time.
2. BACKGROUND2.1. Existing methods
The prior works on post-hoc visual XAI can be divided intothree main groups: ‘backpropagation-based’, ‘perturbation-based’, and ‘CAM-based’ methods. The backpropagation-based methods mainly operate by backpropagating the signalsfrom the output neuron of the model to the inputs [6, 7] or thehidden nodes of the model [8], in order to calculate gradi-ent [7] of relevance [9] terms. Perturbation-based approachesrely on feed-forwarding the model with perturbed copies ofthe input image. They interpret the model’s behavior usingtechniques such as probing the model with random masks[10] or optimizing a perturbation mask for each input [11].Moreover, CAM-based methods are built based on the ClassActivation Mapping (CAM) method [12] and are used specif-ically for CNNs by taking advantage of the phenomenon ofthis type of networks in weak object localization, as stated in[13]. Most of these methods are developed by backpropaga-tion techniques [14, 15] or perturbation techniques [16]. a r X i v : . [ c s . C V ] F e b .2. Semantic Input Sampling for Explanation SISE is a recent explanation method that spans among allthree mentioned visual XAI methods, although it is generallyclassified as a perturbation-based algorithm. SISE employsthe feature information underlying the model’s various depthsto generate a saliency-based high-resolution visual explana-tion map. For a given trained classification model δ : I → R C with N convolutional blocks that outputs a confidence scoreover C classes for each input image I , SISE generates a 2-dimensional explanation map Y I,δ ( λ ) for λ in the domain offeature maps Λ , through its four-phased architecture.In the first phase ( Feature map Extraction ), pooling lay-ers p l of the model for l ∈ { , .., N } are targeted, and theircorresponding feature maps F [ p ] k for k ∈ { , .., M p } are col-lected. As this operation is independent of the classifier part,there would be a lot of irrelevant feature information aboutthe background or other object classes (if present), in addi-tion to the class of interest. The excess information is fil-tered out in the second phase ( Feature map Selection ) basedon their backpropagation scores. Here, the feature maps withpositive gradients towards a particular class are selected andpost-processed to be converted into attribution masks A [ p ] k , viabilinear interpolation followed by normalization in the range [0 , .The generated attribution masks are then scored by weigh-ing based on their classification scores in the third phase ( At-tribution mask Scoring ) and later combined to form a layervisualization map V [ p ] I,δ ( λ ) . These preceding steps are repeatedfor all pooling (down-sampling) layers p l of the network andthen passed to the final phase ( Fusion ) of the algorithm, wherethey are fused in a cascading manner under a series of oper-ations including addition, multiplication, and adaptive bina-rization, to reach the final explanation map.
3. PROBLEM STATEMENT
The gradient-based feature map selection policy in SISE isaimed to distinguish the feature maps containing essentialattributions for the model’s prediction (‘positive-gradient’)against the ones representing outliers or background informa-tion. That was achieved using a threshold parameter µ thatwas set to 0 by default to discard the ones with ‘negative-gradient’ scores.However, most of the elected activation maps with posi-tive gradients are relatively ineffective in the prediction proce-dure, thereby increasing SISE’s computational overhead un-necessarily. We identify that the average gradient distributionof the positive-gradient feature maps follows a pattern, as inFig. 1, where several trivial features are represented with lowgradient values. Thus, only a fraction of the most critical fea-ture maps is passed to the third phase of SISE. Hence, wefocus on developing an adaptive selection policy for the pa-rameter µ of SISE to estimate the least number of required Normalized gradient values C oun t s Histogram of the gradient values of feature maps
Fig. 1 . Histogram of the gradient values recorded from thefeature maps in the last convolutional layer of a ResNet-50.features to generate an explanation map without any notablecompromise (and even in some cases, a slight enhancement)in terms of visual quality.
4. ADAPTIVE MASK SELECTION
Back-propagation
Layer Visualization Map
Input ExplanationMap InputFeature mapsCNN CNN
SISE
Otsu-based Adaptive ThresholdSelection +ve -ve . . .. . . c Cascaded Fusion
Layers
Layer +ve
Ada-SISE
AttributionMasks
Phase 1: Feature map Extraction Phase 2: Feature map SelectionPhase 3: Attribution mask ScoringPhase 4: Fusion
Chosen +ve
Fig. 2 . Architecture of the proposed Ada-SISE XAI method.To tune the strictness of feature map selection adaptivelyfor each of the layers, we employ an Otsu-based framework[17]. For a selected layer p , we reach the set of feature maps F [ p ] k and their corresponding gradient values σ [ p ] k , and deter-mine its maximum as ρ [ p ] . Denoting the normalized gradientvalues for the feature maps as υ [ p ] k = σ [ p ] k ρ [ p ] , we define the setof positive-gradient feature maps as: Υ [ p ] ≡ Υ [ p ]+ = { υ [ p ] k > | k ∈ { , ..., M [ p ] }} (1)where M [ p ] is the number of feature maps extracted fromlayer p . Otsu’s method is applied to the set of positive-gradient feature maps to implement an updated thresholdon them, based on the histogram of their average gradientscores. Assuming Υ [ p ] ( i ) ∀ i ∈ { , ..., | Υ [ p ] |} to be the i -th lgorithm 1: Ada-SISE: Adaptive Semantic InputSampling for Explanation
Input :
An input image I and a trained model δ . η ← post-processing function. ζ ← heatmap fusion function. for n ← , ..., N do Select the pooling layer p and collect featuremaps F [ p ] k ∀ k ∈ { , .., M p } ;Let the domain of the feature maps be Λ [ p ] ; σ [ p ] k = (cid:88) λ [ p ] ∈ c ∂δ ( I ) ∂F [ p ] k ( λ [ p ] ) & ρ [ p ] = max( σ [ p ] k ) ; A [ p ] k ← [] ; Υ [ p ] ← { υ [ p ] k > | k ∈ { , ..., M [ p ] } ; µ [ p ] ← Υ [ p ] (argmax j ∈{ ,..., | Υ [ p ] |} ( τ [ p ] ( j ))) ; foreach k ← { , ..., m p } doif σ [ p ] k ρ [ p ] > µ [ p ] then A [ p ] k ← A [ p ] k ∪ η ( F [ p ] k ) ; else A [ p ] k ← A [ p ] k ; endend V [ p ] I,δ ( λ ) = E A [ p ] [ δ ( I (cid:12) m ) · C m ( λ )] ; end SISE explanation: Y I,δ ( λ ) = ζ ( V [ p ] I,δ ( λ ) ) Output:
A 2D explanation map Y I,δ ( λ ) .value in Υ [ p ] , we can formulate the mean value of the maskswith less/more gradient values than Υ [ p ] ( i ) respectively, asfollows: ω [ p ] L ( i ) = (cid:80) ij =1 (Υ [ p ] ( j )) i × | Υ [ p ] | (2) ω [ p ] H ( i ) = (cid:80) | Υ [ p ] | j = i (Υ [ p ] ( j )) | Υ [ p ] | − i × | Υ [ p ] | (3)If we set µ = Υ [ p ] ( i ) to divide the set of positive-gradientfeature maps into two low and high subsets, the inter-classvariance of these sets are calculated as follows: τ [ p ] ( i ) = ω [ p ] L ( i ) × ω [ p ] H ( i ) × (cid:20) | Υ [ p ] | − i | Υ [ p ] | − i | Υ [ p ] | (cid:21) (4)which can be simplified as: τ [ p ] ( i ) = ω [ p ] L ( i ) × ω [ p ] H ( i ) × (cid:20) | Υ [ p ] | − i | Υ [ p ] | (cid:21) (5)According to [17], minimizing the intra-class variance forboth classes simultaneously is equivalent to maximizing theinter-class variance in equation (5). Hence, we can identifythe most deterministic feature maps in each layer by applying a threshold which maximizes the inter-class variance accord-ingly: µ [ p ] = Υ [ p ] (cid:18) argmax j ∈{ ,..., | Υ [ p ] |} (cid:0) τ [ p ] ( j ) (cid:1)(cid:19) (6)The argmax operation in equation (6) is achieved by asimple search method. If the number of feature maps derivedfrom a layer is not noticeably large, and if some of these fea-ture maps are discarded as negative-gradient activation maps,a simple search method would not add any significant addi-tional complexity to SISE framework. We term our methodAda-SISE and show its architecture in Fig. 2 and its method-ology in Algorithm 1.
5. RESULTS
To compare Ada-SISE’s performance abreast with SISE, ex-periments were performed on the test set of the Pascal VOC2007 dataset [18]. Two pre-trained models, a shallow VGG16(with a test accuracy of 87.18%) and a residual ResNet-50network (with 87.96% test accuracy), are directly loaded fromthe TorchRay library [10] to replicate the original experimen-tation setup. As it was reported in [5] that SISE meets or out-performs most of the state-of-the-art XAI methods like Grad-CAM [14], RISE [10] and Score-CAM [16], we restrict ourcomparisons only with Extremal Perturbation [11] (as it is oneof the sophisticated perturbation-based methods) and SISE. V GG EBPG 61.19
Bbox
Increase% (s) 87.42 5.96 R e s N e t- EBPG (s) 78.37 9.21 . Results of benchmark evaluation of Ada-SISE onpre-trained models on the PASCAL VOC 2007 [18] dataset.Table 1 shows the benchmark evaluation of Ada-SISEconcerning various metrics and their execution time. As thedepicted results are achieved through the same experimentalsetup as SISE paper, the readers can refer to [5] to infer furtherhead-to-head comparison of Ada-SISE with other state-of-the-art methods. Energy-Based Pointing Game (EBPG) [16]nd Bbox [19] use the ground-truth annotations available todetermine the precision of an XAI algorithm. Concurrently,Drop and Increase rates [20] measure the contribution ofpixels captured in the explanation map towards the model’spredictive accuracy. Ada-SISE outperforms SISE in almostall of the metrics while executing about 30% faster. Fig. 3compares the explanation maps qualitatively on a ResNet-50model and shows the ground-truth class and their annotationsalong with the model’s corresponding confidence score foreach image. Ada-SISESISEGround TruthInput Image Aeroplane0.7568Person0.5617Bicycle0.998Person0.0019
Fig. 3 . Comparison of Ada-SISE with SISE [5] on a ResNet-50 model with images from Pascal VOC 2007 dataset [18]demonstrating their class-discriminative explanation ability.
Four phases of SISE and Ada-SISE T i m e ( s ) Run-time comparison
SISEAda-SISE
Fig. 4 . Comparison of the average run-times of Ada-SISEwith SISE on a sample of images with a ResNet-50.The bottleneck in SISE’s run-time is its third phase, wheretoo many positive gradient feature maps are feed-forwarded For each metric in Table 1, the best is shown in bold. Besides Drop%and run-time (in seconds), the higher is better for all other metrics. to compute their weights for scoring. As Ada-SISE choosesonly a fraction of them that it considers crucial, our algo-rithm’s run-time is reduced significantly in the scoring phase.Fig. 4 shows the comparison of run-times, where it can benoted that Ada-SISE executes under 6.3 seconds while SISEtakes about 9.21 seconds. The small rise in the execution timeat the second phase of Ada-SISE is the effect of our proposedadaptive thresholding procedure. The reported numbers arethe average of experimentation performed over 100 randomimages from the Pascal VOC dataset on an NVIDIA Tesla T4GPU with 16 GB of RAM. p p p p p No. of feature
64 256 512 1024 2048 maps availableSISE
31 130 262 515 1008
Ada-SISE
26 114 179 420 551
Table 2 . The number of feature maps chosen by Ada-SISE(on average) over the five pooling layers of a ResNet-50 com-pared with that of SISE and the corresponding number ofavailable maps.The number of feature maps selected for each poolinglayer p l of the network was recorded over a data sample of500 images from the Pascal dataset, averaged, and reported inTable 2. As the deeper layers contribute more feature maps, itcan be noticed that Ada-SISE chooses only a fraction of them,justifying the run-time reduction after the second phase. Thisvalidates our claim that by neglecting comparatively lowergradient values, dominant feature maps that contribute moretowards a prediction can be extracted without compromisingthe explanation quality. Although an ablation study could beperformed to identify a suitable value for µ by fine-tuningSISE through extensive experiments, this solution would beprofoundly dependent on the training data and would be brit-tle when expanded to new unseen data. Therefore, Ada-SISEgeneralizes SISE to be scaled for any application.
6. CONCLUSION
In this work, we propose Ada-SISE as an improvement to therecent SISE method that makes it a fully automated XAI al-gorithm. We also report a reduction in run-time and an overallimprovement in the benchmark analysis without losing its vi-sual explanation quality. Such identified important featureswould be adopted in future works to analyze a model’s be-havior by studying its effect on the model’s prediction whenreplaced with noises or other classes’ attributions. . REFERENCES [1] Fenglei Fan, Jinjun Xiong, and Ge Wang, “On inter-pretability of artificial neural networks,” arXiv preprintarXiv:2001.02522 , 2020.[2] Alejandro Barredo Arrieta, Natalia D´ıaz-Rodr´ıguez,Javier Del Ser, Adrien Bennetot, Siham Tabik, AlbertoBarbado, Salvador Garc´ıa, Sergio Gil-L´opez, DanielMolina, Richard Benjamins, et al., “Explainable arti-ficial intelligence (xai): Concepts, taxonomies, opportu-nities and challenges toward responsible ai,”
Informa-tion Fusion , vol. 58, pp. 82–115, 2020.[3] Julius A Adebayo et al.,
FairML: ToolBox for diag-nosing bias in predictive modeling , Ph.D. thesis, Mas-sachusetts Institute of Technology, 2016.[4] Rachel KE Bellamy, Kuntal Dey, Michael Hind,Samuel C Hoffman, Stephanie Houde, Kalapriya Kan-nan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta,A Mojsilovi´c, et al., “Ai fairness 360: An extensibletoolkit for detecting and mitigating algorithmic bias,”
IBM Journal of Research and Development , vol. 63, no.4/5, pp. 4–1, 2019.[5] Sam Sattarzadeh, Mahesh Sudhakar, Anthony Lem,Shervin Mehryar, KN Plataniotis, Jongseong Jang,Hyunwoo Kim, Yeonjeong Jeong, Sangmin Lee, andKyunghoon Bae, “Explaining convolutional neu-ral networks through attribution-based input samplingand block-wise feature aggregation,” arXiv preprintarXiv:2010.00672 , 2020.[6] Mukund Sundararajan, Ankur Taly, and Qiqi Yan, “Ax-iomatic attribution for deep networks,” arXiv preprintarXiv:1703.01365 , 2017.[7] Karen Simonyan, Andrea Vedaldi, and Andrew Zisser-man, “Deep inside convolutional networks: Visualisingimage classification models and saliency maps,” arXivpreprint arXiv:1312.6034 , 2013.[8] Suraj Srinivas and Franc¸ois Fleuret, “Full-gradient rep-resentation for neural network visualization,” in
Ad-vances in Neural Information Processing Systems , 2019,pp. 4126–4135.[9] Sebastian Bach, Alexander Binder, Gr´egoire Montavon,Frederick Klauschen, Klaus-Robert M¨uller, and Woj-ciech Samek, “On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance prop-agation,”
PloS one , vol. 10, no. 7, pp. e0130140, 2015.[10] Vitali Petsiuk, Abir Das, and Kate Saenko, “Rise: Ran-domized input sampling for explanation of black-boxmodels,” arXiv preprint arXiv:1806.07421 , 2018. [11] Ruth Fong, Mandela Patrick, and Andrea Vedaldi, “Un-derstanding deep networks via extremal perturbationsand smooth masks,” in
Proceedings of the IEEE In-ternational Conference on Computer Vision , 2019, pp.2950–2958.[12] Bolei Zhou, Aditya Khosla, Agata Lapedriza, AudeOliva, and Antonio Torralba, “Learning deep featuresfor discriminative localization,” in
Proceedings of theIEEE conference on computer vision and pattern recog-nition , 2016, pp. 2921–2929.[13] Bolei Zhou, Aditya Khosla, Agata Lapedriza, AudeOliva, and Antonio Torralba, “Object detectors emergein deep scene cnns,” arXiv preprint arXiv:1412.6856 ,2014.[14] Ramprasaath R Selvaraju, Michael Cogswell, AbhishekDas, Ramakrishna Vedantam, Devi Parikh, and DhruvBatra, “Grad-cam: Visual explanations from deep net-works via gradient-based localization,” in
Proceedingsof the IEEE international conference on computer vi-sion , 2017, pp. 618–626.[15] Aditya Chattopadhay, Anirban Sarkar, PrantikHowlader, and Vineeth N Balasubramanian, “Grad-cam++: Generalized gradient-based visual explanationsfor deep convolutional networks,” in . IEEE, 2018, pp. 839–847.[16] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang,Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu,“Score-cam: Score-weighted visual explanations forconvolutional neural networks,” 2020.[17] N. Otsu, “A threshold selection method from gray-levelhistograms,”
IEEE Transactions on Systems, Man, andCybernetics , vol. 9, no. 1, pp. 62–66, 1979.[18] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn,and A. Zisserman, “The PASCAL Visual Object ClassesChallenge 2007 (VOC2007) Results,” 2007.[19] Karl Schulz, Leon Sixt, Federico Tombari, and TimLandgraf, “Restricting the flow: Information bottle-necks for attribution,” arXiv preprint arXiv:2001.00396 ,2020.[20] Harish Guruprasad Ramaswamy et al., “Ablation-cam:Visual explanations for deep convolutional network viagradient-free localization,” in