An investigation of different interpretability methods used to evaluate a prediction from a CNN model

Abstract

In this thesis we investigate different interpretability methods for evaluating predictions from Convolutional Neural Networks. We look at research on several explanation methods with a focus on Local Interpretable Model-agnostic Explanations (LIME) and Layer-wise Relevance Propagation (LRP). Our goal is to investigate different interpretability methods and how robust they are in comparison to each other. We do initial experiments by testing a set of images with Guided Backpropagation, Gradient-weighted Class Activation Mapping (Grad-CAM), LIME and LRP. In the next set of experiments we focus on LRP and LIME. The models we use are VGG16 with and without batchnorm layers. We use rotation and Gaussian noise to transform the input images. To measure the robustness we use Root Mean Square Error (RMSE). The transformation is added to the input and sent through the model. The output from the model is sent through the interpretability method. The resulting heatmap for the transformed image is then compared with the original heatmap to measure the RMSE score. We use a set of small transformations and a set of more extreme transformations. The transformations we use for rotation are between 0.5-10 degrees and 15-40 degrees. For the Gaussian noise we use $\sigma$ between 0.01-0.10 and 0.25-10.0. We observe that LIME focuses on super pixels and will therefore be less robust for transformations compared to LRP. We find that methods which emphasises on both positive and negative contributions, such as LRP and Grad-CAM are more helpful since they highlight the regions that contribute and work against the prediction in the image. When implementing LRP with models using batchnorm layers we find that this give unreliable results. We handle this by merging the batchnorm layers with the corresponding convolutional layer before backpropagating LRP. Our experiments show that the explanation from the interpretability method correlates significantly with the models robustness. Though in some cases the robustness of the model is not reflected in the interpretability method and this is especially noticeable when Gaussian noise are applied to the input in the LIME experiments.