--- kind: concept slug: feature-attribution title: Feature attribution summary: >- Methods that score each input element, or each spatial location of a feature map, by its share of a model output. Covers path-integrated gradient maps, Shapley values by sampling or by propagation, relevance maps for Transformers, class activation maps, counterfactual explanations, and whether rankings hold across models and agree between methods. Author-reported results only. tags: [explainability, attribution, interpretability, saliency, game-theory] papers: ["1803.07517", "1905.00780", "1907.08456", "2007.04131", "2007.15911", "2011.05429", "2011.11036", "2012.09838", "2103.04244", "2105.00108", "2108.04840", "2110.10790", "2202.01602", "2202.05594", "2203.00962", "2303.14126", "2305.02012", "2309.01029", "2410.01917", "2504.10527", "2506.17455", "2605.25305"] related: [explanation-evaluation, explainable-ai, interpretable-machine-learning, causal-inference, counterfactual-explanation, structural-similarity, active-learning, image-super-resolution, semantic-segmentation, image-manipulation-detection, attention-interpretability, vision-transformer, mechanistic-interpretability, credit-assignment, data-valuation] status: draft updated_at: 2026-11-01 --- ## Overview The problem is to say which parts of an input drove a model's output, or which parts of the image support a class. The shared idea is a map over the input that assigns each element a score. Many methods also make the scores add up to the change in the output between a reference and the actual input. This is the completeness property that a path-integrated map is checked against [arxiv:2011.11036 §3.3] and that propagated Shapley values satisfy for each baseline [arxiv:2105.00108 eq. 13]. Shapley values are one way to get such scores from a game, by averaging each input's marginal contribution over orderings [arxiv:2202.05594 §2.1]. Two families differ in what they score. Input-space methods assign credit to input elements, by perturbing the input along a path or by sampling coalitions [arxiv:1803.07517 §5.2.2] [arxiv:2105.00108 §2]. Feature-map methods such as class activation maps use the last feature map, not the input pixels, so they localise evidence rather than attribute to inputs [arxiv:2203.00962 eq. 2]. Counterfactual methods ask a different question: what smallest change to the input would flip the output [arxiv:2103.04244 §4.1]. The field has no single agreed definition of a good explanation. The survey cards use qualitative criteria such as fidelity and clarity [arxiv:1803.07517 §5.1], and the quantitative numbers below come from the authors' own setups. On tabular data, the ranking itself can also change with the classifier fitted to the same data [arxiv:2305.02012 §2.1]. ## Variants ### Path-integrated gradient maps for dense outputs For a super-resolution network, Local Attribution Maps integrate the gradient of a patch score along a path from a blurred input to the real input, using progressively less blur [arxiv:2011.11036 §3.3] [eq. 2]. The authors use 100 path steps [arxiv:2011.11036 §3.3] and report that the diffusion index of these maps correlates with PSNR across 48 networks, with Pearson and Spearman values above 0.8 [arxiv:2011.11036 §4.4] [fig. 7]. Card: [Local Attribution Maps](/papers/2011.11036). ### Shapley propagation through a model chain DeepSHAP explains a series of models, such as a deep network feeding a tree model. It passes attributions back through the chain and rescales each link so the credit still sums to the output change [arxiv:2105.00108 eq. 3] [eq. 13]. Averaging over a baseline set gives the interventional Shapley value [arxiv:2105.00108 §6.2] [eq. 11]. The paper proves exactness only when every model is linear [arxiv:2105.00108 §6.4] and says the efficiency property is approximate for mixed series [arxiv:2105.00108 §6.5]. Card: [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108). ### Sampled Shapley values for tabular models Kernel SHAP estimates Shapley values by fitting a weighted regression to a sample of feature subsets. Its sampling rule is a heuristic, and when the paper was submitted no non-asymptotic accuracy guarantee was known for it below 2^n − 2 samples [arxiv:2410.01917 §1.2]. Leverage SHAP keeps the regression and changes only the sampling. Its leverage scores have a closed form, so all subsets of one size share a score, and the sample can be drawn without listing every subset [arxiv:2410.01917 §3.1]. In the authors' experiments on XGBoost models over eight SHAP-library datasets, at 10n evaluations the mean relative squared error on Adult is 0.00416, against 0.0104 for the SHAP library's Optimized Kernel SHAP [arxiv:2410.01917 tab. 1]. Averaged over the 38 entries of [tab. 2] that are not 1, the authors report about 50% of that baseline's error. A recount of [tab. 2] that includes the ten entries equal to 1 gives about 0.60. Both methods are exact once the budget reaches 2^n, so the gap is limited to smaller budgets [arxiv:2410.01917 tab. 2]. Card: [Provably Accurate Shapley Value Estimation via Leverage Score Sampling](/papers/2410.01917). ### Ranking stability across models and correlated features SHAP is usually fitted to one model at a time, but the same tabular data can give a different ranking for each classifier. On a UK Biobank cohort with four classifiers, the top three SHAP features agree across models and the other seven vary [arxiv:2305.02012 §2.1]. The paper names correlated features as a second source of change, because SHAP assumes the features are independent [arxiv:2305.02012 §2.1]. Its two proposed measures both come from a loop that drops the top-ranked feature and refits. NMR scores how far the ranking moves, and a lower value means a more stable ranking. MIP re-orders a list by each feature's relative rank across rounds, and the paper presents it as a way to allow for multicollinearity [arxiv:2305.02012 §2.1] [§3]. On an airline satisfaction dataset with 22 features, NMR is 0.231 for LightGBM and 0.445 for a decision tree [arxiv:2305.02012 §3]. LightGBM also has the highest accuracy in the authors' table, 0.91 [arxiv:2305.02012 tab. 2]. The paper says that this accuracy does not show its explanation is better, because the models used default parameters [arxiv:2305.02012 §3]. Neither measure has a formula in the paper [arxiv:2305.02012 §2.1]. Each case study uses one dataset and one split, with no seeds or confidence intervals [arxiv:2305.02012 §2.1] [§3]. For LIME, the same paper applies it to one representative subject, and its weights describe that subject only [arxiv:2305.02012 §2.2] [§4]. Card: [A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME](/papers/2305.02012). ### SHAP-ranked feature pruning for a forecaster A forecasting paper ranks 47 lagged and calendar predictors by the mean absolute SHAP value from two models, a random forest and XGBoost. It removes the least important predictor one at a time, refits both models after each removal, and averages their MAE curves. The average is lowest after 36 removals, which leaves 11 predictors [arxiv:2605.25305 §3.2.1] [fig. 7] [tab. 5]. The kept set includes lags 1, 2, 3, 6 and 12, the maximum and minimum temperature, and the maximum wind speed [tab. 5]. A SHAP analysis of one model then shows that the same lags push forecasts in opposite directions at the two campuses [arxiv:2605.25305 §4.2] [fig. 12] [fig. 13]. Each campus has one test year, and the results are author-reported [arxiv:2605.25305 §4.1]. The paper does not say which rows scored the pruning curve [arxiv:2605.25305 §3.2.1], and the card records that the released code fits the SHAP models on the full frame, including the test year. Card: [Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations](/papers/2605.25305). ### Gradient sensitivity of fixed attributes In the EA-LSTM rainfall-runoff model, the 27 static catchment attributes enter only through one input gate [arxiv:1907.08456 eq. 7]. To measure how strongly the output responds to each attribute, the authors take the mean absolute derivative with respect to the standardised attribute, averaged over days. The derivative is computed analytically [arxiv:1907.08456 §2.6.2] [eq. 14–15]. The paper calls this a Morris-style global measure [arxiv:1907.08456 §2.6.2], but the released code computes a local gradient, not a sampled average over the input space, as the card's pitfalls section records. The same paper adds Gaussian noise to the standardised attributes at σ = 0.1 to 1.0, and reports that the validation median NSE falls from 0.71 to 0.48 at σ = 1 [arxiv:1907.08456 §3.3] [fig. 6]. These are author-reported results from one model family. Card: [Towards Learning Universal, Regional, and Local Hydrological Behaviors via Machine-Learning Applied to Large-Sample Datasets](/papers/1907.08456). ### Full-gradient decomposition for ReLU networks For a ReLU network with biases, the output splits exactly into an input-gradient term and a bias-gradient term [arxiv:1905.00780 eq. 1]. FullGrad turns these pieces into one saliency map for a CNN. It passes each piece through a post-processing step ψ (absolute value, rescale to [0, 1], upsample) and adds them up [arxiv:1905.00780 eq. 2] [§4.2]. Because ψ is applied, the map is an approximation and no longer sums to the output [arxiv:1905.00780 §4.2]. On MNIST digit flipping, the authors' table gives 12.93 for the signed variant (no absolute value) and 8.26 for FullGrad, against 11.92 for plain gradients [arxiv:1905.00780 tab. 1]. On MNIST pixel perturbation, IntegratedGradients is lowest at all three RF values, and the paper's own summary is that no method wins on both tests [arxiv:1905.00780 tab. 2] [supp. Digit Flipping]. The ImageNet pixel-perturbation and remove-and-retrain results appear only as plots [arxiv:1905.00780 fig. 2]. All numbers are the authors' own. Card: [Full-Gradient Representation for Neural Network Visualization](/papers/1905.00780). ### Agreement between attribution methods Attribution methods applied to the same prediction can disagree. Krishna et al. define six agreement scores [arxiv:2202.01602 §3.2] [app. A]. Four compare the top-k features by absolute attribution: feature, rank, sign and signed-rank agreement. Two compare a chosen feature set: rank correlation and pairwise rank agreement. Lower values mean more disagreement [arxiv:2202.01602 §3.2]. Their experiments show that disagreement is strongest on text. For an LSTM on AG News at k = 11, most method pairs have rank and signed-rank agreement below 0.1 [arxiv:2202.01602 §4.3.2] [fig. 4]. On ImageNet images with superpixels, LIME and KernelSHAP agree closely, with rank correlation 0.8977 [arxiv:2202.01602 tab. 2]. At pixel level, rank correlation between Integrated Gradients and SmoothGrad is 0.001 [arxiv:2202.01602 §4.3.3]. In a user study, practitioners mostly resolved such conflicts with ad hoc heuristics [arxiv:2202.01602 §5.2.4]. The metrics come from practitioner interviews rather than a theory of which explanation is correct, and the results are author-reported [arxiv:2202.01602 §3.1]. Card: [The Disagreement Problem in Explainable Machine Learning](/papers/2202.01602). ### Counterfactual explanations Counterfactual methods search for a nearby input that changes the prediction. The review groups the algorithms it covers into seven families, such as instance-centric search, constraint solvers, genetic search and Shapley-based methods [arxiv:2103.04244 §5] [tab. 2]. It marks only one algorithm, RECOURSE, as causal, and argues that the rest rely on statistical associations [arxiv:2103.04244 §6.7] [tab. 2]. These are the authors' claims and they run no experiments of their own. Card: [Counterfactuals and Causability in Explainable Artificial Intelligence](/papers/2103.04244). ### Class activation maps as training targets Class activation maps (CAM) weight a feature map by the classifier's weights to show where a class is. ReCAM trains a second classifier on features masked by each class map, and then reads new maps out of the model [arxiv:2203.00962 §4.1] [eq. 5-9]. On PASCAL VOC 2012 seeds, the authors report a rise from 48.8 to 54.8 mIoU for plain CAM versus ReCAM, and from 66.3 to 70.9 with IRN refinement [arxiv:2203.00962 tab. 2]. The seed step costs 1.0 to 1.9 time units on VOC in their table [arxiv:2203.00962 tab. 2]. Card: [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](/papers/2203.00962). ### Grad-CAM for a visual check Grad-CAM is a gradient-weighted CAM. The CIFAKE paper uses it on a few random test images to see whether a synthetic-image detector looks at the main object or at the background [arxiv:2303.14126 §3.4]. The authors judge the maps by eye and report no quantitative explanation metric [arxiv:2303.14126 §4.2]. Card: [CIFAKE](/papers/2303.14126). A dataset paper applies Grad-CAM and LIME in the same qualitative way, to its best and a weaker ImageNet-pretrained classifier on underwater images [arxiv:2506.17455 §4.5] [fig. 7]. It reports no faithfulness test, and the Grad-CAM layer and LIME settings are not stated [arxiv:2506.17455 §4.5]. Card: [AQUA20](/papers/2506.17455). ### Relevance propagation for Transformers Attention weights are often read as an explanation, but they leave out the value path, the feed-forward layers and the skip connections. Chefer et al. instead propagate the score of one class back through each block, using LRP-style rules for the linear layers, the additions and the matrix products [arxiv:2012.09838 §3.1] [§3.2]. Each block contributes its gradient-weighted, head-averaged positive relevance plus an identity term, and the block matrices are multiplied in order [arxiv:2012.09838 eq. 13] [eq. 14]. The [CLS] row, read over the patch tokens, gives the map [arxiv:2012.09838 §3.4]. The method is class-specific by construction and needs no contrasting pass [arxiv:2012.09838 §2]. On ImageNet with ViT-base, the perturbation AUC for the predicted class is 54.16 for negative and 17.03 for positive perturbation, against 53.1 and 20.05 for rollout [arxiv:2012.09838 tab. 1]. The margin over rollout on the negative test is small. On the segmentation test the mIoU is 61.95, against 55.42 for rollout and 32.89 for LRP [arxiv:2012.09838 tab. 2]. The evaluation covers ViT-base and BERT-base only, and the numbers are the authors' own [arxiv:2012.09838 §4]. Card: [Transformer Interpretability Beyond Attention Visualization](/papers/2012.09838). ### Sanity tests with planted defects Adebayo et al. plant known defects and check whether the maps respond. In a birds-versus-dogs CNN where the background predicts the class, the maps of 15 attribution methods are compared with a relaxed ground-truth mask. SSIM against that mask ranges from 0.78 (KernelSHAP) to 0.98 (VGrad), which the authors read as the maps picking up the spurious background [arxiv:2011.05429 §3.1] [tab. 2]. When 10% of training labels are flipped, the maps for a flipped and a correct training input still have SSIM between 0.73 and 0.99, and the authors conclude these methods are unlikely to find mislabelled examples [arxiv:2011.05429 §3.2] [fig. 6]. When the top layers of a pretrained network are re-initialised, the modified back-propagation methods (Guided Backprop, DConvNet, PatternNet, PatternAttribution, LRP-SPAF) stay visually and quantitatively unchanged, so they fail that test [arxiv:2011.05429 §4] [fig. 7]. The relaxed mask and the SSIM values are the authors' own constructions. Card: [Debugging Tests for Model Explanations](/papers/2011.05429). ### Taxonomy and criteria The survey chapter splits explanation methods into rule extraction, attribution and intrinsic methods, and rates them against fidelity, interpretability, generalisability and explanatory power [arxiv:1803.07517 §5.1] [§5.2]. It reports no experiments. It is useful for framing, not for choosing an implementation. Card: [Explanation Methods in Deep Learning](/papers/1803.07517). A survey of the Shapley value in machine learning gives the definition that the sampled and regression methods above approximate. It states the four properties (null player, efficiency, symmetry, linearity) and the sampling and regression approximations, and it reports no experiments [arxiv:2202.05594 §2.2] [§3]. It names the loss of these properties under approximation as the largest open issue [arxiv:2202.05594 §5.1.3]. Card: [The Shapley Value in Machine Learning](/papers/2202.05594). Later surveys frame the same space from other sides. The LLM survey files feature attribution under local explanations of fine-tuned models, in four families: perturbation, gradient-based, surrogate models such as LIME and SHAP, and layer-wise decomposition such as LRP [arxiv:2309.01029 §3.1.1]. It states the raw gradient in full but does not fix the norm used to turn a gradient into a score [arxiv:2309.01029 §3.1.1]. It reports no experiments of its own. Card: [Explainability for Large Language Models: A Survey](/papers/2309.01029). A health-care survey sorts post-hoc methods into model-based, attribution-based and example-based explanations. It keeps only perturbation-based attribution methods and places SHAP and QII among the Shapley-value methods [arxiv:2007.15911 §4.2] [tab. 1]. Its evaluation table gives attribution methods a generic soundness metric, but no metric for completeness [arxiv:2007.15911 tab. 2]. The paper trains no models and has no experiments. Card: [The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911). The NLP survey sorts post-hoc methods by the kind of explanation they give. For input features it covers gradients, Integrated Gradients, LIME, Kernel SHAP and Anchors, and it notes that faithfulness depends on the task [arxiv:2108.04840 §6] [§6.2]. Card: [Post-hoc Interpretability for Neural NLP](/papers/2108.04840). The human-centred chapter pairs each user question with example techniques, such as LIME or SHAP for "why" and counterfactual methods for "why not" [arxiv:2110.10790 tab. 1]. The authors present this mapping as a suggestion, not a validated result. It has no experiments. Card: [Human-Centered Explainable AI](/papers/2110.10790). A review of model-agnostic methods, such as permutation importance, partial dependence and Shapley values, lists ten pitfalls, each with a fix and an open question [arxiv:2007.04131 §2–§9]. Three bear directly on attribution. When features are dependent, perturbation-based scores can evaluate the model where the data has no points [arxiv:2007.04131 §5.1]. Testing 50 unimportant features at α = 0.05 gives a probability of about 0.923 of at least one false positive, a figure that assumes the tests are independent [arxiv:2007.04131 §8.3]. A non-zero importance does not show a cause without causal assumptions [arxiv:2007.04131 §9]. The paper has no benchmark, and its numbers come from simulations. Card: [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](/papers/2007.04131). A semi-systematic review of explainable AI in food quality, built from database searches and snowballing, sorts its studies by data type and explanation type [arxiv:2504.10527 §1.1] [§9]. Pictorial data and visual explanations, mostly CAM-type heatmaps, dominate. Spectral and time-series data are rarely explained, and few studies combine several methods [arxiv:2504.10527 §9] [fig. 8] [fig. 9]. These are the authors' own tallies from a corpus they describe as non-exhaustive. Card: [Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review](/papers/2504.10527). ## When to use which | Situation | Use | Why | |---|---|---| | Dense output (per-pixel) of a differentiable model, you need a map over the input | Path-integrated gradient maps | Designed for image outputs; each map costs one pass per path step, 100 in the paper [arxiv:2011.11036 §3.3] | | Chain of models, some not shareable, each link has its own fast attribution | DeepSHAP-style propagation | Explains the chain without opening every model [arxiv:2105.00108 §4.4]; exact only if all links are linear [arxiv:2105.00108 §6.4] | | You need to know what change would flip the decision | Counterfactual methods | Gives actionable what-if answers [arxiv:2103.04244 §1.3]; causal guarantees only for RECOURSE, which needs an assumed causal model [arxiv:2103.04244 §6.7] | | Only image labels, you want pseudo masks for a segmenter | CAM with a second training stage (ReCAM) | Gains reported over plain CAM on VOC [arxiv:2203.00962 tab. 2]; smaller gains on MS COCO (33.1 to 34.6 mIoU seeds) [arxiv:2203.00962 tab. 2] | | Quick look at what a classifier uses, a few images | Grad-CAM | Cheap and visual; no quantitative test in the source [arxiv:2303.14126 §4.2]; the AQUA20 paper pairs it with LIME, also without a faithfulness test [arxiv:2506.17455 §4.5] | | Class-specific token or patch map for a Transformer, where attention weights are the usual shortcut | Gradient-weighted relevance propagation through the blocks | Beats rollout, attention, GradCAM and LRP on the authors' ViT-base perturbation and segmentation tests [arxiv:2012.09838 tab. 1] [tab. 2]; evaluated on ViT-base and BERT-base only [arxiv:2012.09838 §4] | | Shapley values for a tabular model with many features, and a fixed evaluation budget below 2^n | Leverage SHAP rather than Optimized Kernel SHAP | Lower error than the SHAP library baseline at every budget below 2^n in the authors' eight datasets [arxiv:2410.01917 tab. 1] [tab. 2]; the guarantee covers an independent sampler, not the per-size sampler in the released code [arxiv:2410.01917 §3.2] | | Per-attribute sensitivity of a regression output to fixed input attributes, such as catchment properties | Mean absolute derivative with respect to the standardised attribute, averaged over time | Computed analytically from the model [arxiv:1907.08456 §2.6.2] [eq. 14–15]; the paper's "global" label does not match the local gradient in its code, see the [EA-LSTM card](/papers/1907.08456) before reporting it as global | | Value function observed with noise, such as an expectation over a random baseline | Leverage SHAP, expecting less gain as noise grows | Best at noise up to σ = 0.1 and close to the baseline at σ ≥ 0.5 [arxiv:2410.01917 app. D] [fig. 4] | | Two attribution tools disagree on one prediction and you must report one | Run the six agreement scores on your data first | Shows how far the methods disagree [arxiv:2202.01602 §3.2]; the paper does not say which method is correct [arxiv:2202.01602 §3.1] | | Token-level explanations for a text model | Gradient, Integrated Gradients, LIME or Kernel SHAP, with a faithfulness check for your task | Faithfulness is task dependent [arxiv:2108.04840 §6.2]; SHAP needs a valid masked input [arxiv:2108.04840 §6.4] | | Choosing a method for an end user, starting from their questions | Question-driven selection from the question bank | Mapping is a suggestion, not validated [arxiv:2110.10790 tab. 1] | | Choosing a method family for a review or a compliance document | Taxonomy in the survey | Qualitative criteria only [arxiv:1803.07517 §5.1] | | Checking that an explanation method responds to a damaged model before you rely on it | Re-initialise top layers and compare maps before and after; do not rely on the modified back-propagation family alone | Those methods stayed unchanged under top-layer re-initialisation [arxiv:2011.05429 §4] [fig. 7] | | Finding mislabelled training examples from attribution maps | Do not rely on the maps alone for this | SSIM 0.73 to 0.99 between correct and flipped labels [arxiv:2011.05429 §3.2] [fig. 6] | | Global effect or importance curves when features are dependent or skewed | Quantile or subsampled grids, and conditional or ALE variants | Equidistant grids create unrealistic points; marginal and conditional methods answer different questions [arxiv:2007.04131 §5.1] [§5.3] | | Ranking many features and reporting which are significant | Correct for multiple tests, such as Bonferroni (α/p), or control the false discovery rate | Without correction, 50 unimportant features give about 0.923 chance of a false positive [arxiv:2007.04131 §8.3] | | Tabular model with correlated features, and you want to know whether its SHAP ranking depends on the classifier | Fit several classifiers and compare their SHAP rankings before reporting one | Rankings changed across four classifiers in the authors' UK Biobank case [arxiv:2305.02012 §2.1] [§4] | | Several classifiers on one tabular dataset, and you must report one explanation | Compare NMR across models and prefer the most stable one, with the accuracy caveat | Lowest NMR in the authors' airline case came from the most accurate model, with default parameters and one split [arxiv:2305.02012 §3] [tab. 2] [§4] | | Forecasting model with many lagged and calendar inputs, and you want to drop inputs with SHAP | Backward elimination on averaged mean absolute SHAP, refitting after each removal, with the scoring rows chosen before the test period | The authors' curve is lowest at 36 removals on one campus [arxiv:2605.25305 §3.2.1] [fig. 7]; the [forecasting card](/papers/2605.25305) records test-year rows in the released pruning code | | You must report that an attribution method is sound, in a regulated setting such as health care | Use the artificial ground truth, perturbation or axiomatic tests the survey lists; do not expect a metric for completeness | Soundness has generic metrics, completeness has none [arxiv:2007.15911 §5.2] [tab. 2] | | Global SHAP list with collinear features that you want re-ordered | MIP, with NMR to check stability first | The paper presents MIP as a way to allow for multicollinearity and shows one re-ordering [arxiv:2305.02012 §2.1] [tab. 3]; it gives no formula [§2.1] | | Explaining one prediction on tabular data | Restrict the claim to that instance; the paper suggests MIP or shapr for one instance or a subgroup | LIME results apply to one subject only [arxiv:2305.02012 §4] | | Saliency map for a ReLU CNN, one backward pass per target class, and you can accept an approximate sum | FullGrad, with ψ chosen for your metric | Sums to the output only before ψ [arxiv:1905.00780 §4.2]; the ImageNet pixel-perturbation results exist only as plots [arxiv:1905.00780 fig. 2] | | You need a map that is complete (sums to the output change) and also weakly dependent on the input | Choose one property, not both | The paper proves a saliency map cannot in general have both for piecewise-linear functions [arxiv:1905.00780 prop. 1] | | Deletion-style metric, choosing between the signed and absolute-value variant of FullGrad | Let the metric decide, and report which variant you used | The authors' digit-flipping test prefers the signed variant, pixel perturbation the absolute one [arxiv:1905.00780 tab. 1] [tab. 2] [§6] | | Shapley credit for features, data points, models or agents, where you must state which axioms you rely on | Use the survey's definitions and check each axiom after any approximation | Axioms often fail under approximation [arxiv:2202.05594 §5.1.3]; the value is defined with v(∅) = 0 [§2.1] | The ReCAM cost in [tab. 2] is the authors' own timing table, in their time unit, and is not a benchmark on other hardware [arxiv:2203.00962 tab. 2]. The Leverage SHAP results are the authors' own and were not re-run for this page [arxiv:2410.01917 §5]. ## Implementation sketch Common core of path-integrated attribution, with the completeness check. The path, the score and the number of steps are the only choices that change between variants. ```pseudocode # Path-integrated attribution for one scalar score [arxiv:2011.11036 §3.3] # Input: input x (H, W, C); score(x) -> float; path(x, t) for t in [0, 1], # with path(x, 0) the baseline and path(x, 1) = x; m steps # Output: map of shape (H, W, C); if the path is exact, the entries sum to # score(x) - score(baseline) def path_attribution(x, score, path, m): # Points on the path, from the baseline to the input [§3.3] points = [path(x, k / m) for k in range(m + 1)] attribution = zeros_like(x) for k in range(1, m + 1): # Gradient at the k-th point, times the step from the previous point # [§3.3] [eq. 4] grad = gradient(score, points[k]) attribution += grad * (points[k] - points[k - 1]) return attribution ``` Notes that the cards support: - Use the step from the previous point, and check completeness by summing the map. The printed index in [eq. 4] of the LAM paper does not satisfy this check, so do not copy it literally [arxiv:2011.11036 §3.3] [eq. 4]. - Do not add an extra 1/m factor. An earlier version of the LAM preprint had one, and the later version removed it [arxiv:2011.11036 app. D]. - Blur width and kernel size are not given in the LAM paper, so calibrate them rather than assuming values [arxiv:2011.11036 §3.3]. - For Shapley propagation, each baseline costs one pass through the whole chain, and the result is the mean over baselines [arxiv:2105.00108 eq. 11]. Efficiency is checked per link, not guaranteed for mixed series [arxiv:2105.00108 §6.5]. - For CAM, the map is at feature-map resolution, not the input resolution [arxiv:2203.00962 eq. 2]. - For sampled Shapley values, the Kernel SHAP weight of a subset of size s is 1 / (C(n, s) · s · (n − s)) [arxiv:2410.01917 eq. 4]. Algorithm 1 needs a budget of at least n evaluations [arxiv:2410.01917 alg. 1]. Report the number of evaluations actually run, because released implementations can differ from the paper's algorithm [arxiv:2410.01917 §3.2]. - For agreement checks, feature agreement is trivially 1 when k equals the number of features, so it says nothing at that k [arxiv:2202.01602 app. C.1]. The paper's rank is a position and does not say how ties are ranked [arxiv:2202.01602 app. A.1]. - For perturbed grids in permutation importance and partial dependence, use quantiles or subsamples rather than equidistant values [arxiv:2007.04131 §5.1]. The paper does not say how to draw subsampled values jointly, and drawing each feature separately breaks their dependence [arxiv:2007.04131 §5.1]. - For a re-initialisation test, the authors do not state the initialiser they used, so fix one and record it [arxiv:2011.05429 §4]. - For Transformer relevance maps, the block matrix is the identity plus the positive part of gradient times relevance, averaged over heads, and the blocks are multiplied in block order [arxiv:2012.09838 eq. 13] [eq. 14]. The head average and the positive part can be read in two orders. The released code uses one of them, and the card lists the difference, so check the [Transformer card](/papers/2012.09838) before reimplementing. - The addition rule of the Transformer method divides by x + y and by sums of relevance, and the paper gives no guard against zero [arxiv:2012.09838 eq. 6] [eq. 10]. The card's pitfalls section suggests a guard that returns zero where the denominator is zero. - For ranking-stability checks, refit each model and recompute the full ranking after each removal, then report the split and the formula you used. The paper gives neither measure as a formula, and its numbers come from one split [arxiv:2305.02012 §2.1] [§3]. - For SHAP-ranked pruning, fit the ranking and the refits on training rows only. The paper does not say which rows scored its pruning curve, and the [forecasting card](/papers/2605.25305) records a released pipeline that used the test year [arxiv:2605.25305 §3.2.1]. - For the static-attribute sensitivity, the released code takes the derivatives with respect to standardised values, so the numbers are not in physical units [arxiv:1907.08456 §2.6.2]. Report that scale, or convert before comparing with other attribution values. The [EA-LSTM card](/papers/1907.08456) lists the code check. - For FullGrad, the paper applies ψ to each channel inside the bias-gradient sum [arxiv:1905.00780 eq. 2]. The released code applies it to the whole layer tensor, and that gives a different map. Pick one, and say which in your report. The [FullGrad card](/papers/1905.00780) lists the other differences, including batch-norm layers and the channel sum of the input term. - For FullGrad, check the sum before ψ, not after it. The paper states that only the un-post-processed sum equals the output [arxiv:1905.00780 §4.2]. The paper does not mention a small constant in the rescale; the released code adds 1e-6 to the divisor, as the [FullGrad card](/papers/1905.00780) records. - For Shapley values over models, data points or agents, the survey defines each game through a trained model or a fit, but does not say what v(∅) is. Set it by convention and record it [arxiv:2202.05594 §2.1] [§4.1]. - For permutation sampling, Algorithm 1 as written calls v twice per player and ordering [arxiv:2202.05594 alg. 1]. Caching the predecessor set saves about half the calls, because the set for one player is the input for the next. This is a card observation, not the survey's own analysis; see the [Shapley survey card](/papers/2202.05594). ## Contradictions - **What an attribution means causally.** The counterfactual review argues that post-hoc, model-agnostic explanations capture correlations between features rather than the mechanism behind a prediction [arxiv:2103.04244 §1.2]. DeepSHAP takes the interventional Shapley value, defined with a do-operator on a flat causal graph, and presents that as its target [arxiv:2105.00108 §6.1]. The papers disagree on whether the second counts as causal. This page does not resolve it. The pitfalls review adds that a predictive model's importance does not identify a cause without causal assumptions [arxiv:2007.04131 §9]. - **Attribution as an explanation.** The taxonomy notes that attribution does not say why a factor matters, and that low fidelity makes maps noisy [arxiv:1803.07517 §5.2.2]. The LAM paper uses the diffusion index of its maps as a quantity to correlate with network quality [arxiv:2011.11036 §4.4], and does not test whether the maps explain the outputs to a user. - **Yardsticks for quality.** The agreement paper compares explanations with one another and does not test which is correct [arxiv:2202.01602 §3.2]. The Leverage SHAP paper scores each estimate against exact Tree SHAP values on a model it trained [arxiv:2410.01917 §5]. The two papers measure different things, so their numbers cannot be compared directly. The planted-defect study uses ground truth built into the model and data, and it finds that maps do not reliably flag mislabelled examples [arxiv:2011.05429 §3.2]. - **Attribution as a diagnostic.** The food review proposes XAI as a diagnostic layer for checking and auditing models on food data [arxiv:2504.10527 §2.1]. The planted-defect study finds that the maps it tested reveal a spurious background but not every defect, and that users in its study relied mainly on predicted labels [arxiv:2011.05429 §3.1] [app. H]. The food review reports no test of this kind, so the two are not measured the same way. - **Where LIME belongs.** The health-care survey files LIME and Anchors under model-based local explanations, not attribution [arxiv:2007.15911 §4.1]. The NLP survey and the LLM survey list LIME among input-feature methods [arxiv:2108.04840 §6] [arxiv:2309.01029 §3.1.1]. Surveys disagree on the category, so check a method's category in the paper you cite. - **What "completeness" means.** The health-care survey uses the word for two things. Its definition is that an explanation carries enough information to recompute the output [arxiv:2007.15911 §2.2]. Its attribution metrics use the word for sensitivity-N, which the table marks as having no metric [arxiv:2007.15911 §5.2] [tab. 2]. The sum-to-output check in the Overview is a different test, so read each paper's definition. - **Which faithfulness metric to trust.** The LLM survey notes that the best-ranked explanation under one faithfulness metric can be the worst under another [arxiv:2309.01029 §5.1]. The agreement study finds that methods disagree on which features matter [arxiv:2202.01602 §3.2]. Neither paper settles which metric or method is right, so this page does not either. - **Relevance propagation under two tests.** On ImageNet with ViT-base, the Transformer paper reports LRP and partial LRP below its own relevance maps on the perturbation and segmentation tests [arxiv:2012.09838 tab. 1] [tab. 2]. The planted-defect study reports that LRP-SPAF, one LRP variant, stays visually and quantitatively unchanged after top-layer re-initialisation [arxiv:2011.05429 §4] [fig. 7]. The two studies use different models and different tests, so neither result settles which method to use. This page does not resolve it. - **Completeness and local importance.** The Overview's completeness checks [arxiv:2011.11036 §3.3] [arxiv:2105.00108 eq. 13] are about the sum of the scores. The FullGrad paper argues that, for piecewise-linear functions, a saliency map cannot in general be both complete and weakly dependent on the input [arxiv:1905.00780 prop. 1]. The two statements are about different properties, and this page does not choose between them. - **Shapley axioms and their guarantees.** The survey says the axioms often fail under approximation [arxiv:2202.05594 §5.1.3]. Leverage SHAP is presented with a guarantee on its regression objective, not directly on the Shapley values [arxiv:2410.01917 §3.2], which is a different claim from the axioms. Neither paper tests the other's property, so read each claim for what it covers. ## Papers - [Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges](/papers/1803.07517): a survey and taxonomy of explanation needs and methods, with no experiments. - [Full-Gradient Representation for Neural Network Visualization](/papers/1905.00780): exact full-gradient decomposition for ReLU networks, and the FullGrad saliency map, with author-reported MNIST and ImageNet tests. - [Interpreting Super-Resolution Networks with Local Attribution Maps](/papers/2011.11036): path-integrated gradient maps for super-resolution networks, with a diffusion index. - [Counterfactuals and Causability in Explainable Artificial Intelligence](/papers/2103.04244): a systematic review of counterfactual explanation methods and their causal claims. - [Explaining a Series of Models by Propagating Shapley Values](/papers/2105.00108): DeepSHAP, which propagates Shapley values through a chain of models. - [Class Re-Activation Maps for Weakly-Supervised Semantic Segmentation](/papers/2203.00962): ReCAM, a second training stage that sharpens CAM seeds for segmentation. - [CIFAKE: Image Classification and Explainable Identification of AI-Generated Synthetic Images](/papers/2303.14126): a synthetic-image detector with Grad-CAM heatmaps used for qualitative inspection. - [Transformer Interpretability Beyond Attention Visualization](/papers/2012.09838): class-specific relevance maps for Transformers, propagated through attention and skip connections, with ViT and BERT results. - [Explainability for Large Language Models: A Survey](/papers/2309.01029): a survey of LLM explainability by training paradigm, with feature attribution as one local family and no experiments. - [Post-hoc Interpretability for Neural NLP: A Survey](/papers/2108.04840): a survey of post-hoc NLP explanation families and how each is validated, with no experiments. - [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](/papers/2110.10790): a survey chapter that maps user questions to XAI techniques, with no experiments. - [The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective](/papers/2202.01602): six agreement scores for feature attributions, and a study of how practitioners resolve conflicts. - [Provably Accurate Shapley Value Estimation via Leverage Score Sampling](/papers/2410.01917): Leverage SHAP, a sampling change to Kernel SHAP with a guarantee on the regression objective. - [Towards Learning Universal, Regional, and Local Hydrological Behaviors via Machine-Learning Applied to Large-Sample Datasets](/papers/1907.08456): an LSTM variant whose static-attribute gate supports a gradient sensitivity read-out, with author-reported hydrology results. - [General Pitfalls of Model-Agnostic Interpretation Methods for Machine Learning Models](/papers/2007.04131): a review of ten pitfalls for permutation importance, partial dependence and Shapley values, with simulations and no benchmark. - [Debugging Tests for Model Explanations](/papers/2011.05429): planted-defect tests for attribution maps, and a user study with defective image classifiers. - [The Shapley Value in Machine Learning](/papers/2202.05594): a survey of the Shapley value's definition, approximations and ML applications, with no experiments. - [Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review](/papers/2504.10527): a semi-systematic review of XAI in food quality, with a five-topic taxonomy and no experiments. - [A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME](/papers/2305.02012): SHAP and LIME rankings across classifiers and correlated features, with the NMR and MIP stability measures; author-reported, with no new benchmark. - [The role of explainability in creating trustworthy artificial intelligence for health care](/papers/2007.15911): a survey of explanation types and their evaluation metrics for health-care AI, with no experiments. - [Electricity Consumption Forecasting: An Approach Using Cooperative Ensemble Learning with SHapley Additive exPlanations](/papers/2605.25305): SHAP-ranked backward elimination of forecasting inputs, with author-reported results on two campuses. - [AQUA20: A Benchmark Dataset for Underwater Species Classification under Challenging Conditions](/papers/2506.17455): a 20-class underwater image benchmark, with Grad-CAM and LIME maps read qualitatively for two models; author-reported.