A Perspective on Explainable Artificial Intelligence Methods: SHAP and LIME

In plain words

Tools such as SHAP and LIME try to say which inputs pushed a model toward its answer. This perspective paper asks how far those explanations can be trusted on tabular data, the spreadsheet-like tables common in medicine and business.

The authors find that the answers depend on which model you chose. Train four classifiers on the same data and SHAP can rank the features differently for each one. Features that move together with each other add to the problem, because correlated features can push one another down the ranking. LIME explains one case at a time with a simple local line, so its numbers do not summarise the whole model.

The paper suggests treating a feature ranking as something to check, not a fact. Compare it across several models. Two proxy measures, NMR and MIP, can test how stable the ranking is and adjust it for correlated features. The evidence comes from two example datasets, not from a broad benchmark.

TL;DR

This perspective compares SHAP and LIME, two widely used explanation methods for tabular data, and shows that both are strongly affected by the choice of classifier and by collinearity between features §1 §2 §3. In a UK Biobank myocardial-infarction case with four classifiers, the top features agree but the lower ranks shift §2.1. In an airline-satisfaction case, the most stable list (lowest NMR) comes from the most accurate model, and MIP reorders the SHAP list for that model §3 tab. 2 tab. 3. The paper proposes no new algorithm and no benchmark; its results are author-reported and rest on one dataset per case.

Problem

Explanation methods are meant to build end-user trust, but their outputs can be hard to read and can change with the model used §1. The paper asks whether model dependency and collinearity change the outcome of two popular methods, and how to present their output responsibly §1 §4.

Assumptions the paper states or relies on:

• The explanation is computed for a fixed model and dataset. SHAP's output depends on the model §2.1; LIME's output for one subject depends on the model and the instance §2.2 §4.
• SHAP assumes features are independent. When features are correlated, the scores can be misleading §2.1.
• The explanation is read by an end user who knows the general framework of the method §1 §4.

Method

The paper is a perspective piece. It describes two existing XAI methods, runs two case studies, and makes recommendations §1. Table 1 compares the two methods; it is reproduced here in condensed form tab. 1.

Source of the table: tab. 1 §2.

SHAP. SHAP treats each feature as a player and the model output as the payoff. Each feature's score is its contribution across all feature coalitions §2.1. Exact computation grows with the number of features, so the paper names Kernel SHAP as an approximation §2.1. The paper states that SHAP scores do not measure a feature's weight; the meaning is in the order of the features §2.1.

LIME. LIME approximates the model around one instance with a local, interpretable model. It reports the coefficients of that local linear model as feature weights, and it shows the class probabilities and each feature's contribution per class §2.2.

Case study 1 (model dependency). Four classifiers (decision tree, logistic regression, LightGBM and SVC) were fitted to a UK Biobank cohort to separate individuals with myocardial infarction from controls, using ten features. SHAP was applied to each model, and LIME was applied to one representative subject §2.1 §2.2 fig. 2 fig. 3.

Case study 2 (collinearity and stability). Four classifiers (LightGBM, logistic regression, decision tree and SVC) were fitted to an airline passenger-satisfaction dataset with 22 features §3. The paper shows a correlation heatmap fig. 4, SHAP's top features per model tab. 2, the NMR stability value per model §3, and MIP's reordering of the SHAP list for the most stable model tab. 3 §3.

Recommendations. Present SHAP results alongside their plots, in plain language that states the assumptions (feature independence, model dependence). Compare SHAP across several models when features are collinear. Use NMR to choose the most stable model, MIP to adjust the list globally, and MIP or shapr to explain a single instance or subgroup. LIME results apply to one subject only §4.

Iterated removal is the procedure behind both NMR and MIP. It is described in prose in §2.1 and §3. The sketch below follows that prose. The paper gives no pseudocode, so the scoring details come from the authors' code.

# Iterated removal, the loop behind NMR and MIP  [§2.1] [§3]
# Input:  train (n, F), test (m, F), labels, explain(model, train, test)
# Output: one feature ranking per round, most important first
def iterated_removal_rankings(train, test, labels, explain):
    rankings = []
    # Stop when one feature is left  [code: NMR README.md#L24]
    while number_of_features(train) > 1:
        # Refit on remaining features  [§3] [code: NMR README.md#L35-L36]
        model = fit_classifier(train, labels)
        # Score each feature: mean absolute SHAP value on the test set
        # [code: NMR README.md#L40-L49]
        scores = explain(model, train, test)
        ranking = sort_by_score_descending(train.columns, scores)
        rankings.append(ranking)
        # Remove the top-ranked feature from both sets  [§2.1] [§3]
        train = drop_column(train, ranking[0])
        test = drop_column(test, ranking[0])
    return rankings

NMR (normalized movement rate) measures how much the ranking moves while features are removed. A smaller value means a more stable ranking §2.1 §3. The paper gives no formula. The sketch below is the procedure the authors' code implements.

# NMR: how far the ranking moves as top features are removed  [§2.1]
# Input:  rankings from iterated_removal_rankings
# Output: one value; 0 to 1, where 1 means the order changes sharply
def normalized_movement_rate(rankings):
    # Track each feature's position in each round it appears in
    # [code: NMR README.md#L57-L64]
    positions = positions_by_feature(rankings)
    rates = []
    for each removal step:
        # Sum of absolute position changes after this step
        # [code: NMR README.md#L100-L106]
        movement = total_position_change(positions, step)
        # Largest movement possible with F features: 2 * sum of (F - j)
        # over odd j < F  [code: NMR README.md#L26-L30]
        largest = max_movement(features_in_round(step))
        rates.append(movement / largest)  # [code: NMR README.md#L118]
    return mean(rates)  # [code: NMR README.md#L121]

MIP (modified index position) re-orders a feature list using its ranks across the rounds. The paper describes it as re-ordering "considering the multicollinearity" and gives no formula §2.1. The authors' code computes the score below.

# MIP: re-order features by their relative rank over the rounds  [§2.1]
# Input:  rankings from iterated_removal_rankings
# Output: feature names, most important first
def modified_index_position(rankings):
    totals = {}
    for ranking in rankings:
        for position, feature in enumerate(ranking, start=1):
            # Relative position in this round: rank / features left
            # [code: MIP README.md#L70] [code: MIP README.md#L74]
            relative = position / len(ranking)
            totals[feature] = totals.get(feature, 0) + relative
    # Smaller total ranks first  [code: MIP README.md#L92]
    return sorted(totals, key=totals.get)

Hyperparameters

The paper reports few hyperparameters. Most of its case-study settings are data sizes and the claim that defaults were used §2.1 §3. Repository values are marked, because they are examples, not the paper's runs.

Results

Author-reported. The paper states no confidence intervals, repeats or seeds. Results from each case study come from a single dataset and a single split §2.1 §3.

Case study 2 accuracy and top SHAP feature per model tab. 2:

The table header reads "AC" for logistic regression; the value is read as accuracy. The paper's text says accuracy alone does not establish that LightGBM is better, because the models used default parameters §3.

NMR per model, lower means more stable §3:

MIP on LightGBM, the most stable model tab. 3 §3: "Ease of Online booking" moves from 15th in the SHAP list to 5th in the MIP list. The other reorderings are visible in tab. 3.

Case study 1, UK Biobank, qualitative only §2.1 §2.2:

• SHAP: the top three features agree across the four models. The remaining seven vary. Body mass index is the least important feature in the decision tree and logistic regression, third in LightGBM and seventh in SVC. For the decision tree, the last five features have SHAP values close to zero fig. 2 §2.1.
• LIME, one representative subject: the probability of the MI class differs by model, with LightGBM the most certain and the decision tree the least. Alcohol consumption pushes towards the MI class in logistic regression and SVC, and towards the control class in the decision tree and LightGBM fig. 3 §2.2.
• Accuracy is described as comparable across the four models. The accuracy values are shown in the figure, not in the text §2.1 fig. 2.

Popularity: a GitHub star count for ten XAI methods shows SHAP and LIME with the most stars. The paper uses this to justify its choice of methods. The collection date is not stated fig. 1 §1.

Paper vs code

Code: the paper names https://github.com/amaa11/NMR for the code of the perspective §2.1. At code_commit the repository contains only a README with an NMR example. It has no LICENSE file, no case-study script and no data. The MIP example in https://github.com/amaa11/MIP (commit cb467e4f0d37049203186183dad31b4d8fba3e56) is also README-only, with no LICENSE file. We fetched it to check the MIP details below; it is cited, not copied. Searches: the paper's own links only; no further GitHub search was needed.

• Case study code is missing. The paper says the code of the perspective is at the NMR link §2.1. The repository has only the NMR example. The four-classifier UK Biobank pipeline and the airline pipeline are not in either repository, so the reported tables cannot be regenerated from the code NMR README.md.
• Classifier parameters. Case study 2 uses default parameters §3. The NMR and MIP examples set logistic regression to C = 0.1 NMR README.md MIP README.md. Which setting produced the paper's numbers is not stated.
• SHAP explainer. The paper cites SHAP's GitHub code §2.1 without naming the explainer per model. The repository example uses a linear explainer with an independent masker for logistic regression only NMR README.md. The masker choice matches the paper's independence assumption §2.1.
• MIP formula. The paper gives no formula. The code ranks each feature by its relative position in each round, sums these, and sorts ascending MIP README.md MIP README.md MIP README.md.
• NMR formula. The paper gives no formula. The code normalises the summed movement by a largest-possible-movement value and averages the rates NMR README.md NMR README.md NMR README.md. The index bookkeeping (pop, popitem and del on the rate and movement lists) is intricate NMR README.md NMR README.md.
• Name. The paper calls MIP the "Modified Index Position" §2.1. The MIP repository calls it the "Modified Informative Position" MIP README.md. The method is the same; the name differs.
• Direction. The NMR README says the value runs from 0 to 1 and that 1 means the list is unstable NMR README.md. This matches the paper's "smaller is more stable" §2.1.
• In-place edits. The NMR and MIP snippets drop columns in place from the same DataFrames NMR README.md MIP README.md. Re-running a snippet on the same objects gives different results. Copy the data before each run.

Pitfalls

• A perspective, not a method. The abstract says the paper proposes "a framework for interpretation" [abstract]. In the body this is a set of recommendations in prose §4; no equation or algorithm defines it.
• Scores are not weights. The paper says the SHAP score does not measure a feature's weight; the ranking carries the meaning §2.1. Reading the score magnitudes as effect sizes is a misreading the paper warns against.
• LIME coefficients are local. A LIME weight holds for one subject and cannot be read as a general effect §2.2 §4.
• Feature independence. SHAP's independence assumption is violated by correlated features, which the paper flags as a risk for the scores §2.1. The paper cites an earlier source that marginalising features is only sensible if they are uncorrelated §2.
• Unrealistic instances. When features are correlated, the Shapley marginalisation can create data points that do not occur in practice §2.
• Model dependence. The ranking can change with the classifier §2.1 §3. The paper recommends comparing several models and stating this in the explanation §4.
• Single split, no uncertainty. No seed, repeat or confidence interval is reported for any case study §2.1 §3. Rankings and NMR values can move with the split.
• Accuracy is not evidence of the best explanation. The paper says LightGBM's higher accuracy does not establish that its explanation is better, given default parameters §3.
• Timing. The table says LIME is faster, especially for tree models tab. 1 §2. The paper reports no timing measurements.
• Stars are not evidence. Star counts are used to justify the choice of methods fig. 1. They say nothing about correctness, and the collection date is not stated.
• Reproduction. Neither linked repository contains the case-study code or data. The NMR and MIP examples are snippets that depend on prepared DataFrames and an unstated data split.
• Typos in tables. "AC" versus "ACC" in tab. 2; "Modified Index Position" versus "Modified Informative Position" (see Paper vs code).

Lineage

• Builds on: SHAP (Lundberg and Lee, 2017) and LIME (Ribeiro et al., 2016) §2. Their arXiv ids are not in this wiki, so neither is linked.
• Builds on: NMR, a stability measure for XAI rankings under collinearity (Salih et al., ICIP 2022) §3 §4. Not in this wiki.
• Builds on: MIP, a post-hoc re-ordering of XAI feature lists for collinear features (Salih et al., IEEE JBHI 2024) §3 §4. Not in this wiki.
• Compared against: shapr, an approximation of SHAP that accounts for collinearity (Aas et al., 2021) §2.1 §4; the shapley cohort refinement approach of Mase et al. (2019) §2.1; GraphLIME for graph-based models §4.
• Superseded by: not stated.