Explanation Methods in Deep Learning: Users, Values, Concerns and Challenges

In plain words

Deep neural networks make decisions that are hard to trace. Because of that, the people involved ask why a given output came out the way it did. This chapter sorts those people into groups: the engineers who build the network, the developers who put it into products, the owners and end users of those products, the people the decisions are about, and outside stakeholders. It then looks at what European data protection law asks for, namely meaningful information about the logic behind automated decisions.

Existing explanation techniques fall into three families. Some pull out rules that approximate the network. Some measure which inputs matter by changing them and watching the result. Some build interpretability into the network itself. The authors conclude that explanations of what influenced an output are possible, but explanations that a non-expert can use are still missing. They also argue that bias in training data and unfair outcomes are hard to fix with explanations alone.

TL;DR

This is a book-chapter survey, not a new method. It gives a taxonomy of explanation methods for deep neural networks (DNNs) and checks them against the concerns of different users and against GDPR requirements §1, §5. Its three classes are rule extraction, attribution and intrinsic methods §5.2. It concludes that current methods are aimed at expert users, and that bias and unfair outcomes are not solved by explanations alone §7. Use it for framing and requirements, not for implementation details.

Problem

Trained DNNs cannot be explained by inspecting the inference process the way decision trees or Bayesian networks can. Their many parameters and learned internal representations are hard to analyse, so the paper treats them as black boxes §1.3. Once DNN decisions affect people, users need to know whether the system can be trusted and whether it is fair §1.2, §2. The GDPR requires that data subjects get meaningful information about the logic involved in automated decisions §3. The paper asks what explanation methods can and cannot deliver for that purpose §1.3, §7.

Method

The paper is analytic. It has no equations, no algorithms, no tables and no experiments. Its only figure is a diagram of the four components of explainability: users, laws and regulations, explanations and algorithms fig. 1 §1.1. The steps of its argument are:

1. Identify users. Expert users are DNN engineers, who study the theory and improve the models, and DNN developers, who build applications from them. Lay users are the owner, the end user, the data subject and stakeholders. One person can belong to several groups §2. A self-driving car case study walks through these roles §2.
2. Map the GDPR concepts. The relevant terms are personal data, data subject, data processor and data controller. The law's definition of profiling and its rights to meaningful information and to plain-language information are also covered §3.
3. Split explanation needs in two. A DNN may dysfunction, meaning it does not work as intended, or misfunction, meaning it produces undesired outcomes §4. The first kind is about the system's basic processes and is needed by developers and expert users. The second kind is about "local explanations" for a particular outcome and is needed by owners, end users, data subjects and stakeholders §4.
4. Define the criteria. An explainer should have high fidelity, interpretability (split into clarity and parsimony), generalizability and explanatory power. The criteria are summarised below §5.1.
5. Classify the methods. The taxonomy is shown below §5.2. The paper says it is provisional and may need revision as the field changes §5.2.
6. Assess each class against the criteria §5.2.1–§5.2.3.
7. List the general concerns that model builders should address §6.

Desirable properties of explainers §5.1

Taxonomy of explanation methods §5.2

Rule extraction has three subtypes §5.2.1:

• Decompositional: works on the network's architecture and/or outputs, often layer by layer, then merges the rules. The paper cites DeepRED §5.2.1.
• Pedagogical: treats the network's input-output behaviour as a learning target. The paper calls it inherently model-agnostic §5.2.1.
• Eclectic: uses internal architecture or weights to support a symbolic learner §5.2.1.

Assessment against the criteria, as the paper gives it §5.2:

• Rule extraction: local explanations are more faithful than global ones. The paper rates it highly clear, because the rules are a formal language. Its parsimony depends on how large the ruleset is. Generalizability depends on the subtype §5.2.1.
• Attribution: fidelity is weakened by ambiguity, and low fidelity produces noisy maps. Methods that only use input and output are the most generalizable. Attribution does not say why a factor is important §5.2.2.
• Intrinsic: these methods do not explain anything by themselves. They improve fidelity, clarity and parsimony of attribution methods §5.2.3.

Classifier sketch. The paper gives no decision order for assigning a method to a class, so this order is our own rendering of the class definitions above:

# Taxonomy decision order  [§5.2]
# Input:  a method description: what it changes and what it perturbs
# Output: one class name; the paper gives no order, so this is our choice

def classify_explainer(method):
    # Intrinsic: changes architecture or loss to make representations
    # interpretable  [§5.2.3]
    if method.changes_architecture_or_loss:
        return "intrinsic"

    # Attribution: perturbs inputs or internal components and records the
    # effect on output  [§5.2.2]
    if method.perturbs_inputs_or_internals:
        return "attribution"

    # Rule extraction: approximates the network with human-readable rules
    # from its inputs, outputs or internals  [§5.2.1]
    if method.produces_rules:
        return "rule-extraction"

    # Not covered by the paper's taxonomy
    return "unclassified"

Hyperparameters

Not applicable. The paper trains no model and runs no experiments, so it states no hyperparameters §1 §7.

Results

No experimental results. The paper gives no quantitative results and no benchmarks. Its only dated values are the GDPR's approval date (April 14, 2016) and its enforcement date (May 25, 2018) §3. Its findings are qualitative claims about the three classes and the criteria §5.2, §7.

Paper vs code

No official code.

The source has no code links. The footnotes point to the GDPR text and to a Dutch website about a Dutch system §3, §6. The arXiv metadata has no code link. A GitHub repository search for the title words returned zero results.

Pitfalls

• Nothing here can be implemented directly. The criteria are qualitative ratings ("high", "low") with no measurement procedure, so do not treat them as scores §5.1, §5.2.
• The taxonomy is provisional. The authors say it may need to change §5.2.
• The paper gives no decision order for classifying a method. The pseudocode below uses one order (intrinsic, attribution, then rules). This is our choice, not the paper's. Attribution and intrinsic methods are described as complementary, so a real method may fit more than one class §5.2.3, §7.
• Attribution has many names, and some of them are used loosely. Check whether two sources mean the same method §5.2.2.
• The paper says explanation methods for lay users do not exist, as far as the authors know §7. That is their survey claim at the time of writing, not a current status check.
• Bias in datasets and unfair outcomes are not solved by explanation methods. The paper says explanations can help only by showing learned features for separate bias analysis, and that other measures such as regulation are needed §6, §7.
• Legal points are the authors' summary of the GDPR articles §3. The paper does not give legal advice.
• The paper's bibliography is inconsistent in places. Papernot et al. is keyed as 2016 but listed as 2017. Lockett et al. has no year [bibliography].
• The case study is illustrative and includes no measurements §2.

Lineage

• Builds on: the paper draws on earlier surveys and taxonomies, such as Samek et al. (arXiv:1708.08296) and Montavon et al. §1.3, §5.2. It also discusses attribution work including Simonyan et al. (arXiv:1312.6034) §5.2.2.
• Compared against: the paper compares its taxonomy to prior classification schemes for visual methods §5.2.
• Other cited works with arXiv ids: Bojarski et al. on end-to-end driving (arXiv:1704.07911) §1.2; Szegedy et al. on adversarial images (arXiv:1312.6199) §6; Kindermans et al. on attribution reliability (arXiv:1711.00867) §5.2.2.
• Superseded by: not known from content in this wiki.