Human-Centered Explainable AI (XAI): From Algorithms to User Experiences

In plain words

Explainable AI (XAI) is about making an AI model understandable to the people who use or are affected by it. There are many explanation methods, and this chapter argues that the right choice depends on who needs the explanation and why. It sorts the methods by the questions users ask, such as "why this answer?" or "what would have to change to get another answer?"

The chapter also looks at what happens when people actually see explanations. Explanations can make people trust an AI more, even when the explanation says little. Many studies have not found clear gains in people's decisions from explanations. The authors treat explanation as a design problem as much as a technical one, and they borrow ideas from social science about how people explain things to each other.

TL;DR

• A survey chapter with no experiments of its own. It is written from a human-centered HCI and UX view of XAI §1.
• It proposes that users' explainability needs, expressed as questions, should drive the choice of XAI technique, and it gives a question-to-technique mapping §3 tab. 1.
• It names two sources of pitfalls: explanations that do not serve users' real goals, and assumptions about how people process explanations that do not hold §4.
• It ends with theory-driven directions from social science and cognition §5.

Problem

XAI algorithms are numerous, but many were built for model developers or AI researchers, not for end users, so their fit to real explanation needs is unclear §3. Whether an explanation is good depends on the receiver's knowledge and goals, not only on the model §1. The chapter asks what human-centered approaches contribute to XAI, and it reviews recent HCI and social-science work on that question §1 §4 §5.

Method

The chapter is a survey and specifies no algorithm. Its structure and its actionable parts are:

Technical landscape §2. XAI techniques fall into two camps: directly interpretable models (for example decision trees, rule-based models and linear models), or opaque models explained after training with a post-hoc technique §2. Post-hoc explanations are grouped as global model explanations, outcome explanations and counterfactual inspection, and each can be feature-based or example-based §2. The chapter notes that most post-hoc methods approximate the model, and that approximations can be unfaithful §2.

Stakeholders §3. Five groups are listed: model developers, business owners or administrators, decision-makers, impacted groups and regulatory bodies §3. The chapter adds that a single group can need different explanations at different times, for example an overview at onboarding and case-level reasons at decision time §3.

XAI Question Bank §3. The authors interviewed 20 designers and collected common user questions across 16 ML applications. The result is more than 50 questions in nine categories: How (global), Why (a given prediction), Why Not (a different prediction), How to Be That (a different prediction), How to Still Be This (the current prediction), What if, Performance, Data and Output §3. The last three categories are broader needs that are not about the model's internal process tab. 1. The chapter's footnote separates Why Not, a comparison question, from How to Be That, a question about recourse §3.

Question-driven design §3. The chapter proposes starting from key user questions found through user research, then using them to choose XAI techniques and to iterate on the design. The steps are sketched below. The chapter does not state a stopping rule for the iteration.

# Question-driven user-centered design of XAI  [§3]
# Input:  users, question categories (nine, see above)
# Output: an XAI experience, refined over several rounds

def question_driven_design(users, question_categories):
    # 1. Find the questions real users ask, through user research  [§3]
    questions = user_research(users, question_categories)
    # 2. Map each question to candidate techniques  [§3] [tab. 1]
    candidates = [techniques_for(q) for q in questions]
    # 3. Choose techniques with designers and data scientists together  [§3]
    techniques = collaborative_choice(candidates)
    # 4. Build the experience and evaluate it with users  [§3]
    design = build_experience(techniques, questions)
    # Stopping rule for iteration: not stated  [§3]
    while not stop_condition_not_stated(design):
        design = refine(design, evaluate(design, users))
    return design

Mapping of questions to techniques tab. 1. The chapter pairs each question category with the ways an explanation can answer it, and with example methods that appear in open-source XAI toolkits. The mapping is a suggestion for discussion, not a validated result §3 tab. 1.

Pitfalls framework §4. The chapter names two disconnects. (a) Technical XAI does not serve users' end goals: the goal is an actionable understanding that is sufficient for a task, and proxy tasks such as simulation tests may not predict real effectiveness §4. Several HCI studies report that uncertainty information about single predictions helps people rely on AI more appropriately than local explanations do §4. (b) XAI assumes people attend to every explanation, but people often use fast heuristics (dual-process theory, System 1 and System 2) §4. Responses named include cognitive forcing functions, longer interaction time, progressive disclosure and multi-modal presentation §4. In one study, AI novices gained less from explanations and were more likely to feel illusory satisfaction; in another, adding explanations lowered satisfaction for people low in Need for Cognition §4.

Model comparison tool fig. 1. Figure 1 shows Model LineUpper, a tool the authors built with collaborators. It lets data scientists compare candidate models by their feature-importance explanations at global, region and instance levels, using a metrics table, a feature-importance comparison view and a probability scatterplot matrix fig. 1 §4.

Fairness auditing fig. 2. Figure 2 compares four explanation types in a study of fairness judgments on a recidivism model. The chapter reports that contrastive explanations best reveal individual-fairness problems fig. 2 §4.

Social transparency §4 fig. 3. The chapter argues that users need to see the social and organizational context of an AI system. It describes a design that shows past users' decisions with the AI, using What, Who, Why and When features next to technical explanations fig. 3 §4.

Theory-driven XAI §5. Miller's review of social science gives four properties of human explanations: they are contrastive, selected by the explainer, social, and statistical explanations are often less effective §5. Malle's theory splits explanation into an information process and an impression-management process §5. Conversational models (Hilton, Antaki, Walton) and Grice's maxims are proposed as bases for interactive XAI §5. A framework by Wang et al. links human reasoning dimensions to XAI methods and to cognitive biases §5.

Hyperparameters

Not applicable. The chapter specifies no models, training runs or evaluation settings. The only quantities it reports are the study sizes of its own question-bank work: 20 designers, 16 ML applications, more than 50 questions and 9 categories §3.

Results

No new results. The chapter reports no experiments. Descriptive counts from its own prior work are the study sizes listed under Hyperparameters §3. Table 1 is a mapping, not a measured result tab. 1. The empirical findings in Section 4 are summaries of cited studies and are not re-run here §4.

Paper vs code

No official code. The chapter is a survey and has no implementation. The paper text contains no repository or project-page link; its only URLs are for LaTeX packages and an ACM classification tool. Searched: (1) the paper and its arXiv metadata for links, none found; (2) GitHub repository search for "human-centered explainable AI" (no results); (3) GitHub repository search for "human-centered xai" (19 results, of which 17 were listed; none owned by an author or by IBM or Microsoft Research; the remaining two were not checked).

Pitfalls

• Draft source. The LaTeX source is labelled "Book Chapter Draft 4/2022" in red in the abstract. The arXiv version is v5, updated 2022-04-19 (arXiv metadata). The published book chapter may differ.
• Unused abstract file. The source package includes abstract.tex, which the main file does not include. It describes a different study (conditional delegation in content moderation). Do not take its content for this chapter. This card uses only the main main.tex file.
• Figure and table numbers are inferred. The source has one table and three figure environments, and this card numbers them in order: Table 1 is the mapping, Figure 1 is Model LineUpper, Figure 2 the fairness comparison and Figure 3 the social transparency design. A commented-out duplicate figure is not counted. Check the compiled PDF before quoting numbers.
• Section citations. The chapter has numbered sections but no subsections; this card cites sections 1 to 6 §1 to §6.
• Self-reported examples. Several cited studies are described as the authors' own work with collaborators (the question bank, Model LineUpper, social transparency, and studies on heuristics and cognitive load) §3 §4. The chapter summarises these; it gives no independent replication.
• Mixed evidence is stated, not resolved. The chapter says many studies found positive effects of XAI on understanding, and also that many did not find performance gains §1 §4. Read the cited studies before relying on either claim.
• Mapping table is a suggestion. Table 1 lists techniques available in open-source toolkits at the time of writing. It does not say which technique works best for a given question tab. 1.
• Broad definition of XAI. The chapter uses a wide definition that includes data, performance and output information, not only model internals §2 §3. Narrow definitions in other work will give different scopes.
• Stopping rule not stated. The question-driven design loop has no stated end condition §3. The pseudocode marks this.

Lineage

• Builds on: the XAI question bank and question-driven design work by the authors with collaborators §3; Miller's social-science account of explanations §5; Malle's theory of explanation §5; the stakeholder knowledge and objectives typology of Suresh et al. §3.
• Compared against: not applicable; the chapter reports no comparison.
• Superseded by: not stated.