wikillm

Paper · 2021-04-02 · cs.LG, cs.AI

Explainable Artificial Intelligence (XAI) on Time Series Data: A Survey

Survey of explainable AI for time-series models. It sorts methods for CNNs (CAM, Gradient*Input, occlusion), attention models, SAX and shapelets by scope and audience, then covers links to stability, robustness and confidence, and how explanations are evaluated. It proposes no method and runs no experiments; Table I lists 31 methods.

Explainable AIFeature attributionExplanation evaluation (judging whether an XAI explanation is good)Time series classificationexplainabilitytime-seriessurveyinterpretability

arXivPDFMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many of the most accurate models for sensor readings, heart traces or motion data are too complicated for a person to read. This survey looks at ways to explain such models. Some methods show which part of an input series pushed the prediction one way. Others show what the model has learned in general, or offer a rough measure of how sure the model is.

The authors sort these methods in a few ways: whether the explanation is built into the model or added after training, whether it covers one prediction or the whole model, and who it is for, such as the engineer who built the model, the person who decides whether to act on it, or the person affected by the result. They also point out that an explanation does not by itself make a model stable against small changes in its input, and that nobody has agreed on a standard way to score explanations. The paper is a map of the field, not a recipe for building one of these explainers.

TL;DR

A 2021 survey, not a new method. It groups explainable methods for time-series models, mostly deep networks, into post-hoc explainers for CNNs, attention models, data-mining models (SAX, fuzzy logic) and example-based models (shapelets) §III. It also asks whether explanations can support stability, robustness and confidence, a link it finds little studied §II.A §V.B. It has no experiments and no shared evaluation metric; it favours quantitative tests for time series §VI. Use it to find the families of methods, and read the original papers for any formula.

Problem

  • Deep CNNs and RNNs work well on time series but are hard to interpret. The methods built for images and text do not carry over directly, because people do not read raw signals by eye §I.
  • High accuracy does not show that a model is stable (its output does not change under small input or model perturbations), robust (it withstands deliberate attacks such as adversarial perturbations) or confident (its uncertainty is known) §II.A.1 §II.A.2 §II.A.3.
  • The authors state that most reviewed explanation methods cannot assert stability, robustness or confidence §II. They also state that no metric is globally recognised for judging explanation quality §VI.
  • The paper's examples come from medicine (ECG, ICU mortality, surgical motion data) and autonomous driving §I §II.A.1 §III.A.1 §III.A.2.
  • The survey does not say how it selected the reviewed works [not stated].

Method

The survey has no algorithm of its own. Its contribution is a taxonomy, and each reviewed method is classified along these axes. Section numbers follow the paper's IEEE numbering: Roman numeral for the section, letter for the subsection, digit for the subsubsection. For example, §III.A.1 is "Backpropagation-based methods".

  1. Ante-hoc versus post-hoc. Ante-hoc methods build explainability into the model structure, so the explanation exists when training ends. Post-hoc methods approximate a trained model after the fact. Post-hoc methods can be model-agnostic or model-specific §III intro tab. 1.
  2. Method family §III:
    • Backpropagation-based, for CNNs §III.A.1. These need one forward and one backward pass. Class activation mapping (CAM) uses the global average pooling layer that follows the convolutions. GradientInput multiplies the derivative of a layer's activations with respect to the input by the input itself. GradientInput is used by Strodthoff, Siddiqui and Cho et al. fig. 4.
    • Perturbation-based, for CNNs §III.A.2. The method alters or removes part of the input, runs a forward pass on the altered input, and measures the change in output. Occlusion sensitivity is one example, used by ConvTimeNet fig. 5. Tonekaboni et al. replace each observation with a generated one and measure the output change.
    • Attention, for RNNs and Transformers §III.B. Attention weights assign importance to time steps, and some models also weight variables. The survey covers temporal and variable attention in LSTM-based models fig. 6. It also covers Temporal Fusion Transformers, whose attention can point to globally important variables and persistent temporal patterns §III.B. SHAP is used as a model-agnostic method on a recurrent model §III.B.
    • Data-mining, for time series §III.C. Symbolic Aggregate approXimation (SAX) turns a series into a string of symbols, and SAX-based classifiers use the resulting features. Fuzzy-logic models give outputs as degrees between 0 and 1, and the paper describes fuzzy rule and fuzzy cognitive map models for prediction §III.C.
    • Example-based §III.D. Shapelets are subsequences that best represent a class, found by searching for the split that maximises information gain. Later work adds a regulariser for interpretable shapelets and speeds up the search with PAA or a new search method §III.D fig. 7.
  3. Scope §IV. An explanation is local when it applies to one sample and global when it applies to a set of samples or the whole dataset.
    • CAM and occlusion are local by default §IV.A. Oviedo et al. average CAM over all training samples of a class to get a global map §IV.B.
    • Tsviz clusters filters by their activation patterns, and Cho et al. cluster input subsequences fig. 8 fig. 9 §IV.B.
    • For RNNs, scope depends on whether internal states reset after each prediction. A single-instance state gives a local explanation; a state covering several instances gives a global one §IV.A.
    • SAX and shapelets are global, because the search runs over the dataset §IV.B.
  4. Audience §V.A. Developers get technical, mostly global explanations. End users get local explanations about their own case. Decision makers, who carry responsibility without being machine-learning experts, need a measure of confidence more than the mechanism §V.A.1 §V.A.2 §V.A.3. Table I puts all 31 methods into these axes, with their evaluation status tab. 1.
  5. Confidence §V.B. Four approaches are reported:
    • Hartl et al. measure how easily each feature can cause a misclassification and defend by dropping the most manipulable features §V.B.
    • Tsviz perturbs the most salient part of the input and reports a large drop in accuracy. Cho et al. perturb the least salient part and expect small changes in activations. The two papers are read as two sides of one idea §V.B.
    • Gee et al. add a diversity penalty to learn more diverse prototypes, which the authors link to stability on samples far from the training distribution §V.B.
    • Ates et al. combine two adversarial training procedures and generate counterfactual visual explanations (RATIO) §V.B.
  6. Evaluation §VI. Qualitative evaluation uses expert assessment. Quantitative evaluation uses perturbation tests, sanity checks with data and model randomisation, or sequence-level tests. Arnout et al. propose Swap Time Points and Mean Time Points, which test trends and patterns that pointwise perturbation misses §VI.B. Table II lists the evaluation approach of each method tab. 2.

Pseudocode, as described in the text. The survey gives these methods in words only; no equation is numbered, so the code below is a reading of its description, not a verified implementation.

Pseudocode
# Class activation mapping (CAM) for a 1-D CNN  [§III.A.1]
# feature_maps: (N, T'), output of the last convolutional layer
# class_weights: (n_classes, N), one weight per filter; learned by a dense
#   layer with softmax, which the text describes but does not specify  [§III.A.1]

def class_activation_maps(feature_maps, class_weights, input_length):
    # Weighted sum over the N filters, one heatmap per class  [§III.A.1]
    heatmaps = class_weights @ feature_maps          # (n_classes, T')
    # Up-sample each heatmap to the input length to locate sub-sequences  [§III.A.1]
    maps = []
    for heatmap in heatmaps:
        maps.append(upsample(heatmap, input_length))  # (T,)
    return maps
Pseudocode
# Gradient*Input attribution for one instance  [§III.A.1] [fig. 4]
# Layer, and whether signed or absolute values are kept: not stated

def gradient_times_input(model, series, layer):
    # Partial derivative of the chosen layer's activations w.r.t. the input  [§III.A.1]
    gradient = partial_derivative(layer_output(model, series, layer), series)
    # Multiply element-wise by the input itself  [§III.A.1]
    return gradient * series                          # (T,)
Pseudocode
# SAX: symbolic aggregate approXimation  [§III.C]
# Input normalisation before this step: not stated  [§III.C]

def sax(series, n_segments, n_symbols):
    # Piecewise aggregate approximation: equal-length segments, each averaged  [§III.C]
    segment_means = paa(series, n_segments)
    # Breakpoints giving equal-probability regions under a Gaussian  [§III.C]
    breakpoints = gaussian_breakpoints(n_symbols)
    # Give each segment the symbol of the region its mean falls in  [§III.C]
    symbols = [region_index(m, breakpoints) for m in segment_means]
    return symbols

Hyperparameters

The survey is not an experiment, so it sets no training values. None of the settings an implementer would need is given in the text; the closest passages are §III.A.1, §III.A.2, §III.C, §III.D and §VI.B.

NameValueWhereNotes
CAM: number of filters N, and filter lengthnot stated§III.A.1Depends on the reviewed model; not given
CAM: how class weights are learnednot stated§III.A.1Dense layer with softmax, no further detail
Occlusion: window size, stride, replacement valuenot stated§III.A.2ConvTimeNet's settings are not in this paper
Tonekaboni: generator for replacement observationsnot stated§III.A.2Only "a generated one"
SAX: number of segments, alphabet sizenot stated§III.CNo values given
SAX: input normalisationnot stated§III.CGaussian assumption is named, no procedure
Shapelets: candidate length and countnot stated§III.DInformation-gain criterion only
Evaluation: perturbation fraction or sample countnot stated§VI.BTests described, no settings

Results

No experiments of its own, and no benchmark numbers for any reviewed method appear in the text. What the paper does report, counted from its tables by us:

  • Table I lists 31 methods: 13 post-hoc and 18 ante-hoc tab. 1. Each entry carries a scope (local, global or both), a target audience and an evaluation tag. Entries are reviewed works, not measured results.
  • Table II lists 12 methods by evaluation type: 5 quantitative only, 6 qualitative only, and 1 (Tsviz) both tab. 2.
  • Setup facts the text gives for cited work: Wang et al. use a little over 80 UCR datasets §III.A.1; ConvTimeNet uses 85 UCR datasets §III.A.2; Tonekaboni et al. use the MIMIC ICU time-series data for mortality prediction §III.A.2; Cho et al. use UWaveGestureLibraryAll and a smartphone activity dataset §III.A.1.

Author-reported claims without numbers: shapelets are described as more interpretable, faster and more accurate than k-nearest neighbours §III.D; overlapping SAX words contribute little individually, which the authors read as lower sensitivity to noise §V.B. The paper does not give the values behind either claim.

Paper vs code

No official code. The survey is a review, so there is no implementation to compare. Searched: the LaTeX source (no code or project link, only DOI and reference URLs); the arXiv metadata in raw/arxiv/2104.00950.json (abstract only, no project link); and a GitHub repository search via the API for "explainable time series survey XAI", which returned 0 results. No discrepancies were checked against code. Internal consistency: the methods that Table I attributes to Wang, Fawaz and Oviedo et al. match the CAM description in the text §III.A.1 tab. 1.

Pitfalls

  • Coverage is not described. The paper does not state how it selected the 31 methods in Table I, which databases it searched, or what it excluded tab. 1. Treat the table as the authors' selection, not as the full field.
  • Descriptions, not formulas. The CAM, Gradient*Input, occlusion and SAX descriptions are verbal, and none is numbered as an equation. The CAM weighting is ambiguous: the text does not say which layer's activations are weighted, or how the softmax is applied §III.A.1. Take exact formulas from the original papers, and check them before implementing.
  • Gradient*Input details. The text names "the current layer" without saying which layer, and does not say whether signed or absolute values are used §III.A.1.
  • SAX preprocessing. The text describes segmenting, averaging and assigning symbols under a Gaussian assumption, but does not describe normalisation before PAA §III.C. An implementation without it will give different symbols.
  • Occlusion and perturbation settings. Window size, stride, replacement value and perturbation count are not stated for any method §III.A.2 §VI.B. Results from perturbation tests depend heavily on these choices, and the paper does not show how sensitive its examples are to them.
  • Attention is treated as explanation. The survey reads attention weights as the importance of time steps §III.B. It also says that trust in such explanations can be questioned, because time series are hard to read §VII.
  • Claims without numbers. The shapelet comparison with k-NN and the SAX noise-sensitivity point are stated without figures §III.D §V.B.
  • "First attempts" claim. The authors say, to their knowledge, the confidence work in §V.B is the first attempt to give epistemic confidence using explanations §V.B. The survey does not check this.
  • Local or global for RNNs depends on state handling. The rule in §IV.A holds only if one knows whether internal states reset. This is not stated for most reviewed models.
  • Third-party figures. Several figures are reproduced from other authors' work "with authorization" (credited to Cho et al., Kashiparekh et al., Schockaert et al., Li et al. and Siddiqui et al.) fig. 4 fig. 5 fig. 6 fig. 7 fig. 8 fig. 9. The paper's CC BY 4.0 license does not cover those figures. This card has no figures and describes them in words only.
  • Version and scope. The record holds arXiv v1 (2021-04-02). Later work is not covered. The arXiv metadata title reads "TimeSeries", and the paper's title reads "Time Series"; the card uses the paper's title.

Lineage

  • Builds on: CAM, occlusion sensitivity, SHAP, SAX and shapelets, which the survey reviews and cites by name §III; the paper gives no arXiv ids for them.
  • Compared against: LIME, layer-wise relevance propagation, sensitivity analysis and feature occlusion, used as baselines in evaluation §VI.B.
  • Superseded by: not stated. The paper is a 2021 review and does not discuss later methods.

Source and license

Adapted from “Explainable Artificial Intelligence (XAI) on Time Series Data: A Survey” by Thomas Rojat, Raphaël Puget, David Filliat, Javier Del Ser, Rodolphe Gelin, Natalia Díaz-Rodríguez (arXiv:2104.00950), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.