Error consistency

Overview

Accuracy says how often a classifier is right, but two classifiers with the same accuracy can fail on different inputs and choose different wrong labels. Error consistency asks whether two systems make the same kind of mistake. The usual comparison is a model against human decisions on the same images 2105.07197 §3.2. The shared idea is to compare error patterns, not scores: a measure is applied to the same trials for both systems, and the result says how closely their errors line up.

The paper defines three such measures. They look at different things:

• Right or wrong per trial: Cohen's kappa, which only checks whether both systems are right or both are wrong on each trial 2105.07197 §3.
• Where the errors fall: a Jensen-Shannon (JS) distance between the distributions of errors over 16 entry-level categories 2105.07197 §3.1.
• Which wrong label replaces which: a JS distance over the 240 off-diagonal cells of the 16 × 16 confusion matrix 2105.07197 §3.1.

The paper applies them to ResNet and Vision Transformer (ViT) models against humans on Stylized ImageNet cue-conflict images 2105.07197 §3.2. A related quantity, shape bias, measures whether a model's correct-looking answers follow shape or texture, and the paper uses it to explain some of the error results 2105.07197 §4.

Variants

Cohen's kappa on correctness

Each trial counts as agreement when the two systems are both right or both wrong. The observed agreement is corrected for the agreement expected by chance, given each system's accuracy 2105.07197 §3. Kappa ignores which wrong class a system picks, and the paper notes that it is hard to interpret 2105.07197 §3. Papers that use it are Are Convolutional Neural Networks or Transformers more like human vision?.

Class-wise JS distance

Each system's errors are turned into a distribution over the 16 categories: for each true category, count the off-diagonal entries of its row, then normalise. The distance between two such distributions is the JS distance, which compares each one with their mixture 2105.07197 §3.1. The paper says this distance depends only on where errors fall, not on overall accuracy 2105.07197 §3.1. Card: Are Convolutional Neural Networks or Transformers more like human vision?.

Inter-class JS distance

The same JS distance, but applied to the 240 off-diagonal cells of the confusion matrix instead of the 16 categories. Each cell is one kind of error (true category to predicted category), so the measure compares which confusions occur 2105.07197 §3.1. Card: Are Convolutional Neural Networks or Transformers more like human vision?.

When to use which

Notes on the evidence:

• The paper reports its comparison results mainly in figures. The text gives directions, not numbers, so this page reports only directions 2105.07197 §3.3 2105.07197 fig. 4.
• All results come from one comparison: ResNet and ViT models against humans on SIN 2105.07197 §3.2. Transfer of these rankings to other data sets or to other architectures is not tested in the paper.
• For fine-tuning, the paper fine-tunes only the smallest ResNet and the smallest ViT 2105.07197 §5.1, so its fine-tuning results do not say anything about larger models.

Implementation sketch

The three measures share one input: the true labels and the predictions of each system on the same trials, with human decisions for the human comparison. Kappa uses per-trial correctness. The two JS measures need a 16 × 16 confusion matrix per system, with rows as true categories and columns as predicted categories.

# Cohen's kappa on correctness  [arxiv:2105.07197 §3]
# Input: labels, preds_a, preds_b, one entry per trial
def kappa_consistency(labels, preds_a, preds_b):
    right_a = preds_a == labels
    right_b = preds_b == labels
    # Share of trials where both are right or both are wrong
    observed = mean(right_a == right_b)
    acc_a = mean(right_a)
    acc_b = mean(right_b)
    # Agreement expected by chance from the two accuracies
    expected = acc_a * acc_b + (1 - acc_a) * (1 - acc_b)
    return (observed - expected) / (1 - expected)

# Error distributions from a 16 x 16 confusion matrix  [arxiv:2105.07197 §3.1]
# conf[i][j]: trials with true category i predicted as category j

def category_errors(conf):
    # Errors made per true category: off-diagonal row sums
    errors = row_sums(conf) - diagonal(conf)
    return errors / sum(errors)

def confusion_errors(conf):
    # Each off-diagonal cell is one error type (16 * 15 = 240)
    cells = conf[off_diagonal_mask]
    return cells / sum(cells)

# JS distance between two error distributions  [arxiv:2105.07197 §3.1]
def js_distance(p, q):
    m = (p + q) / 2
    # Average KL to the mixture, in both directions
    divergence = (kl(p, m) + kl(q, m)) / 2
    # The distance is the square root of that divergence
    return sqrt(divergence)

class_wise = js_distance(category_errors(conf_a),
                         category_errors(conf_b))
inter_class = js_distance(confusion_errors(conf_a),
                          confusion_errors(conf_b))

Pitfalls for an implementation:

• The paper's JS formula has no logarithm base 2105.07197 §3.1. Absolute values change with the base, so compare only distances computed with the same base.
• The paper's measures assume the two systems answer the same trials. Do not mix trial sets when comparing systems.
• The paper says nothing about the sample sizes behind the 240 cells. If you work with few trials, expect sparse cells; the paper does not address this.

Contradictions

• Which model is more human-like depends on the measure. On kappa and on class-wise JS, ViT is more consistent with humans than ResNet. On inter-class JS, the paper reports the reverse 2105.07197 §3.3 2105.07197 fig. 4.
• Summary versus detail. The paper's conclusion says Transformers are largely more consistent with human errors 2105.07197 §6. The inter-class result in §3.3 goes the other way for ViT. The paper partly reconciles this by pointing to the shape-bias analysis 2105.07197 §4.
• Class-wise JS and kappa move together; inter-class JS does not. Across models, class-wise JS is closely inversely correlated with kappa 2105.07197 §3.3 2105.07197 fig. 3. The paper reports no correlation between inter-class JS and kappa 2105.07197 §3.1 2105.07197 fig. 3.
• Fine-tuning effects differ by measure and model. After fine-tuning on augmented ImageNet, ResNet becomes less human-like on kappa and inter-class JS, which the paper calls significant. ViT shows no significant change on these measures 2105.07197 §5.2 2105.07197 fig. 6.
• Paper and released code differ on training settings. The fine-tuning schedules, the noise level and the augmentation stacking in the paper do not match the released code. Those differences matter for anyone who tries to reproduce the fine-tuning results. See the paper card's Paper vs code section: Are Convolutional Neural Networks or Transformers more like human vision?.

Papers

• Are Convolutional Neural Networks or Transformers more like human vision?: compares ResNet and ViT with human decisions on SIN using kappa and two JS distances over error distributions, and tests shape bias after fine-tuning on augmented ImageNet.