---
kind: concept
slug: error-consistency
title: Error consistency
summary: >-
  Measures whether two classifiers, usually a model and humans, fail on the same inputs and in the same ways, beyond accuracy. Variants are Cohen's kappa over right and wrong trials and two Jensen-Shannon distances over error distributions, one over categories and one over confusion cells. The measures can disagree about which system is more human-like.
tags: [evaluation, human-alignment]
papers: ["2105.07197"]
related: [inductive-bias, vision-transformer, convolutional-neural-network, inter-annotator-agreement, human-evaluation]
status: draft
updated_at: 2026-11-06
---

## Overview

Accuracy says how often a classifier is right, but two classifiers with the
same accuracy can fail on different inputs and choose different wrong labels.
Error consistency asks whether two systems make the same kind of mistake. The
usual comparison is a model against human decisions on the same images
[arxiv:2105.07197 §3.2]. The shared idea is to compare error patterns, not
scores: a measure is applied to the same trials for both systems, and the
result says how closely their errors line up.

The paper defines three such measures. They look at different things:

- **Right or wrong per trial:** Cohen's kappa, which only checks whether both
  systems are right or both are wrong on each trial [arxiv:2105.07197 §3].
- **Where the errors fall:** a Jensen-Shannon (JS) distance between the
  distributions of errors over 16 entry-level categories
  [arxiv:2105.07197 §3.1].
- **Which wrong label replaces which:** a JS distance over the 240
  off-diagonal cells of the 16 × 16 confusion matrix
  [arxiv:2105.07197 §3.1].

The paper applies them to ResNet and Vision Transformer (ViT) models against
humans on Stylized ImageNet cue-conflict images [arxiv:2105.07197 §3.2]. A
related quantity, shape bias, measures whether a model's correct-looking
answers follow shape or texture, and the paper uses it to explain some of the
error results [arxiv:2105.07197 §4].

## Variants

### Cohen's kappa on correctness

Each trial counts as agreement when the two systems are both right or both
wrong. The observed agreement is corrected for the agreement expected by
chance, given each system's accuracy [arxiv:2105.07197 §3]. Kappa ignores which
wrong class a system picks, and the paper notes that it is hard to interpret
[arxiv:2105.07197 §3]. Papers that use it are [Are Convolutional Neural Networks or Transformers more like human vision?](/papers/2105.07197).

### Class-wise JS distance

Each system's errors are turned into a distribution over the 16 categories:
for each true category, count the off-diagonal entries of its row, then
normalise. The distance between two such distributions is the JS distance,
which compares each one with their mixture [arxiv:2105.07197 §3.1]. The paper
says this distance depends only on where errors fall, not on overall accuracy
[arxiv:2105.07197 §3.1]. Card: [Are Convolutional Neural Networks or Transformers more like human vision?](/papers/2105.07197).

### Inter-class JS distance

The same JS distance, but applied to the 240 off-diagonal cells of the
confusion matrix instead of the 16 categories. Each cell is one kind of
error (true category to predicted category), so the measure compares which
confusions occur [arxiv:2105.07197 §3.1]. Card: [Are Convolutional Neural Networks or Transformers more like human vision?](/papers/2105.07197).

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need one number for how often two systems fail on the same trials, and you only have right or wrong per trial | Cohen's kappa | It needs no category structure; the paper uses it as the main consistency statistic [arxiv:2105.07197 §3] |
| You want to know whether two systems spread their errors over the same categories | Class-wise JS over 16 categories | It uses only the error distribution, so overall accuracy does not enter [arxiv:2105.07197 §3.1] |
| You want to know whether two systems make the same specific confusions | Inter-class JS over 240 cells | It compares which true-to-predicted pairs occur, which the other two measures do not [arxiv:2105.07197 §3.1] |
| You care about the kind of mistake, not just the rate | Report class-wise and inter-class JS together | The paper's ViT and ResNet ranking differs between them [arxiv:2105.07197 §3.3] |
| You have no access to model internals, only labels and predictions | Any of the three | Each is computed from predicted labels on a shared set of trials [arxiv:2105.07197 §3] [arxiv:2105.07197 §3.1] |
| You need human decisions for the comparison | A stimulus set with human decisions on the same images | The paper compares humans and models on SIN cue-conflict images [arxiv:2105.07197 §3.2] |

Notes on the evidence:

- The paper reports its comparison results mainly in figures. The text gives
  directions, not numbers, so this page reports only directions
  [arxiv:2105.07197 §3.3] [arxiv:2105.07197 fig. 4].
- All results come from one comparison: ResNet and ViT models against humans
  on SIN [arxiv:2105.07197 §3.2]. Transfer of these rankings to other data sets
  or to other architectures is not tested in the paper.
- For fine-tuning, the paper fine-tunes only the smallest ResNet and the
  smallest ViT [arxiv:2105.07197 §5.1], so its fine-tuning results do not say
  anything about larger models.

## Implementation sketch

The three measures share one input: the true labels and the predictions of
each system on the same trials, with human decisions for the human
comparison. Kappa uses per-trial correctness. The two JS measures need a
16 × 16 confusion matrix per system, with rows as true categories and columns
as predicted categories.

```pseudocode
# Cohen's kappa on correctness  [arxiv:2105.07197 §3]
# Input: labels, preds_a, preds_b, one entry per trial
def kappa_consistency(labels, preds_a, preds_b):
    right_a = preds_a == labels
    right_b = preds_b == labels
    # Share of trials where both are right or both are wrong
    observed = mean(right_a == right_b)
    acc_a = mean(right_a)
    acc_b = mean(right_b)
    # Agreement expected by chance from the two accuracies
    expected = acc_a * acc_b + (1 - acc_a) * (1 - acc_b)
    return (observed - expected) / (1 - expected)
```

```pseudocode
# Error distributions from a 16 x 16 confusion matrix  [arxiv:2105.07197 §3.1]
# conf[i][j]: trials with true category i predicted as category j

def category_errors(conf):
    # Errors made per true category: off-diagonal row sums
    errors = row_sums(conf) - diagonal(conf)
    return errors / sum(errors)

def confusion_errors(conf):
    # Each off-diagonal cell is one error type (16 * 15 = 240)
    cells = conf[off_diagonal_mask]
    return cells / sum(cells)

# JS distance between two error distributions  [arxiv:2105.07197 §3.1]
def js_distance(p, q):
    m = (p + q) / 2
    # Average KL to the mixture, in both directions
    divergence = (kl(p, m) + kl(q, m)) / 2
    # The distance is the square root of that divergence
    return sqrt(divergence)

class_wise = js_distance(category_errors(conf_a),
                         category_errors(conf_b))
inter_class = js_distance(confusion_errors(conf_a),
                          confusion_errors(conf_b))
```

Pitfalls for an implementation:

- The paper's JS formula has no logarithm base [arxiv:2105.07197 §3.1].
  Absolute values change with the base, so compare only distances computed
  with the same base.
- The paper's measures assume the two systems answer the same trials. Do not
  mix trial sets when comparing systems.
- The paper says nothing about the sample sizes behind the 240 cells. If you
  work with few trials, expect sparse cells; the paper does not address this.

## Contradictions

- **Which model is more human-like depends on the measure.** On kappa and on
  class-wise JS, ViT is more consistent with humans than ResNet. On inter-class
  JS, the paper reports the reverse [arxiv:2105.07197 §3.3] [arxiv:2105.07197 fig. 4].
- **Summary versus detail.** The paper's conclusion says Transformers are
  largely more consistent with human errors [arxiv:2105.07197 §6]. The
  inter-class result in §3.3 goes the other way for ViT. The paper partly
  reconciles this by pointing to the shape-bias analysis [arxiv:2105.07197 §4].
- **Class-wise JS and kappa move together; inter-class JS does not.** Across
  models, class-wise JS is closely inversely correlated with kappa
  [arxiv:2105.07197 §3.3] [arxiv:2105.07197 fig. 3]. The paper reports no
  correlation between inter-class JS and kappa [arxiv:2105.07197 §3.1]
  [arxiv:2105.07197 fig. 3].
- **Fine-tuning effects differ by measure and model.** After fine-tuning on
  augmented ImageNet, ResNet becomes less human-like on kappa and inter-class
  JS, which the paper calls significant. ViT shows no significant change on
  these measures [arxiv:2105.07197 §5.2] [arxiv:2105.07197 fig. 6].
- **Paper and released code differ on training settings.** The fine-tuning
  schedules, the noise level and the augmentation stacking in the paper do not
  match the released code. Those differences matter for anyone who tries to
  reproduce the fine-tuning results. See the paper card's Paper vs code
  section: [Are Convolutional Neural Networks or Transformers more like human vision?](/papers/2105.07197).

## Papers

- [Are Convolutional Neural Networks or Transformers more like human vision?](/papers/2105.07197): compares ResNet and ViT with human decisions on SIN using kappa and two JS distances over error distributions, and tests shape bias after fine-tuning on augmented ImageNet.