Are Convolutional Neural Networks or Transformers more like human vision?

In plain words

Image classifiers are usually judged by how often they are right. Two classifiers with the same score can still fail on different pictures and in different ways. This paper asks whether a classic convolutional network (CNN) or a Transformer that looks at images as patches is more like a person in the mistakes it makes.

The authors compare what each system gets wrong, not only how often it is wrong. They use a test set of images whose shape and texture point to different object types, and they ask which kind of cue each system follows. They also check whether training on images with added changes (rotations, blur, noise and similar) moves each system toward what people do.

The main finding is that the Transformer's errors are closer to human errors on the main consistency measure, while a second, more detailed measure gives the opposite answer. Training with the added changes makes both systems rely more on shape.

TL;DR

• Compares ResNet (BiT-M-R50x1) and ViT models with human decisions on the Stylized ImageNet (SIN) cue-conflict images §3.2.
• Measures error consistency with Cohen's kappa and with two Jensen-Shannon (JS) distances over error distributions, one over 16 categories and one over 240 inter-class error types §3 §3.1.
• ViT is more consistent with humans than ResNet on kappa and class-wise JS; for inter-class JS, the paper reports the reverse §3.3.
• Fine-tuning on augmented ImageNet increases shape bias in both models; ResNet loses more ImageNet accuracy than ViT §5.3 §5.4.

Problem

Accuracy alone can hide different decision strategies. Cohen's kappa compares whether two systems are right or wrong on the same trials, but it ignores which wrong class is chosen and is hard to interpret §3. Accuracy on the training task also does not show whether a model relies on shape or texture §1 §4. CNNs are reported to lean on texture, while people lean on shape §1 §4. The paper treats this as an unknown: whether the Transformer's weaker built-in locality assumption produces more human-like decisions §2.

Method

The paper defines measures and then applies them. Equations are not numbered in the source, so they are cited by section.

1. Observed error overlap and Cohen's kappa §3. For two systems i and j, the observed overlap is the share of trials on which both are right or both are wrong. The expected overlap by chance uses each system's accuracy as an independent binomial probability, p_i p_j + (1 - p_i)(1 - p_j). Kappa normalises the observed overlap by the expected one: (c_obs - c_exp) / (1 - c_exp).

2. Confusion matrix on 16 entry-level categories §3.1. ImageNet's 1000 classes are grouped into 16 entry-level categories taken from the WordNet hierarchy §3.1. The 16 category labels are the folder names of the SIN cue-conflict stimuli, sorted vit_jax_fine-tune_imagenet.ipynb cell-32. Each model's class probabilities are summed over the ImageNet classes of each category, giving a 16 × 16 confusion matrix per model §3.1. The mapping is also reflected in the code (see Paper vs code).

3. Error distribution §3.1. For a system, the error count of true class i is the sum of its off-diagonal entries in row i. Dividing by the total number of errors gives a probability distribution p over the 16 classes.

4. Class-wise JS distance §3.1. The distance between two error distributions p and q uses their mixture m = (p + q) / 2. It is the square root of the average of the two Kullback-Leibler divergences D(p || m) and D(q || m). It depends only on where errors fall, not on overall accuracy §3.1.

5. Inter-class JS distance §3.1. The same JS distance applied to the 240 off-diagonal cells of the 16 × 16 matrix, so it compares what is confused with what.

6. Shape bias on SIN §4. On SIN cue-conflict images, take the trials where the prediction matches either the shape label or the texture label. Shape bias is the share of these trials where the prediction matches the shape §4.

7. Fine-tuning on augmented ImageNet §5.1. Augmentations are applied to ImageNet and the models are fine-tuned on the augmented data: rotations of ±90° and 180°; random cutout of 2 × 2 px up to half the image width; Sobel filtering; Gaussian blur with a 3 × 3 px kernel; colour jitter with probability 0.8 and colour drop with probability 0.2; and Gaussian noise with standard deviation 0.196 on the normalised image §5.1. Only the smallest ViT (ViT-B/32) and the smallest ResNet (BiT-M-R50x1) are fine-tuned §5.1. Training settings are in Hyperparameters.

# Cohen's kappa between two systems  [§3]
# Input:  labels (N), predictions_a (N), predictions_b (N)
# Output: kappa (scalar)

def cohens_kappa(labels, predictions_a, predictions_b):
    # Share of trials where both are right or both are wrong  [§3]
    agree = (predictions_a == labels) == (predictions_b == labels)
    observed_overlap = mean(agree)
    # Overlap expected by chance from each system's accuracy  [§3]
    accuracy_a = mean(predictions_a == labels)
    accuracy_b = mean(predictions_b == labels)
    expected_overlap = (accuracy_a * accuracy_b
                        + (1 - accuracy_a) * (1 - accuracy_b))
    return (observed_overlap - expected_overlap) / (1 - expected_overlap)

# Class-wise and inter-class JS distances  [§3.1]
# Input:  confusion_a (16, 16), confusion_b (16, 16)
#         rows: true category, columns: predicted category
# Output: class_wise (scalar), inter_class (scalar)

def class_wise_errors(confusion):
    # Errors per true class: off-diagonal row sums  [§3.1]
    errors = row_sums(confusion) - diagonal(confusion)
    return errors / sum(errors)  # distribution over 16 classes

def inter_class_errors(confusion):
    # All 16 * 15 = 240 off-diagonal cells as error types  [§3.1]
    errors = off_diagonal_cells(confusion)
    return errors / sum(errors)  # distribution over 240 types

def js_distance(p, q):
    # Mixture of the two error distributions  [§3.1]
    mixture = (p + q) / 2
    # Symmetrised KL to the mixture; distance is its square root  [§3.1]
    divergence = (kl(p, mixture) + kl(q, mixture)) / 2
    return sqrt(divergence)

class_wise = js_distance(
    class_wise_errors(confusion_a),
    class_wise_errors(confusion_b),
)
inter_class = js_distance(
    inter_class_errors(confusion_a),
    inter_class_errors(confusion_b),
)

# Shape bias on cue-conflict images  [§4]
# Input:  predicted16 (N), shape_labels (N), texture_labels (N)
# Output: shape_bias (scalar)

def shape_bias(predicted16, shape_labels, texture_labels):
    # Trials where the prediction matches the shape or the texture  [§4]
    matches_shape = predicted16 == shape_labels
    matches_either = matches_shape | (predicted16 == texture_labels)
    # Share of those trials that follow shape  [§4]
    return sum(matches_shape) / sum(matches_either)

Figures: the error-consistency comparison on SIN is fig. 4; the JS-versus-kappa and JS-versus-accuracy plots are fig. 3; the shape-bias bars are fig. 5; the before and after fine-tuning consistency is fig. 6; shape bias before and after fine-tuning is fig. 7; shape bias and ImageNet accuracy after fine-tuning is fig. 8. The stimulus example is fig. 1 and the schematic of convolution and attention is fig. 2.

Hyperparameters

Results

Author-reported. Setup: SIN cue-conflict images, ViT and ResNet models pretrained on ImageNet-21K and ILSVRC-2012, compared with human decisions §3.2 §3.3. The paper gives no numeric results in its text; all comparisons below are qualitative and read from the figures. Values were not transcribed from the plots.

Headline numbers in the text: 16 categories, a 16 × 16 confusion matrix, 240 inter-class error types, 95% confidence intervals §3.1 §3.2. The conclusion states that Transformers "maintain their accuracy" while gaining shape bias §6; the numbers behind that claim are in fig. 8, not in the text.

Paper vs code

Code: the official repository shikhartuli/cnn_txf_bias, linked from the paper at §5.1, owned by the first author, at commit 36e95d8ad9d3fe1f5561e8bd26c432d11cf7fb13. It contains Jupyter notebooks, with the fine-tuning and analysis logic in notebooks. Several sub-folders are vendored from Google projects with their own Apache-2.0 LICENSE files (big_transfer/, simclr/, vision_transformer/). The repository root has no LICENSE file. Notebook references below cite the cell id, because notebooks have no line numbers.

• ViT fine-tune length and learning rate. Paper: 100 epochs, initial learning rate 0.3, cosine step decay §5.1. Code: total_steps = 50, warmup_steps = 5, decay_type = 'cosine', base_lr = 0.03 vit_jax_fine-tune_imagenet.ipynb cell-28.
• ResNet fine-tune schedule. Paper: linear step decay, 5000 epochs, initial learning rate 0.03 §5.1. The BiT notebook uses a step schedule whose boundaries depend on dataset size (for datasets above 500,000 examples: steps 500, 10,000, 12,000, 18,000, 20,000), dividing the learning rate by 10 at each boundary, with a default base rate of 0.003 [code: big_transfer/colabs/big_transfer_jax_fine-tune_imagenet.ipynb, functions get_schedule and get_lr_ft]. No 5000-epoch or 0.03 setting was found in that notebook. The ResNet run configuration was not located elsewhere in the repository.
• Gaussian noise. Paper: standard deviation 0.196 "for normalized image" §5.1. Code: training images are scaled to [-1, 1] input_pipeline.py; noise with standard deviation 50/255 ≈ 0.196 is added and the result is clipped to [0, 1] vit_jax_fine-tune_imagenet.ipynb cell-32. The clip sets every value below 0 to 0, which the paper does not mention.
• Augmentation composition. The paper's text does not say whether the six augmentations are combined §5.1. The code trains each stage on top of the previous stage's weights and optimiser state, and names the checkpoints "Baseline+Rotate+Cutout+…" vit_jax_fine-tune_imagenet.ipynb cell-32.
• Colour distortion strength. Paper: probabilities only §5.1. Code: SimCLR's random_color_jitter, applied with probability 0.8 and then greyscale with probability 0.2, with strength 0.2 (brightness, contrast and saturation 0.16; hue 0.04) data_util.py.
• Gaussian blur. Paper: 3 × 3 px kernel §5.1. Code: calls tfa.image.gaussian_filter2d with its explicit filter arguments commented out, so library defaults apply vit_jax_fine-tune_imagenet.ipynb cell-32.
• Sobel filtering. Paper names the filter §5.1 but not its output scaling. Code calls tfio.experimental.filter.sobel and writes the result back without rescaling vit_jax_fine-tune_imagenet.ipynb cell-32.
• Rotation and cutout. Consistent with the paper: rotations are 90°, 180° or 270° via rot90 with k in {1, 2, 3}; cutout uses random even sizes from 2 to 190 px per axis vit_jax_fine-tune_imagenet.ipynb cell-32.
• 16-category mapping. Consistent with the paper: softmax probabilities are summed over fixed ImageNet indices for each of the 16 categories vit_jax_fine-tune_imagenet.ipynb cell-32.
• Shape bias. Code computes shape match divided by (shape match + texture match) vit_jax_fine-tune_imagenet.ipynb cell-32. With cue-conflict images, the two labels differ, so this equals the paper's definition §4.
• JS distance. Paper: formula with no logarithm base §3.1. Code: scipy.spatial.distance.jensenshannon(p, q), which normalises its inputs and uses the natural logarithm by default confusion_matrices.ipynb. The notebook also defines a Wasserstein distance that the paper does not describe.
• Cohen's kappa. The analysis notebook reads precomputed kappa values from a CSV file rather than computing them confusion_matrices.ipynb. The computation was not checked here.
• Human data. The paper's active text does not say where the human decisions come from §3.2. The repository contains error-consistency/raw-data and cifar-10h/; these were not reviewed for this card.

No discrepancies were checked for the Results figures, because the plotted values were not extracted.

Pitfalls

• Unnumbered equations. The paper's equations are unnumbered in the source. Cite them by section (for example §3 for kappa and §3.1 for JS) and not by equation number.
• Epochs, steps and learning rate do not match. The paper says 100 epochs at learning rate 0.3 for ViT; the code runs 50 steps at 0.03 §5.1 vit_jax_fine-tune_imagenet.ipynb cell-28. An implementation that follows the paper will not reproduce the released runs.
• Clipped noise. The paper's noise is specified for normalised images. The code clips to [0, 1] on [-1, 1] data, which removes all negative values vit_jax_fine-tune_imagenet.ipynb cell-32 input_pipeline.py.
• Augmentation stacking is undocumented in the paper. The code stacks augmentations in a fixed order and carries weights across stages vit_jax_fine-tune_imagenet.ipynb cell-32. A reader who fine-tunes each augmentation separately gets a different experiment.
• Parameters the code fixes but the paper omits. Colour-jitter strength, Gaussian blur kernel, Sobel output scaling, batch size, input resolution, gradient clipping and accumulation are not in the paper's text §5.1.
• Log base of the JS distance. The paper's formula has no base. The code uses the natural logarithm via SciPy §3.1 confusion_matrices.ipynb. Absolute JS values change with the base, although the ordering of distances does not.
• Full-distribution result contradicts the summary. The paper's conclusion says Transformers are "largely more consistent with human errors" §6, but its inter-class result shows ViT less consistent than ResNet §3.3. The paper itself reconciles this only partly by pointing to the shape-bias analysis §4.
• Statistical tests. The paper reports 95% confidence intervals on cross-validated test error §3.2 and significance statements for fine-tuning §5.2, but does not describe the test or the cross-validation. The statistics are not reproducible from the text.
• Model size. Fine-tuning uses only the smallest ViT and ResNet §5.1; the paper says effects of parameter count are left to future work. Do not extend the fine-tuning results to larger models.

Lineage

• Builds on: Geirhos et al. 2018 (Stylized ImageNet, the SIN stimuli) §1; Geirhos et al. 2020 (error consistency with Cohen's kappa and the shape-bias definition) §3 §4; Dosovitskiy et al. 2021 (Vision Transformer) §1; Kolesnikov et al. 2020 (BiT, source of the BiT-M-R50x1 ResNet) §3.2; Hermann et al. 2019 (augmentation effects on texture bias) §5. None of these have wiki cards yet.
• Compared against: ResNet (BiT-M-R50x1) and ViT-B/16, ViT-B/32, ViT-L/16, ViT-L/32 §3.2; humans §3.3.
• Superseded by: not stated.