CINIC-10 is not ImageNet or CIFAR-10

In plain words

Image classifiers are compared on benchmark datasets. CIFAR-10 is the small standard one: ten classes of tiny pictures, but with no proper validation split. ImageNet is the large standard one, and a single training run on it can take days. The authors wanted something in between, with equal-sized splits.

CINIC-10 keeps CIFAR-10's tiny pictures and class names and adds many more pictures taken from ImageNet and shrunk to the same size. Models can swap it in for CIFAR-10 unchanged. It has three equal-sized train, validation and test sets, so results are easier to compare.

The ImageNet part is noisier, because those pictures were not checked by hand, and the two sources do not look identically distributed. The authors treat that gap as a feature: it shows how a model trained on CIFAR-style pictures copes with ImageNet-style ones. A few standard networks are scored on it as a reference point.

TL;DR

• CINIC-10 is 270,000 32×32 colour images in ten classes: all 60,000 CIFAR-10 images plus 210,000 downsampled ImageNet images §2.
• Three equal splits of 90,000 images each (9,000 per class) make it a drop-in CIFAR-10 replacement §2 README.md.
• This is a dataset report with no new method. Its only results are test errors for six CIFAR-style CNNs: 8.55 to 18.00 tab. 1.
• The paper omits the training recipe. The repository README gives one (300 epochs, momentum 0.9) README.md.
• Use it when a harder, CIFAR-sized benchmark is needed. Check the pitfalls first: the ImageNet part is unvetted, its sampling code is not in the repository, and the dataset licence is not stated.

Problem

• CIFAR-10 has 6,000 examples per class and CIFAR-100 has 600, which leaves a large gap in difficulty. The authors argue CIFAR-100 is harder than ImageNet §1.
• ImageNet is unwieldy: over a million large images, and a single training run can take days without large compute §1.
• ImageNet-32 already exists as a CIFAR alternative, but its downsampled images carry less information, so it is harder than ImageNet per image §1.
• Most benchmark datasets have uneven train, validation and test splits. CIFAR has no validation split at all. Equal split sizes give a fairer view of generalisation §1.
• Implicit assumption: a benchmark between CIFAR-10 and ImageNet in difficulty, with equal splits, is useful §1. The paper argues this but does not measure difficulty beyond the error table tab. 1.

Method

CINIC-10 is CIFAR-10 plus ImageNet images selected for the same ten classes. The CIFAR-10 images are kept in full: 60,000 of them, split into three equal sets §2. Each CIFAR-10 class also receives 21,000 ImageNet images, drawn from ImageNet synonym sets (synsets) that were mapped to that class §2. Together that is 270,000 images. The three sets are 9,000 images per class each §2. The paper gives the steps in prose, and the repository holds the synset mapping README.md.

1. Reformat CIFAR-10 as PNG files. A file name records the class, the CIFAR-10 origin set (train or test) and the original index, so CIFAR-10 can be rebuilt from the names §2 README.md.
2. Build the CINIC-10 test set from the full CIFAR-10 test set plus randomly chosen CIFAR-10 train images. Split the remaining CIFAR-10 train images at random into CINIC-10 train and validation. Each set holds 2,000 CIFAR-10 images per class §2.
3. For each CIFAR-10 class, collect the ImageNet synsets that match it, from the Fall 2011 ImageNet release §2. The synset groups are listed in synsets-to-cifar-10-classes.txt and the mapping in imagenet-contributors.csv README.md.
4. Download the synsets with ImageNet Utils. Some archives failed with 0-byte downloads even after retries. The authors judge this not very harmful, since only a subset of the images was used §2 README.md.
5. Decode each JPEG with Pillow and resize it to 32×32 with the Box filter, the same procedure as ImageNet32 §2.
6. For each class, randomly select 21,000 images from its ImageNet pool. The smallest class-relevant pool was 21,939 images, for the truck class §2.
7. Split the 21,000 at random into the three sets, 7,000 per set per class (derived: 21,000 / 3). Store each as set/class/synset_number.png, so the source of every image is recoverable from its name §2 README.md.
8. The final counts are 270,000 images, equal across sets, with 9,000 per class per set §2.

Key equations: none. The paper has no equations.

# Build CINIC-10 from CIFAR-10 and ImageNet  [§2]
# Input:  cifar_train, cifar_test: labelled 32x32 images
#         imagenet_pool: per class, the downloaded ImageNet JPEGs
# Output: train, valid, test: 9000 images per class in each set

def build_cinic10(cifar_train, cifar_test, imagenet_pool):
    # Test set = all CIFAR-10 test images plus random train images [§2]
    cifar_extra = random_sample(cifar_train, 10000)  # derived: 20000 - 10000
    cifar_test_all = cifar_test + cifar_extra
    # Remaining CIFAR-10 train images are split at random [§2]
    cifar_rest = remove(cifar_train, cifar_extra)
    cifar_train_split, cifar_valid_split = random_split(
        cifar_rest, [20000, 20000])
    # Group each CIFAR-10 split by class: 2000 per class per set [§2]
    cifar_train_split = by_class(cifar_train_split)
    cifar_valid_split = by_class(cifar_valid_split)
    cifar_test_all = by_class(cifar_test_all)

    for class_name in CLASSES:
        # Downsample each JPEG to 32x32 with the Box filter [§2]
        small = [box_resize(jpeg, 32, 32)
                 for jpeg in imagenet_pool[class_name]]
        # Draw 21000 at random per class group; no replacement stated [§2]
        sampled = random_sample(small, 21000)
        # Split into three sets, 7000 each (derived) [§2]
        train_imagenet, valid_imagenet, test_imagenet = random_split(
            sampled, [7000, 7000, 7000])
        # Each set: 2000 CIFAR-10 plus 7000 ImageNet per class [§2]
        train[class_name] = cifar_train_split[class_name] + train_imagenet
        valid[class_name] = cifar_valid_split[class_name] + valid_imagenet
        test[class_name] = cifar_test_all[class_name] + test_imagenet

    return train, valid, test

Data loading: the paper's example uses torchvision ImageFolder with per-channel mean and standard deviation, and a batch size of 128 §3. The normalisation constants match the README README.md.

Hyperparameters

The paper states the construction numbers and the normalisation constants. It does not state a training recipe, so training values below come from the repository README and are marked as such.

Results

Author-reported. Setup: CINIC-10; models are CIFAR-10 architectures from kuangliu/pytorch-cifar §5. The paper does not state which training split produced its table tab. 1; the README gives both configurations README.md. Hardware: not stated. Baseline: no other method is compared, only the six architectures against each other §5.

Sources: the paper column is tab. 1. The README columns are README.md and README.md. The README labels both tables "Validation Error", but its text says the test subset is used README.md. See Pitfalls.

Paper vs code

• Paper's table column is "Test Error" tab. 1. The README labels the same numbers "Validation Error" README.md. Its text says the train+validation models are tested on the test subset README.md. Read the numbers as test error, and treat the README label as a labelling error.
• The paper reports only the train+validation numbers. The README adds a train-only table README.md, which is not in the paper.
• The paper's table has no ResNet-18 (preact) row tab. 1. The README includes it at 10.10 ± 0.08 README.md.
• The README's train-only table omits DenseNet-121 README.md.
• Batch size: the paper's loader example uses 128 §3. The README benchmark uses 64 README.md.
• The ± in the paper's table is not defined tab. 1. The README says these are five runs with different seeds README.md.
• The paper does not state a training recipe §5. The README gives one README.md. No training script is in the repository, only notebooks for extraction, enlarging the train set and recovering CIFAR-10 .
• The construction code is not in the repository. The notebooks cover recovering CIFAR-10, extracting ImageNet and CIFAR-10 subsets, and enlarging the train set. A search of those notebooks found no resize, random-sampling or seed calls. So the 21,000-per-class sampling and the Box resize are described but not implemented in code here imagenet-extraction.ipynb.
• No discrepancy was found for the normalisation constants: paper and README agree §3 README.md.

Pitfalls

• ImageNet part is unvetted. The authors say ImageNet constituent samples were not vetted and the set is noisier than CIFAR-10 README.md. The paper notes some images contain elements less class-distinct than CIFAR-10, for instance cows and goats in the deer class §4.2 fig. 6.
• Test set shares images with CIFAR-10 train. CINIC-10's test set includes random CIFAR-10 training images §2. A model trained on CIFAR-10 train and evaluated on CINIC-10 test sees some of its training images. The paper does not discuss this (inference from §2).
• Distribution shift is built in. CIFAR-10 and ImageNet images are not identically distributed by design §1. The authors say colour distributions are very similar but not identical, with no numeric test fig. 1 §4.1.
• Sampling is per class group, not per synset. The paper says 21,000 images are drawn "from each synset-group" §2. The synset groups map to CIFAR-10 classes, so this is 21,000 per class. The 21,939 figure is the smallest class-relevant pool, not the number kept per synset §2.
• Sampling replacement and seed are not stated. Neither the paper nor the code says whether draws are with replacement or what the seed is §2 .
• Some ImageNet downloads failed. Some archives were 0-byte downloads, and the paper does not say which classes lost images §2 README.md. The file names let you recover the source of each kept image README.md.
• Training recipe is not in the paper. The README's recipe (300 epochs, learning rate 0.1, momentum 0.9, weight decay 0.0001, batch 64, cosine to zero) is the only one given README.md. The paper does not say whether its table used it tab. 1.
• Table numbers cannot be matched exactly. Hardware, run time and augmentation are not stated §5, so an exact rerun is not possible from the paper alone.
• Dataset licence is not stated. The paper and repository do not state a licence for the dataset itself. The dataset contains images selected from ImageNet, whose terms are separate. The repository's MIT licence covers the code only LICENSE. The dataset DOI and download location are in the README README.md §3.
• Small model set. Six CIFAR-style models are scored, and no other method is compared tab. 1 §5. Numbers from a different training recipe are not comparable to these.

Lineage

• Builds on: CIFAR-10 and ImageNet, both cited in §1 and §2 and not cards in this wiki. Nothing from their content is used here.
• Same downsampling as: ImageNet32 (arXiv:1707.08819), which the paper cites for its resize procedure §2.
• Compared against: ImageNet-32, CIFAR-10 and CIFAR-100 as benchmarks §1.
• Superseded by: not stated.