Dataset distillation

Overview

Dataset distillation asks how small a training set can become while still training a good model 2107.13034 §1. Instead of keeping real examples, the method learns a synthetic set. The shared mechanism is gradient descent on the synthetic images, with a predictor's loss measured on real target data 2107.13034 §2 eq. 1. Coresets and low-dimensional projections keep natural subsets or coarse summaries instead, and trade accuracy for size 2107.13034 §1.

This page currently rests on one paper card. It describes one family, kernel regression with an infinite-width kernel, and does not survey the field.

Variants

KIP with a convolutional kernel

Kernel Inducing Points (KIP) minimises the kernel ridge regression loss over the synthetic images, and optionally their labels, with gradient steps 2107.13034 §2 eq. 1. Each step uses a fresh batch of real target images 2107.13034 §2. The kernel is the neural tangent kernel of a 4-layer ConvNet with average pooling and no normalisation layers, so that the infinite-width limit exists 2107.13034 §2 app. A. The paper's convolutional extension of earlier fully connected work is the main contribution 2107.13034 §1. Card: Dataset Distillation with Infinitely Wide Convolutional Networks. Author-reported: one image per CIFAR-10 class reaches 64.7 ± 0.2 kernel regression accuracy without augmentation 2107.13034 tab. 1.

Label Solve

Label Solve keeps the synthetic images fixed and solves once for the labels that minimise the same loss over the whole target set, in the least-norm sense 2107.13034 §2 eq. 2. The support set must be smaller than the target set 2107.13034 §2. Its ConvNet-kernel results are lower than KIP's in the paper's table, for example 26.1% on CIFAR-10 with one image per class 2107.13034 tab. 1.

Transfer to finite networks

The distilled sets can also train ordinary finite networks, here of width 1,024 with standard parameterisation 2107.13034 app. A. This is a use of the variants above, not a separate method. On CIFAR-10 with one image per class, the paper reports a finite-network accuracy of 49.9% for KIP sets, against 28.8% for the better of DC and DSA 2107.13034 tab. 2. Relative to a kernel regression number the paper does not list, the same row shows a drop of 9.2 points 2107.13034 tab. 2. Across the table, transfer generally loses accuracy relative to the kernel setting.

When to use which

Ablation results suggest the first-layer difference changes CIFAR-10 kernel results by small amounts 2107.13034 app. C.4, so the architecture gap is a caveat rather than the main explanation for differences.

Implementation sketch

Language-agnostic pseudocode of the KIP step. Variants differ in whether the labels are trained, in whether the target batch is augmented, and in whether the kernel is sampled 2107.13034 §2 2107.13034 §3.2.

# One KIP update on the synthetic images  [arxiv:2107.13034 §2] [eq. 1]
# Input:  synthetic images x_s, labels y_s, real dataset, kernel function K
# Output: updated x_s (and y_s, if label training is on)

def kip_step(x_s, y_s, real_data, K):
    # Fresh class-balanced target batch from real data  [app. A]
    x_t, y_t = sample_balanced(real_data, 5000)
    # Kernel blocks between synthetic and target images  [§2]
    k_ss = K(x_s, x_s)                           # (S, S)
    k_ts = K(x_t, x_s)                           # (T, S)
    # Ridge term scaled by the mean kernel diagonal  [app. A]
    lam = 1e-6 * trace(k_ss) / S
    # Kernel ridge prediction on the target batch  [eq. 1]
    pred = k_ts @ solve(k_ss + lam * I, y_s)
    # Squared error over the target batch  [eq. 1]
    loss = 0.5 * sum((y_t - pred) ** 2)
    # Gradient step on the images; learning rate is set per dataset  [app. A]
    x_s = adam_update(x_s, grad(loss, x_s))
    if label_training:
        y_s = adam_update(y_s, grad(loss, y_s))
    return x_s, y_s

For large convolutional kernels, the kernel blocks are computed in tiles and summed, and gradients are formed tile by tile 2107.13034 §2.1. The paper does not give code-level detail for this 2107.13034 §2.1.

Contradictions

• Abstract versus table. The paper's abstract describes over 65% CIFAR-10 accuracy from 10 images. The highest CIFAR-10 entry with one image per class is 64.7 ± 0.2 2107.13034 tab. 1. The paper's own tables do not show a 65% entry for that setting.
• Size of the improvement. The introduction says gains over prior art on CIFAR-10 exceed 25 points 2107.13034 §1. Against KIP-FC the gap is 24.2 points, and against DSA it is 35.9 2107.13034 tab. 1.
• Paper versus released code. The released code does not reproduce the distributed trainer. Its Label Solve regulariser also differs from the paper's single λ definition, which the paper applies to both kernel regression and label solving 2107.13034 app. A. Details are on the card.

No disagreements between papers are known, since one card exists.

Papers

• Dataset Distillation with Infinitely Wide Convolutional Networks: extends kernel-based distillation (KIP and Label Solve) to convolutional kernels, with a distributed computation scheme, and tests the learned sets on finite networks.