--- kind: concept slug: dataset-distillation title: Dataset distillation summary: >- Learns a few synthetic training images, optionally with labels, that stand in for a large dataset. The kernel variant optimises them against kernel ridge regression with the NTK of an infinitely wide ConvNet. Trade-off: the convolutional kernel is costly to compute at scale, and the learned sets lose accuracy when used to train finite networks. tags: [training, data-efficiency] papers: ["2107.13034"] related: [neural-tangent-kernel, neural-network-gaussian-process, meta-learning, dataset-curation, knowledge-distillation] status: draft updated_at: 2026-11-08 --- ## Overview Dataset distillation asks how small a training set can become while still training a good model [arxiv:2107.13034 §1]. Instead of keeping real examples, the method learns a synthetic set. The shared mechanism is gradient descent on the synthetic images, with a predictor's loss measured on real target data [arxiv:2107.13034 §2] [eq. 1]. Coresets and low-dimensional projections keep natural subsets or coarse summaries instead, and trade accuracy for size [arxiv:2107.13034 §1]. This page currently rests on one paper card. It describes one family, kernel regression with an infinite-width kernel, and does not survey the field. ## Variants ### KIP with a convolutional kernel Kernel Inducing Points (KIP) minimises the kernel ridge regression loss over the synthetic images, and optionally their labels, with gradient steps [arxiv:2107.13034 §2] [eq. 1]. Each step uses a fresh batch of real target images [arxiv:2107.13034 §2]. The kernel is the neural tangent kernel of a 4-layer ConvNet with average pooling and no normalisation layers, so that the infinite-width limit exists [arxiv:2107.13034 §2] [app. A]. The paper's convolutional extension of earlier fully connected work is the main contribution [arxiv:2107.13034 §1]. Card: [Dataset Distillation with Infinitely Wide Convolutional Networks](/papers/2107.13034). Author-reported: one image per CIFAR-10 class reaches 64.7 ± 0.2 kernel regression accuracy without augmentation [arxiv:2107.13034 tab. 1]. ### Label Solve Label Solve keeps the synthetic images fixed and solves once for the labels that minimise the same loss over the whole target set, in the least-norm sense [arxiv:2107.13034 §2] [eq. 2]. The support set must be smaller than the target set [arxiv:2107.13034 §2]. Its ConvNet-kernel results are lower than KIP's in the paper's table, for example 26.1% on CIFAR-10 with one image per class [arxiv:2107.13034 tab. 1]. ### Transfer to finite networks The distilled sets can also train ordinary finite networks, here of width 1,024 with standard parameterisation [arxiv:2107.13034 app. A]. This is a use of the variants above, not a separate method. On CIFAR-10 with one image per class, the paper reports a finite-network accuracy of 49.9% for KIP sets, against 28.8% for the better of DC and DSA [arxiv:2107.13034 tab. 2]. Relative to a kernel regression number the paper does not list, the same row shows a drop of 9.2 points [arxiv:2107.13034 tab. 2]. Across the table, transfer generally loses accuracy relative to the kernel setting. ## When to use which | Situation | Use | Why | |---|---|---| | Your predictor is a kernel regressor and you want the highest few-image accuracy | KIP with the ConvNet kernel, label training on | The KIP ConvNet row has the highest CIFAR-10 kernel accuracies at one and ten images per class [arxiv:2107.13034 tab. 1] | | The images are already chosen, and only labels can change | Label Solve | It solves for labels in closed form over the target set, with fixed images [arxiv:2107.13034 §2] [eq. 2] | | The target model is a finite network trained by gradient descent | Measure transfer directly; do not assume the kernel number | Finite-network results trail the kernel results in the paper's table [arxiv:2107.13034 tab. 2] | | Compute is limited to a few devices | Budget carefully, or pick a smaller kernel | Each step is reported at about 460 s on 18 × 18 tiles of a V100, and the paper uses hundreds of GPUs per run [arxiv:2107.13034 app. B] [arxiv:2107.13034 §2.1] | | Comparisons with the DC or DSA baselines | Check the architecture gap first | The paper's ConvNet has four layers and no normalisation, while DC and DSA use three layers with instance normalisation [arxiv:2107.13034 app. C.4] | Ablation results suggest the first-layer difference changes CIFAR-10 kernel results by small amounts [arxiv:2107.13034 app. C.4], so the architecture gap is a caveat rather than the main explanation for differences. ## Implementation sketch Language-agnostic pseudocode of the KIP step. Variants differ in whether the labels are trained, in whether the target batch is augmented, and in whether the kernel is sampled [arxiv:2107.13034 §2] [arxiv:2107.13034 §3.2]. ```pseudocode # One KIP update on the synthetic images [arxiv:2107.13034 §2] [eq. 1] # Input: synthetic images x_s, labels y_s, real dataset, kernel function K # Output: updated x_s (and y_s, if label training is on) def kip_step(x_s, y_s, real_data, K): # Fresh class-balanced target batch from real data [app. A] x_t, y_t = sample_balanced(real_data, 5000) # Kernel blocks between synthetic and target images [§2] k_ss = K(x_s, x_s) # (S, S) k_ts = K(x_t, x_s) # (T, S) # Ridge term scaled by the mean kernel diagonal [app. A] lam = 1e-6 * trace(k_ss) / S # Kernel ridge prediction on the target batch [eq. 1] pred = k_ts @ solve(k_ss + lam * I, y_s) # Squared error over the target batch [eq. 1] loss = 0.5 * sum((y_t - pred) ** 2) # Gradient step on the images; learning rate is set per dataset [app. A] x_s = adam_update(x_s, grad(loss, x_s)) if label_training: y_s = adam_update(y_s, grad(loss, y_s)) return x_s, y_s ``` For large convolutional kernels, the kernel blocks are computed in tiles and summed, and gradients are formed tile by tile [arxiv:2107.13034 §2.1]. The paper does not give code-level detail for this [arxiv:2107.13034 §2.1]. ## Contradictions - **Abstract versus table.** The paper's abstract describes over 65% CIFAR-10 accuracy from 10 images. The highest CIFAR-10 entry with one image per class is 64.7 ± 0.2 [arxiv:2107.13034 tab. 1]. The paper's own tables do not show a 65% entry for that setting. - **Size of the improvement.** The introduction says gains over prior art on CIFAR-10 exceed 25 points [arxiv:2107.13034 §1]. Against KIP-FC the gap is 24.2 points, and against DSA it is 35.9 [arxiv:2107.13034 tab. 1]. - **Paper versus released code.** The released code does not reproduce the distributed trainer. Its Label Solve regulariser also differs from the paper's single λ definition, which the paper applies to both kernel regression and label solving [arxiv:2107.13034 app. A]. Details are on the [card](/papers/2107.13034). No disagreements between papers are known, since one card exists. ## Papers - [Dataset Distillation with Infinitely Wide Convolutional Networks](/papers/2107.13034): extends kernel-based distillation (KIP and Label Solve) to convolutional kernels, with a distributed computation scheme, and tests the learned sets on finite networks.