---
kind: concept
slug: knowledge-distillation
title: Knowledge distillation
summary: >-
  A student is trained to match a teacher's outputs, not only labels. Targets can be class probabilities, stored top-K probabilities, feature maps, pairwise relations, EMA-teacher patch tokens, density at 3D points, few-step sampler outputs or written rationales. Trade-off: a teacher pass or stored targets, often an extra training run, and gains that depend on teacher and benchmark.
tags: [training, compression, distillation]
papers: ["1908.09791", "2003.11539", "2004.05937", "2004.09813", "2004.13513", "2006.07242", "2011.11390", "2012.07356", "2101.02388", "2103.04257", "2103.13744", "2103.16302", "2104.07713", "2104.10858", "2104.10972", "2104.14294", "2105.10056", "2105.13093", "2106.03097", "2107.07651", "2111.07832", "2111.09679", "2203.08679", "2203.11473", "2205.10536", "2206.08129", "2207.10666", "2212.03533", "2212.08059", "2301.12726", "2305.05003", "2307.09283", "2309.11932", "2311.11883", "2311.17042", "2311.18828", "2312.03203", "2402.03216", "2403.05530", "2403.06977", "2403.07563", "2407.14514", "2408.00118", "2409.06857", "2410.11845", "2412.07772", "2412.09858", "2504.08165", "2504.09685", "2504.17432", "2505.13975", "2506.02153", "2511.23429", "2601.03290", "2604.05081", "2605.07865", "2605.10889"]
related: [on-policy-distillation, monocular-depth-estimation, membership-inference, implicit-bias-of-gradient-descent, policy-gradient, baseline-subtraction, continual-learning, catastrophic-forgetting, few-shot-learning, self-training, anomaly-detection, masked-image-modeling, efficient-vision-backbone, vision-transformer, hierarchical-vision-transformer, convolutional-neural-network, neural-radiance-field, empty-space-skipping, diffusion-models, generative-adversarial-network, chain-of-thought-prompting, instruction-tuning, contrastive-learning, kv-cache-management, large-reasoning-models, federated-learning, federated-aggregation, data-augmentation, in-context-learning, multimodal-language-model, language-model-pretraining, multi-label-classification, self-supervised-learning, vision-language-pretraining, transfer-learning, network-pruning, post-training-quantization, neural-architecture-search, text-embedding, multilingual-evaluation, text-to-image-generation, llm-agent, gaussian-splatting, open-vocabulary-segmentation, promptable-segmentation, state-space-model, imitation-learning, vision-language-action-model, implicit-neural-representation, feed-forward-reconstruction]
status: draft
updated_at: 2026-11-01
---

## Overview

A teacher model supplies targets, and a student is trained to match them. The
choice of target is the main design decision. The variants on this page use a
predictive distribution over classes [arxiv:2003.11539 §3.3], relations
between the entries of those distributions [arxiv:2205.10536 §3.1] [arxiv:2205.10536
§3.2], pooled or raw intermediate feature maps [arxiv:2004.13513 §3.1]
[arxiv:2011.11390 §3.2] [arxiv:2103.04257 §3.2], patch-token distributions
from a momentum teacher [arxiv:2111.07832 §3.1], density and colour at
sampled 3D points [arxiv:2103.13744 §3.3], top-K probabilities computed
once before training [arxiv:2207.10666 §3.1], the output of a deterministic
multi-step sampler [arxiv:2101.02388 §3.1], the top-5 token probabilities of a
large language model's solutions [arxiv:2301.12726 §3], and a score difference
from a bidirectional video teacher [arxiv:2412.07772 §3.2]. Embedding models
also distil soft scores from a cross-encoder [arxiv:2212.03533 §4.2]. A dense
long-context multimodal model is distilled from its larger sibling during
training [arxiv:2403.05530 §3.2]. A multilingual sentence encoder is
trained to reproduce a monolingual teacher's vector for each sentence and for
its translation [arxiv:2004.09813 §2]. A diffusion teacher denoises
re-noised student samples, and the one-to-four-step student matches that
estimate [arxiv:2311.17042 §3.3]. The ImageNet-21K pretraining paper adds a
distillation term per level of its label hierarchy [arxiv:2104.10972 §3.4].
Two variants use targets that are not probability vectors: a teacher's written
explanations, used as text [arxiv:2305.05003 §4.2.2], and a teacher's
similarity structure over a batch [arxiv:2504.17432 §3.2]. In federated
learning, a client matches only the not-true-class part of the frozen global
model's output [arxiv:2106.03097 §3]. The token-labeling paper uses dense
per-patch labels from an offline annotator and contrasts this with
distillation [arxiv:2104.10858 §3.2]. The newer cards add targets of other
kinds. A federated server distils from the averaged logits of client models on
unlabelled batches [arxiv:2006.07242 §3]. In a data-free variant, a generator
trained on the clients' prediction layers supplies the samples instead of
data [arxiv:2105.10056 §3.1]. A retrieval model's teacher is its own combined
score over three heads [arxiv:2402.03216 §3.3]. Three cards add feature-level
targets. A 2D foundation model's per-pixel features are distilled into the
features of 3D Gaussians [arxiv:2312.03203 §3.1] [arxiv:2312.03203 §3.2]. A
generalizable feature field appends a frozen teacher's 2D features to its
latent and is trained to match them on novel views [arxiv:2403.07563
§3.2] [arxiv:2403.07563 eq. 3]. A smaller trained video model supplies the
final features that a larger one aligns to [arxiv:2403.06977 §4.1]. In one
robot-policy card the teacher is a set of reinforcement-learning experts, and
its successful rollouts are stored as the student's training data instead of
being queried during training [arxiv:2412.09858 §3]. Two further cards use distilled models outside compression. A lightweight depth network is trained to match the disparity of a larger one [arxiv:2012.07356 §3.5]. A membership-inference attack trains models on the target's own outputs to estimate a record's loss without that record [arxiv:2111.09679 §4.4].

Most teachers are fixed while the student trains. The old model in incremental
learning is the frozen teacher [arxiv:2004.13513 §3.1], and the anomaly
detector's teacher is never updated [arxiv:2103.04257 §3.2]. The teacher gets
no gradient in any variant here. Two variants differ in how the teacher is
used. iBOT's teacher is an exponential moving average of the student and
changes during training [arxiv:2111.07832 §2.2]. DINO and ALBEF use the same
kind of teacher [arxiv:2104.14294 §3.1] [arxiv:2107.07651 §3.3]. TinyViT goes further and runs
the teacher only before training, so the stored targets are the only teacher
output the student sees [arxiv:2207.10666 §3.1]. In the video student the
teacher stays frozen, but a second network, the fake score, is trained online
on the student's own outputs [arxiv:2412.07772 §3.2].

Where labels exist, the distillation term is usually added to the label loss
rather than replacing it [arxiv:2003.11539 §3.3] [arxiv:2205.10536 §3.2]
[arxiv:2212.03533 §4.2] [arxiv:2106.03097 §3]. The anomaly detector, the NeRF distillation, TinyViT's
pretraining, the diffusion student and the top-5 language-model student use no
labels in the student loss [arxiv:2103.04257 §3.2] [arxiv:2103.13744 §3.3]
[arxiv:2207.10666 §3.1] [arxiv:2101.02388 §3.1] [arxiv:2301.12726 §3]. The
feature term in Feature 3DGS is also label-free, and its colour term is a
photometric loss on the training images [arxiv:2312.03203 §3.2]. The
language-model paper uses the gold answer only to keep correct teacher
solutions [arxiv:2301.12726 §3]. VideoMamba adds its alignment term to the
classification loss [arxiv:2403.06977 §4.1]. Label-free and label-mixing cases
differ.
DINO uses no labels at all [arxiv:2104.14294 §3.1]. The federated server in
FedDF uses labels only for validation and early stopping [arxiv:2006.07242
§4.1]. ALBEF mixes the one-hot label with the teacher's distribution
[arxiv:2107.07651 §3.3].

The variants below cover several uses, not the whole field. Other distillation
methods are not represented on this page yet. A shared small-data pretraining
challenge also lists teacher–student distillation among the approaches with
the strongest gains, without per-approach numbers [arxiv:2504.08165 §7.4].

## Variants

### Soft-label self-distillation

A fresh network of the same architecture is trained on the merged labelled
data with a weighted sum of cross-entropy and KL divergence to the previous
generation's predictions [arxiv:2003.11539 §3.3]. The student is re-initialised,
not warm-started from the teacher, and each generation is a full training run
[arxiv:2003.11539 §4.1]. Card: [Rethinking Few-Shot Image Classification](/papers/2003.11539).

### Relation-based logit distillation from a stronger teacher

DIST uses only the output logits. Instead of matching the exact probabilities,
it matches Pearson correlations in two directions: across the classes of each
instance, and across the instances of each class [arxiv:2205.10536 §3.1]
[arxiv:2205.10536 §3.2]. The authors argue that exact matching is a poor target
when the teacher is much stronger than the student [arxiv:2205.10536 §2.1]. On
ImageNet the total loss weights the relation terms with β=2 and γ=2
[arxiv:2205.10536 §4.1]. Card: [Knowledge Distillation from A Stronger Teacher](/papers/2205.10536).

### Decoupled logit distillation

DKD rewrites the classic KD loss as a target-class term plus a non-target term
multiplied by one minus the teacher's probability for the correct class
[arxiv:2203.08679 §3.1] [arxiv:2203.08679 eq. 6]. It drops that multiplier and
weights the two terms with separate constants, α and β [arxiv:2203.08679 §3.3]
[arxiv:2203.08679 eq. 7]. The authors read the multiplier as suppressing the
non-target term when the teacher is confident [arxiv:2203.08679 §3.2]. The loss
adds no parameters and has about the same cost as KD [arxiv:2203.08679 §4.2].
On CIFAR-100 the paper uses α = 1.0 and T = 4, with β set per teacher and no
general rule [arxiv:2203.08679 app. A.3]. Card: [Decoupled Knowledge Distillation](/papers/2203.08679).

### Pooled-activation distillation for incremental learning

While a class-incremental model learns new classes, its intermediate stage
outputs and final embedding are kept close to those of the frozen previous
model. The loss compares sums of squared activations along width and along
height, which tolerates small local shifts while keeping the overall layout
[arxiv:2004.13513 §3.1]. A second, flat term matches the normalised final
embedding [arxiv:2004.13513 §3.1]. The term is scaled by an adaptive factor
that grows with the number of seen classes [arxiv:2004.13513 §6.2]. Card: [PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning](/papers/2004.13513).

### Multi-scale pooled distillation with pseudo-labels for continual segmentation

PLOP's Local POD applies the same width and height pooling over grid regions
at several scales, so it keeps both large-scale layout and small details
[arxiv:2011.11390 §3.2]. Its pseudo-labels are a separate mechanism: an old-class
guess replaces a "background" label only where the old model is confident,
judged by per-class entropy thresholds [arxiv:2011.11390 §3.3]. The paper's
ablation reports the two effects separately. Card: [PLOP: Learning without Forgetting for Continual Semantic Segmentation](/papers/2011.11390).

### Not-true-class distillation in federated learning

Each client trains on its own labelled data with cross-entropy, plus a term
that matches the frozen global model only on the classes that are not the
true label [arxiv:2106.03097 §3]. Both softmaxes drop the true class, so the
true-class logit gets no gradient from the distillation term [arxiv:2106.03097
§3]. The authors attribute poor accuracy under non-IID splits mainly to
forgetting of classes a client does not hold, and say they treat this as a
diagnostic rather than a proven cause [arxiv:2106.03097 §2]. The reported runs
use β = τ = 1 [arxiv:2106.03097 app. Algorithm Implementation Details], and a
too-small temperature "significantly drops the accuracy" [arxiv:2106.03097 app.
Effect of FedNTD Hyperparameters]. An MSE version of the term gets worse as β
rises and diverges at β = 0.3, so the KL form is the tested one [arxiv:2106.03097
app. MSE Loss for Not-True Distillation]. Card: [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](/papers/2106.03097).

### Class-relation distillation in federated class-incremental learning

Clients receive new classes over time and keep only a small exemplar memory.
GLFC adds a relation-distillation term to each client's loss. The target is the
one-hot label vector with its old-class entries replaced by the sigmoid outputs
of the stored previous model, and the loss is a KL divergence to that target
[arxiv:2203.11473 §4.1 eq. 5]. The term is combined with a gradient-compensated
classification loss, with weights λ1 = 1 and λ2 = 0 on the first task and
λ1 = λ2 = 0.5 afterwards [arxiv:2203.11473 §4.1 eq. 6]. The server
rebuilds prototype samples from shared gradients and keeps the best models of
the last and current task; each client uses either the previous task's model or
the current one as the old model, depending on whether it detected a new task
[arxiv:2203.11473 §4.2] [arxiv:2203.11473 §4.3].
The authors report 66.9 average accuracy on CIFAR-100 with 10 tasks, against
61.5 for the best adapted baseline, and 65.8 when the relation term is removed
[arxiv:2203.11473 tab. 1]. Their numbers are author-reported. Several settings
in the official code differ from the paper, including the distillation loss
and the reconstruction optimiser; see the card. Card: [Federated Class-Incremental Learning](/papers/2203.11473).

### Ensemble distillation for federated model fusion

FedDF replaces the weight-averaging step of FedAvg with distillation. The
server takes a data-size-weighted average of the returned client models as its
starting point, then trains it to match the unweighted mean of all sampled
clients' logits on unlabelled or generated batches, using a KL loss without a
temperature in the equation [arxiv:2006.07242 alg. 1] [arxiv:2006.07242 eq. 1].
Client models may differ in architecture, size or precision, because only
their outputs are combined [arxiv:2006.07242 app. B]. In the CIFAR-10 setting
with 20 clients, reaching 80% top-1 takes 20 ± 4 rounds against 104 ± 52 for
FedAvg [arxiv:2006.07242 tab. 1]. Card: [Ensemble Distillation for Robust Model Fusion in Federated Learning](/papers/2006.07242).

### Data-free generator distillation for federated learning

The server trains a small conditional generator from the clients' prediction
layers, with no data and no proxy set [arxiv:2105.10056 §3.1]. The generator is
sent to each client, which adds a term to its local loss asking that the
generator's samples be classified correctly by its own predictor
[arxiv:2105.10056 eq. 5]. The authors call these samples an inductive bias for
local learning [arxiv:2105.10056 §4.1]. Only the predictor layers are needed
from each client [arxiv:2105.10056 §3.1]. On MNIST with Dirichlet α = 0.05 the
authors report 91.30 against 87.70 for FedAvg [arxiv:2105.10056 tab. 1]. Card:
[Data-Free Knowledge Distillation for Heterogeneous Federated Learning](/papers/2105.10056).

### Feature-pyramid distillation for anomaly detection

STFPM trains a same-architecture student on defect-free images only. It
matches L2-normalised ResNet-18 features at three levels (conv2_x to conv4_x)
with equal weights [arxiv:2103.04257 §3.2]. At test time the per-level
distance maps are upsampled and multiplied, and the image score is the maximum
of the result [arxiv:2103.04257 §3.3]. Card: [Student-Teacher Feature Pyramid Matching for Anomaly Detection](/papers/2103.04257).

### Self-distillation with an EMA teacher for masked image modelling

iBOT's teacher reads unmasked views and is an exponential moving average of
the student [arxiv:2111.07832 §2.2]. Its per-patch output distributions are the
targets at masked positions, and a [CLS] term aligns the summary token across
two views [arxiv:2111.07832 §3.1] [arxiv:2111.07832 eq. 2] [arxiv:2111.07832 eq. 3].
The targets are softmax distributions over K=8192 outputs, not hard labels.
The paper's ablation shows lower linear probing with hard targets [arxiv:2111.07832
tab. 18]. Card: [iBOT: Image BERT Pre-Training with Online Tokenizer](/papers/2111.07832).

### Label-free self-distillation with an EMA teacher

DINO trains a ViT without labels. A student matches the softmax outputs of a
teacher across several crops. The teacher sees only the two global crops and is
an exponential moving average of the student, with momentum following a cosine
schedule from 0.996 to 1 [arxiv:2104.14294 §3.1]. Centring and sharpening the
teacher's outputs prevent collapse, so the method needs no contrastive
negatives and no predictor [arxiv:2104.14294 §3.1]. Using the previous epoch as
the teacher gives 66.6 k-NN top-1 against 72.8 for the momentum teacher
[arxiv:2104.14294 fig. 6]. The paper's teacher-temperature warm-up from 0.04 to
0.07 is not the code default [arxiv:2104.14294 app. D]. Card: [Emerging Properties in Self-Supervised Vision Transformers](/papers/2104.14294).

### Momentum distillation for vision-language pretraining

ALBEF aligns image and text features with a contrastive loss before cross-modal
fusion [arxiv:2107.07651 §3.2]. Its momentum teacher is an exponential moving
average with momentum 0.995, and its similarities give soft targets
[arxiv:2107.07651 §3.3]. For the contrastive loss, the target mixes the
teacher's distribution with the one-hot label at weight α [arxiv:2107.07651
eq. 6]. The masked-language loss takes the same mix over the vocabulary
[arxiv:2107.07651 eq. 7]. α ramps linearly from 0 to 0.4 in the first epoch
[arxiv:2107.07651 §3.5]. At 4M images, adding momentum distillation to both
losses raises the Flickr30K text-retrieval average from 97.01 to 97.47 and VQA
test-dev from 73.81 to 74.42 [arxiv:2107.07651 tab. 1]. Card: [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](/papers/2107.07651).

### Similarity-distribution matching for strong augmentations

Contrastive learning normally pulls each augmented view toward its own
positive key, which can fail when the view is heavily distorted. CLSA keeps the
usual contrastive loss on gentle views and adds a divergence term. The weakly
augmented query gives a softmax distribution over its positive and a queue of
negatives; this distribution is the target, with gradients stopped, and the
strongly augmented query is trained to match it over the same candidates
[arxiv:2104.07713 §3.2] [arxiv:2104.07713 eq. 3] [arxiv:2104.07713 eq. 6]. The
total loss is the contrastive term plus β times the divergence, with β = 1.0
[arxiv:2104.07713 eq. 7]. The teacher here is the weak view itself, so the term
is a self-distillation step inside a contrastive method. The authors report
69.4 ImageNet linear top-1 for single-crop CLSA against 67.5 for MoCo v2 at
200 epochs, and 76.2 for multi-crop CLSA* at 800 epochs against 76.5 for
supervised training [arxiv:2104.07713 tab. 2] [arxiv:2104.07713 tab. 3]. These
numbers are author-reported. The paper's own ablation shows that adding strong views
naively gains little: 67.7 with a strong query and 67.0 with a strong query and
key, against 67.5 for MoCo v2. Replacing the divergence with a contrastive loss
gives 68.0 [arxiv:2104.07713 tab. 5]. Card: [Contrastive Learning with Stronger Augmentations](/papers/2104.07713).

### Teacher-to-cell density distillation

A NeRF teacher is queried at random points inside each cell of a grid, and
each small per-cell student is trained to match the teacher's opacity and
colour there. No training images are used in this stage [arxiv:2103.13744
§3.3]. Fine-tuning on the images follows, because distillation alone leaves
artefacts in free space [arxiv:2103.13744 §3.3] [arxiv:2103.13744 §4.3]. Card: [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](/papers/2103.13744).

### Offline top-K teacher labels for pretraining small transformers

TinyViT runs a large teacher once over each augmented ImageNet-21k view, before
any student is trained. It keeps the K largest teacher probabilities and their
class indices, plus a 4-byte seed that replays the same augmentation later
[arxiv:2207.10666 §3.1]. The student minimises cross-entropy to the stored
distribution, with no ground-truth labels [arxiv:2207.10666 §3.1] [arxiv:2207.10666
eq. 1]. The probability left outside the top K is spread evenly over the other
classes [arxiv:2207.10666 eq. 2]. The paper uses K = 100 for ImageNet-21k and
K = 10 for ImageNet-1k [arxiv:2207.10666 §5.2]. Changing the augmentation
pipeline after labelling breaks the seed replay, so the stored labels fix the
training views. Card: [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](/papers/2207.10666).

### Dense per-patch labels from an offline annotator

Every training image gets one label per patch token, produced once before
training by an ImageNet-only NFNet-F6 classifier (86.3% top-1) [arxiv:2104.10858
§3.2] [arxiv:2104.10858 §4.1]. The objective adds the class loss to a weighted
average of per-token cross-entropies [arxiv:2104.10858 eq. 2] [arxiv:2104.10858
eq. 3]. To save space, only the five largest values per cell are kept, in half
precision, which the authors report as about 10 GB instead of 2 TB [arxiv:2104.10858
§4.1]. MixToken mixes patch embeddings rather than pixels, and the token labels
are mixed with the same mask [arxiv:2104.10858 §3.3] [arxiv:2104.10858 eq. 6].
The paper contrasts this with distillation, which it says needs a teacher
running during training [arxiv:2104.10858 §3.2]. Card: [All Tokens Matter: Token Labeling for Training Better Vision Transformers](/papers/2104.10858).

### Hard distillation as a training ingredient

EfficientFormerV2 names a RegNetY-16GF teacher and hard distillation for
training [arxiv:2212.08059 §5.1]. The paper does not state the distillation
weight or temperature [arxiv:2212.08059 §5.1], so these must be checked
against the released code before reproducing. Distillation is one ingredient
of the recipe, not the main contribution. Card: [Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059).

RepViT uses the same teacher, RegNetY-16GF [arxiv:2307.09283 §4.1], for the
numbers in its Table 1. The paper does not say whether the teacher term is hard
or soft. The released code gives the student a second head for the teacher and
mixes the label loss with the teacher loss at weight 0.5, which the card records
from the code, not the paper. Table 2 reports the same models without
distillation [arxiv:2307.09283 tab. 2]. Card: [RepViT: Revisiting Mobile CNN From ViT Perspective](/papers/2307.09283).

Pooling-based ViT (PiT) also trains a distillation token against a RegNetY-16GF
teacher [arxiv:2103.16302 §4.1]. The paper does not say whether that loss is
soft or hard. Relative to the DeiT recipe without distillation, distillation
adds 1.0 point for PiT-S (80.9 to 81.9) and 1.4 points for ViT-S (79.8 to 81.2)
[arxiv:2103.16302 tab. 2]. Card: [Rethinking Spatial Dimensions of Vision Transformers](/papers/2103.16302).

### Distillation from the largest sub-network in elastic training

Once-for-all trains one super-network whose sub-networks vary in depth, width,
kernel size and input size [arxiv:1908.09791 §3.2]. During progressive
shrinking, each sampled sub-network gets a soft-label term from the largest
network, added to a smoothed hard-label term [arxiv:1908.09791 §3.3]. The paper
does not state the weight of either term [arxiv:1908.09791 §3.3]. Without
fine-tuning, the authors report 76.0% top-1 at 230M MACs, against 75.2% for
MobileNetV3-Large [arxiv:1908.09791 tab. 1]. The 80.0% headline needs 75 epochs
of fine-tuning [arxiv:1908.09791 tab. 1]. Card: [Once-for-All: Train One Network and Specialize it for Efficient Deployment](/papers/1908.09791).

### One-step student from a deterministic diffusion sampler

The teacher is a deterministic DDIM sampler with 100 steps on CIFAR-10 and 50
steps on LSUN [arxiv:2101.02388 §4]. For many initial noise vectors, the
teacher's final image is recorded once, giving about 1.024M pairs per dataset
[arxiv:2101.02388 §3.1]. The student starts from the teacher's network weights
and is trained to produce that final image from the same noise in one network
evaluation. The loss is a squared distance, and the paper reports L1 as better
on CIFAR-10 [arxiv:2101.02388 eq. 11] [arxiv:2101.02388 app. B]. No adversarial
term is used [arxiv:2101.02388 §3.1]. Only deterministic teachers qualify:
score-based and energy-based samplers that use stochastic MCMC cannot serve as
teachers, and a GAN is deterministic but already fast [arxiv:2101.02388 §2.1].
Author-reported CIFAR-10 FID is 9.36 at one step, against 4.16 for the
teacher [arxiv:2101.02388 tab. 1]. Card: [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](/papers/2101.02388).

### Top-5 distribution matching for a small language model

The teacher is a large model used through its outputs: 40 sampled solutions per
GSM8K training question, of which the ones that reach the gold answer are kept
[arxiv:2301.12726 §3] [arxiv:2301.12726 §4.1]. The students are T5 and FlanT5
models from 250M to 11B parameters [arxiv:2301.12726 §4]. The main objective is
a KL divergence between the student's next-token distribution and the teacher's
top-5 probabilities at each step, with the other tokens set to zero. The two
models use different tokenisers, so their token sequences are aligned before
the distributions are compared [arxiv:2301.12726 §3]. Four data formats are
mixed, in-context and zero-shot, each with or without the chain of thought
[arxiv:2301.12726 §3]. Checkpoints are chosen on held-out maths sets, not on
the training data [arxiv:2301.12726 §4.3]. Card: [Specializing Smaller Language Models towards Multi-Step Reasoning](/papers/2301.12726).

### Teacher-pruned student reasoning traces for a reasoning model

The student writes its own long chain of thought first. A teacher then splits
that trace into steps with skill labels, chooses for each step to keep, delete,
or compress it, and rewrites the survivors into one trace and answer
[arxiv:2505.13975 §3.1] [arxiv:2505.13975 §3.2]. The student is fine-tuned on
these pairs with LoRA [arxiv:2505.13975 §3.3] [arxiv:2505.13975 §4.4]. The
target is the student's own style, shortened, not a concise teacher's style.
The paper's ablation finds that direct distillation from GPT-4o's concise
traces falls below the base student in all four 7B settings, while DRP matches or
beats it [arxiv:2505.13975 tab. 4]. Training data is 8,000 maths samples from GSM8K and
PRM12K [arxiv:2505.13975 app. B]. Author-reported, 7B student on GSM8K: 91.7% to 94.1%
accuracy, with 917 to 328 tokens [arxiv:2505.13975 tab. 1]. Card: [DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models](/papers/2505.13975).

### Token-level distillation in recent language and medical models

Gemma 2 trains its 2B and 9B students to match the teacher's full next-token
distribution, with a cross-entropy summed over the whole vocabulary
[arxiv:2408.00118 §3.2]. The 2B ablation uses a 7B teacher [arxiv:2408.00118
tab. 6]. Temperature, top-k truncation and the token budget per step are not
stated [arxiv:2408.00118 §3.2]. MedGemma 1.5 4B also post-trains by distillation.
Each token keeps 256 teacher logits, weighted by teacher probability, and the
student is trained with cross-entropy on them. The teacher is an instruction-tuned
model plus supplementary teachers, whose training the paper does not describe
[arxiv:2604.05081 §2.2]. Cards: [Gemma 2: Improving Open Language Models at a Practical Size](/papers/2408.00118) and [MedGemma 1.5 Technical Report](/papers/2604.05081).

### Online distillation during training of a long-context multimodal model

Gemini 1.5 Flash is a dense Transformer that the report describes as
distilled from Pro while it trains, so it learns from Pro as well as from its
fixed dataset [arxiv:2403.05530 §3.2] [arxiv:2403.05530 app. model card]. The
report gives no loss, schedule or sampling rule for this step, so the
teacher–student mechanism cannot be reproduced from it [arxiv:2403.05530 §3.2].
Flash is also trained with a higher-order preconditioned optimiser, which is not
named [arxiv:2403.05530 §3.2]. Card: [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](/papers/2403.05530).

### Teacher-written explanations as fine-tuning targets for extraction

GPT-3 (175B) writes one explanation per training example, given hand-written
examples and the gold triplets [arxiv:2305.05003 §4.2.2]. Flan-T5 Large (760M)
is then fine-tuned on the gold triplets together with that explanation
[arxiv:2305.05003 §4.2.2] [arxiv:2305.05003 fig. 3]. The targets are text, so
the student needs no access to the teacher's probabilities. For NYT, the
authors generated explanations for 25k of the 56k training examples, because
of cost [arxiv:2305.05003 §4.2.2]. A variant where GPT-3 also writes the
labels is reported only for CoNLL04 [arxiv:2305.05003 tab. 2]. Card: [Revisiting Relation Extraction in the era of Large Language Models](/papers/2305.05003).

### Distribution matching for one-step image generation

DMD trains a one-step generator so that its samples match the teacher's
distribution, rather than reproducing the teacher's noise-to-image mapping
[arxiv:2311.18828 §1]. The generator's update direction is the difference
between the teacher's score and the score of a fake-score network, which is
trained alongside on the generator's own outputs [arxiv:2311.18828 §3.2]
[arxiv:2311.18828 eq. 7]. A regression term, LPIPS between the generator's
output and the teacher's output for paired noise, keeps the modes covered
[arxiv:2311.18828 §3.3] [arxiv:2311.18828 eq. 9]. Its default weight is 0.25
[arxiv:2311.18828 §3.3]. Without it, CIFAR-10 FID rises from 2.66 to 5.58
[arxiv:2311.18828 tab. 2]. Card: [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828).

### Adversarial hinge loss plus score distillation for one to four steps

ADD starts the student from the weights of a pretrained text-to-image diffusion
U-Net, and one student evaluation produces the sample [arxiv:2311.17042 §3.1].
Frozen ViT features feed lightweight heads, which are trained with a hinge loss
to separate real images from student samples. The student is trained to fool
them [arxiv:2311.17042 §3.2] [arxiv:2311.17042 eq. 2] [arxiv:2311.17042 eq. 3].
A frozen diffusion teacher denoises the student's sample after re-noising, and
the student matches that clean estimate with a squared L2 term whose target is
detached [arxiv:2311.17042 §3.3]. The total loss is the adversarial term plus
λ = 2.5 times the distillation term, with no regression term
[arxiv:2311.17042 §4] [arxiv:2311.17042 eq. 1]. In the ablation, the adversarial
term alone gives FID 20.8, the distillation term alone gives 315.6, and random
student initialisation gives 293.6 against 20.6 for the default
[arxiv:2311.17042 tab. 1]. The paper gives no learning rate, batch size or
iteration count for its main models [arxiv:2311.17042 §4]. Among SD1.5-based
few-step methods, the one-step ADD-M reaches COCO FID-5k 19.7 and CLIP 0.326,
against 22.5 for UFOGen at one step [arxiv:2311.17042 tab. 2]. Card:
[Adversarial Diffusion Distillation](/papers/2311.17042).

### Distribution matching from a bidirectional teacher for causal video

A bidirectional video diffusion transformer, which needs about 50 denoising
steps, is the teacher. The student generates frames chunk by chunk, attending
to earlier chunks through a KV cache, and uses four denoising steps
[arxiv:2412.07772 §4.1] [arxiv:2412.07772 §4.4]. Training has two stages. First
the student is regressed from noisy states to the clean endpoints of teacher
ODE trajectories [arxiv:2412.07772 §4.3]. Then asymmetric distribution matching
distillation: the update direction is the difference between a fake score,
trained online on the student's outputs, and the teacher's guided score. The
fake score is updated five times per student update [arxiv:2412.07772 §3.2]
[arxiv:2412.07772 alg. 1] [arxiv:2412.07772 §5]. Author-reported: VBench-Long
total 84.27 and 9.4 FPS streaming on one H100 [arxiv:2412.07772 app. A]
[arxiv:2412.07772 tab. 3]. Card: [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](/papers/2412.07772).

The same recipe appears in a 14B interactive game world model
[arxiv:2511.23429 §4.2.3]. A bidirectional teacher is distilled with
Self-Forcing into a few-step causal generator that uses a sink-token KV cache
and block-sparse attention over earlier blocks. A later stage trains with a
DMD loss on randomised rollouts, with a fake-score model on the student's own
frames [arxiv:2511.23429 §4.2.4] [arxiv:2511.23429 alg. 1]. The authors report
16 FPS, but the hardware behind that number is not stated [arxiv:2511.23429
§4.4] [arxiv:2511.23429 tab. 4]. Card: [Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model](/papers/2511.23429).

### Cross-encoder soft labels in embedding fine-tuning

E5 is first pre-trained without labels on about 270M web and forum text pairs,
with a contrastive loss over in-batch negatives [arxiv:2212.03533 §4.1]
[arxiv:2212.03533 tab. 9]. Distillation appears only in the optional second
stage. There the student's probabilities over a query's candidate passages are
pushed toward those of a cross-encoder teacher with a KL term, plus α = 0.2
times a contrastive loss on the hard labels [arxiv:2212.03533 §4.2]
[arxiv:2212.03533 eq. 3] [arxiv:2212.03533 tab. 11]. The teacher scores are
reused from SimLM, and each example has seven hard negatives [arxiv:2212.03533
§4.2] [arxiv:2212.03533 §5.1]. The paper calls the loss a linear interpolation,
but its equation has no weight on the KL term, so the weighting is unclear
[arxiv:2212.03533 §4.2] [arxiv:2212.03533 eq. 3]. Card: [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533).

### Self-knowledge distillation across retrieval heads

M3-Embedding has one encoder with three retrieval heads: dense, sparse and
multi-vector [arxiv:2402.03216 §3.2]. Each head gets an InfoNCE loss. The
teacher is the integrated score, a weighted sum of the three heads, and each head
is trained with a soft cross-entropy towards it [arxiv:2402.03216 §3.3]
[arxiv:2402.03216 eq. 3] [arxiv:2402.03216 eq. 5]. The paper does not say
whether gradients pass through the teacher [arxiv:2402.03216 §3.3]. Distillation
starts after about 6,000 warm-up steps [arxiv:2402.03216 app. B.1]. On MIRACL
dev, the sparse head reaches 53.9 nDCG@10 with self-distillation and 36.7
without, while the dense head moves from 68.7 to 69.2 [arxiv:2402.03216 tab. 5].
Card: [M3-Embedding](/papers/2402.03216).

### Batch similarity distillation into a multimodal LLM's text pathway

UniME's first stage trains the language model of a multimodal LLM on text
only. The teacher is NV-Embed V2, whose sentence embeddings are computed
offline [arxiv:2504.17432 §3.2]. The loss is a KL divergence between the
student's and the teacher's batch-wise similarity distributions at temperature
τ, summed over the batch [arxiv:2504.17432 §3.2] [arxiv:2504.17432 eq. 2]. Only
the LLM is trained, with LoRA [arxiv:2504.17432 §3.2]. Stage 1 uses about 273K
NLI sentence pairs, batch 768 through gradient accumulation, and two epochs
[arxiv:2504.17432 §4.1] [arxiv:2504.17432 tab. 6]. On Phi-3.5-V, stage 1 alone
gives an MMEB average of 40.3, against 25.3 with neither stage and 64.2 with
both [arxiv:2504.17432 tab. 3]. Card: [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](/papers/2504.17432).

### Per-token diagnostic for on-policy distillation

This is an analysis, not a training recipe. On-policy distillation trains the
student on its own rollouts, with a per-token signal from a teacher. The paper
asks, for each token at each branch point, whether the teacher's gradient points
toward answers that succeed [arxiv:2605.10889 §1]. The student samples many
answers. At each branch point the empirical success rate of each next token
defines an ideal gradient, and the GKD gradient is scored by its cosine
similarity to it, from −1 to +1 [arxiv:2605.10889 §2.2] [arxiv:2605.10889 §2.4]
[arxiv:2605.10889 eq. 2.12]. Targeted rollouts keep this affordable:
about 45K to 200K per question [arxiv:2605.10889 §2.5] [arxiv:2605.10889 §3].
On Qwen3 students the mean alignment is small, about +0.03 on BoolQ
[arxiv:2605.10889 §4]. It is higher on incorrect rollouts than on correct ones
in all four settings, and significant at p ≤ 0.0001 in three of them, with the
1.7B MMLU gap not significant [arxiv:2605.10889 fig. 3] [arxiv:2605.10889 app.
E.6]. Keeping only positively aligned tokens raises the signal, but that
selection uses the true alignment, which is unknown at training time
[arxiv:2605.10889 app. E.3]. The best teacher depends on student size and task
[arxiv:2605.10889 §4.2]. Card: [Unmasking On-Policy Distillation](/papers/2605.10889).

### Baseline-subtracted on-policy distillation (vOPD)

Same setting as the diagnostic above: the student samples its own responses,
and a teacher's per-token log-probabilities give the reward. The paper subtracts
a baseline from that reward, which is the negative reverse KL between student
and teacher at the same context [arxiv:2605.07865 §3.1] [arxiv:2605.07865 eq. 10].
The baseline comes from the forward pass already run, so no critic or extra
rollouts are needed [arxiv:2605.07865 §3.1]. It is detached, so the gradient
stays unbiased [arxiv:2605.07865 §2.2] [arxiv:2605.07865 app. A.1]. A top-k
version computes the KL only over the student's k most likely tokens, which is
cheaper and still unbiased for any k [arxiv:2605.07865 §3.2] [arxiv:2605.07865 eq. 14].
The paper's setting uses k = 20 [arxiv:2605.07865 §4.1]. Author-reported,
Qwen3-1.7B on the math average: 29.8 for vanilla OPD, 33.0 for top-k vOPD and
33.1 for full-vocabulary vOPD [arxiv:2605.07865 tab. 2]. Full-vocabulary OPD
scores 33.0 there, so the top-k baseline matches the costly reference
[arxiv:2605.07865 tab. 2]. The variance argument is a heuristic that the paper
motivates but does not prove [arxiv:2605.07865 app. A.3]. Experiments stop at
7B parameters, and only one size-mismatched teacher pair is tested
[arxiv:2605.07865 §4.2] [arxiv:2605.07865 §6]. Card:
[KL for a KL: On-Policy Distillation with Control Variate Baseline](/papers/2605.07865).

### Expert-rollout data distillation into a generalist robot policy

The teacher is not queried by the student. For each task, a reinforcement-learning
expert is trained on the real robot until it succeeds reliably, then rolled out,
and only its successful episodes are kept [arxiv:2412.09858 §3.1]
[arxiv:2412.09858 app. B]. The generalist, OpenVLA or Octo, is fine-tuned on
these episodes with the negative log-likelihood of the recorded actions
[arxiv:2412.09858 §3.3] [arxiv:2412.09858 eq. 2]. The data is the only channel
from the experts to the generalist [arxiv:2412.09858 §3.2]. For a multi-stage
task, the expert covers only the critical phase, and human demonstrations
supply the rest [arxiv:2412.09858 §3.2]. The expert's reward is a binary
success classifier trained on teleoperated positives and negatives
[arxiv:2412.09858 app. B]. On precise connector insertion, the authors report
gains over fine-tuning on the same number of human-demonstration episodes of
23% for OpenVLA and 37% for Octo. The paper does not say whether these are
absolute or relative gains [arxiv:2412.09858 §4.2] [arxiv:2412.09858 fig. 4].
On four unseen connectors, OpenVLA trained on the RL data succeeds 73 times out
of 80, against 49 for the best single-connector RL policy [arxiv:2412.09858
§4.3]. The trial counts are small, and the released code omits the generalist
stage, so the results cannot be reproduced from the release. Card:
[RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning](/papers/2412.09858).

### Feature-map distillation into a 3D Gaussian field

Feature 3DGS gives each 3D Gaussian an N-channel feature and renders it with
the colour, using the same splatting [arxiv:2312.03203 §3.1]. The student's
rendered feature is compared with the teacher's feature map, from LSeg or SAM,
by an L1 term with weight γ = 1.0 [arxiv:2312.03203 eq. 4] [arxiv:2312.03203
§3.2]. The teacher has 512 channels for LSeg and 256 for SAM, so a speed-up
renders fewer channels and lifts them with a 1x1 convolution [arxiv:2312.03203
§3.2] [arxiv:2312.03203 app. A]. The paper describes this module as not
affecting downstream results, but its sweeps show that the rendered size
matters at the low end [arxiv:2312.03203 tab. A]. A text or point query compares
a feature with each Gaussian's feature by cosine similarity [arxiv:2312.03203
eq. 5]. Card: [Feature 3DGS](/papers/2312.03203).

### Teacher features appended to a generalizable feature field

GeFF is a scene-level feature field that one feed-forward pass builds from
posed RGB-D frames, with no per-scene optimisation [arxiv:2403.07563 §3.2].
The encoder's latent is the encoder output concatenated with the frozen
teacher's 2D features of the same input views [arxiv:2403.07563 eq. 3]. The
semantic decoder is trained with an L2 loss to the teacher's features on
ground-truth novel views, rendered through the same volume rendering as depth
[arxiv:2403.07563 §3.2] [arxiv:2403.07563 eq. 2] [arxiv:2403.07563 eq. 5]. The
paper names MaskCLIP as the teacher in its method section, but its training
details name a ViT-L CLIP model [arxiv:2403.07563 §3.2] [arxiv:2403.07563
§4.1]. Its auxiliary-input ablation on the robot is the only test of the
teacher term here. Removing it lowers the object-level successes in the meeting-room and
kitchen trials from 41 to 25 out of 66 [arxiv:2403.07563 tab. 2]. The paper gives no
distillation-only result, and it reports no confidence intervals. Card:
[Learning Generalizable Feature Fields for Mobile Manipulation](/papers/2403.07563).

### Self-distillation from a smaller trained video model

VideoMamba scales a stack of bidirectional Mamba blocks over spatiotemporal
tokens. For each size, a smaller trained model is the teacher for the larger
one. The M model uses a pretrained S model [arxiv:2403.06977 §4.1]. The
student's final features are aligned to the teacher's, and that term is added
to the classification loss [arxiv:2403.06977 §4.1]. The authors report that
this overcomes the overfitting seen when the pure Mamba model is scaled up
[arxiv:2403.06977 §3.3] [arxiv:2403.06977 fig. 6]. Video pretraining uses a
different target: unmasked tokens are aligned to CLIP ViT-B features, at the
final layer only, with 80% masking [arxiv:2403.06977 §3.4] [arxiv:2403.06977
§4.2]. Card: [VideoMamba](/papers/2403.06977).

### Embedding regression from an English teacher across languages

The student is a multilingual encoder, XLM-R in the paper, trained on parallel
sentence pairs [arxiv:2004.09813 §3]. For each pair, it is trained to reproduce
the teacher's English vector for the source sentence and for its translation,
with a squared-distance loss [arxiv:2004.09813 §2]. The teacher is an English
SBERT model. The best reported student uses a teacher trained on paraphrases
[arxiv:2004.09813 §4.1] [arxiv:2004.09813 tab. 1]. Author-reported, the
student averages 84.6 on STS 2017 monolingual and 83.7 cross-lingual, against
76.4 and 73.5 for LaBSE [arxiv:2004.09813 tab. 1] [arxiv:2004.09813 tab. 2].
On BUCC bitext mining it trails LASER and LaBSE, at 88.6 F1 against 93.0 and
93.5. The paper attributes this to targeting semantic similarity rather than
exact translation [arxiv:2004.09813 tab. 3] [arxiv:2004.09813 §4.2]. Card:
[Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813).

### Per-depth distillation in semantic-softmax pretraining

ImageNet-21K labels sit at different levels of the WordNet hierarchy. The
semantic-softmax student has one softmax per depth, and an image trains every
depth at which its ancestor labels exist [arxiv:2104.10972 §3.3]. The
distillation term compares, depth by depth, the student's and the teacher's
distributions over that depth's classes, with a mean squared error
[arxiv:2104.10972 §3.4] [arxiv:2104.10972 eq. 4]. Each depth is weighted by the
teacher's probability mass on its top 5% classes [arxiv:2104.10972 eq. 5]
[arxiv:2104.10972 app. F]. The paper does not name the teacher, and it does not
give the weight between this term and the classification loss
[arxiv:2104.10972 §3.4]. Author-reported gains over semantic softmax without
distillation: ImageNet-1K top-1 81.4 to 82.2, CIFAR-100 90.4 to 91.7 and
Kinetics-200 83.0 to 84.4 [arxiv:2104.10972 tab. 2]. The distillation code is
not in the released repository, so these numbers cannot be reproduced from it.
Card: [ImageNet-21K Pretraining for the Masses](/papers/2104.10972).

### Logit distillation from a ViT teacher after LLM-guided architecture search

The student is a five-stage CNN that a language model proposes and a feedback
loop selects [arxiv:2504.09685 §3.1] [arxiv:2504.09685 §3.2]. Only the chosen
network is distilled. It is trained for 50 epochs against a ViT-B/16 teacher at
temperature T = 10, with a cross-entropy term and a KL term scaled by T²
[arxiv:2504.09685 §4.1.3] [arxiv:2504.09685 eq. 10] [arxiv:2504.09685 eq. 11].
The paper does not give the optimiser, learning rate or batch size for this
stage [arxiv:2504.09685 §4.1.3]. Its mixing weight α rises from 0.4 to 0.8 over
the run [arxiv:2504.09685 eq. 12]. The prose describes the opposite direction,
with the distillation weight rising [arxiv:2504.09685 §3.3]. See Contradictions.
Author-reported, the same distillation step adds 1.32 points to LMaNet-Core,
1.40 to QwNet-Core and 1.35 to LMaNet-Elite on CIFAR-100
[arxiv:2504.09685 tab. 4]. The paper gives no seed variance for these gains.
The released code ranks candidates on test accuracy, so the headline 74.50%
is not a held-out score; the card's Paper vs code section has the details.
Card: [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685).

### Teacher-compressed search space for battery-free microcontrollers

The paper keeps a neural architecture search (NAS) loop for a microcontroller
that runs on harvested power, and changes its starting point. A large MCUNet
teacher is distilled into a smaller student, which is pruned by more than 80%
of its parameters, weight-shared and then shrunk along kernel, depth and
channel dimensions before the search [arxiv:2407.14514 §4.2]. The teacher is
MCUNetV1-Int2 in float16 [arxiv:2407.14514 §5.3]. The paper does not give the
distillation loss, temperature or training settings [arxiv:2407.14514 §4.2]. The
search itself is the iNAS loop, with an RNN controller and a design explorer
that handles power cycles [arxiv:2407.14514 §4.1] [arxiv:2407.14514 fig. 6].
Author-reported, the combined method beats HW-NAS by +19.1% on CIFAR-10 and
+12.0% on Tiny ImageNet, with 1.33× and 1.27× the peak memory, and is 12% slower
than iNAS on CIFAR-10 [arxiv:2407.14514 §5.4] [arxiv:2407.14514 §5.5]
[arxiv:2407.14514 §5.6]. The paper does not say whether the accuracy gains are
percentage points, and the card records no result for the distillation step
alone. Card: [Accelerate Intermittent Deep Inference](/papers/2407.14514).

### Theory: linear teacher and student trained on soft labels

A linear teacher and a linear student, with the student trained on the
teacher's soft outputs under gradient flow. With at least as many transfer
inputs as dimensions, the student's weight vector equals the teacher's
[arxiv:2105.13093 thm. 1]. With fewer inputs, it converges to the projection
of the teacher onto the span of the inputs [arxiv:2105.13093 thm. 1]. The
expected transfer risk is bounded by a term that depends on the distribution
of angles between the teacher and the inputs. That bound gives exponential
rates for tasks with an angular margin and polynomial rates otherwise
[arxiv:2105.13093 thm. 3] [arxiv:2105.13093 cor. 1] [arxiv:2105.13093 cor. 2].
The proofs cover linear models only, and the nonlinear case is left open
[arxiv:2105.13093 §6]. The angle distribution is unknown in practice, and the
paper gives no way to estimate it from data [arxiv:2105.13093 §4.2]. The
statement and proof of the polynomial-rate corollary differ by a constant, and
the source does not reconcile them [arxiv:2105.13093 cor. 2]. The experiments
are synthetic or MNIST digits 0 and 1, and they do not state the learning rate
or step count [arxiv:2105.13093 §5]. Card: [Towards Understanding Knowledge Distillation](/papers/2105.13093).

### Expert imitation with action and feature targets for driving

A single front-camera driving network learns from a reinforcement-learning
expert that has privileged map and actor information [arxiv:2206.08129 §3.1].
The student is trained on the expert's actions, its latent features and its
value estimate, not on the expert's output alone [arxiv:2206.08129 §3.3]. The
trajectory target is the expert's waypoints [arxiv:2206.08129 §3.1]. The
network has a waypoint branch and a control branch. The control branch attends
over the image grid, guided by the waypoint branch, and the two outputs are
blended at test time [arxiv:2206.08129 §3.2] [arxiv:2206.08129 §3.4]. Author-
reported on the CARLA leaderboard: 69.714 driving score for one model and
75.137 for an ensemble, against 61.846 for LAV, which uses four cameras and a
LiDAR [arxiv:2206.08129 tab. 1]. In the ablation, the control-only model scores
32.45, and fusion gives the largest single step, to 57.01 [arxiv:2206.08129
tab. 3]. The gain from guided attention (42.87 to 46.08) is smaller than the
reported spread [arxiv:2206.08129 tab. 3]. The results come from one simulator
and the paper's own leaderboard entries. The released repository's LICENSE
(Apache-2.0) conflicts with the supplement's CC BY-NC 4.0 statement, and the
repository has no TCP-SB or TCP-Ens model. Card: [Trajectory-guided Control Prediction for End-to-end Autonomous Driving](/papers/2206.08129).

### Teacher disparity for a lightweight depth network

A larger self-supervised depth network is trained first and serves as the
teacher. A MobileNetV3 student is then trained to match the teacher's
disparity map with a norm the paper does not state [arxiv:2012.07356 §3.5]
[arxiv:2012.07356 eq. 10]. The teacher is not named, and the paper does not
say whether the student also gets the self-supervised loss. Card: [HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation](/papers/2012.07356).

### Distilled models as reference data for membership inference

Here distillation is an auditing tool, not a compression step. Attack D
trains several models on random population subsets, labels each subset with
the target model's outputs, and uses their losses on the target record. The
threshold is a smoothed percentile of those losses at a fixed false-positive
rate [arxiv:2111.09679 §4.4] [arxiv:2111.09679 app. B.3]. The aim is to
approximate leave-one-out retraining without the other training records
[arxiv:2111.09679 §4.4] [arxiv:2111.09679 §4.5]. Card: [Enhanced Membership Inference Attacks against Machine Learning Models](/papers/2111.09679).

### Surveys and position papers

A review of knowledge distillation and student-teacher learning for vision
sorts methods by teacher count, knowledge type, data availability, online and
self distillation, and application [arxiv:2004.05937 fig. 2]. It has no
experiments and no numeric results of its own, and it lists open questions
[arxiv:2004.05937 §4.1.1]. Card: [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](/papers/2004.05937).

A 2024 survey of small models lists distillation as one way LLMs help smaller
models. It splits this into full data synthesis, rationale distillation, data
augmentation, and representation distillation, which is logit-based or
feature-based [arxiv:2409.06857 §3.2]. Its one experiment is a figure-only
embedding comparison [arxiv:2409.06857 §4.1]. Card: [Small Models in the LLM Era (survey)](/papers/2409.06857).

A position paper on agents includes distilling from the LLM as an optional step
in its conversion plan, next to LoRA or full fine-tuning [arxiv:2506.02153 §6].
It has no experiments of its own. Its figures are cited from other work or are
author estimates [arxiv:2506.02153 §3.1] [arxiv:2506.02153 app. B]. Card:
[Small Language Models are the Future of Agentic AI](/papers/2506.02153).

A PRISMA review of TinyML from the machine-learning side covers 172 eligible
papers from 2018 to 2023 [arxiv:2309.11932 §3]. It places knowledge
distillation under model optimization, with response-, feature- and
relation-based knowledge and offline, online and self-distillation schemes
[arxiv:2309.11932 §5.1.3]. Its popularity table gives distillation the top
label and 11 cited works in 2020–22 [arxiv:2309.11932 tab. 1]. The paper does
not define how the labels are assigned, so the table shows where the authors
put their attention, not a measured adoption rate. Card:
[A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932).

A review of efficient neural networks for Arm Cortex-M microcontrollers
describes distillation as a student trained on a loss that mixes its own task
loss with a divergence from the teacher's softened outputs
[arxiv:2311.11883 §4.1] [arxiv:2311.11883 eq. 5]. It runs no experiments.
Its α and temperature are not stated, and the direction of the KL term is not
specified [arxiv:2311.11883 §4.1]. It compares published flash size against
accuracy rather than its own distillation runs [arxiv:2311.11883 §6]. Card:
[Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](/papers/2311.11883).

A survey of edge large language models lists distillation as an offline
compression step, with white-box methods that use the teacher's internal
features and logits, and black-box methods that use only teacher outputs,
often as chain-of-thought rationales [arxiv:2410.11845 §3.3]. Its
distillation figures, such as a student of about 0.3B parameters from a teacher
above 100B, are quoted from cited works [arxiv:2410.11845 §3.3.2]. The paper
runs no experiments [arxiv:2410.11845 §1] [arxiv:2410.11845 §7]. Card:
[A Review on Edge Large Language Models: Design, Execution, and Applications](/papers/2410.11845).

A survey of lightweight transformers for edge devices gives a two-stage
distillation recipe, general pre-training then task-specific, with a reported
1.5–2.5% gain over one stage, and a teacher-to-student size ratio of 4–6x
[arxiv:2601.03290 §VIII-G]. The paper describes no experiment behind these
ranges, and it gives no optimiser, learning rate or batch size
[arxiv:2601.03290 §VIII-G]. Card:
[Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Labels available, few-shot with a frozen embedding, want a stronger model without a new architecture | Soft-label self-distillation, one or two generations | On miniImageNet with ResNet-12, the distilled model gains 2.8 points at 1-shot and 2.5 at 5-shot over the simple frozen baseline [arxiv:2003.11539 tab. 1]. Each generation costs a full training run [arxiv:2003.11539 §4.1]. |
| Same few-shot setup, but on Meta-Dataset with a linear probe | Optional; expect little average gain | Mean accuracy rises only from 68.02 to 68.28, and the distilled probe is lower on five of ten domains [arxiv:2003.11539 §5]. |
| Labels available, strong teacher (larger model or stronger training recipe), little extra compute wanted | Relation-based logit distillation (DIST) | ResNet-18 student from a ResNet-34 teacher: 72.07 against 70.66 for KD; from ResNet-152: 72.24 against 71.12 [arxiv:2205.10536 tab. 2] [arxiv:2205.10536 tab. 3]. Training speed is close to KD, 14.19 against 14.28 batches per second in one setup [arxiv:2205.10536 tab. 11]. The gain over KD shrinks to 0.33 for a ResNet-34 student with a ResNet-50 teacher [arxiv:2205.10536 tab. 3]. Its distillation weight totals 4 against KD's 1, and the paper does not separate weight from design [arxiv:2205.10536 §4.1]. The paper says it is less effective on binary classification [arxiv:2205.10536 app. A.6]. |
| Class-incremental learning with a small per-class exemplar memory | Pooled-activation distillation from the previous model | On CIFAR100 with 50 one-class steps and 20 exemplars per class, the authors report 61.40 against 48.57 for UCIR (NME) [arxiv:2004.13513 tab. 1]. The comparison uses the authors' protocol. Each step needs a teacher forward pass [arxiv:2004.13513 §3.1]. |
| Continual segmentation, old training images not stored | Local POD with confident pseudo-labels (PLOP) | Pascal VOC 2012, 15-1 overlapped: 54.64 mIoU against 29.29 for the authors' MiB re-implementation [arxiv:2011.11390 tab. voc_sota_full]. On a 20% validation split, pseudo-labels alone reach 19.74 and adding Local POD reaches 50.41 [arxiv:2011.11390 tab. ablations]. Each step needs a frozen old-model pass and a pass over training pixels to fit thresholds [arxiv:2011.11390 §3.3]. |
| Anomaly detection with only normal images and no defect labels | Feature-pyramid distillation (STFPM) | Image AUC-ROC 0.955 on MVTec AD, against 0.921 for Patch-SVDD, as the authors report [arxiv:2103.04257 tab. 2]. The teacher must be pretrained: ImageNet gives 0.955, CIFAR-10 0.826 and MNIST 0.619 [arxiv:2103.04257 tab. 4]. Matching only two levels (conv2_x and conv3_x) gives 0.849, against 0.955 for all three [arxiv:2103.04257 tab. 3]. |
| Pretraining a ViT without a separate tokenizer, want patch-level and image-level features | EMA self-distillation (iBOT) | ViT-B/16 fine-tuned on ImageNet-1K: 84.0 against 83.4 for BEiT [arxiv:2111.07832 tab. 2]. iBOT's 1600 effective epochs are twice BEiT's 800, so the comparison is not compute-matched [arxiv:2111.07832 tab. 2]. ViT-S for 800 epochs with multi-crop: 193.4 h against 180.0 h for DINO [arxiv:2111.07832 tab. 19]. |
| Pretraining a small transformer on a large labelled dataset with a strong teacher, where disk space can hold the labels and many student runs reuse them | Offline top-K teacher labels with seed replay (TinyViT) | For DeiT-Ti, DeiT-S and Swin-T, ImageNet-21k pretraining without distillation gains 0.6 to 0.8 points over ImageNet-1k training alone, and with distillation 2.1 to 2.2 points [arxiv:2207.10666 tab. 2]. The paper's appendix gives 92.4 GPU days to store Florence labels and 140.0 to train with them, against 330.9 for conventional distillation [arxiv:2207.10666 app. B]. Storage is 481 GB for the ImageNet-21k run [arxiv:2207.10666 §5.2]. |
| Expensive NeRF queried at any point, bounded scene with known bounds, no training images for the first stage | Teacher-to-cell density distillation, then fine-tuning | The authors report 26 to 91 ms rendering against 37 to 183 s for NeRF [arxiv:2103.13744 tab. 1]. The method assumes a bounded scene with a known box [arxiv:2103.13744 §3.2]. |
| Small mobile transformer trained with a strong teacher | Hard distillation as in EfficientFormerV2 | At 300 epochs, the same architectures score 75.7, 79.0 and 81.6 for S0, S1 and S2 with distillation, against 73.7, 77.9 and 80.4 without it [arxiv:2212.08059 tab. 2] [arxiv:2212.08059 app.]. |
| Small mobile CNN, latency measured on the target phone, and a strong teacher can be run during training | Distillation from a RegNetY teacher, as in RepViT | At 300 epochs, distilled against not distilled: 78.7 against 77.4 at 0.9 ms, 80.0 against 78.6 at 1.0 ms, 83.3 against 82.5 at 2.3 ms [arxiv:2307.09283 tab. 1] [arxiv:2307.09283 tab. 2]. The gap across the five models is 0.8 to 1.4 points. Latency is for Core ML on an iPhone 12 and does not transfer to other devices [arxiv:2307.09283 §3.1]. |
| Diffusion sampler too slow to serve, a deterministic teacher exists, and about a million teacher samples can be generated and stored | One-step regression from a deterministic teacher | CIFAR-10 FID 9.36 in one step against 4.16 for the 100-step teacher [arxiv:2101.02388 tab. 1]. Time to generate 50k images on one V100: 51.5 s against 4.96 × 10³ s for the teacher [arxiv:2101.02388 tab. 2]. Each training pair costs a full teacher chain of 50 to 100 network evaluations [arxiv:2101.02388 §3.1]. The authors report blurry textures on 256 × 256 LSUN [arxiv:2101.02388 §4.1]. |
| Small model must do multi-step maths, general reasoning can be lost, and teacher solutions or top-5 probabilities can be stored | Top-5 distribution matching (Specializing FlanT5) | FlanT5-XL (3B) on GSM8K: 13.5 becomes 22.4 after specialisation; FlanT5-Base (250M): 3.0 becomes 13.4 [arxiv:2301.12726 tab. 1]. At 3B, starting from FlanT5 beats starting from raw T5: 23.8 against 20.6 on GSM8K validation [arxiv:2301.12726 tab. 2]. BBH chain-of-thought accuracy falls to between 0.0 and 0.3 at every size [arxiv:2301.12726 tab. 1]. Distribution matching needs about |V| times more stored data than sample matching [arxiv:2301.12726 §3]. |
| Streaming or interactive video from a bidirectional video diffusion model, with a few-step budget | Causal student with distribution matching from a bidirectional teacher | 1.3 s latency and 9.4 FPS, against 219.2 s for the bidirectional teacher on a 10 s, 120-frame clip [arxiv:2412.07772 tab. 3]. Training used about 400K internal videos and about 2 days on 64 H100 GPUs [arxiv:2412.07772 §5]. The paper says the objective trades diversity for quality [arxiv:2412.07772 §6], and quality degrades over very long videos [arxiv:2412.07772 §6]. |
| Retrieval embeddings with no labels, or labels plus a cross-encoder reranker | Contrastive pre-training first, then optional distillation fine-tuning (E5) | Unsupervised E5-PT-base averages 42.9 nDCG@10 on 15 BEIR datasets, against 41.7 for BM25 [arxiv:2212.03533 tab. 1]. Fine-tuned E5-base reaches 60.4 on 56 MTEB English datasets [arxiv:2212.03533 tab. 3]. Pre-training ran one to two days on 16 to 64 V100 GPUs [arxiv:2212.03533 tab. 11]. The distillation term is not isolated in any table [arxiv:2212.03533 tab. 6]. |
| Federated clients with strongly non-IID label splits, one extra forward pass per batch acceptable, no extra communication wanted | Not-true-class distillation from the frozen global model (FedNTD) | Sharded CIFAR-10 with two shards per client: 52.61 against 40.14 for FedAvg [arxiv:2106.03097 tab. 1]. SCAFFOLD is higher on the five-shard CIFAR-10 split, 65.74 against 64.93 [arxiv:2106.03097 tab. 1]. Each client batch needs a forward pass of the global model, and the paper does not report the compute cost [arxiv:2106.03097 §4.2]. |
| Pretraining a ViT with a strong annotator, where the dense label maps can be computed once and about 10 GB of stored top-5 values is acceptable | Dense per-patch labels from an offline annotator (token labeling) | LV-ViT-S at 224 px: 83.3 top-1 with 26M parameters [arxiv:2104.10858 tab. 1]. With the same MixToken setup, token labels give 83.3 against 82.5 for hard labels [arxiv:2104.10858 tab. 2]. Using only 20% of tokens for the token loss costs 0.5 points on LV-ViT-S [arxiv:2104.10858 §4.2]. |
| Information extraction with a small seq2seq student, a large LLM can explain each training example, and text targets are acceptable | Teacher-written explanations as fine-tuning targets (Flan-T5) | CoNLL04 micro-F1: 80.76 against 75.28 for Flan-T5 Large fine-tuned on gold labels alone; ADE 92.17 against 83.15; NYT 95.23 against 91.03 [arxiv:2305.05003 tab. 2]. The explanations are the main cost: generated for only 25k of 56k NYT examples [arxiv:2305.05003 §4.2.2]. |
| Multimodal embedder, text-only data available, and a strong text embedder can produce targets offline | Text-only batch similarity distillation as stage 1 (UniME) | MMEB average 40.3 after stage 1 alone, against 25.3 with neither stage and 64.2 with both [arxiv:2504.17432 tab. 3]. Stage 1 uses about 273K NLI pairs and about 1 h on 8×A100 for Phi-3.5-V [arxiv:2504.17432 §4.1] [arxiv:2504.17432 app. A.1]. Batch sampling details beyond the table values are not stated [arxiv:2504.17432 §4.1]. |
| Before an expensive on-policy run, want to know whether a teacher's per-token guidance helps | Per-token alignment diagnostic (an analysis, not a training method) | The full suite takes about 72 A100-days [arxiv:2605.10889 §3]. The authors present the method as an offline evaluation tool, not a training signal [arxiv:2605.10889 app. C]. Mean alignment is near +0.03 on BoolQ [arxiv:2605.10889 §4]. |
| On-policy distillation of a student that already reasons well, a teacher's log-probabilities are accessible, and the full-vocabulary KL is too costly | Baseline-subtracted reverse-KL advantage with top-k baseline (vOPD) | Qwen3-1.7B on the math average: 29.8 for vanilla OPD against 33.0 for top-k vOPD and 33.0 for full-vocabulary OPD [arxiv:2605.07865 tab. 2]. Wall-clock time is reduced by up to 57.7% against full-vocabulary OPD [arxiv:2605.07865 §1]. Its variance gain is argued, not measured as a bound [arxiv:2605.07865 app. A.3]. The paper gives no seed variance, so differences of about one point are not a ranking [arxiv:2605.07865 tab. 2]. |
| Long chain-of-thought student that wastes tokens on repeated steps, a teacher can label and rewrite its traces offline, and a LoRA run on pruned traces is acceptable | Teacher-pruned student traces as the SFT target (DRP) | 7B student on GSM8K: 91.7% at 917 tokens against 94.1% at 328 tokens; AIME24 15/30 at 8674 tokens against 15/30 at 4966 [arxiv:2505.13975 tab. 1]. Direct distillation from concise GPT-4o traces lowers accuracy in all four 7B settings [arxiv:2505.13975 tab. 4]. With GPT-4o as teacher, the GSM8K result is the best of the four teachers tested; the others give 91.2% to 93.2% [arxiv:2505.13975 tab. 3]. |
| Many deployment targets from one trained network, each with its own latency or MAC budget, and no training per target | Elastic super-network with progressive shrinking; soft labels from the largest sub-network (Once-for-All) | Without fine-tuning, 76.0% top-1 at 230M MACs against 75.2% for MobileNetV3-Large [arxiv:1908.09791 tab. 1]. Progressive shrinking lifts one sub-network from 71.5% to 74.8% [arxiv:1908.09791 §4.1]. The 80.0% headline uses 75 epochs of fine-tuning [arxiv:1908.09791 tab. 1]. |
| Label-free ViT pretraining, want frozen features for nearest-neighbour lookup or retrieval | Self-distillation with an EMA teacher, centring and sharpening (DINO) | k-NN top-1 74.5 for ViT-S/16 and 78.3 for ViT-S/8; linear 80.1 for ViT-B/8 [arxiv:2104.14294 tab. 2]. ViT-S/8 runs at 180 images per second against 1007 for ViT-S/16 [arxiv:2104.14294 tab. 2]. |
| Vision-language pretraining on noisy web pairs, no object detector wanted | Align before fusion, with momentum distillation on the contrastive and masked-language losses (ALBEF) | At 4M images, momentum distillation on both losses raises the Flickr30K text-retrieval average from 97.01 to 97.47 and VQA test-dev from 73.81 to 74.42 [arxiv:2107.07651 tab. 1]. The 14M-image model reaches VQA test-std 76.04 [arxiv:2107.07651 tab. 4]. |
| Self-supervised ImageNet pretraining in a MoCo-style setup, where strong augmentations are wanted and a longer run per epoch is acceptable | Similarity-distribution matching on strong views (CLSA) | Single-crop CLSA at 200 epochs: 69.4 against 67.5 for MoCo v2 [arxiv:2104.07713 tab. 2]. Wall-clock time is 70 h against 53 h for MoCo v2 [arxiv:2104.07713 tab. 6], and multi-crop takes about 166 h [arxiv:2104.07713 §4.2]. Adding a strong query alone gives 67.7 and strong query plus strong key gives 67.0, so the divergence term is what carries the gain [arxiv:2104.07713 tab. 5]. Single-crop CLSA sits below InfoMin Aug (70.1) in the same table [arxiv:2104.07713 tab. 2]. The code's loss form differs from the paper's equations, so check the card before reimplementing [card](/papers/2104.07713). |
| Federated clients receiving new classes over time, each keeping a small exemplar memory, and a server that can rebuild samples from shared gradients | Relation distillation from the stored old model, plus gradient-compensated loss and a best-model selection on the server (GLFC) | CIFAR-100, 10 tasks: 66.9 against 61.5 for GeoDL + iCaRL + FL, and 65.8 without the relation term [arxiv:2203.11473 tab. 1]. ImageNet-Subset, 10 tasks: 57.0 against 52.6 for PODNet + FL [arxiv:2203.11473 tab. 2]. Server reconstruction runs 200 L-BFGS iterations per received gradient [arxiv:2203.11473 §4.2] [arxiv:2203.11473 app. A.2]. Every client runs a forward pass and exemplar update each round [arxiv:2203.11473 §4.3] [arxiv:2203.11473 app. B]. |
| Federated clients with different architectures or precisions, and unlabelled or generated data on the server | Ensemble distillation of averaged client logits into the fused model (FedDF) | CIFAR-10, ResNet-8, 20 clients, 40% sampled, 40 local epochs: 20 ± 4 rounds to 80% against 104 ± 52 for FedAvg [arxiv:2006.07242 tab. 1]. Each distillation step runs every sampled client model on the batch, so server cost grows with participants [arxiv:2006.07242 eq. 1]. |
| Federated clients with skewed labels and no shared public data | Data-free generator trained on client prediction layers, samples as a local regulariser (FedGen) | MNIST, Dirichlet α = 0.05: 91.30 against 87.70 for FedAvg [arxiv:2105.10056 tab. 1]. At EMNIST α = 0.05 the proxy-data FedDFusion scores 70.40 against 68.53 for FedGen [arxiv:2105.10056 tab. 1]. Local training takes 57 to 60 ms against 48 ms for FedAvg [arxiv:2105.10056 tab. 3]. |
| Logit-only distillation at KD's cost, teacher logits available, no extra feature modules | Decoupled KD with separate weights α and β (DKD) | CIFAR-100, ResNet32x4 to ResNet8x4: 76.32 against 73.33 for KD [arxiv:2203.08679 tab. 6]. ImageNet, ResNet-34 to ResNet-18: 71.70 against 70.66 for the KD row from other work, and 71.03 for the authors' own KD run [arxiv:2203.08679 tab. 8]. On COCO, DKD alone trails ReviewKD for ResNet-101 to ResNet-18 (35.05 against 36.75 AP) [arxiv:2203.08679 tab. 10]. β is per teacher, from 2.0 to 8.0 on CIFAR-100 [arxiv:2203.08679 app. A.3]. |
| One encoder must serve dense, sparse and multi-vector retrieval over many languages and long inputs | Self-knowledge distillation from the weighted sum of the three heads (M3-Embedding) | On MIRACL dev, sparse nDCG@10 is 53.9 with self-distillation and 36.7 without; dense is 69.2 against 68.7 [arxiv:2402.03216 tab. 5]. Distillation starts after about 6,000 warm-up steps [arxiv:2402.03216 app. B.1]. |
| Diffusion teacher too slow to serve, one network evaluation required, and the regression pairs from the teacher are affordable | Distribution matching with a fake-score network and LPIPS regression (DMD) | ImageNet 64×64: FID 2.62 in one forward pass, against 2.32 for the EDM teacher with 512 [arxiv:2311.18828 tab. 1]. Zero-shot COCO-30k: FID 11.49 at 0.09 s against 8.78 at 2.59 s for SD v1.5 [arxiv:2311.18828 tab. 3]. Dropping the regression term raises CIFAR-10 FID from 2.66 to 5.58 [arxiv:2311.18828 tab. 2]. |
| Pretraining a small LLM with a large teacher and a large token budget | Full-vocabulary token distillation from a larger model (Gemma 2 2B and 9B) | A 2B student distilled from a 7B teacher averages 67.7 on three benchmarks, against 60.3 from scratch, both on 500B tokens [arxiv:2408.00118 tab. 6]. Validation perplexity is lower at 200M, 400M and 1B parameters [arxiv:2408.00118 tab. 7]. Temperature and token budget per step are not stated [arxiv:2408.00118 §3.2]. |
| An agent makes many narrow, repetitive LLM calls and a cheaper specialist is wanted | Per-task specialist model, optionally distilled from the LLM (the position paper's conversion plan) | The paper has no experiments. Its replaceable-call shares of 40% to 70% are the authors' estimates for three agents, with no stated method [arxiv:2506.02153 app. B]. It suggests 10k to 100k examples per task as a rule of thumb [arxiv:2506.02153 §6]. |
| Queryable 3D scene from 2D foundation-model features, novel views needed, and a teacher can run on each training view | Feature-map distillation into 3D Gaussians (Feature 3DGS) | On Replica, segmentation mIoU 0.782 at 14.55 FPS, against 0.636 at 5.38 FPS for NeRF-DFF [arxiv:2312.03203 tab. 2]. The comparison is with NeRF-DFF, not with a run without distillation. Rendering 8 channels drops LSeg mIoU to 0.354, and 128 channels train in 19:55 against 1:29:42 at 512 [arxiv:2312.03203 tab. A]. The official code is research-only, as the card records. |
| Video model that overfits when scaled up, a smaller trained model of the same family exists, and an extra training run is acceptable | Self-distillation from the smaller model (VideoMamba) | The distillation ablation is plotted only, with no numbers in the text [arxiv:2403.06977 fig. 6]. The M model reaches 84.0% ImageNet-1K top-1 at 576 px, against 83.8 for ConvNeXt-B [arxiv:2403.06977 tab. 2]. That row does not isolate the distillation step. The authors report that early stopping did not help [arxiv:2403.06977 §4.1]. |
| Sentence embeddings for many languages, parallel sentence pairs available, and a strong English encoder to copy from | Embedding regression from the English teacher (multilingual student) | Cross-lingual STS 2017 average 83.7, against 73.5 for LaBSE [arxiv:2004.09813 tab. 2]. On BUCC mining the method trails LaBSE and LASER, 88.6 against 93.5 and 93.0 F1 [arxiv:2004.09813 tab. 3]. The paper's reading is that the target is semantic similarity, so exact-translation mining is a weaker fit [arxiv:2004.09813 §4.2]. |
| Pretraining on ImageNet-21K or another label hierarchy, a clean fixed split is wanted, and the KD term is optional | Semantic softmax on the cleaned set, with or without per-depth distillation | All 30 model-dataset cells favour the cleaned ImageNet-21K pretraining over ImageNet-1K pretraining, for example MS-COCO mAP rising from 79.5 to 82.2 for TResNet-M [arxiv:2104.10972 tab. 4]. Adding the distillation term raises ImageNet-1K top-1 from 81.4 to 82.2 over plain semantic softmax [arxiv:2104.10972 tab. 2]. The distillation term has no released training code, so the gain is not reproducible from the release [Card](/papers/2104.10972). |
| Few-step text-to-image generation from a large diffusion teacher, one-step or four-step budget, and a discriminator and GPU run affordable | Adversarial hinge loss plus score distillation (ADD) | ADD-M at one step reaches FID-5k 19.7 against 22.5 for UFOGen at one step [arxiv:2311.17042 tab. 2]. One-step ADD-XL beats four-step LCM-XL in human preference, and four-step ADD-XL beats SDXL at 50 steps in most comparisons [arxiv:2311.17042 §4.2]. Released SDXL-Turbo weights carry a non-commercial licence, as the [card](/papers/2311.17042) records. |
| Microcontroller on harvested power with about 2 KB of runtime SRAM, and a large compact teacher is available to shrink | Teacher-compressed search space before intermittent NAS (accelerated iNAS) | Author-reported on CIFAR-10: +19.1% over HW-NAS, with 1.33× the peak memory [arxiv:2407.14514 §5.4] [arxiv:2407.14514 §5.5]. The unit of the accuracy gain is not defined, and the card records no distillation-only result [arxiv:2407.14514 fig. 11]. The distillation loss and training settings are not stated [arxiv:2407.14514 §4.2]. |
| Microcontroller classifier under a 320 KB SRAM budget, LLM-proposed candidates, and a ViT teacher that can be run for 50 epochs | Logit distillation after the search, with α rising from 0.4 to 0.8 (LMaNet) | Author-reported on CIFAR-100: +1.32 points for LMaNet-Core, against 74.20 with distillation [arxiv:2504.09685 tab. 4] [arxiv:2504.09685 tab. 2]. The released code ranks candidates on test accuracy, so the headline figures are not held-out scores [card](/papers/2504.09685). The paper's text and equation disagree on the α direction [arxiv:2504.09685 §3.3] [arxiv:2504.09685 eq. 11]. |
| Scene queries for a mobile robot in a room-scale space, one forward pass needed instead of per-scene fitting, and a frozen 2D teacher can run on the input views | Teacher features appended to a generalizable feature field (GeFF) | Author-reported overall score 62.4% against 43.2% for LERF and 41.9% for ConceptGraphs (offline) [arxiv:2403.07563 tab. 1]. GeFF runs in 0.39 s on the robot, against about 2 h for LERF offline [arxiv:2403.07563 tab. 1]. The results come from 17 objects and three part categories, with small trial counts, and no official code was found [arxiv:2403.07563 §4.1] [GeFF](/papers/2403.07563). |
| Precise contact task on a robot, a generalist policy already exists, a per-task RL expert can reach full success in the real world, and human demonstrations are noisy or costly | Successful RL-expert rollouts as fine-tuning data (RLDG) | Connector insertion: 23% higher success than human-demo fine-tuning for OpenVLA and 37% for Octo, as stated [arxiv:2412.09858 §4.2] [arxiv:2412.09858 fig. 4]. Unseen connectors, OpenVLA: 73 of 80 against 49 of 80 for the best RL policy [arxiv:2412.09858 §4.3]. The RL experts need 1 to 3 hours of real-world training each, which the comparison does not count [arxiv:2412.09858 app. B]. |
| Explaining or budgeting distillation data for a linear or near-linear model, before running experiments | Theory of linear teacher and student (Phuong and Lampert) | With at least as many transfer inputs as dimensions, the student recovers the teacher exactly; with fewer, the transfer-risk bound depends on the angle distribution [arxiv:2105.13093 thm. 1] [arxiv:2105.13093 thm. 3]. The results are proofs for linear models, with no benchmark numbers [arxiv:2105.13093 §6]. |
| End-to-end driving from one camera, a privileged RL expert exists in simulation, and the policy must be trained with action and feature targets | Expert imitation with action, latent-feature and value targets, plus trajectory-guided attention (TCP) | Author-reported: 69.714 driving score for one model and 75.137 for the ensemble, against 61.846 for LAV, a four-camera LiDAR entry [arxiv:2206.08129 tab. 1]. The control-only ablation baseline scores 32.45 [arxiv:2206.08129 tab. 3]. Results are from one simulator, and the repository does not contain the leaderboard models [card](/papers/2206.08129). |
| Lightweight monocular depth model of about 3M parameters that must hold up at high input resolution, and a larger self-supervised depth model can be trained first | Disparity distillation into a MobileNetV3 student (Lite-HR-Depth) | At 1024x320, Abs rel 0.105 with the teacher against 0.111 without, and 0.115 for Monodepth2 with 14.84M parameters [arxiv:2012.07356 tab. 3]. At 640x192 without the teacher, the student is slightly worse than Monodepth2 (0.116 against 0.115) [arxiv:2012.07356 tab. 3]. The teacher is not identified [arxiv:2012.07356 §3.5]. |
| Black-box membership audit of a trained classifier, where shadow-model attacks miss atypical records and about 1,000 extra models per target are affordable | Distilled reference models trained on the target's outputs (Attack D) | Purchase100 setup II AUC: 0.831 for D, 0.808 for reference-model attack R and 0.754 for shadow-model attack S, author-reported [arxiv:2111.09679 app. C]. At low false-positive rates, R and D reach more than ten times the true-positive rate of S and P on Purchase100 [arxiv:2111.09679 §5.2]. Purchase100 setups I–III use 999 distilled models per target [arxiv:2111.09679 app. B.2]. Whether distilled models are shared across records is not stated [arxiv:2111.09679 §4.4]. |

## Implementation sketch

Common core: the student's loss is a supervised term, when labels exist, plus
a distance to the teacher's output. The variants differ in the teacher output
and the distance. The score-difference update for distribution matching is
the exception, shown last.

```pseudocode
# Shared training step: fixed or EMA teacher, trainable student
# [arxiv:2003.11539 §3.3] [arxiv:2004.13513 §3.1] [arxiv:2111.07832 §2.2]
def train_step(student, teacher, x, y, alpha, beta):
    # Teacher outputs carry no gradient
    with no_grad():
        t_out = teacher(x)
    s_out = student(x)
    # Supervised term, omitted when there are no labels
    # Weights alpha and beta are given in [arxiv:2003.11539 §4.1]
    loss = alpha * cross_entropy(s_out.logits, y)
    loss = loss + beta * distance(s_out, t_out)
    return loss

# Soft-label variant: KL to the teacher's predictive distribution
# [arxiv:2003.11539 §3.3]
def distance_soft(s_out, t_out):
    return kl_divergence(softmax(s_out.logits), softmax(t_out.logits))

# Relation variant: correlate the rows of the probability matrices
# [arxiv:2205.10536 §3.1] [arxiv:2205.10536 §3.2]
def distance_relation(s_logits, t_logits):
    s_p = softmax(s_logits, dim=1)   # (B, C)
    t_p = softmax(t_logits, dim=1)
    # Across classes, one correlation per instance  [arxiv:2205.10536 eq. 8]
    inter = 1 - mean(pearson(s_p, t_p))
    # Across instances, one correlation per class  [arxiv:2205.10536 eq. 9]
    intra = 1 - mean(pearson(s_p.T, t_p.T))
    # Weighted in the total loss  [arxiv:2205.10536 eq. 10]
    return inter, intra

# Pooled-activation variant: width and height sums, normalised
# [arxiv:2004.13513 §3.1] [arxiv:2004.13513 §6.2]
def distance_pooled(s_map, t_map):
    # Square before pooling, then sum along each axis separately
    s_sq, t_sq = s_map ** 2, t_map ** 2
    s_vec = concat(sum(s_sq, axis=W), sum(s_sq, axis=H))
    t_vec = concat(sum(t_sq, axis=W), sum(t_sq, axis=H))
    # Normalise the pooled vectors before comparing
    return squared_norm(l2_normalize(s_vec) - l2_normalize(t_vec))

# Momentum teacher: exponential moving average of the student
# [arxiv:2111.07832 §2.2]
def update_teacher(teacher, student, momentum):
    for t, s in zip(teacher.params, student.params):
        # The momentum schedule is not fixed in this sketch
        t.data = momentum * t.data + (1 - momentum) * s.data

# Density variant: match opacity and colour at random points in one cell
# [arxiv:2103.13744 §3.3]
def distill_cell(student, teacher, cell, iters):
    for step in range(iters):
        # Random 3D point inside the cell, random view direction
        x = sample_in_cell(cell)
        d = sample_unit_sphere()
        # Density to opacity needs a spacing delta, not given in the paper
        a_t = opacity(teacher.sigma(x), delta)
        a_s = opacity(student.sigma(x), delta)
        rgb_t, rgb_s = teacher.rgb(x, d), student.rgb(x, d)
        loss = mse(a_s, a_t) + mse(rgb_s, rgb_t)
        update(student, loss)

# Offline variant: sparse teacher targets stored before training
# [arxiv:2207.10666 §3.1] [arxiv:2207.10666 eq. 2]
def distance_offline_topk(s_logits, values, indices, C):
    # Stored probabilities: K values and K class indices per view
    k = indices.shape[-1]
    # Spread the mass outside the top K evenly over the other classes
    # (the top-5 language-model variant sets it to zero instead)
    rest = (1 - sum(values, axis=-1, keepdims=True)) / (C - k)
    t = scatter(broadcast(rest, C), indices, values)
    # Soft cross-entropy against the stored target; teacher not run
    return -sum(t * log_softmax(s_logits), axis=-1)

# Distribution-matching step for a one-pass generator  [§3.2] [arxiv:2412.07772 alg. 1]
def dmd_step(student, teacher_score, fake_score, x_clean, schedule):
    # Student produces a clean estimate from a noised input
    x_hat = student(add_noise(x_clean, sample_step(schedule)))
    # Re-noise the estimate at a random timestep
    t = sample_timestep()
    x_t = add_noise(x_hat, t)
    # Update direction: fake score minus the teacher's guided score
    grad = fake_score(x_t, t) - teacher_score(x_t, t)
    # Only the student moves along this direction; the teacher is frozen
    update(student, mse(x_hat, (x_hat - grad).detach()))
    # The fake score learns the denoising loss on the student's own outputs
    fresh = add_noise(x_hat.detach(), sample_timestep())
    update(fake_score, denoising_loss(fake_score, fresh))
```

## Contradictions

Several results point in different directions, and each is stated with its
source. No direct conflict on the same claim across the cards, except where
noted; the last item is within one card.

- **Teacher temperature.** DIST's default KD temperature is given as τ=4, but
  the ImageNet runs use τ=1, and the detection table labels KD as τ=1
  [arxiv:2205.10536 §4.1] [arxiv:2205.10536 tab. 6]. Compare KD baselines only
  at the same temperature.
- **Mixed gains.** On Meta-Dataset the distilled linear probe scores lower
  than the plain one on five of ten domains, for example Quick Draw at 60.83
  against 62.48 [arxiv:2003.11539 §5]. In PODNet's pooling study, POD-pixels
  (49.74) and perceptual-style features (51.01) score below the
  no-distillation baseline (53.29), while GradCam maps (54.13) are slightly
  above it [arxiv:2004.13513 tab. 3].
- **Pooling for segmentation.** PODNet's classification ablation favours the
  POD-spatial term [arxiv:2004.13513 tab. 3]. PLOP argues that plain pooling
  blurs small objects in segmentation and adds local grids for that reason
  [arxiv:2011.11390 §3.2]. These are different tasks, not a direct contradiction.
- **Ground-truth term.** Several variants here add a label loss to the
  distillation term [arxiv:2003.11539 §3.3] [arxiv:2205.10536 §3.2]. TinyViT
  reports that adding ground-truth labels to its distillation loss lowered
  ImageNet-1k top-1 by 0.2 points with the CLIP teacher and 0.7 with Florence
  [arxiv:2207.10666 app. C]. The settings differ, so the results are not
  directly comparable, and the paper gives this as its reason for a label-free
  loss.
- **Logit vs feature targets.** In PLOP's ablation with the same pseudo-label
  loss, logit-level KD reaches 23.69 on all classes, POD reaches 34.62 and
  Local POD reaches 52.31 [arxiv:2011.11390 tab. ablation_classif_distillation].
  The ablation is on a validation split, not the main protocol.
- **Student against teacher.** In the one-step diffusion case the student stays
  well behind its teacher, with FID 9.36 against 4.16 [arxiv:2101.02388 tab. 1].
  In the video case the four-step student scores above its own many-step
  bidirectional teacher on the paper's ablation: 94.7, 64.4 and 30.1 against
  94.6, 62.7 and 29.6 on temporal quality, frame quality and text alignment
  [arxiv:2412.07772 tab. 4]. The metrics and settings differ, so the two
  results are not comparable.
- **Mass outside the top K.** TinyViT spreads the probability outside its top K
  evenly over the other classes [arxiv:2207.10666 eq. 2]. The top-5
  language-model variant sets it to zero, and its five probabilities usually sum
  to close to one [arxiv:2301.12726 §3]. Gemma 2 matches the full vocabulary
  [arxiv:2408.00118 §3.2], and MedGemma 1.5 keeps 256 teacher logits per token
  [arxiv:2604.05081 §2.2], which does not say how the rest is treated. Decide which
  convention to use before comparing numbers across these two pages.
- **Live or stored teacher.** The token-labeling paper says distillation needs
  a teacher running during training [arxiv:2104.10858 §3.2]. TinyViT stores
  the teacher's top-K outputs before training [arxiv:2207.10666 §3.1], and the
  causal video student queries its teacher during training [arxiv:2412.07772
  §3.2], and RLDG's generalist never queries its RL experts, learning only
  from stored rollouts [arxiv:2412.09858 §3.2]. "Distillation" covers all of
  these practices here, so say which one a source means.
- **UniME's gain over VLM2Vec.** The text reports a 10.3% gain with LLaVA-1.6
  [arxiv:2504.17432 §4.3]. Table 1 gives 66.6 against 63.3, a 3.3-point gap
  [arxiv:2504.17432 tab. 1]. Both are author-reported and the page does not
  choose between them.
- **FedDF against FedGen.** FedGen's table places FedDFusion, which the FedGen
  card ties to the ensemble distillation of [arxiv:2006.07242 §3], above FedGen
  at EMNIST α = 0.05 (70.40 against 68.53), and below it at MNIST α = 0.05 (90.02
  against 91.30) [arxiv:2105.10056 tab. 1]. The FedDF card reports rounds to a
  target accuracy on CIFAR-10 [arxiv:2006.07242 tab. 1], not the same metric, so
  the two papers' rankings are not directly comparable.
- **The KD baseline changes the DKD gain.** On ImageNet, ResNet-34 to ResNet-18,
  DKD is +1.04 top-1 over the KD row copied from other work (70.66), but +0.67
  over the authors' own KD run (71.03) [arxiv:2203.08679 tab. 8].
- **Logit-only results vary by task.** DKD alone is below ReviewKD on MS-COCO for
  ResNet-101 to ResNet-18 (35.05 against 36.75 AP) [arxiv:2203.08679 tab. 10].
  On ImageNet it is 0.09 ahead for ResNet-34 to ResNet-18 (71.70 against 71.61)
  [arxiv:2203.08679 tab. 8], and 0.51 behind for ResNet-50 to MobileNet-V1
  (72.05 against 72.56) [arxiv:2203.08679 tab. 9].
- **Distance in VideoMamba's self-distillation.** The paper calls the alignment
  an L2 loss [arxiv:2403.06977 §4.1]. The card's Paper vs code section records
  that the released code uses the squared distance between L2-normalised
  features, which equals 2 − 2·cos for unit vectors. Check which form an
  implementation needs before comparing scales ([VideoMamba](/papers/2403.06977)).
- **Distillation weighting in ADD.** The exponential weighting gives the best
  FID in the ablation, 20.6, while NFSD gives the higher CLIP score, 0.327
  against 0.319 [arxiv:2311.17042 tab. 1]. The paper's final model uses NFSD
  and its ablation default is the exponential weighting [arxiv:2311.17042 §4.1].
  The NFSD formula is not given in the paper, so the choice cannot be
  reproduced from the paper alone.
- **Regression term in one-step distribution matching.** DMD reports that
  dropping its LPIPS regression term raises CIFAR-10 FID from 2.66 to 5.58
  [arxiv:2311.18828 tab. 2]. ADD's objective has no regression term
  [arxiv:2311.17042 eq. 1], and its ablations do not test one
  [arxiv:2311.17042 tab. 1]. The datasets and losses differ, so the two results
  are not directly comparable.
- **Weight direction in one LLM-guided search.** The paper's prose says
  cross-entropy dominates early and the distillation weight rises
  [arxiv:2504.09685 §3.3]. Equation 11 with α from 0.4 to 0.8 does the reverse
  [arxiv:2504.09685 eq. 11] [arxiv:2504.09685 eq. 12]. The paper does not
  resolve this, so an implementation should follow the equations and say so.

## Papers

- [Rethinking Few-Shot Image Classification](/papers/2003.11539): soft-label self-distillation on top of a frozen embedding, with reported few-shot gains.
- [Knowledge Distillation from A Stronger Teacher](/papers/2205.10536): relation-based logit distillation with Pearson correlation, for strong teachers.
- [PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning](/papers/2004.13513): pooled-activation distillation between old and new model for class-incremental learning.
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](/papers/2011.11390): multi-scale pooled feature distillation with confident pseudo-labels for continual segmentation.
- [Student-Teacher Feature Pyramid Matching for Anomaly Detection](/papers/2103.04257): feature-pyramid distillation trained on normal images for anomaly detection.
- [TinyViT: Fast Pretraining Distillation for Small Vision Transformers](/papers/2207.10666): offline top-K teacher labels with seed replay for pretraining small hierarchical transformers.
- [iBOT: Image BERT Pre-Training with Online Tokenizer](/papers/2111.07832): EMA-teacher self-distillation for masked patch prediction in Vision Transformers.
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](/papers/2103.13744): teacher-to-cell distillation of NeRF density and colour, followed by fine-tuning.
- [Rethinking Vision Transformers for MobileNet Size and Speed](/papers/2212.08059): hard distillation as one ingredient of an efficient hybrid transformer recipe.
- [Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed](/papers/2101.02388): one-step student regressed onto a deterministic DDIM teacher's outputs.
- [Specializing Smaller Language Models towards Multi-Step Reasoning](/papers/2301.12726): top-5 distribution matching of chain-of-thought maths solutions into small T5 and FlanT5 students.
- [From Slow Bidirectional to Fast Autoregressive Video Diffusion Models](/papers/2412.07772): causal few-step video student trained with distribution matching from a bidirectional teacher.
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](/papers/2212.03533): contrastive pre-training of E5, then optional cross-encoder distillation during fine-tuning.
- [Unmasking On-Policy Distillation](/papers/2605.10889): training-free per-token diagnostic of whether on-policy distillation gradients point toward success.
- [All Tokens Matter: Token Labeling for Training Better Vision Transformers](/papers/2104.10858): dense per-patch labels from an offline annotator, as an alternative to distillation with a live teacher.
- [Preservation of the Global Knowledge by Not-True Distillation in Federated Learning](/papers/2106.03097): distillation from the frozen global model restricted to the not-true classes, for non-IID federated training.
- [Revisiting Relation Extraction in the era of Large Language Models](/papers/2305.05003): GPT-3 explanations used as fine-tuning targets for a Flan-T5 Large relation extractor.
- [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](/papers/2504.17432): text-only batch similarity distillation as stage 1 of a multimodal LLM embedder.
- [Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model](/papers/2511.23429): distilled few-step causal generator for interactive game video, with a DMD-based long-video stage.
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](/papers/2504.08165): shared-budget challenge report; teacher–student distillation is one of the approaches it groups with the strongest gains, without per-approach numbers.
- [RepViT: Revisiting Mobile CNN From ViT Perspective](/papers/2307.09283): mobile CNN whose headline numbers use distillation from a RegNetY teacher; distillation gap reported against the same models without it.
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](/papers/1908.09791): elastic super-network trained with soft labels from its largest sub-network, then specialised without retraining.
- [Federated Class-Incremental Learning](/papers/2203.11473): GLFC, with a KL relation-distillation term against the stored old model and server-side prototype reconstruction.
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](/papers/2006.07242): server-side distillation of averaged client logits into the fused model, in place of weight averaging.
- [Emerging Properties in Self-Supervised Vision Transformers](/papers/2104.14294): label-free self-distillation with an EMA teacher, centring and sharpening (DINO).
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](/papers/2105.10056): generator trained on client prediction layers, with its samples used as a regulariser in local training (FedGen).
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](/papers/2107.07651): momentum distillation from an EMA teacher for the contrastive and masked-language losses (ALBEF).
- [Contrastive Learning with Stronger Augmentations](/papers/2104.07713): divergence loss matching a strong view's similarity distribution to its weak twin's, added to MoCo-style contrastive pretraining.
- [Decoupled Knowledge Distillation](/papers/2203.08679): logit loss split into target-class and non-target terms, with separate weights.
- [One-step Diffusion with Distribution Matching Distillation](/papers/2311.18828): one-step image generator trained to match the teacher's distribution, with a fake-score network and a regression term (DMD).
- [M3-Embedding](/papers/2402.03216): self-knowledge distillation from the combined score of dense, sparse and multi-vector heads.
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](/papers/2403.05530): dense Flash model described as online-distilled from Pro; no loss or schedule given.
- [Gemma 2: Improving Open Language Models at a Practical Size](/papers/2408.00118): 2B and 9B models trained by matching a larger teacher's full next-token distribution.
- [Small Models in the LLM Era (survey)](/papers/2409.06857): survey placing distillation among small-model and LLM relationships; no method of its own.
- [Small Language Models are the Future of Agentic AI](/papers/2506.02153): position paper with distillation from the LLM as an optional conversion step; no experiments.
- [MedGemma 1.5 Technical Report](/papers/2604.05081): post-training by distilling 256 teacher logits per token, with supplementary teachers.
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](/papers/2312.03203): per-Gaussian features distilled from LSeg or SAM by splatting, with an optional channel speed-up.
- [VideoMamba: State Space Model for Efficient Video Understanding](/papers/2403.06977): self-distillation from a smaller trained video model, plus masked alignment to CLIP ViT-B features.
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](/papers/2004.09813): English sentence encoder copied into a multilingual student by regressing translation pairs onto its vectors.
- [Rethinking Spatial Dimensions of Vision Transformers](/papers/2103.16302): pooling-based ViT whose distillation token is trained against a RegNetY-16GF teacher, with the gain reported per recipe.
- [ImageNet-21K Pretraining for the Masses](/papers/2104.10972): semantic-softmax pretraining on a cleaned ImageNet-21K, with a per-depth distillation term whose code is not released.
- [Learning Generalizable Feature Fields for Mobile Manipulation](/papers/2403.07563): feed-forward feature field with a frozen 2D teacher's features appended to its latent, for navigation and grasping.
- [RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning](/papers/2412.09858): per-task RL experts' successful rollouts used as fine-tuning data for OpenVLA and Octo.
- [Adversarial Diffusion Distillation](/papers/2311.17042): one-to-four-step diffusion student trained with a hinge-loss discriminator and a frozen diffusion teacher's score distillation.
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](/papers/2004.05937): survey of knowledge distillation and student-teacher learning for vision; no experiments.
- [DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models](/papers/2505.13975): teacher splits and prunes the student's own reasoning traces, which are then used for LoRA fine-tuning.
- [KL for a KL: On-Policy Distillation with Control Variate Baseline](/papers/2605.07865): detached negative reverse-KL baseline for on-policy distillation, with a cheaper top-k version.
- [Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?](/papers/2504.09685): LLM-proposed microcontroller CNNs, filtered by MAC and SRAM estimates, then distilled from a ViT-B/16 teacher.
- [Accelerate Intermittent Deep Inference](/papers/2407.14514): distilled and shrunk MCUNet teacher as the starting space for intermittent-aware NAS on about 2 KB of SRAM.
- [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932): PRISMA review of TinyML with distillation as one model-optimization technique; no experiments.
- [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](/papers/2311.11883): review of distillation, pruning, quantization and tooling for Cortex-M microcontrollers; no new experiments.
- [A Review on Edge Large Language Models: Design, Execution, and Applications](/papers/2410.11845): edge-LLM survey placing white-box and black-box distillation among offline compression methods; no experiments.
- [Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290): transformer survey with a two-stage distillation recipe whose ranges have no described experiment.
- [Towards Understanding Knowledge Distillation](/papers/2105.13093): linear teacher and student with soft labels; recovery and transfer-risk bounds, with synthetic and MNIST experiments.
- [Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline](/papers/2206.08129): single-camera driving network that learns from a privileged RL expert's actions and features, with trajectory-guided attention.
- [HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation](/papers/2012.07356): lightweight MobileNetV3 depth student trained to match the disparity of a larger self-supervised model.
- [Enhanced Membership Inference Attacks against Machine Learning Models](/papers/2111.09679): membership-inference attack whose per-record threshold comes from distilled models labelled by the target (Attack D).