Knowledge distillation

Overview

A teacher model supplies targets, and a student is trained to match them. The choice of target is the main design decision. The variants on this page use a predictive distribution over classes 2003.11539 §3.3, relations between the entries of those distributions 2205.10536 §3.1 [arxiv:2205.10536 §3.2], pooled or raw intermediate feature maps 2004.13513 §3.1 2011.11390 §3.2 2103.04257 §3.2, patch-token distributions from a momentum teacher 2111.07832 §3.1, density and colour at sampled 3D points 2103.13744 §3.3, top-K probabilities computed once before training 2207.10666 §3.1, the output of a deterministic multi-step sampler 2101.02388 §3.1, the top-5 token probabilities of a large language model's solutions 2301.12726 §3, and a score difference from a bidirectional video teacher 2412.07772 §3.2. Embedding models also distil soft scores from a cross-encoder 2212.03533 §4.2. A dense long-context multimodal model is distilled from its larger sibling during training 2403.05530 §3.2. A multilingual sentence encoder is trained to reproduce a monolingual teacher's vector for each sentence and for its translation 2004.09813 §2. A diffusion teacher denoises re-noised student samples, and the one-to-four-step student matches that estimate 2311.17042 §3.3. The ImageNet-21K pretraining paper adds a distillation term per level of its label hierarchy 2104.10972 §3.4. Two variants use targets that are not probability vectors: a teacher's written explanations, used as text 2305.05003 §4.2.2, and a teacher's similarity structure over a batch 2504.17432 §3.2. In federated learning, a client matches only the not-true-class part of the frozen global model's output 2106.03097 §3. The token-labeling paper uses dense per-patch labels from an offline annotator and contrasts this with distillation 2104.10858 §3.2. The newer cards add targets of other kinds. A federated server distils from the averaged logits of client models on unlabelled batches 2006.07242 §3. In a data-free variant, a generator trained on the clients' prediction layers supplies the samples instead of data 2105.10056 §3.1. A retrieval model's teacher is its own combined score over three heads 2402.03216 §3.3. Three cards add feature-level targets. A 2D foundation model's per-pixel features are distilled into the features of 3D Gaussians 2312.03203 §3.1 2312.03203 §3.2. A generalizable feature field appends a frozen teacher's 2D features to its latent and is trained to match them on novel views [arxiv:2403.07563 §3.2] 2403.07563 eq. 3. A smaller trained video model supplies the final features that a larger one aligns to 2403.06977 §4.1. In one robot-policy card the teacher is a set of reinforcement-learning experts, and its successful rollouts are stored as the student's training data instead of being queried during training 2412.09858 §3. Two further cards use distilled models outside compression. A lightweight depth network is trained to match the disparity of a larger one 2012.07356 §3.5. A membership-inference attack trains models on the target's own outputs to estimate a record's loss without that record 2111.09679 §4.4.

Most teachers are fixed while the student trains. The old model in incremental learning is the frozen teacher 2004.13513 §3.1, and the anomaly detector's teacher is never updated 2103.04257 §3.2. The teacher gets no gradient in any variant here. Two variants differ in how the teacher is used. iBOT's teacher is an exponential moving average of the student and changes during training 2111.07832 §2.2. DINO and ALBEF use the same kind of teacher 2104.14294 §3.1 2107.07651 §3.3. TinyViT goes further and runs the teacher only before training, so the stored targets are the only teacher output the student sees 2207.10666 §3.1. In the video student the teacher stays frozen, but a second network, the fake score, is trained online on the student's own outputs 2412.07772 §3.2.

Where labels exist, the distillation term is usually added to the label loss rather than replacing it 2003.11539 §3.3 2205.10536 §3.2 2212.03533 §4.2 2106.03097 §3. The anomaly detector, the NeRF distillation, TinyViT's pretraining, the diffusion student and the top-5 language-model student use no labels in the student loss 2103.04257 §3.2 2103.13744 §3.3 2207.10666 §3.1 2101.02388 §3.1 2301.12726 §3. The feature term in Feature 3DGS is also label-free, and its colour term is a photometric loss on the training images 2312.03203 §3.2. The language-model paper uses the gold answer only to keep correct teacher solutions 2301.12726 §3. VideoMamba adds its alignment term to the classification loss 2403.06977 §4.1. Label-free and label-mixing cases differ. DINO uses no labels at all 2104.14294 §3.1. The federated server in FedDF uses labels only for validation and early stopping [arxiv:2006.07242 §4.1]. ALBEF mixes the one-hot label with the teacher's distribution 2107.07651 §3.3.

The variants below cover several uses, not the whole field. Other distillation methods are not represented on this page yet. A shared small-data pretraining challenge also lists teacher–student distillation among the approaches with the strongest gains, without per-approach numbers 2504.08165 §7.4.

Variants

Soft-label self-distillation

A fresh network of the same architecture is trained on the merged labelled data with a weighted sum of cross-entropy and KL divergence to the previous generation's predictions 2003.11539 §3.3. The student is re-initialised, not warm-started from the teacher, and each generation is a full training run 2003.11539 §4.1. Card: Rethinking Few-Shot Image Classification.

Relation-based logit distillation from a stronger teacher

DIST uses only the output logits. Instead of matching the exact probabilities, it matches Pearson correlations in two directions: across the classes of each instance, and across the instances of each class 2205.10536 §3.1 2205.10536 §3.2. The authors argue that exact matching is a poor target when the teacher is much stronger than the student 2205.10536 §2.1. On ImageNet the total loss weights the relation terms with β=2 and γ=2 2205.10536 §4.1. Card: Knowledge Distillation from A Stronger Teacher.

Decoupled logit distillation

DKD rewrites the classic KD loss as a target-class term plus a non-target term multiplied by one minus the teacher's probability for the correct class 2203.08679 §3.1 2203.08679 eq. 6. It drops that multiplier and weights the two terms with separate constants, α and β 2203.08679 §3.3 2203.08679 eq. 7. The authors read the multiplier as suppressing the non-target term when the teacher is confident 2203.08679 §3.2. The loss adds no parameters and has about the same cost as KD 2203.08679 §4.2. On CIFAR-100 the paper uses α = 1.0 and T = 4, with β set per teacher and no general rule 2203.08679 app. A.3. Card: Decoupled Knowledge Distillation.

Pooled-activation distillation for incremental learning

While a class-incremental model learns new classes, its intermediate stage outputs and final embedding are kept close to those of the frozen previous model. The loss compares sums of squared activations along width and along height, which tolerates small local shifts while keeping the overall layout 2004.13513 §3.1. A second, flat term matches the normalised final embedding 2004.13513 §3.1. The term is scaled by an adaptive factor that grows with the number of seen classes 2004.13513 §6.2. Card: PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning.

Multi-scale pooled distillation with pseudo-labels for continual segmentation

PLOP's Local POD applies the same width and height pooling over grid regions at several scales, so it keeps both large-scale layout and small details 2011.11390 §3.2. Its pseudo-labels are a separate mechanism: an old-class guess replaces a "background" label only where the old model is confident, judged by per-class entropy thresholds 2011.11390 §3.3. The paper's ablation reports the two effects separately. Card: PLOP: Learning without Forgetting for Continual Semantic Segmentation.

Not-true-class distillation in federated learning

Each client trains on its own labelled data with cross-entropy, plus a term that matches the frozen global model only on the classes that are not the true label 2106.03097 §3. Both softmaxes drop the true class, so the true-class logit gets no gradient from the distillation term [arxiv:2106.03097 §3]. The authors attribute poor accuracy under non-IID splits mainly to forgetting of classes a client does not hold, and say they treat this as a diagnostic rather than a proven cause 2106.03097 §2. The reported runs use β = τ = 1 2106.03097 app. Algorithm Implementation Details, and a too-small temperature "significantly drops the accuracy" [arxiv:2106.03097 app. Effect of FedNTD Hyperparameters]. An MSE version of the term gets worse as β rises and diverges at β = 0.3, so the KL form is the tested one [arxiv:2106.03097 app. MSE Loss for Not-True Distillation]. Card: Preservation of the Global Knowledge by Not-True Distillation in Federated Learning.

Class-relation distillation in federated class-incremental learning

Clients receive new classes over time and keep only a small exemplar memory. GLFC adds a relation-distillation term to each client's loss. The target is the one-hot label vector with its old-class entries replaced by the sigmoid outputs of the stored previous model, and the loss is a KL divergence to that target 2203.11473 §4.1 eq. 5. The term is combined with a gradient-compensated classification loss, with weights λ1 = 1 and λ2 = 0 on the first task and λ1 = λ2 = 0.5 afterwards 2203.11473 §4.1 eq. 6. The server rebuilds prototype samples from shared gradients and keeps the best models of the last and current task; each client uses either the previous task's model or the current one as the old model, depending on whether it detected a new task 2203.11473 §4.2 2203.11473 §4.3. The authors report 66.9 average accuracy on CIFAR-100 with 10 tasks, against 61.5 for the best adapted baseline, and 65.8 when the relation term is removed 2203.11473 tab. 1. Their numbers are author-reported. Several settings in the official code differ from the paper, including the distillation loss and the reconstruction optimiser; see the card. Card: Federated Class-Incremental Learning.

Ensemble distillation for federated model fusion

FedDF replaces the weight-averaging step of FedAvg with distillation. The server takes a data-size-weighted average of the returned client models as its starting point, then trains it to match the unweighted mean of all sampled clients' logits on unlabelled or generated batches, using a KL loss without a temperature in the equation 2006.07242 alg. 1 2006.07242 eq. 1. Client models may differ in architecture, size or precision, because only their outputs are combined 2006.07242 app. B. In the CIFAR-10 setting with 20 clients, reaching 80% top-1 takes 20 ± 4 rounds against 104 ± 52 for FedAvg 2006.07242 tab. 1. Card: Ensemble Distillation for Robust Model Fusion in Federated Learning.

Data-free generator distillation for federated learning

The server trains a small conditional generator from the clients' prediction layers, with no data and no proxy set 2105.10056 §3.1. The generator is sent to each client, which adds a term to its local loss asking that the generator's samples be classified correctly by its own predictor 2105.10056 eq. 5. The authors call these samples an inductive bias for local learning 2105.10056 §4.1. Only the predictor layers are needed from each client 2105.10056 §3.1. On MNIST with Dirichlet α = 0.05 the authors report 91.30 against 87.70 for FedAvg 2105.10056 tab. 1. Card: Data-Free Knowledge Distillation for Heterogeneous Federated Learning.

Feature-pyramid distillation for anomaly detection

STFPM trains a same-architecture student on defect-free images only. It matches L2-normalised ResNet-18 features at three levels (conv2_x to conv4_x) with equal weights 2103.04257 §3.2. At test time the per-level distance maps are upsampled and multiplied, and the image score is the maximum of the result 2103.04257 §3.3. Card: Student-Teacher Feature Pyramid Matching for Anomaly Detection.

Self-distillation with an EMA teacher for masked image modelling

iBOT's teacher reads unmasked views and is an exponential moving average of the student 2111.07832 §2.2. Its per-patch output distributions are the targets at masked positions, and a [CLS] term aligns the summary token across two views 2111.07832 §3.1 2111.07832 eq. 2 2111.07832 eq. 3. The targets are softmax distributions over K=8192 outputs, not hard labels. The paper's ablation shows lower linear probing with hard targets [arxiv:2111.07832 tab. 18]. Card: iBOT: Image BERT Pre-Training with Online Tokenizer.

Label-free self-distillation with an EMA teacher

DINO trains a ViT without labels. A student matches the softmax outputs of a teacher across several crops. The teacher sees only the two global crops and is an exponential moving average of the student, with momentum following a cosine schedule from 0.996 to 1 2104.14294 §3.1. Centring and sharpening the teacher's outputs prevent collapse, so the method needs no contrastive negatives and no predictor 2104.14294 §3.1. Using the previous epoch as the teacher gives 66.6 k-NN top-1 against 72.8 for the momentum teacher 2104.14294 fig. 6. The paper's teacher-temperature warm-up from 0.04 to 0.07 is not the code default 2104.14294 app. D. Card: Emerging Properties in Self-Supervised Vision Transformers.

Momentum distillation for vision-language pretraining

ALBEF aligns image and text features with a contrastive loss before cross-modal fusion 2107.07651 §3.2. Its momentum teacher is an exponential moving average with momentum 0.995, and its similarities give soft targets 2107.07651 §3.3. For the contrastive loss, the target mixes the teacher's distribution with the one-hot label at weight α [arxiv:2107.07651 eq. 6]. The masked-language loss takes the same mix over the vocabulary 2107.07651 eq. 7. α ramps linearly from 0 to 0.4 in the first epoch 2107.07651 §3.5. At 4M images, adding momentum distillation to both losses raises the Flickr30K text-retrieval average from 97.01 to 97.47 and VQA test-dev from 73.81 to 74.42 2107.07651 tab. 1. Card: Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.

Similarity-distribution matching for strong augmentations

Contrastive learning normally pulls each augmented view toward its own positive key, which can fail when the view is heavily distorted. CLSA keeps the usual contrastive loss on gentle views and adds a divergence term. The weakly augmented query gives a softmax distribution over its positive and a queue of negatives; this distribution is the target, with gradients stopped, and the strongly augmented query is trained to match it over the same candidates 2104.07713 §3.2 2104.07713 eq. 3 2104.07713 eq. 6. The total loss is the contrastive term plus β times the divergence, with β = 1.0 2104.07713 eq. 7. The teacher here is the weak view itself, so the term is a self-distillation step inside a contrastive method. The authors report 69.4 ImageNet linear top-1 for single-crop CLSA against 67.5 for MoCo v2 at 200 epochs, and 76.2 for multi-crop CLSA* at 800 epochs against 76.5 for supervised training 2104.07713 tab. 2 2104.07713 tab. 3. These numbers are author-reported. The paper's own ablation shows that adding strong views naively gains little: 67.7 with a strong query and 67.0 with a strong query and key, against 67.5 for MoCo v2. Replacing the divergence with a contrastive loss gives 68.0 2104.07713 tab. 5. Card: Contrastive Learning with Stronger Augmentations.

Teacher-to-cell density distillation

A NeRF teacher is queried at random points inside each cell of a grid, and each small per-cell student is trained to match the teacher's opacity and colour there. No training images are used in this stage [arxiv:2103.13744 §3.3]. Fine-tuning on the images follows, because distillation alone leaves artefacts in free space 2103.13744 §3.3 2103.13744 §4.3. Card: KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs.

Offline top-K teacher labels for pretraining small transformers

TinyViT runs a large teacher once over each augmented ImageNet-21k view, before any student is trained. It keeps the K largest teacher probabilities and their class indices, plus a 4-byte seed that replays the same augmentation later 2207.10666 §3.1. The student minimises cross-entropy to the stored distribution, with no ground-truth labels 2207.10666 §3.1 [arxiv:2207.10666 eq. 1]. The probability left outside the top K is spread evenly over the other classes 2207.10666 eq. 2. The paper uses K = 100 for ImageNet-21k and K = 10 for ImageNet-1k 2207.10666 §5.2. Changing the augmentation pipeline after labelling breaks the seed replay, so the stored labels fix the training views. Card: TinyViT: Fast Pretraining Distillation for Small Vision Transformers.

Dense per-patch labels from an offline annotator

Every training image gets one label per patch token, produced once before training by an ImageNet-only NFNet-F6 classifier (86.3% top-1) [arxiv:2104.10858 §3.2] 2104.10858 §4.1. The objective adds the class loss to a weighted average of per-token cross-entropies 2104.10858 eq. 2 [arxiv:2104.10858 eq. 3]. To save space, only the five largest values per cell are kept, in half precision, which the authors report as about 10 GB instead of 2 TB [arxiv:2104.10858 §4.1]. MixToken mixes patch embeddings rather than pixels, and the token labels are mixed with the same mask 2104.10858 §3.3 2104.10858 eq. 6. The paper contrasts this with distillation, which it says needs a teacher running during training 2104.10858 §3.2. Card: All Tokens Matter: Token Labeling for Training Better Vision Transformers.

Hard distillation as a training ingredient

EfficientFormerV2 names a RegNetY-16GF teacher and hard distillation for training 2212.08059 §5.1. The paper does not state the distillation weight or temperature 2212.08059 §5.1, so these must be checked against the released code before reproducing. Distillation is one ingredient of the recipe, not the main contribution. Card: Rethinking Vision Transformers for MobileNet Size and Speed.

RepViT uses the same teacher, RegNetY-16GF 2307.09283 §4.1, for the numbers in its Table 1. The paper does not say whether the teacher term is hard or soft. The released code gives the student a second head for the teacher and mixes the label loss with the teacher loss at weight 0.5, which the card records from the code, not the paper. Table 2 reports the same models without distillation 2307.09283 tab. 2. Card: RepViT: Revisiting Mobile CNN From ViT Perspective.

Pooling-based ViT (PiT) also trains a distillation token against a RegNetY-16GF teacher 2103.16302 §4.1. The paper does not say whether that loss is soft or hard. Relative to the DeiT recipe without distillation, distillation adds 1.0 point for PiT-S (80.9 to 81.9) and 1.4 points for ViT-S (79.8 to 81.2) 2103.16302 tab. 2. Card: Rethinking Spatial Dimensions of Vision Transformers.

Distillation from the largest sub-network in elastic training

Once-for-all trains one super-network whose sub-networks vary in depth, width, kernel size and input size 1908.09791 §3.2. During progressive shrinking, each sampled sub-network gets a soft-label term from the largest network, added to a smoothed hard-label term 1908.09791 §3.3. The paper does not state the weight of either term 1908.09791 §3.3. Without fine-tuning, the authors report 76.0% top-1 at 230M MACs, against 75.2% for MobileNetV3-Large 1908.09791 tab. 1. The 80.0% headline needs 75 epochs of fine-tuning 1908.09791 tab. 1. Card: Once-for-All: Train One Network and Specialize it for Efficient Deployment.

One-step student from a deterministic diffusion sampler

The teacher is a deterministic DDIM sampler with 100 steps on CIFAR-10 and 50 steps on LSUN 2101.02388 §4. For many initial noise vectors, the teacher's final image is recorded once, giving about 1.024M pairs per dataset 2101.02388 §3.1. The student starts from the teacher's network weights and is trained to produce that final image from the same noise in one network evaluation. The loss is a squared distance, and the paper reports L1 as better on CIFAR-10 2101.02388 eq. 11 2101.02388 app. B. No adversarial term is used 2101.02388 §3.1. Only deterministic teachers qualify: score-based and energy-based samplers that use stochastic MCMC cannot serve as teachers, and a GAN is deterministic but already fast 2101.02388 §2.1. Author-reported CIFAR-10 FID is 9.36 at one step, against 4.16 for the teacher 2101.02388 tab. 1. Card: Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed.

Top-5 distribution matching for a small language model

The teacher is a large model used through its outputs: 40 sampled solutions per GSM8K training question, of which the ones that reach the gold answer are kept 2301.12726 §3 2301.12726 §4.1. The students are T5 and FlanT5 models from 250M to 11B parameters 2301.12726 §4. The main objective is a KL divergence between the student's next-token distribution and the teacher's top-5 probabilities at each step, with the other tokens set to zero. The two models use different tokenisers, so their token sequences are aligned before the distributions are compared 2301.12726 §3. Four data formats are mixed, in-context and zero-shot, each with or without the chain of thought 2301.12726 §3. Checkpoints are chosen on held-out maths sets, not on the training data 2301.12726 §4.3. Card: Specializing Smaller Language Models towards Multi-Step Reasoning.

Teacher-pruned student reasoning traces for a reasoning model

The student writes its own long chain of thought first. A teacher then splits that trace into steps with skill labels, chooses for each step to keep, delete, or compress it, and rewrites the survivors into one trace and answer 2505.13975 §3.1 2505.13975 §3.2. The student is fine-tuned on these pairs with LoRA 2505.13975 §3.3 2505.13975 §4.4. The target is the student's own style, shortened, not a concise teacher's style. The paper's ablation finds that direct distillation from GPT-4o's concise traces falls below the base student in all four 7B settings, while DRP matches or beats it 2505.13975 tab. 4. Training data is 8,000 maths samples from GSM8K and PRM12K 2505.13975 app. B. Author-reported, 7B student on GSM8K: 91.7% to 94.1% accuracy, with 917 to 328 tokens 2505.13975 tab. 1. Card: DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models.

Token-level distillation in recent language and medical models

Gemma 2 trains its 2B and 9B students to match the teacher's full next-token distribution, with a cross-entropy summed over the whole vocabulary 2408.00118 §3.2. The 2B ablation uses a 7B teacher [arxiv:2408.00118 tab. 6]. Temperature, top-k truncation and the token budget per step are not stated 2408.00118 §3.2. MedGemma 1.5 4B also post-trains by distillation. Each token keeps 256 teacher logits, weighted by teacher probability, and the student is trained with cross-entropy on them. The teacher is an instruction-tuned model plus supplementary teachers, whose training the paper does not describe 2604.05081 §2.2. Cards: Gemma 2: Improving Open Language Models at a Practical Size and MedGemma 1.5 Technical Report.

Online distillation during training of a long-context multimodal model

Gemini 1.5 Flash is a dense Transformer that the report describes as distilled from Pro while it trains, so it learns from Pro as well as from its fixed dataset 2403.05530 §3.2 2403.05530 app. model card. The report gives no loss, schedule or sampling rule for this step, so the teacher–student mechanism cannot be reproduced from it 2403.05530 §3.2. Flash is also trained with a higher-order preconditioned optimiser, which is not named 2403.05530 §3.2. Card: Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.

Teacher-written explanations as fine-tuning targets for extraction

GPT-3 (175B) writes one explanation per training example, given hand-written examples and the gold triplets 2305.05003 §4.2.2. Flan-T5 Large (760M) is then fine-tuned on the gold triplets together with that explanation 2305.05003 §4.2.2 2305.05003 fig. 3. The targets are text, so the student needs no access to the teacher's probabilities. For NYT, the authors generated explanations for 25k of the 56k training examples, because of cost 2305.05003 §4.2.2. A variant where GPT-3 also writes the labels is reported only for CoNLL04 2305.05003 tab. 2. Card: Revisiting Relation Extraction in the era of Large Language Models.

Distribution matching for one-step image generation

DMD trains a one-step generator so that its samples match the teacher's distribution, rather than reproducing the teacher's noise-to-image mapping 2311.18828 §1. The generator's update direction is the difference between the teacher's score and the score of a fake-score network, which is trained alongside on the generator's own outputs 2311.18828 §3.2 2311.18828 eq. 7. A regression term, LPIPS between the generator's output and the teacher's output for paired noise, keeps the modes covered 2311.18828 §3.3 2311.18828 eq. 9. Its default weight is 0.25 2311.18828 §3.3. Without it, CIFAR-10 FID rises from 2.66 to 5.58 2311.18828 tab. 2. Card: One-step Diffusion with Distribution Matching Distillation.

Adversarial hinge loss plus score distillation for one to four steps

ADD starts the student from the weights of a pretrained text-to-image diffusion U-Net, and one student evaluation produces the sample 2311.17042 §3.1. Frozen ViT features feed lightweight heads, which are trained with a hinge loss to separate real images from student samples. The student is trained to fool them 2311.17042 §3.2 2311.17042 eq. 2 2311.17042 eq. 3. A frozen diffusion teacher denoises the student's sample after re-noising, and the student matches that clean estimate with a squared L2 term whose target is detached 2311.17042 §3.3. The total loss is the adversarial term plus λ = 2.5 times the distillation term, with no regression term 2311.17042 §4 2311.17042 eq. 1. In the ablation, the adversarial term alone gives FID 20.8, the distillation term alone gives 315.6, and random student initialisation gives 293.6 against 20.6 for the default 2311.17042 tab. 1. The paper gives no learning rate, batch size or iteration count for its main models 2311.17042 §4. Among SD1.5-based few-step methods, the one-step ADD-M reaches COCO FID-5k 19.7 and CLIP 0.326, against 22.5 for UFOGen at one step 2311.17042 tab. 2. Card: Adversarial Diffusion Distillation.

Distribution matching from a bidirectional teacher for causal video

A bidirectional video diffusion transformer, which needs about 50 denoising steps, is the teacher. The student generates frames chunk by chunk, attending to earlier chunks through a KV cache, and uses four denoising steps 2412.07772 §4.1 2412.07772 §4.4. Training has two stages. First the student is regressed from noisy states to the clean endpoints of teacher ODE trajectories 2412.07772 §4.3. Then asymmetric distribution matching distillation: the update direction is the difference between a fake score, trained online on the student's outputs, and the teacher's guided score. The fake score is updated five times per student update 2412.07772 §3.2 2412.07772 alg. 1 2412.07772 §5. Author-reported: VBench-Long total 84.27 and 9.4 FPS streaming on one H100 2412.07772 app. A 2412.07772 tab. 3. Card: From Slow Bidirectional to Fast Autoregressive Video Diffusion Models.

The same recipe appears in a 14B interactive game world model 2511.23429 §4.2.3. A bidirectional teacher is distilled with Self-Forcing into a few-step causal generator that uses a sink-token KV cache and block-sparse attention over earlier blocks. A later stage trains with a DMD loss on randomised rollouts, with a fake-score model on the student's own frames 2511.23429 §4.2.4 2511.23429 alg. 1. The authors report 16 FPS, but the hardware behind that number is not stated [arxiv:2511.23429 §4.4] 2511.23429 tab. 4. Card: Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model.

Cross-encoder soft labels in embedding fine-tuning

E5 is first pre-trained without labels on about 270M web and forum text pairs, with a contrastive loss over in-batch negatives 2212.03533 §4.1 2212.03533 tab. 9. Distillation appears only in the optional second stage. There the student's probabilities over a query's candidate passages are pushed toward those of a cross-encoder teacher with a KL term, plus α = 0.2 times a contrastive loss on the hard labels 2212.03533 §4.2 2212.03533 eq. 3 2212.03533 tab. 11. The teacher scores are reused from SimLM, and each example has seven hard negatives [arxiv:2212.03533 §4.2] 2212.03533 §5.1. The paper calls the loss a linear interpolation, but its equation has no weight on the KL term, so the weighting is unclear 2212.03533 §4.2 2212.03533 eq. 3. Card: Text Embeddings by Weakly-Supervised Contrastive Pre-training.

Self-knowledge distillation across retrieval heads

M3-Embedding has one encoder with three retrieval heads: dense, sparse and multi-vector 2402.03216 §3.2. Each head gets an InfoNCE loss. The teacher is the integrated score, a weighted sum of the three heads, and each head is trained with a soft cross-entropy towards it 2402.03216 §3.3 2402.03216 eq. 3 2402.03216 eq. 5. The paper does not say whether gradients pass through the teacher 2402.03216 §3.3. Distillation starts after about 6,000 warm-up steps 2402.03216 app. B.1. On MIRACL dev, the sparse head reaches 53.9 nDCG@10 with self-distillation and 36.7 without, while the dense head moves from 68.7 to 69.2 2402.03216 tab. 5. Card: M3-Embedding.

Batch similarity distillation into a multimodal LLM's text pathway

UniME's first stage trains the language model of a multimodal LLM on text only. The teacher is NV-Embed V2, whose sentence embeddings are computed offline 2504.17432 §3.2. The loss is a KL divergence between the student's and the teacher's batch-wise similarity distributions at temperature τ, summed over the batch 2504.17432 §3.2 2504.17432 eq. 2. Only the LLM is trained, with LoRA 2504.17432 §3.2. Stage 1 uses about 273K NLI sentence pairs, batch 768 through gradient accumulation, and two epochs 2504.17432 §4.1 2504.17432 tab. 6. On Phi-3.5-V, stage 1 alone gives an MMEB average of 40.3, against 25.3 with neither stage and 64.2 with both 2504.17432 tab. 3. Card: Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs.

Per-token diagnostic for on-policy distillation

This is an analysis, not a training recipe. On-policy distillation trains the student on its own rollouts, with a per-token signal from a teacher. The paper asks, for each token at each branch point, whether the teacher's gradient points toward answers that succeed 2605.10889 §1. The student samples many answers. At each branch point the empirical success rate of each next token defines an ideal gradient, and the GKD gradient is scored by its cosine similarity to it, from −1 to +1 2605.10889 §2.2 2605.10889 §2.4 2605.10889 eq. 2.12. Targeted rollouts keep this affordable: about 45K to 200K per question 2605.10889 §2.5 2605.10889 §3. On Qwen3 students the mean alignment is small, about +0.03 on BoolQ 2605.10889 §4. It is higher on incorrect rollouts than on correct ones in all four settings, and significant at p ≤ 0.0001 in three of them, with the 1.7B MMLU gap not significant 2605.10889 fig. 3 [arxiv:2605.10889 app. E.6]. Keeping only positively aligned tokens raises the signal, but that selection uses the true alignment, which is unknown at training time 2605.10889 app. E.3. The best teacher depends on student size and task 2605.10889 §4.2. Card: Unmasking On-Policy Distillation.

Baseline-subtracted on-policy distillation (vOPD)

Same setting as the diagnostic above: the student samples its own responses, and a teacher's per-token log-probabilities give the reward. The paper subtracts a baseline from that reward, which is the negative reverse KL between student and teacher at the same context 2605.07865 §3.1 2605.07865 eq. 10. The baseline comes from the forward pass already run, so no critic or extra rollouts are needed 2605.07865 §3.1. It is detached, so the gradient stays unbiased 2605.07865 §2.2 2605.07865 app. A.1. A top-k version computes the KL only over the student's k most likely tokens, which is cheaper and still unbiased for any k 2605.07865 §3.2 2605.07865 eq. 14. The paper's setting uses k = 20 2605.07865 §4.1. Author-reported, Qwen3-1.7B on the math average: 29.8 for vanilla OPD, 33.0 for top-k vOPD and 33.1 for full-vocabulary vOPD 2605.07865 tab. 2. Full-vocabulary OPD scores 33.0 there, so the top-k baseline matches the costly reference 2605.07865 tab. 2. The variance argument is a heuristic that the paper motivates but does not prove 2605.07865 app. A.3. Experiments stop at 7B parameters, and only one size-mismatched teacher pair is tested 2605.07865 §4.2 2605.07865 §6. Card: KL for a KL: On-Policy Distillation with Control Variate Baseline.

Expert-rollout data distillation into a generalist robot policy

The teacher is not queried by the student. For each task, a reinforcement-learning expert is trained on the real robot until it succeeds reliably, then rolled out, and only its successful episodes are kept 2412.09858 §3.1 2412.09858 app. B. The generalist, OpenVLA or Octo, is fine-tuned on these episodes with the negative log-likelihood of the recorded actions 2412.09858 §3.3 2412.09858 eq. 2. The data is the only channel from the experts to the generalist 2412.09858 §3.2. For a multi-stage task, the expert covers only the critical phase, and human demonstrations supply the rest 2412.09858 §3.2. The expert's reward is a binary success classifier trained on teleoperated positives and negatives 2412.09858 app. B. On precise connector insertion, the authors report gains over fine-tuning on the same number of human-demonstration episodes of 23% for OpenVLA and 37% for Octo. The paper does not say whether these are absolute or relative gains 2412.09858 §4.2 2412.09858 fig. 4. On four unseen connectors, OpenVLA trained on the RL data succeeds 73 times out of 80, against 49 for the best single-connector RL policy [arxiv:2412.09858 §4.3]. The trial counts are small, and the released code omits the generalist stage, so the results cannot be reproduced from the release. Card: RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning.

Feature-map distillation into a 3D Gaussian field

Feature 3DGS gives each 3D Gaussian an N-channel feature and renders it with the colour, using the same splatting 2312.03203 §3.1. The student's rendered feature is compared with the teacher's feature map, from LSeg or SAM, by an L1 term with weight γ = 1.0 2312.03203 eq. 4 [arxiv:2312.03203 §3.2]. The teacher has 512 channels for LSeg and 256 for SAM, so a speed-up renders fewer channels and lifts them with a 1x1 convolution [arxiv:2312.03203 §3.2] 2312.03203 app. A. The paper describes this module as not affecting downstream results, but its sweeps show that the rendered size matters at the low end 2312.03203 tab. A. A text or point query compares a feature with each Gaussian's feature by cosine similarity [arxiv:2312.03203 eq. 5]. Card: Feature 3DGS.

Teacher features appended to a generalizable feature field

GeFF is a scene-level feature field that one feed-forward pass builds from posed RGB-D frames, with no per-scene optimisation 2403.07563 §3.2. The encoder's latent is the encoder output concatenated with the frozen teacher's 2D features of the same input views 2403.07563 eq. 3. The semantic decoder is trained with an L2 loss to the teacher's features on ground-truth novel views, rendered through the same volume rendering as depth 2403.07563 §3.2 2403.07563 eq. 2 2403.07563 eq. 5. The paper names MaskCLIP as the teacher in its method section, but its training details name a ViT-L CLIP model 2403.07563 §3.2 [arxiv:2403.07563 §4.1]. Its auxiliary-input ablation on the robot is the only test of the teacher term here. Removing it lowers the object-level successes in the meeting-room and kitchen trials from 41 to 25 out of 66 2403.07563 tab. 2. The paper gives no distillation-only result, and it reports no confidence intervals. Card: Learning Generalizable Feature Fields for Mobile Manipulation.

Self-distillation from a smaller trained video model

VideoMamba scales a stack of bidirectional Mamba blocks over spatiotemporal tokens. For each size, a smaller trained model is the teacher for the larger one. The M model uses a pretrained S model 2403.06977 §4.1. The student's final features are aligned to the teacher's, and that term is added to the classification loss 2403.06977 §4.1. The authors report that this overcomes the overfitting seen when the pure Mamba model is scaled up 2403.06977 §3.3 2403.06977 fig. 6. Video pretraining uses a different target: unmasked tokens are aligned to CLIP ViT-B features, at the final layer only, with 80% masking 2403.06977 §3.4 [arxiv:2403.06977 §4.2]. Card: VideoMamba.

Embedding regression from an English teacher across languages

The student is a multilingual encoder, XLM-R in the paper, trained on parallel sentence pairs 2004.09813 §3. For each pair, it is trained to reproduce the teacher's English vector for the source sentence and for its translation, with a squared-distance loss 2004.09813 §2. The teacher is an English SBERT model. The best reported student uses a teacher trained on paraphrases 2004.09813 §4.1 2004.09813 tab. 1. Author-reported, the student averages 84.6 on STS 2017 monolingual and 83.7 cross-lingual, against 76.4 and 73.5 for LaBSE 2004.09813 tab. 1 2004.09813 tab. 2. On BUCC bitext mining it trails LASER and LaBSE, at 88.6 F1 against 93.0 and 93.5. The paper attributes this to targeting semantic similarity rather than exact translation 2004.09813 tab. 3 2004.09813 §4.2. Card: Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation.

Per-depth distillation in semantic-softmax pretraining

ImageNet-21K labels sit at different levels of the WordNet hierarchy. The semantic-softmax student has one softmax per depth, and an image trains every depth at which its ancestor labels exist 2104.10972 §3.3. The distillation term compares, depth by depth, the student's and the teacher's distributions over that depth's classes, with a mean squared error 2104.10972 §3.4 2104.10972 eq. 4. Each depth is weighted by the teacher's probability mass on its top 5% classes 2104.10972 eq. 5 2104.10972 app. F. The paper does not name the teacher, and it does not give the weight between this term and the classification loss 2104.10972 §3.4. Author-reported gains over semantic softmax without distillation: ImageNet-1K top-1 81.4 to 82.2, CIFAR-100 90.4 to 91.7 and Kinetics-200 83.0 to 84.4 2104.10972 tab. 2. The distillation code is not in the released repository, so these numbers cannot be reproduced from it. Card: ImageNet-21K Pretraining for the Masses.

Logit distillation from a ViT teacher after LLM-guided architecture search

The student is a five-stage CNN that a language model proposes and a feedback loop selects 2504.09685 §3.1 2504.09685 §3.2. Only the chosen network is distilled. It is trained for 50 epochs against a ViT-B/16 teacher at temperature T = 10, with a cross-entropy term and a KL term scaled by T² 2504.09685 §4.1.3 2504.09685 eq. 10 2504.09685 eq. 11. The paper does not give the optimiser, learning rate or batch size for this stage 2504.09685 §4.1.3. Its mixing weight α rises from 0.4 to 0.8 over the run 2504.09685 eq. 12. The prose describes the opposite direction, with the distillation weight rising 2504.09685 §3.3. See Contradictions. Author-reported, the same distillation step adds 1.32 points to LMaNet-Core, 1.40 to QwNet-Core and 1.35 to LMaNet-Elite on CIFAR-100 2504.09685 tab. 4. The paper gives no seed variance for these gains. The released code ranks candidates on test accuracy, so the headline 74.50% is not a held-out score; the card's Paper vs code section has the details. Card: Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?.

Teacher-compressed search space for battery-free microcontrollers

The paper keeps a neural architecture search (NAS) loop for a microcontroller that runs on harvested power, and changes its starting point. A large MCUNet teacher is distilled into a smaller student, which is pruned by more than 80% of its parameters, weight-shared and then shrunk along kernel, depth and channel dimensions before the search 2407.14514 §4.2. The teacher is MCUNetV1-Int2 in float16 2407.14514 §5.3. The paper does not give the distillation loss, temperature or training settings 2407.14514 §4.2. The search itself is the iNAS loop, with an RNN controller and a design explorer that handles power cycles 2407.14514 §4.1 2407.14514 fig. 6. Author-reported, the combined method beats HW-NAS by +19.1% on CIFAR-10 and +12.0% on Tiny ImageNet, with 1.33× and 1.27× the peak memory, and is 12% slower than iNAS on CIFAR-10 2407.14514 §5.4 2407.14514 §5.5 2407.14514 §5.6. The paper does not say whether the accuracy gains are percentage points, and the card records no result for the distillation step alone. Card: Accelerate Intermittent Deep Inference.

Theory: linear teacher and student trained on soft labels

A linear teacher and a linear student, with the student trained on the teacher's soft outputs under gradient flow. With at least as many transfer inputs as dimensions, the student's weight vector equals the teacher's 2105.13093 thm. 1. With fewer inputs, it converges to the projection of the teacher onto the span of the inputs 2105.13093 thm. 1. The expected transfer risk is bounded by a term that depends on the distribution of angles between the teacher and the inputs. That bound gives exponential rates for tasks with an angular margin and polynomial rates otherwise 2105.13093 thm. 3 2105.13093 cor. 1 2105.13093 cor. 2. The proofs cover linear models only, and the nonlinear case is left open 2105.13093 §6. The angle distribution is unknown in practice, and the paper gives no way to estimate it from data 2105.13093 §4.2. The statement and proof of the polynomial-rate corollary differ by a constant, and the source does not reconcile them 2105.13093 cor. 2. The experiments are synthetic or MNIST digits 0 and 1, and they do not state the learning rate or step count 2105.13093 §5. Card: Towards Understanding Knowledge Distillation.

Expert imitation with action and feature targets for driving

A single front-camera driving network learns from a reinforcement-learning expert that has privileged map and actor information 2206.08129 §3.1. The student is trained on the expert's actions, its latent features and its value estimate, not on the expert's output alone 2206.08129 §3.3. The trajectory target is the expert's waypoints 2206.08129 §3.1. The network has a waypoint branch and a control branch. The control branch attends over the image grid, guided by the waypoint branch, and the two outputs are blended at test time 2206.08129 §3.2 2206.08129 §3.4. Author- reported on the CARLA leaderboard: 69.714 driving score for one model and 75.137 for an ensemble, against 61.846 for LAV, which uses four cameras and a LiDAR 2206.08129 tab. 1. In the ablation, the control-only model scores 32.45, and fusion gives the largest single step, to 57.01 [arxiv:2206.08129 tab. 3]. The gain from guided attention (42.87 to 46.08) is smaller than the reported spread 2206.08129 tab. 3. The results come from one simulator and the paper's own leaderboard entries. The released repository's LICENSE (Apache-2.0) conflicts with the supplement's CC BY-NC 4.0 statement, and the repository has no TCP-SB or TCP-Ens model. Card: Trajectory-guided Control Prediction for End-to-end Autonomous Driving.

Teacher disparity for a lightweight depth network

A larger self-supervised depth network is trained first and serves as the teacher. A MobileNetV3 student is then trained to match the teacher's disparity map with a norm the paper does not state 2012.07356 §3.5 2012.07356 eq. 10. The teacher is not named, and the paper does not say whether the student also gets the self-supervised loss. Card: HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation.

Distilled models as reference data for membership inference

Here distillation is an auditing tool, not a compression step. Attack D trains several models on random population subsets, labels each subset with the target model's outputs, and uses their losses on the target record. The threshold is a smoothed percentile of those losses at a fixed false-positive rate 2111.09679 §4.4 2111.09679 app. B.3. The aim is to approximate leave-one-out retraining without the other training records 2111.09679 §4.4 2111.09679 §4.5. Card: Enhanced Membership Inference Attacks against Machine Learning Models.

Surveys and position papers

A review of knowledge distillation and student-teacher learning for vision sorts methods by teacher count, knowledge type, data availability, online and self distillation, and application 2004.05937 fig. 2. It has no experiments and no numeric results of its own, and it lists open questions 2004.05937 §4.1.1. Card: Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks.

A 2024 survey of small models lists distillation as one way LLMs help smaller models. It splits this into full data synthesis, rationale distillation, data augmentation, and representation distillation, which is logit-based or feature-based 2409.06857 §3.2. Its one experiment is a figure-only embedding comparison 2409.06857 §4.1. Card: Small Models in the LLM Era (survey).

A position paper on agents includes distilling from the LLM as an optional step in its conversion plan, next to LoRA or full fine-tuning 2506.02153 §6. It has no experiments of its own. Its figures are cited from other work or are author estimates 2506.02153 §3.1 2506.02153 app. B. Card: Small Language Models are the Future of Agentic AI.

A PRISMA review of TinyML from the machine-learning side covers 172 eligible papers from 2018 to 2023 2309.11932 §3. It places knowledge distillation under model optimization, with response-, feature- and relation-based knowledge and offline, online and self-distillation schemes 2309.11932 §5.1.3. Its popularity table gives distillation the top label and 11 cited works in 2020–22 2309.11932 tab. 1. The paper does not define how the labels are assigned, so the table shows where the authors put their attention, not a measured adoption rate. Card: A Machine Learning-oriented Survey on Tiny Machine Learning.

A review of efficient neural networks for Arm Cortex-M microcontrollers describes distillation as a student trained on a loss that mixes its own task loss with a divergence from the teacher's softened outputs 2311.11883 §4.1 2311.11883 eq. 5. It runs no experiments. Its α and temperature are not stated, and the direction of the KL term is not specified 2311.11883 §4.1. It compares published flash size against accuracy rather than its own distillation runs 2311.11883 §6. Card: Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review.

A survey of edge large language models lists distillation as an offline compression step, with white-box methods that use the teacher's internal features and logits, and black-box methods that use only teacher outputs, often as chain-of-thought rationales 2410.11845 §3.3. Its distillation figures, such as a student of about 0.3B parameters from a teacher above 100B, are quoted from cited works 2410.11845 §3.3.2. The paper runs no experiments 2410.11845 §1 2410.11845 §7. Card: A Review on Edge Large Language Models: Design, Execution, and Applications.

A survey of lightweight transformers for edge devices gives a two-stage distillation recipe, general pre-training then task-specific, with a reported 1.5–2.5% gain over one stage, and a teacher-to-student size ratio of 4–6x 2601.03290 §VIII-G. The paper describes no experiment behind these ranges, and it gives no optimiser, learning rate or batch size 2601.03290 §VIII-G. Card: Lightweight Transformer Architectures for Edge Devices in Real-Time Applications.

When to use which

Implementation sketch

Common core: the student's loss is a supervised term, when labels exist, plus a distance to the teacher's output. The variants differ in the teacher output and the distance. The score-difference update for distribution matching is the exception, shown last.

# Shared training step: fixed or EMA teacher, trainable student
# [arxiv:2003.11539 §3.3] [arxiv:2004.13513 §3.1] [arxiv:2111.07832 §2.2]
def train_step(student, teacher, x, y, alpha, beta):
    # Teacher outputs carry no gradient
    with no_grad():
        t_out = teacher(x)
    s_out = student(x)
    # Supervised term, omitted when there are no labels
    # Weights alpha and beta are given in [arxiv:2003.11539 §4.1]
    loss = alpha * cross_entropy(s_out.logits, y)
    loss = loss + beta * distance(s_out, t_out)
    return loss

# Soft-label variant: KL to the teacher's predictive distribution
# [arxiv:2003.11539 §3.3]
def distance_soft(s_out, t_out):
    return kl_divergence(softmax(s_out.logits), softmax(t_out.logits))

# Relation variant: correlate the rows of the probability matrices
# [arxiv:2205.10536 §3.1] [arxiv:2205.10536 §3.2]
def distance_relation(s_logits, t_logits):
    s_p = softmax(s_logits, dim=1)   # (B, C)
    t_p = softmax(t_logits, dim=1)
    # Across classes, one correlation per instance  [arxiv:2205.10536 eq. 8]
    inter = 1 - mean(pearson(s_p, t_p))
    # Across instances, one correlation per class  [arxiv:2205.10536 eq. 9]
    intra = 1 - mean(pearson(s_p.T, t_p.T))
    # Weighted in the total loss  [arxiv:2205.10536 eq. 10]
    return inter, intra

# Pooled-activation variant: width and height sums, normalised
# [arxiv:2004.13513 §3.1] [arxiv:2004.13513 §6.2]
def distance_pooled(s_map, t_map):
    # Square before pooling, then sum along each axis separately
    s_sq, t_sq = s_map ** 2, t_map ** 2
    s_vec = concat(sum(s_sq, axis=W), sum(s_sq, axis=H))
    t_vec = concat(sum(t_sq, axis=W), sum(t_sq, axis=H))
    # Normalise the pooled vectors before comparing
    return squared_norm(l2_normalize(s_vec) - l2_normalize(t_vec))

# Momentum teacher: exponential moving average of the student
# [arxiv:2111.07832 §2.2]
def update_teacher(teacher, student, momentum):
    for t, s in zip(teacher.params, student.params):
        # The momentum schedule is not fixed in this sketch
        t.data = momentum * t.data + (1 - momentum) * s.data

# Density variant: match opacity and colour at random points in one cell
# [arxiv:2103.13744 §3.3]
def distill_cell(student, teacher, cell, iters):
    for step in range(iters):
        # Random 3D point inside the cell, random view direction
        x = sample_in_cell(cell)
        d = sample_unit_sphere()
        # Density to opacity needs a spacing delta, not given in the paper
        a_t = opacity(teacher.sigma(x), delta)
        a_s = opacity(student.sigma(x), delta)
        rgb_t, rgb_s = teacher.rgb(x, d), student.rgb(x, d)
        loss = mse(a_s, a_t) + mse(rgb_s, rgb_t)
        update(student, loss)

# Offline variant: sparse teacher targets stored before training
# [arxiv:2207.10666 §3.1] [arxiv:2207.10666 eq. 2]
def distance_offline_topk(s_logits, values, indices, C):
    # Stored probabilities: K values and K class indices per view
    k = indices.shape[-1]
    # Spread the mass outside the top K evenly over the other classes
    # (the top-5 language-model variant sets it to zero instead)
    rest = (1 - sum(values, axis=-1, keepdims=True)) / (C - k)
    t = scatter(broadcast(rest, C), indices, values)
    # Soft cross-entropy against the stored target; teacher not run
    return -sum(t * log_softmax(s_logits), axis=-1)

# Distribution-matching step for a one-pass generator  [§3.2] [arxiv:2412.07772 alg. 1]
def dmd_step(student, teacher_score, fake_score, x_clean, schedule):
    # Student produces a clean estimate from a noised input
    x_hat = student(add_noise(x_clean, sample_step(schedule)))
    # Re-noise the estimate at a random timestep
    t = sample_timestep()
    x_t = add_noise(x_hat, t)
    # Update direction: fake score minus the teacher's guided score
    grad = fake_score(x_t, t) - teacher_score(x_t, t)
    # Only the student moves along this direction; the teacher is frozen
    update(student, mse(x_hat, (x_hat - grad).detach()))
    # The fake score learns the denoising loss on the student's own outputs
    fresh = add_noise(x_hat.detach(), sample_timestep())
    update(fake_score, denoising_loss(fake_score, fresh))

Contradictions

Several results point in different directions, and each is stated with its source. No direct conflict on the same claim across the cards, except where noted; the last item is within one card.

• Teacher temperature. DIST's default KD temperature is given as τ=4, but the ImageNet runs use τ=1, and the detection table labels KD as τ=1 2205.10536 §4.1 2205.10536 tab. 6. Compare KD baselines only at the same temperature.
• Mixed gains. On Meta-Dataset the distilled linear probe scores lower than the plain one on five of ten domains, for example Quick Draw at 60.83 against 62.48 2003.11539 §5. In PODNet's pooling study, POD-pixels (49.74) and perceptual-style features (51.01) score below the no-distillation baseline (53.29), while GradCam maps (54.13) are slightly above it 2004.13513 tab. 3.
• Pooling for segmentation. PODNet's classification ablation favours the POD-spatial term 2004.13513 tab. 3. PLOP argues that plain pooling blurs small objects in segmentation and adds local grids for that reason 2011.11390 §3.2. These are different tasks, not a direct contradiction.
• Ground-truth term. Several variants here add a label loss to the distillation term 2003.11539 §3.3 2205.10536 §3.2. TinyViT reports that adding ground-truth labels to its distillation loss lowered ImageNet-1k top-1 by 0.2 points with the CLIP teacher and 0.7 with Florence 2207.10666 app. C. The settings differ, so the results are not directly comparable, and the paper gives this as its reason for a label-free loss.
• Logit vs feature targets. In PLOP's ablation with the same pseudo-label loss, logit-level KD reaches 23.69 on all classes, POD reaches 34.62 and Local POD reaches 52.31 2011.11390 tab. ablation_classif_distillation. The ablation is on a validation split, not the main protocol.
• Student against teacher. In the one-step diffusion case the student stays well behind its teacher, with FID 9.36 against 4.16 2101.02388 tab. 1. In the video case the four-step student scores above its own many-step bidirectional teacher on the paper's ablation: 94.7, 64.4 and 30.1 against 94.6, 62.7 and 29.6 on temporal quality, frame quality and text alignment 2412.07772 tab. 4. The metrics and settings differ, so the two results are not comparable.
• Mass outside the top K. TinyViT spreads the probability outside its top K evenly over the other classes 2207.10666 eq. 2. The top-5 language-model variant sets it to zero, and its five probabilities usually sum to close to one 2301.12726 §3. Gemma 2 matches the full vocabulary 2408.00118 §3.2, and MedGemma 1.5 keeps 256 teacher logits per token 2604.05081 §2.2, which does not say how the rest is treated. Decide which convention to use before comparing numbers across these two pages.
• Live or stored teacher. The token-labeling paper says distillation needs a teacher running during training 2104.10858 §3.2. TinyViT stores the teacher's top-K outputs before training 2207.10666 §3.1, and the causal video student queries its teacher during training [arxiv:2412.07772 §3.2], and RLDG's generalist never queries its RL experts, learning only from stored rollouts 2412.09858 §3.2. "Distillation" covers all of these practices here, so say which one a source means.
• UniME's gain over VLM2Vec. The text reports a 10.3% gain with LLaVA-1.6 2504.17432 §4.3. Table 1 gives 66.6 against 63.3, a 3.3-point gap 2504.17432 tab. 1. Both are author-reported and the page does not choose between them.
• FedDF against FedGen. FedGen's table places FedDFusion, which the FedGen card ties to the ensemble distillation of 2006.07242 §3, above FedGen at EMNIST α = 0.05 (70.40 against 68.53), and below it at MNIST α = 0.05 (90.02 against 91.30) 2105.10056 tab. 1. The FedDF card reports rounds to a target accuracy on CIFAR-10 2006.07242 tab. 1, not the same metric, so the two papers' rankings are not directly comparable.
• The KD baseline changes the DKD gain. On ImageNet, ResNet-34 to ResNet-18, DKD is +1.04 top-1 over the KD row copied from other work (70.66), but +0.67 over the authors' own KD run (71.03) 2203.08679 tab. 8.
• Logit-only results vary by task. DKD alone is below ReviewKD on MS-COCO for ResNet-101 to ResNet-18 (35.05 against 36.75 AP) 2203.08679 tab. 10. On ImageNet it is 0.09 ahead for ResNet-34 to ResNet-18 (71.70 against 71.61) 2203.08679 tab. 8, and 0.51 behind for ResNet-50 to MobileNet-V1 (72.05 against 72.56) 2203.08679 tab. 9.
• Distance in VideoMamba's self-distillation. The paper calls the alignment an L2 loss 2403.06977 §4.1. The card's Paper vs code section records that the released code uses the squared distance between L2-normalised features, which equals 2 − 2·cos for unit vectors. Check which form an implementation needs before comparing scales (VideoMamba).
• Distillation weighting in ADD. The exponential weighting gives the best FID in the ablation, 20.6, while NFSD gives the higher CLIP score, 0.327 against 0.319 2311.17042 tab. 1. The paper's final model uses NFSD and its ablation default is the exponential weighting 2311.17042 §4.1. The NFSD formula is not given in the paper, so the choice cannot be reproduced from the paper alone.
• Regression term in one-step distribution matching. DMD reports that dropping its LPIPS regression term raises CIFAR-10 FID from 2.66 to 5.58 2311.18828 tab. 2. ADD's objective has no regression term 2311.17042 eq. 1, and its ablations do not test one 2311.17042 tab. 1. The datasets and losses differ, so the two results are not directly comparable.
• Weight direction in one LLM-guided search. The paper's prose says cross-entropy dominates early and the distillation weight rises 2504.09685 §3.3. Equation 11 with α from 0.4 to 0.8 does the reverse 2504.09685 eq. 11 2504.09685 eq. 12. The paper does not resolve this, so an implementation should follow the equations and say so.

Papers

• Rethinking Few-Shot Image Classification: soft-label self-distillation on top of a frozen embedding, with reported few-shot gains.
• Knowledge Distillation from A Stronger Teacher: relation-based logit distillation with Pearson correlation, for strong teachers.
• PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning: pooled-activation distillation between old and new model for class-incremental learning.
• PLOP: Learning without Forgetting for Continual Semantic Segmentation: multi-scale pooled feature distillation with confident pseudo-labels for continual segmentation.
• Student-Teacher Feature Pyramid Matching for Anomaly Detection: feature-pyramid distillation trained on normal images for anomaly detection.
• TinyViT: Fast Pretraining Distillation for Small Vision Transformers: offline top-K teacher labels with seed replay for pretraining small hierarchical transformers.
• iBOT: Image BERT Pre-Training with Online Tokenizer: EMA-teacher self-distillation for masked patch prediction in Vision Transformers.
• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs: teacher-to-cell distillation of NeRF density and colour, followed by fine-tuning.
• Rethinking Vision Transformers for MobileNet Size and Speed: hard distillation as one ingredient of an efficient hybrid transformer recipe.
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed: one-step student regressed onto a deterministic DDIM teacher's outputs.
• Specializing Smaller Language Models towards Multi-Step Reasoning: top-5 distribution matching of chain-of-thought maths solutions into small T5 and FlanT5 students.
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models: causal few-step video student trained with distribution matching from a bidirectional teacher.
• Text Embeddings by Weakly-Supervised Contrastive Pre-training: contrastive pre-training of E5, then optional cross-encoder distillation during fine-tuning.
• Unmasking On-Policy Distillation: training-free per-token diagnostic of whether on-policy distillation gradients point toward success.
• All Tokens Matter: Token Labeling for Training Better Vision Transformers: dense per-patch labels from an offline annotator, as an alternative to distillation with a live teacher.
• Preservation of the Global Knowledge by Not-True Distillation in Federated Learning: distillation from the frozen global model restricted to the not-true classes, for non-IID federated training.
• Revisiting Relation Extraction in the era of Large Language Models: GPT-3 explanations used as fine-tuning targets for a Flan-T5 Large relation extractor.
• Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs: text-only batch similarity distillation as stage 1 of a multimodal LLM embedder.
• Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model: distilled few-step causal generator for interactive game video, with a DMD-based long-video stage.
• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora: shared-budget challenge report; teacher–student distillation is one of the approaches it groups with the strongest gains, without per-approach numbers.
• RepViT: Revisiting Mobile CNN From ViT Perspective: mobile CNN whose headline numbers use distillation from a RegNetY teacher; distillation gap reported against the same models without it.
• Once-for-All: Train One Network and Specialize it for Efficient Deployment: elastic super-network trained with soft labels from its largest sub-network, then specialised without retraining.
• Federated Class-Incremental Learning: GLFC, with a KL relation-distillation term against the stored old model and server-side prototype reconstruction.
• Ensemble Distillation for Robust Model Fusion in Federated Learning: server-side distillation of averaged client logits into the fused model, in place of weight averaging.
• Emerging Properties in Self-Supervised Vision Transformers: label-free self-distillation with an EMA teacher, centring and sharpening (DINO).
• Data-Free Knowledge Distillation for Heterogeneous Federated Learning: generator trained on client prediction layers, with its samples used as a regulariser in local training (FedGen).
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation: momentum distillation from an EMA teacher for the contrastive and masked-language losses (ALBEF).
• Contrastive Learning with Stronger Augmentations: divergence loss matching a strong view's similarity distribution to its weak twin's, added to MoCo-style contrastive pretraining.
• Decoupled Knowledge Distillation: logit loss split into target-class and non-target terms, with separate weights.
• One-step Diffusion with Distribution Matching Distillation: one-step image generator trained to match the teacher's distribution, with a fake-score network and a regression term (DMD).
• M3-Embedding: self-knowledge distillation from the combined score of dense, sparse and multi-vector heads.
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context: dense Flash model described as online-distilled from Pro; no loss or schedule given.
• Gemma 2: Improving Open Language Models at a Practical Size: 2B and 9B models trained by matching a larger teacher's full next-token distribution.
• Small Models in the LLM Era (survey): survey placing distillation among small-model and LLM relationships; no method of its own.
• Small Language Models are the Future of Agentic AI: position paper with distillation from the LLM as an optional conversion step; no experiments.
• MedGemma 1.5 Technical Report: post-training by distilling 256 teacher logits per token, with supplementary teachers.
• Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields: per-Gaussian features distilled from LSeg or SAM by splatting, with an optional channel speed-up.
• VideoMamba: State Space Model for Efficient Video Understanding: self-distillation from a smaller trained video model, plus masked alignment to CLIP ViT-B features.
• Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation: English sentence encoder copied into a multilingual student by regressing translation pairs onto its vectors.
• Rethinking Spatial Dimensions of Vision Transformers: pooling-based ViT whose distillation token is trained against a RegNetY-16GF teacher, with the gain reported per recipe.
• ImageNet-21K Pretraining for the Masses: semantic-softmax pretraining on a cleaned ImageNet-21K, with a per-depth distillation term whose code is not released.
• Learning Generalizable Feature Fields for Mobile Manipulation: feed-forward feature field with a frozen 2D teacher's features appended to its latent, for navigation and grasping.
• RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning: per-task RL experts' successful rollouts used as fine-tuning data for OpenVLA and Octo.
• Adversarial Diffusion Distillation: one-to-four-step diffusion student trained with a hinge-loss discriminator and a frozen diffusion teacher's score distillation.
• Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks: survey of knowledge distillation and student-teacher learning for vision; no experiments.
• DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Models: teacher splits and prunes the student's own reasoning traces, which are then used for LoRA fine-tuning.
• KL for a KL: On-Policy Distillation with Control Variate Baseline: detached negative reverse-KL baseline for on-policy distillation, with a cheaper top-k version.
• Can LLMs Revolutionize the Design of Explainable and Efficient TinyML Models?: LLM-proposed microcontroller CNNs, filtered by MAC and SRAM estimates, then distilled from a ViT-B/16 teacher.
• Accelerate Intermittent Deep Inference: distilled and shrunk MCUNet teacher as the starting space for intermittent-aware NAS on about 2 KB of SRAM.
• A Machine Learning-oriented Survey on Tiny Machine Learning: PRISMA review of TinyML with distillation as one model-optimization technique; no experiments.
• Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review: review of distillation, pruning, quantization and tooling for Cortex-M microcontrollers; no new experiments.
• A Review on Edge Large Language Models: Design, Execution, and Applications: edge-LLM survey placing white-box and black-box distillation among offline compression methods; no experiments.
• Lightweight Transformer Architectures for Edge Devices in Real-Time Applications: transformer survey with a two-stage distillation recipe whose ranges have no described experiment.
• Towards Understanding Knowledge Distillation: linear teacher and student with soft labels; recovery and transfer-risk bounds, with synthetic and MNIST experiments.
• Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline: single-camera driving network that learns from a privileged RL expert's actions and features, with trajectory-guided attention.
• HR-Depth: High Resolution Self-Supervised Monocular Depth Estimation: lightweight MobileNetV3 depth student trained to match the disparity of a larger self-supervised model.
• Enhanced Membership Inference Attacks against Machine Learning Models: membership-inference attack whose per-record threshold comes from distilled models labelled by the target (Attack D).