Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

In plain words

Large neural networks are accurate but heavy, and they usually need a lot of labelled data. Knowledge distillation trains a small "student" network to copy a large "teacher" network, instead of learning only from the correct answers. The teacher's softened output probabilities also say which wrong classes look similar to the right one, and that extra signal helps the student.

This survey collects the many ways researchers have done this for computer vision. It groups methods by what gets copied (final outputs, middle-layer features, or relations between samples), how many teachers are used, whether labels or data are available, and whether a network teaches itself. It also covers use in segmentation, detection, domain adaptation and video.

The survey is a map of the field, not a benchmark. It reports no new experiments, and it says that open questions remain about why distillation works.

TL;DR

• A review, not a new method. It covers knowledge distillation (KD) and student-teacher (S-T) learning, mainly for visual tasks §1.
• The core KD loss is a weighted sum of a ground-truth term and a term that matches the teacher's temperature-softened outputs eq. 2.
• Methods are organised by teacher count, knowledge type (logits or features), data format, online and self distillation, learning metric, label use and application fig. 2.
• No code and no experiments of its own. Use it for the landscape and the open questions, not for numbers.

Problem

Deep models have millions or billions of parameters, which demands heavy computation and blocks deployment on edge devices. Performance also depends heavily on large amounts of labelled data §1 abstract. KD transfers what one model has learned to another, to compress models and to reduce the need for data §1.

Two limits the survey raises:

• When the teacher's softmax is almost one-hot, its soft targets add little beyond the ground-truth label §2.
• The idea that "any student can learn from any teacher" is hard to achieve on some datasets, such as ImageNet, even with regularisation or strictness techniques. This holds when the student has low capacity §4.1.1, open challenges.

Logit-based KD is limited to the softmax loss and depends on the number of classes. It cannot be applied when there are no labels, for example in low-level vision §4.1.1, summary.

Method

The survey defines the S-T framework and the KD loss, then sorts the literature. There is no single method to implement.

Soft targets and temperature. Given logits z from a network, the survey defines the class probability with a temperature \rho eq. 1. With \rho = 1 this is the standard softmax. A larger \rho gives a softer distribution, and the survey calls the information in it "dark knowledge" §2. The student's logits are softened with the same \rho when computing the distillation term §2.

p_i = \frac{\exp(z_i / \rho)}{\sum_j \exp(z_j / \rho)} eq. 1 (the typeset sum index is corrected here; see Pitfalls)

Overall KD loss. The survey recommends training the student on both the ground-truth labels and the teacher's soft labels §2. The total loss is a student term and a distillation term eq. 2:

\mathcal{L}_{KD} = \alpha\, H(y, \sigma(z_s)) + \beta\, H(\sigma(z_t; \rho),\ \sigma(z_s; \rho)) eq. 2

• H is a loss function, y is the ground-truth label, \sigma is the softmax with temperature \rho, \rho \neq 1 for the distillation term.
• \alpha and \beta are coefficients. z_s and z_t are the student and teacher logits §2.
• The survey expands the distillation term as a KL divergence between the softened distributions plus the teacher's entropy eq. 2.

# Logit-based KD step, as the survey writes it  [eq. 1] [eq. 2]
# Input:  images x, labels y, frozen teacher, student, alpha, beta, rho
# Output: scalar loss L_KD

def distillation_loss(x, y, teacher, student, alpha, beta, rho):
    # 1. Compute both sets of logits for the batch  [§2]
    teacher_logits = teacher(x)                    # (B, C)
    student_logits = student(x)                    # (B, C)

    # 2. Ground-truth term at temperature 1  [§2] [eq. 2]
    student_loss = cross_entropy(softmax(student_logits, 1.0), y)

    # 3. Soft-target term; both sides use the same rho  [§2] [eq. 1]
    teacher_soft = softmax(teacher_logits, rho)    # (B, C)
    student_soft = softmax(student_logits, rho)    # (B, C)
    distill_loss = cross_entropy(student_soft, teacher_soft)

    # 4. Weighted sum  [§2] [eq. 2]
    return alpha * student_loss + beta * distill_loss

Feature hints. Feature-based distillation matches a teacher layer (the hint) to a student layer (the guided layer). Each side may pass through a transform function, and a distance function compares them §4.1.2:

\mathcal{L}(F_T, F_S) = D\big(TF_t(F_T),\ TF_s(F_S)\big) eq. 10

• F_T and F_S are the selected teacher and student layers. TF_t and TF_s are the transform (regressor) functions. D is a distance, for example the \ell_2 norm §4.1.2.
• Choosing, transforming and matching the layers is the main design question. Hint layers act as a regulariser, so a pair must be chosen so that the student is not over-regularised §4.1.2.

Variants of the logit loss. The survey writes four variants, each a modification of eq. 2:

• Distilling from the teacher's output at iteration l into the student's output at iteration l-1, so the teacher is a previous state of the same network (one-generation distillation) eq. 6, §4.1.1.
• Adding adversarial perturbations to the inputs, so the student keeps robustness eq. 7, §4.1.1.
• Adding a margin term that asks the teacher to rank the other classes, from the "strictness" group eq. 8, §4.1.1.
• Averaging the loss over several exits or ensemble members eq. 9, §4.1.1.

Taxonomy (sections of the survey).

• §4 KD by number of teachers: one teacher (logits and intermediate layers, §4.1) and multiple teachers (ensembles of logits or features, unifying data sources, one teacher to several sub-teachers, heterogeneous teachers, mutual learning among peers; §4.2) fig. 4 tab. 3.
• §5 KD by data format: data-free distillation via metadata, class similarity and generators tab. 4; few-sample distillation via pseudo examples and layer-wise estimation (§5.2); cross-modal distillation, supervised and unsupervised fig. 5 tab. 5.
• §6 Online and teacher-free distillation: individual student peers, sharing blocks among peers, ensembles of peers (§6.1) fig. 6; self-distillation via born-again networks, deep supervision, data augmentation and architecture transformation (§6.2) fig. 7 tab. 6.
• §7 Label-required and label-free distillation: original labels versus pseudo labels (§7.1), and label-free settings such as cross-modal learning (§7.2).
• §8 Novel learning metrics: adversarial learning fig. 8 tab. 7; graph representations fig. 10 tab. 9; semi- and self-supervised learning; few-shot learning; incremental learning; reinforcement learning (§8.1–§8.6).
• §9 Applications: segmentation and motion (§9.1), detection and tracking (§9.2), domain adaptation (§9.3), depth and scene flow (§9.4), image translation (§9.5), video (§9.6).
• §10 Discussion and §11 new outlooks: open questions and future directions.

Theory. The survey argues that KD can be read as maximising mutual information between teacher and student representations §3, eq. 3. It derives a lower bound on that quantity, assuming a sufficiently expressive model q of the conditional distribution eq. 4, eq. 5. The survey presents this as a sketch of intuition and says there is no commonly agreed theory of KD §3.

Hyperparameters

The survey is not an experiment. The table lists the symbols that appear in the losses it writes out; the paper gives no values.

Results

No experiments of its own. The survey reports no benchmark numbers. Its tables classify methods by mechanism and are not results tables. They are:

Table 8 is a notation summary for §8, not a result.

Paper vs code

No official code. I checked the paper's links and its bibliography. The three GitHub links are awesome-list repositories and a documentation page: FLHonker/Awesome-Knowledge-Distillation, dkozlov/awesome-knowledge-distillation and the Intel Distiller docs page, which the survey cites for the temperature formula §2. None is the authors' code, so none is recorded. A GitHub repository search for the survey title returned no results. The paper has no method name, so there was nothing else to search for.

Pitfalls

• No numbers to reproduce. The survey is a review. Every number or claim about a method comes from the cited work, and this card has not checked those works.
• Typo in the softmax. The typeset denominator in eq. 1 is \sum_j \exp(z_i/\rho), with the summand written in z_i. The sum must range over z_j. The card uses the corrected form.
• Typeset parenthesis in eq. 2. The first term has an unclosed parenthesis. The survey writes the distillation term as a KL divergence plus teacher entropy, so H is read here as cross-entropy. The survey does not state a \rho^2 scaling factor on the distillation term, and the card does not add one. Check the source paper of any method before copying a scale.
• Undefined values. \alpha, \beta, \rho, \tau and K have no values anywhere in the survey (see Hyperparameters).
• Teacher-free and self-distillation. The survey says online self-distillation converges to flat minima, avoids vanishing gradients and yields more discriminative features §10. These are the authors' reading of cited work, not results they measured.
• Bigger teachers can be worse. The survey cites the finding that a larger, more accurate teacher does not always give a better student, and that the student's accuracy can fall as teacher capacity grows §10, citing cho2019efficacy. Use the cited work for details.
• Theory gap. The bound in §3 assumes a sufficiently expressive q and gives no conditions for it. It is not a convergence or generalisation guarantee.
• Data-free KD. The survey says there is no theory for why data-free methods work and that they are mostly limited to classification, with generators that are hard to train and expensive §10.
• Dated. The text is the arXiv v7 version (TeX dated 2021-06-17). Later methods are not covered.

Lineage

• Builds on: Bucilua et al. (2006) for model compression, and Hinton et al. (2015) for softened targets and temperature §2. Romero et al. (2014, FitNets) for hint-based feature learning §4.1.2. Not linked as cards on this wiki.
• Compared against: not applicable; it is a survey.
• Superseded by: not stated.