Adaptive Multi-Teacher Multi-level Knowledge Distillation

In plain words

Knowledge distillation trains a small "student" network to copy a larger "teacher" network. This paper asks what happens when the student has several teachers. Giving every teacher the same say is a weak choice, because one teacher may be the better guide for one image and a poor guide for another. The authors let the student learn, for each image, how much to trust each teacher, and use those weights to blend the teachers' answers into one target.

The method also passes along information from inside the teachers, not only their final answers. Each teacher's deep features serve as a hint for one part of the student. This extends an earlier single-teacher hint method to several teachers. A further term asks the student to keep the angles formed by triples of examples close to the teachers' angles, which captures how the examples relate to one another. The teachers matter only during training, so the student is what gets deployed.

TL;DR

• A ResNet20 student is trained with several teachers at once. The weight of each teacher is set per image by a small learned adapter.
• Intermediate hints: each teacher supervises one group of student layers. An angle-based relational loss is added to the usual soft-target loss.
• Author-reported gains over the multi-teacher baseline AvgMKD, on CIFAR-10, CIFAR-100 and Tiny-ImageNet tab. 2 tab. 7.
• An official repository exists, but it differs from the paper in several loss terms, the pooling step and the training schedule. Implement from the paper only with the Paper vs code section next to you.

Problem

• Single-teacher distillation uses one teacher and so ignores the others §1 §2.
• Averaging several teachers treats each as equally important for every example, so per-example differences in teacher quality are lost §2.
• Feature hints in FitNet-style methods come from one teacher §1 §3.4.
• Multi-teacher work on multi-task or multi-domain settings assumes one teacher per task or domain. The paper targets one task in one domain with several teachers §2.
• Teachers are needed only during training and add nothing at deployment §1.

Method

The student is trained with the sum of three losses: a standard distillation loss with integrated soft targets, an angle-based loss, and a multi-group hint loss §3.1 eq. 3 fig. 2.

Adaptive teacher weights §3.2. Each teacher t has a learned latent vector theta_t of size d. The student's last convolutional map for image i is max-pooled over its full spatial extent, giving a vector delta_i of size C (the paper sets d = C, e.g. 128) §3.2 eq. 4. A learned global vector nu scores each teacher by an element-wise product with delta_i, and a softmax over teachers gives the weights eq. 5 eq. 6. The integrated soft target is the weighted sum of the teachers' soft targets eq. 7.

High-level knowledge §3.3.1. The standard distillation loss uses the integrated target in place of a single teacher's target eq. 8. Soft targets use temperature T = 5.0 §4.3.

Structural knowledge §3.3.2. For a triplet of images (i, j, k), the angle at x_j is measured as the inner product of the two normalised difference vectors eq. 9. The loss is a Huber loss between teacher and student angles, summed over triplets. The paper does not say which triplets eq. 10.

Multi-group hint §3.4. Each teacher is assigned to one group of student layers through a fixed one-to-one mapping f. The teacher's last feature map is matched to the student group's output through a single FitNet-style regression layer per teacher eq. 11. The paper's default mapping gives the better teachers to the higher student groups, and it reports only a small gain over the other two mappings it tried §3.4.

Key equations, as in the paper:

\mathcal{L} = \mathcal{L}_{KD} + \alpha\,\mathcal{L}_{Angle} + \beta\,\mathcal{L}_{HT} eq. 3

\delta_i = \operatorname{MaxPooling}(\mathbf{B}_i, s), \quad s = H \times W eq. 4

\gamma_{t,i} = \boldsymbol{\nu}^{\mathrm{T}}\left(\boldsymbol{\theta}_t \odot \boldsymbol{\delta}_i\right), \qquad w_{t,i} = \frac{\exp(\gamma_{t,i})}{\sum_{t'=1}^{m}\exp(\gamma_{t',i})} eq. 5 eq. 6

\tilde{\mathbf{y}}^T_i = \sum_{t=1}^{m} w_{t,i}\,\tilde{\mathbf{y}}^T_{t,i} eq. 7

\mathcal{L}_{KD} = \sum_i \mathcal{H}(\mathbf{y}_i, \mathbf{y}_i^S) + \lambda \sum_i \mathcal{D}_{KL}(\tilde{\mathbf{y}}^T_i, \tilde{\mathbf{y}}^S_i) eq. 8

\Delta(x_i, x_j, x_k) = \langle \mathbf{e}^{ij}, \mathbf{e}^{kj} \rangle, \quad \mathbf{e}^{ij} = \frac{x_i - x_j}{\lVert x_i - x_j \rVert_2} eq. 9

\mathcal{L}_{Angle} = \sum_{(i,j,k)} l_\delta\big(\Delta(\tilde{\mathbf{y}}^T_i, \tilde{\mathbf{y}}^T_j, \tilde{\mathbf{y}}^T_k),\ \Delta(\tilde{\mathbf{y}}^S_i, \tilde{\mathbf{y}}^S_j, \tilde{\mathbf{y}}^S_k)\big) eq. 10

\mathcal{L}_{HT} = \sum_{t} \lVert \mathbf{u}_t - \mathbf{F}_t(\mathbf{v}_{f(t)}) \rVert^2 eq. 11

Pseudocode, written from the equations above. The paper gives no algorithm box. Where the code resolves a point the paper leaves open, see Paper vs code.

# Inputs: images (B, 3, H, W), labels (B,), m frozen teachers
# Output: scalar loss for the student (paper's full objective, eq. 3)

def amtml_loss(images, labels):
    # 1. Student forward: layer maps, pooled feature, logits  [§3.2] [§3.4]
    maps, pooled, logits = student(images)  # pooled: (B, C)

    # 2. Teacher soft-targets at temperature T, no gradient  [§4.3]
    soft_targets = []
    for teacher in teachers:
        soft_targets.append(softmax(teacher(images) / T))
    soft_targets = stack(soft_targets, dim=1)  # (B, m, K)

    # 3. Instance-level teacher weights from the adapter  [eq. 5] [eq. 6]
    scores = []
    for t in range(m):
        scores.append(dot(nu, theta[t] * pooled))  # (B,)
    weights = softmax(stack(scores, dim=1), dim=1)  # (B, m)

    # 4. Integrated soft-target: weighted sum over teachers  [eq. 7]
    integrated = (weights[:, :, None] * soft_targets).sum(dim=1)  # (B, K)

    # 5. Standard KD: hard-label loss plus KL to integrated target  [eq. 8]
    # Student soft-target temperature is not stated; T is assumed here.
    student_soft = softmax(logits / T)
    hard_loss = cross_entropy(logits, labels)
    kd_loss = hard_loss + lam * kl(integrated, student_soft)

    # 6. Huber loss between teacher and student triplet angles  [eq. 10]
    # angles(x)[i, j, k] = cosine of the angle at x_j  [eq. 9]
    angle_loss = huber(angles(integrated), angles(student_soft))

    # 7. Multi-group hint: teacher t to its assigned student group  [eq. 11]
    hint_loss = 0
    for t in range(m):
        predicted = regressors[t](maps[group_of[t]])
        hint_loss = hint_loss + squared_norm(teacher_feats[t] - predicted)

    # 8. Combine with the paper's weights (alpha = 1, beta = 2)  [eq. 3] [§4.3]
    return kd_loss + alpha * angle_loss + beta * hint_loss

Hyperparameters

Results

Author-reported. Setup: CIFAR-10 and CIFAR-100, each with 50,000 training and 10,000 test images §4.1.1. Tiny-ImageNet with 200 classes at 64×64 §4.1.2. Metric: top-1 accuracy §5. Baselines: OKD, FitNet, RKD, AvgMKD, and DML with three students §4.2. The text reports no error bars and no repeated runs.

Model sizes tab. 1, tab. 6. Compression rate is relative to VGG-19 §5.1.1.

Student Stu1 (ResNet20) on CIFAR, single-teacher and multi-teacher methods tab. 2:

Three students on CIFAR, multi-teacher methods tab. 3:

Number of teachers, CIFAR-10 tab. 4:

Cost, ResNet20 on CIFAR-100 tab. 5:

Tiny-ImageNet teacher and student accuracies tab. 6:

Tiny-ImageNet, Stu1 tab. 7:

Tiny-ImageNet, three students tab. 8:

Ablation, CIFAR-100 and Tiny-ImageNet tab. 9:

Figures, described in words. fig. 1 shows one image (an orchid) with soft targets from three teachers over four similar classes; the first teacher's targets carry the most information and receive the largest weight §3.1. fig. 2 gives the overview: teacher soft targets and layer representations feed the adapter, which weights the teachers for the KD and angle losses, while a multi-group hint loss transfers intermediate features §3.2. fig. 3 compares multi-teacher methods on CIFAR-100 as the number of teachers changes; its values are only in the figure. fig. 4 shows the learned teacher weights for example images from CIFAR-10 and MNIST, and states that a wrong teacher can be outweighed.

Paper vs code

Repository: https://github.com/FLHonker/AMTML-KD-code, MIT licence, cloned at commit 926ab3c65e995cd97dc98bfcdb8c1bc62994329c. The paper has no code link. The arXiv abstract and the LaTeX source carry no link either. A GitHub search for "AMTML-KD" returned this repository. Its README names all three authors and links the first author's homepage (flhonker.github.io) README.md, which ties it to the authors. The README also points to arXiv 2012.00573 as the paper PDF and cites the Neurocomputing 2020 version README.md. This card covers 2103.04062 v1, so the arXiv identifiers differ.

The repository is a set of notebooks, not a script. LR_adaptive_learning.ipynb is the most complete variant. It has the adaptive weights, the angle and distance losses, and a hint loss in the first stage only. LR_adaptive_Hint.ipynb is a second variant with pooled teacher hints and no angle loss. Other notebooks cover the single-teacher and multi-student baselines. Line numbers in the notebooks refer to the raw JSON file.

Discrepancies:

• Teachers. The paper's CIFAR teachers are ResNet110, VGG-19 and DenseNet121 §5.1.1. The code's default teachers are ResNet32, ResNet56 and ResNet110, and VGG-19 is not one of the selectable teachers LR_adaptive_learning.ipynb.
• Distillation loss form. The paper uses cross-entropy plus λ times KL eq. 8. The code uses 2αT² times KL plus (1−α) times cross-entropy, with α = 0.7 LR_adaptive_learning.ipynb. Cross-entropy is also computed on temperature-scaled logits LR_adaptive_learning.ipynb.
• Reduction. The paper's sums over images eq. 8. The code uses PyTorch's default mean reduction for KL and cross-entropy LR_adaptive_learning.ipynb, so the relative size of the terms changes.
• Angle loss input. The paper applies it to student soft targets eq. 10. The code applies it to log-softmax of student logits divided by T LR_adaptive_learning.ipynb LR_adaptive_learning.ipynb, and to the same weighted teacher probabilities as the KD term.
• Distance loss. The code adds an RKD distance loss, with weight 1 and not in the paper LR_adaptive_learning.ipynb loss.py. The hint-variant notebook comments it out LR_adaptive_Hint.ipynb.
• Triplets. The angle loss uses every triplet in the batch, which means N³ angles for a batch of N loss.py. The paper does not say which triplets it uses.
• Pooling. The paper uses max pooling over the full spatial map eq. 4. The student's pooled feature is an average pool over the last stage resnet20.py. The adapter's max-pool kernel is set from a dummy 128×128 input, where the pooled map is 1×1, so the max-pool has kernel size 1 and does nothing LR_adaptive_learning.ipynb LR_adaptive_learning.ipynb LR_adaptive_learning.ipynb. The inline comment says 8, which does not match this input. This is our reading of the shapes; we did not run the code.
• Hint features. The paper says the teacher's last feature map is matched eq. 11. The main notebook matches the teacher's second-stage map, 16×16, to the student's second stage, with three 1×1 regressors of width 32 LR_adaptive_learning.ipynb LR_adaptive_learning.ipynb. The hint notebook uses the teacher's pooled 1×1 feature LR_adaptive_Hint.ipynb. The regressor is a 1×1 convolution applied to the student's full map and compared with that vector, so the comparison broadcasts over the student's spatial positions loss.py.
• Group mapping. The hint notebook maps the teacher with index j to student stage j+1, and only for the first m-1 teachers LR_adaptive_Hint.ipynb. With the default teacher order, ResNet110 receives no hint at all in that notebook LR_adaptive_learning.ipynb. The paper's preferred mapping, better teachers to higher groups §3.4, is not implemented; the code uses the order of the list.
• Hint weight. The paper sets β = 2 §4.3. The notebooks add the hint loss with weight 1 LR_adaptive_learning.ipynb.
• Hint regressors are never trained. The regressor modules are not in any optimiser; the student optimiser covers only the student's parameters LR_adaptive_learning.ipynb. Their weights keep the uniform ±0.005 initialisation loss.py. Their gradients are computed but never applied. This is our reading.
• Adapter regularisation. The code adds 0.1 times the squared norms of ν and θ during adapter training LR_adaptive_learning.ipynb. The paper does not mention it.
• Teacher weights are not detached. adapter.eval() calls theta.detach() but discards the result LR_adaptive_learning.ipynb. The KD term's gradient therefore reaches the student's pooled features through the weights. This is our reading.
• Adapter is frozen after stage one. The main training loop steps only the student optimiser, so θ and ν stop updating LR_adaptive_learning.ipynb. The paper does not say.
• Student trains during stage one. The 80 adapter epochs also update the student with the hint loss included LR_adaptive_learning.ipynb LR_adaptive_learning.ipynb. The student's learning-rate schedule runs only in the main loop LR_adaptive_learning.ipynb. The paper does not describe a two-stage schedule in its active text.
• Hint loss in the main stage is disabled. The FitNet term is commented out in the main training loop LR_adaptive_learning.ipynb.
• Model selection. The paper tunes hyperparameters on validation sets §4.3. The code has no validation split and selects the best epoch on the test set LR_adaptive_learning.ipynb.
• Tiny-ImageNet. The paper reports Tiny-ImageNet results §5.2. The adaptive notebooks list only CIFAR-10 and CIFAR-100 as dataset choices LR_adaptive_learning.ipynb. Tiny-ImageNet appears in other notebooks (nets_training.ipynb, original_KD.ipynb) and a loader exists at dataset/tinyimagenet.py, but no adaptive Tiny-ImageNet run was found.
• Reported numbers. The saved outputs of the adaptive notebooks contain no final accuracy line, so the paper's tables cannot be matched to a run in the repository.
• Teacher training. Teacher checkpoints are not in the repository. train_resnet.py trains a ResNet20 with learning rate 0.1, weight decay 1e-4 and 200 epochs, with a schedule at 100 and 150 train_resnet.py train_resnet.py train_resnet.py. Schedules for the other teachers are not in the repository.
• Model size. The code's ResNet20 uses widths 16, 32 and 64 with three stages resnet20.py resnet20.py. We did not count parameters, so the paper's ~1.13M figure for Stu1 is unverified.
• Augmentation. The code adds random cropping, which the paper does not mention §4.1.2 LR_adaptive_learning.ipynb.

Pitfalls

• Teacher-to-layer mapping is ambiguous. The paper gives a default ("better teachers to higher groups") but no way to rank teachers. The code orders them by the --teachers list. The hint notebook ignores the last teacher, so a reimplementation following the paper's mapping will differ from both notebooks.
• Hint features are a design choice, not a given. The paper's last-feature hint, the main notebook's second-stage hint and the hint notebook's pooled hint are three different choices. The paper does not say which one its numbers use.
• Angle loss cost grows cubically. The angle loss builds N³ angles per batch. At batch size 128 that is 2,097,152 entries per term, for teacher and student each loss.py. Reduce the batch or restrict the triplets.
• Teacher forward passes are repeated. Teachers are run on every training batch with no cache, so training cost scales with the number of teachers. The paper's cost table gives AMTML-KD (3T) at 75.8 min against AvgMKD (3T) at 73.4 min on ResNet20 tab. 5.
• Numerically sensitive terms. Soft targets use T = 5. The code scales KL by T² times 2α, and its KL and cross-entropy use batch means. Both choices change the weight of the KD term relative to the angle and hint terms, which the paper's λ, α and β do not describe.
• Gradients into the weights. Since teacher weights depend on the student's pooled feature and are not detached, the KD term also updates the student through the weights. A version that detaches them would behave differently.
• Two-stage training. Stage one trains the student at a constant learning rate with hints for 80 epochs. Whether the reported numbers used this schedule is unknown; the paper's active text describes neither stage.
• Test-set selection may inflate reported numbers. Because the best epoch is picked on the test set, reported accuracies may be optimistic. Compare only against baselines run the same way.
• Small gains. Gaps between AMTML-KD and AvgMKD are 0.63 points on CIFAR-10 tab. 2 and 0.75 on CIFAR-100 tab. 2. The text reports no error bars or repeated runs.
• Teacher count. More teachers help, but the gain shrinks §5.1.3 tab. 4. The 5T setting adds VGG-19 and DenseNet121 to the 3T set tab. 4, which means more teacher forward passes per batch.

Lineage

• Builds on: knowledge distillation (Hinton et al., arXiv:1503.02531, per the paper's bibliography) §1; FitNets, Romero et al., ICLR 2015, for intermediate hints §1 §3.4; relational knowledge distillation, Park et al., CVPR 2019, for the angle loss §2 §3.3.2. None has a card in this wiki.
• Compared against: OKD, FitNet, RKD, AvgMKD (You et al., AAAI 2018, as cited in the paper), and DML (Zhang et al., CVPR 2018) §4.2.
• Superseded by: not known from content in this wiki.