Teacher-student learning

Overview

In a teacher-student setting, a teacher network with known architecture generates the labels, and a student network of the same shape is fitted to them. The question this page covers is whether training can recover the teacher's parameters, not just a student that predicts similarly 1706.03175 §3. The invariant the analysis relies on is noiseless labels from a hidden one-layer teacher with k units, on inputs x ~ N(0, I) in R^d with k ≤ d 1706.03175 §3.

Only one card in the corpus is filed under this slug so far. For using a teacher to train a smaller or otherwise cheaper model, see Knowledge distillation.

Variants

Parameter recovery for one-hidden-layer teachers

The teacher's labels are a weighted sum of activations of k first-layer projections 1706.03175 §3. Recovery has two stages: a tensor-based moment estimate lands the student near the teacher, then gradient descent on fresh samples refines it 1706.03175 §5, §6, alg. 1, alg. 2. The paper reports sample size roughly linear in d for a fixed accuracy, with polynomial factors in k, the condition number and other constants that it does not state explicitly 1706.03175 §5, §6. Card: Recovery Guarantees for One-hidden-layer Neural Networks.

Within this variant the activation class matters:

• Smooth, homogeneous activations (e.g. squared ReLU in the paper's experiments) get a linear-convergence result for gradient descent on fresh samples 1706.03175 §4, app. D.
• Piecewise-linear activations (ReLU, leaky ReLU) get the tensor initialisation and local Hessian bound, but not the linear-rate theorem 1706.03175 §4, §6.

When to use which

Implementation sketch

Tensor initialisation and gradient descent with fresh samples, following the structure the paper gives. Constants and sample splits are not stated in the paper and must be chosen by the implementer.

# Recovery of one-hidden-layer teacher weights  [arxiv:1706.03175 §6, alg. 2]
# Input:  samples S (linear in d), width k, number of steps T
# Output: W (d, k), v (k,)
def recover_teacher(samples, k, T):
    # Split into T + 1 disjoint subsets; the first one is for initialisation
    # [arxiv:1706.03175 alg. 2]
    parts = partition(samples, T + 1)
    # Tensor initialisation on the first subset  [arxiv:1706.03175 alg. 1]
    weights, v = tensor_initialise(parts[0], k)
    # Output weights are fixed after initialisation  [arxiv:1706.03175 alg. 2]
    for q in range(T):
        # Fresh batch every step (resampling)  [arxiv:1706.03175 §4, §6]
        grad = empirical_gradient(weights, v, parts[q + 1])  # (d, k)
        # Step size bound from the theorem; constant is not stated
        # [arxiv:1706.03175 §6]
        weights = weights - eta * grad
    return weights, v

Where variants differ: the initialisation uses a power-method subspace and a non-orthogonal tensor decomposition, and the choice of moment order depends on which empirical moments are non-zero [arxiv:1706.03175 alg. 1, alg. 4, app. E]. The paper does not say how that zero test is made on finite samples 1706.03175 alg. 4.

Contradictions

• Resampling and step size. The theory needs fresh samples at each step and a step size of at most 1/(k v_max² σ₁^{2p}) 1706.03175 §4, §6. The experiments use a fixed step of 0.02 with no resampling, and the paper says this "still works well" 1706.03175 §7.
• Homogeneity exponent. The body describes scaling by an exponent 1/p, with p the degree of homogeneity 1706.03175 §4. The appendix gives the degree as p + 1 and uses 1/(p+1) in the norm recovery formula 1706.03175 app. E, alg. 4. The two forms do not agree, and the paper does not say which is correct.
• Sample count. The sample bound is written with poly(·) factors in the informal theorem 1706.03175 §5, §6 and with explicit powers of d and k in the appendix proof 1706.03175 app. E. The exponents differ between these forms and are not reconciled.

Papers

• Recovery Guarantees for One-hidden-layer Neural Networks: tensor initialisation plus resampled gradient descent recovers a one-hidden-layer teacher's weights from about d·log(1/ε) samples, up to polynomial factors; theory with synthetic checks.