---
kind: concept
slug: teacher-student-learning
title: Teacher-student learning
summary: >-
  A fixed teacher model labels the data and a same-shape student is trained on it. Question: can the student recover the teacher's weights, not only its predictions? For one-hidden-layer nets with Gaussian inputs and noiseless labels, tensor initialisation plus gradient descent does so with samples roughly linear in input dimension. Trade-off: strong assumptions, theory only.
tags: [theory, learning-theory, parameter-recovery]
papers: ["1706.03175"]
related: [knowledge-distillation, spectral-methods, tensor-decomposition]
status: draft
updated_at: 2026-11-01
---

## Overview

In a teacher-student setting, a teacher network with known architecture
generates the labels, and a student network of the same shape is fitted to
them. The question this page covers is whether training can recover the
teacher's parameters, not just a student that predicts similarly
[arxiv:1706.03175 §3]. The invariant the analysis relies on is noiseless labels
from a hidden one-layer teacher with k units, on inputs x ~ N(0, I) in R^d
with k ≤ d [arxiv:1706.03175 §3].

Only one card in the corpus is filed under this slug so far. For using a
teacher to train a smaller or otherwise cheaper model, see
[Knowledge distillation](/concepts/knowledge-distillation).

## Variants

### Parameter recovery for one-hidden-layer teachers

The teacher's labels are a weighted sum of activations of k first-layer
projections [arxiv:1706.03175 §3]. Recovery has two stages: a tensor-based
moment estimate lands the student near the teacher, then gradient descent on
fresh samples refines it [arxiv:1706.03175 §5, §6, alg. 1, alg. 2]. The paper
reports sample size roughly linear in d for a fixed accuracy, with polynomial
factors in k, the condition number and other constants that it does not state
explicitly [arxiv:1706.03175 §5, §6]. Card: [Recovery Guarantees for One-hidden-layer Neural Networks](/papers/1706.03175).

Within this variant the activation class matters:

- **Smooth, homogeneous activations** (e.g. squared ReLU in the paper's
  experiments) get a linear-convergence result for gradient descent on fresh
  samples [arxiv:1706.03175 §4, app. D].
- **Piecewise-linear activations** (ReLU, leaky ReLU) get the tensor
  initialisation and local Hessian bound, but not the linear-rate theorem
  [arxiv:1706.03175 §4, §6].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need the teacher's weights, not only its outputs | This variant's recovery method | It is the only setting here with a recovery guarantee [arxiv:1706.03175 §3–§6] |
| Teacher has one hidden layer, inputs are standard Gaussian, labels are noiseless, k ≤ d | Tensor init, then gradient descent | Conditions under which the theorems apply [arxiv:1706.03175 §3] |
| Activation is smooth and homogeneous | Expect linear convergence | Only smooth activations have the rate proof [arxiv:1706.03175 §4, app. D] |
| Activation is ReLU or leaky ReLU | Use the initialisation and Hessian results | The linear-rate theorem does not cover them [arxiv:1706.03175 §6] |
| Activation is sigmoid or tanh at large scale | Expect very loose bounds | ρ(σ) falls to 5.1e-5 for sigmoid at σ = 10, and the sample bound grows with 1/ρ² [arxiv:1706.03175 tab. 1, app. D] |
| Labels are noisy, inputs have unknown covariance, or the teacher is deeper | Outside the stated results | Noise is not covered; general covariance is only whitened "with some loss of accuracy" [arxiv:1706.03175 §3] |
| You want a smaller model that predicts well on real data | Not covered here | The paper has only synthetic experiments [arxiv:1706.03175 §7] |

## Implementation sketch

Tensor initialisation and gradient descent with fresh samples, following the
structure the paper gives. Constants and sample splits are not stated in the
paper and must be chosen by the implementer.

```pseudocode
# Recovery of one-hidden-layer teacher weights  [arxiv:1706.03175 §6, alg. 2]
# Input:  samples S (linear in d), width k, number of steps T
# Output: W (d, k), v (k,)
def recover_teacher(samples, k, T):
    # Split into T + 1 disjoint subsets; the first one is for initialisation
    # [arxiv:1706.03175 alg. 2]
    parts = partition(samples, T + 1)
    # Tensor initialisation on the first subset  [arxiv:1706.03175 alg. 1]
    weights, v = tensor_initialise(parts[0], k)
    # Output weights are fixed after initialisation  [arxiv:1706.03175 alg. 2]
    for q in range(T):
        # Fresh batch every step (resampling)  [arxiv:1706.03175 §4, §6]
        grad = empirical_gradient(weights, v, parts[q + 1])  # (d, k)
        # Step size bound from the theorem; constant is not stated
        # [arxiv:1706.03175 §6]
        weights = weights - eta * grad
    return weights, v
```

Where variants differ: the initialisation uses a power-method subspace and a
non-orthogonal tensor decomposition, and the choice of moment order depends on
which empirical moments are non-zero [arxiv:1706.03175 alg. 1, alg. 4,
app. E]. The paper does not say how that zero test is made on finite samples
[arxiv:1706.03175 alg. 4].

## Contradictions

- **Resampling and step size.** The theory needs fresh samples at each step and
  a step size of at most 1/(k v_max² σ₁^{2p}) [arxiv:1706.03175 §4, §6]. The
  experiments use a fixed step of 0.02 with no resampling, and the paper says
  this "still works well" [arxiv:1706.03175 §7].
- **Homogeneity exponent.** The body describes scaling by an exponent 1/p, with
  p the degree of homogeneity [arxiv:1706.03175 §4]. The appendix gives the
  degree as p + 1 and uses 1/(p+1) in the norm recovery formula
  [arxiv:1706.03175 app. E, alg. 4]. The two forms do not agree, and the paper
  does not say which is correct.
- **Sample count.** The sample bound is written with poly(·) factors in the
  informal theorem [arxiv:1706.03175 §5, §6] and with explicit powers of d and k
  in the appendix proof [arxiv:1706.03175 app. E]. The exponents differ between
  these forms and are not reconciled.

## Papers

- [Recovery Guarantees for One-hidden-layer Neural Networks](/papers/1706.03175): tensor initialisation plus resampled gradient descent recovers a one-hidden-layer teacher's weights from about d·log(1/ε) samples, up to polynomial factors; theory with synthetic checks.