---
kind: concept
slug: greedy-layerwise-training
title: Greedy layer-wise training
summary: >-
  Trains a deep model in stages, one block or layer at a time. Supervised variants give each block a shallow auxiliary classifier and freeze earlier blocks, with no end-to-end backprop (88.7% top-5 on ImageNet with 3-hidden-layer heads, author-reported). Unsupervised variants pre-train first layers with RBMs or autoencoders, then fine-tune end-to-end.
tags: [training, deep-learning, cnn, pretraining]
papers: ["1812.11446", "1601.02376"]
related: [convolutional-neural-network, linear-probing, transfer-learning, self-supervised-learning, multi-layer-perceptron, denoising-autoencoder, factorization-machine]
status: draft
updated_at: 2026-10-19
---

## Overview

Instead of training all layers at once, the network is trained in stages. Two
families are covered here, and they differ in what happens after each stage.

**Supervised block-wise training.** Block j is fitted together with a small
auxiliary classifier on the outputs of the blocks before it. Earlier parameters
stay fixed while block j is trained, then block j is frozen and its output
becomes the input of the next block [arxiv:1812.11446 §3.2] [arxiv:1812.11446 alg. 1]. The invariant of this family is that the error signal of a block never
reaches earlier blocks.

**Unsupervised pre-training, then fine-tuning.** The first layers are
initialised one at a time with an RBM or an autoencoder, and then all layers
are fine-tuned end-to-end with backpropagation [arxiv:1601.02376 §3.1]
[arxiv:1601.02376 §3.2]. Here the invariant is the opposite: the greedy stage
only sets initial weights, and the final model is trained as one network.

The supervised family argues for itself on two grounds. End-to-end backprop
defines only the output objective, so the role of middle layers is indirect.
It also stores most intermediate activations, which costs memory
[arxiv:1812.11446 §1]. The idea itself is older: the paper cites greedy
layer-wise pretraining of deep generative and supervised networks (Bengio et
al. 2007; Hinton et al. 2006) as its origin [arxiv:1812.11446 §2]. The
pre-train-then-fine-tune family follows that older recipe more closely
[arxiv:1601.02376 §3.1].

## Variants

Two papers are cited. The first family is covered by one card and the second
by another. All results are author-reported, and the two families report
different tasks and metrics, so their numbers are not comparable.

### Family A: supervised blocks with auxiliary heads

Source: [Greedy Layerwise Learning Can Scale to ImageNet](/papers/1812.11446).

#### Shallow auxiliary head (k = 1)

Each block has a head that pools its features into a 2×2 map and applies one
linear layer [arxiv:1812.11446 eq. 2]. The block is then fitted as a
one-hidden-layer problem, which the authors call the case with the clearest
theory [arxiv:1812.11446 §3.3]. With eight blocks on ImageNet it reaches 79.7%
single-crop top-5 (80.8% ensemble), above the 79.1% AlexNet reference
[arxiv:1812.11446 tab. 3]. On CIFAR-10 it reaches 88.3% [arxiv:1812.11446 tab. 2].
Card: [Greedy Layerwise Learning Can Scale to ImageNet](/papers/1812.11446).

#### Deeper auxiliary head (k = 2 or 3)

The head has k−1 extra convolutions of constant width before pooling. Only the
first convolution of each sub-problem is kept in the block
[arxiv:1812.11446 §3]. Batch norm is used in these auxiliary problems
[arxiv:1812.11446 §4.3]. On ImageNet the reported top-5 is 86.3% for k = 2 and
88.7% for k = 3 (single-crop) [arxiv:1812.11446 tab. 3]. The final auxiliary
network is part of the k = 3 model [arxiv:1812.11446 §4.3].

#### Downsampling operator

A block may downsample with an invertible 2×2 space-to-depth step rather than
pooling [arxiv:1812.11446 §3.1]. On CIFAR-10 with k = 1 the reported accuracies
are 87.8% for strided convolution, 88.3% for the invertible step, 87.6% for
average pooling and 88.0% for max pooling [arxiv:1812.11446 tab. 7]. The gap is
at most 0.7 points on that one dataset.

#### Compressed layer-wise model

Each new block's filters are pruned from 128 to 64 and the auxiliary head is
fine-tuned [arxiv:1812.11446 §4.3]. On CIFAR-10 this gives 87.6%, against 87.5%
for an end-to-end five-layer network of width 64 [arxiv:1812.11446 §4.3].

#### Reduced final auxiliary width (k = 3)

Shrinking the final auxiliary width from 2048 to 512 costs 0.2 points of
top-5 on ImageNet (88.5% against 88.7%) [arxiv:1812.11446 tab. 1]. The
reduced model has about 46M parameters, against about 102M
[arxiv:1812.11446 tab. 6].

### Family B: unsupervised pre-training, then end-to-end fine-tuning

Source: [arXiv:1601.02376](/papers/1601.02376), a card on ad click prediction.
The task is click prediction on one-hot categorical fields, scored by AUC.
The card's [Paper vs code](/papers/1601.02376) notes say the public code does
not implement the hidden-layer pre-training described below for the FNN model.

#### Hidden layers pre-trained by RBM (FNN)

The hidden layers sit above a factorisation-machine embedding, and their
initial weights come from layer-wise RBM pre-training with contrastive
divergence. Fine-tuning then runs back-propagation over all weights
[arxiv:1601.02376 §3.1]. The paper's FNN reaches 70.70% AUC on the pooled
data, against 68.81% for logistic regression [arxiv:1601.02376 tab. 1]. The
card records no run without this pre-training, so the gain cannot be attributed
to it alone.

#### Sparse bottom layer pre-trained on sampled negatives (SNN)

The bottom layer is a sigmoid layer over all one-hot inputs. For pre-training,
each field keeps its active unit and `m` sampled negatives, and the rest are
ignored. Two samplers are used: an RBM (SNN-RBM) and a denoising autoencoder
(SNN-DAE) [arxiv:1601.02376 §3.2]. The point of the sampling is cost: a dense
first layer over about 1M inputs would need about 100 million weights at 100
units [arxiv:1601.02376 §1]. The paper reports `m = 2` as best in most cases
[arxiv:1601.02376 §4.3]. The SNN models reach 69.15% AUC on the pooled data
[arxiv:1601.02376 tab. 1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Activation memory is the binding limit | Block-wise training, k = 1 or reduced-width k = 3 | The authors report that the end-to-end CIFAR-10 model needs about five times the GPU memory of k = 1 training [arxiv:1812.11446 §4.1]. No memory figures are given for ImageNet runs. |
| Parameter budget is tight | k = 1, or k = 3 with reduced final width | The k = 3 ImageNet model has about 102M parameters, 46M when reduced [arxiv:1812.11446 tab. 6]. For k = 2, 3 most parameters sit in the auxiliary networks [arxiv:1812.11446 app. B]. |
| Highest reported ImageNet accuracy matters more than simplicity | k = 3 | 88.7% top-5 against 79.7% for k = 1 [arxiv:1812.11446 tab. 3]. |
| Simplest head and easiest analysis | k = 1 | Clearest theory [arxiv:1812.11446 §3.3]; the head is a pooling step and one linear layer [arxiv:1812.11446 eq. 2]. |
| Reusing frozen features for transfer | Deeper k = 1 blocks | On Caltech-101 with 30 training images per class, a linear model on layer 8 reaches 82.6 ± 0.9% against 46 ± 1.7% for a ConvNet trained from scratch [arxiv:1812.11446 app. C.3] [arxiv:1812.11446 tab. 8]. This is one dataset. |
| Implementation effort is a concern | Expect one training stage per block | The CIFAR-10 schedule is 50 epochs per layer [arxiv:1812.11446 §4]. Each block needs its own head and its own fit. |
| Very wide sparse one-hot inputs and a dense first layer is too large | Sampled-negative pre-training of a sparse bottom layer (SNN) | Only the active unit and `m` sampled negatives per field enter pre-training [arxiv:1601.02376 §3.2]. The paper reports SNN-DAE or SNN-RBM as best on some advertisers, but FNN as best on the pooled data [arxiv:1601.02376 tab. 1]. |
| Need the reported FNN behaviour in practice | Implement the pre-training step yourself | The public FNN code does not pre-train its hidden layers; the card's Paper vs code notes list this and other gaps. Only the SNN scripts pre-train. |

Do not treat the ImageNet numbers as a guarantee for other datasets or
architectures, and do not compare them with the AUC figures from Family B. All
figures are author-reported. No official code for the paper in Family A was
found.

## Implementation sketch

The common core of Family A is the same for every variant. Choices of head
depth, downsampling and width are marked below the sketch.

```pseudocode
# Greedy layer-wise training  [arxiv:1812.11446 alg. 1]
# Input:  labelled samples (x, y); number of blocks J
# Output: frozen blocks 0 .. J-1, each with a trained auxiliary head

def greedy_layerwise(samples, J):
    # Block inputs start as the raw data  [arxiv:1812.11446 eq. 1]
    inputs = samples.x
    blocks = []
    for j in range(J):
        # New block (3x3 conv, bias, ReLU) and its auxiliary head  [arxiv:1812.11446 §3.1]
        block, head = new_block(j), new_aux_head(j)
        # Fit only this block and its head; earlier blocks stay frozen  [arxiv:1812.11446 §3.2]
        fit(block, head, inputs, samples.y)
        # Keep the best iterate seen during fitting  [arxiv:1812.11446 §3.2]
        block.freeze()
        blocks.append(block)
        # Frozen block output is the next block's input  [arxiv:1812.11446 eq. 1]
        inputs = block.forward(inputs)
    return blocks
```

Where variants differ:

- **Head depth:** k = 1 pools and applies one linear layer. k = 2 or 3 adds
  k−1 constant-width convolutions before pooling [arxiv:1812.11446 eq. 2].
- **Downsampling:** an invertible space-to-depth step, applied at chosen
  depths, is used in place of pooling [arxiv:1812.11446 §3.1]. The paper's
  ImageNet k = 1 model doubles channel width at those depths [arxiv:1812.11446 tab. 5].
- **Depths:** the paper's ImageNet models use 8 blocks and downsample at
  blocks 2, 4 and 6 (k = 2, 3) or 2, 3, 4 and 6 (k = 1) [arxiv:1812.11446 tab. 4] [arxiv:1812.11446 tab. 5].

Family B follows a different loop. For the sparse bottom layer, pre-train it
on the sampled input (the active unit and `m` sampled negatives per field),
then move on to the upper layers. Once all layers have been pre-trained, train
the whole network with back-propagation of cross-entropy [arxiv:1601.02376 §3.2] [arxiv:1601.02376 eq. 7]. The stopping rule and learning rate are chosen
on a validation set [arxiv:1601.02376 §4.3].

## Contradictions

No disagreement between papers is known. Within the papers:

- BoostResNet on CIFAR-10 is 82.0% in the text [arxiv:1812.11446 §4.1] and
  82.1% in the table [arxiv:1812.11446 tab. 2].
- The introduction's single-block figures (61% on CIFAR-10, 12.3% on ImageNet)
  [arxiv:1812.11446 §4] do not match the headline k = 1 results of 88.3% and
  79.7% [arxiv:1812.11446 tab. 2] [arxiv:1812.11446 tab. 3]. The text does not
  say which configuration the introduction figures come from.
- Family B's text says FNN and SNN beat logistic regression and the
  factorisation machine on all tested datasets [arxiv:1601.02376 §4.2]. Table 1
  does not support this for SNN: both SNN variants are below both baselines on
  advertisers 2259 and 3386 [arxiv:1601.02376 tab. 1].
- Family B's text says `m = 2` is best [arxiv:1601.02376 §4.3], and the card
  notes that the public code samples one negative per positive. This is a gap
  between paper and code, not between two papers; see the card's Paper vs code notes.

## Papers

- [Greedy Layerwise Learning Can Scale to ImageNet](/papers/1812.11446): trains CNN blocks with shallow auxiliary heads, without end-to-end backpropagation, and reports ImageNet and CIFAR-10 results against end-to-end and other layer-wise baselines.
- [arXiv:1601.02376](/papers/1601.02376): on ad click data, pre-trains a sparse sigmoid layer with sampled negatives (RBM or denoising autoencoder) and initialises FNN hidden layers by RBM pre-training, then fine-tunes end-to-end on ad click data.