Greedy layer-wise training

Overview

Instead of training all layers at once, the network is trained in stages. Two families are covered here, and they differ in what happens after each stage.

Supervised block-wise training. Block j is fitted together with a small auxiliary classifier on the outputs of the blocks before it. Earlier parameters stay fixed while block j is trained, then block j is frozen and its output becomes the input of the next block 1812.11446 §3.2 1812.11446 alg. 1. The invariant of this family is that the error signal of a block never reaches earlier blocks.

Unsupervised pre-training, then fine-tuning. The first layers are initialised one at a time with an RBM or an autoencoder, and then all layers are fine-tuned end-to-end with backpropagation 1601.02376 §3.1 1601.02376 §3.2. Here the invariant is the opposite: the greedy stage only sets initial weights, and the final model is trained as one network.

The supervised family argues for itself on two grounds. End-to-end backprop defines only the output objective, so the role of middle layers is indirect. It also stores most intermediate activations, which costs memory 1812.11446 §1. The idea itself is older: the paper cites greedy layer-wise pretraining of deep generative and supervised networks (Bengio et al. 2007; Hinton et al. 2006) as its origin 1812.11446 §2. The pre-train-then-fine-tune family follows that older recipe more closely 1601.02376 §3.1.

Variants

Two papers are cited. The first family is covered by one card and the second by another. All results are author-reported, and the two families report different tasks and metrics, so their numbers are not comparable.

Family A: supervised blocks with auxiliary heads

Source: Greedy Layerwise Learning Can Scale to ImageNet.

Shallow auxiliary head (k = 1)

Each block has a head that pools its features into a 2×2 map and applies one linear layer 1812.11446 eq. 2. The block is then fitted as a one-hidden-layer problem, which the authors call the case with the clearest theory 1812.11446 §3.3. With eight blocks on ImageNet it reaches 79.7% single-crop top-5 (80.8% ensemble), above the 79.1% AlexNet reference 1812.11446 tab. 3. On CIFAR-10 it reaches 88.3% 1812.11446 tab. 2. Card: Greedy Layerwise Learning Can Scale to ImageNet.

Deeper auxiliary head (k = 2 or 3)

The head has k−1 extra convolutions of constant width before pooling. Only the first convolution of each sub-problem is kept in the block 1812.11446 §3. Batch norm is used in these auxiliary problems 1812.11446 §4.3. On ImageNet the reported top-5 is 86.3% for k = 2 and 88.7% for k = 3 (single-crop) 1812.11446 tab. 3. The final auxiliary network is part of the k = 3 model 1812.11446 §4.3.

Downsampling operator

A block may downsample with an invertible 2×2 space-to-depth step rather than pooling 1812.11446 §3.1. On CIFAR-10 with k = 1 the reported accuracies are 87.8% for strided convolution, 88.3% for the invertible step, 87.6% for average pooling and 88.0% for max pooling 1812.11446 tab. 7. The gap is at most 0.7 points on that one dataset.

Compressed layer-wise model

Each new block's filters are pruned from 128 to 64 and the auxiliary head is fine-tuned 1812.11446 §4.3. On CIFAR-10 this gives 87.6%, against 87.5% for an end-to-end five-layer network of width 64 1812.11446 §4.3.

Reduced final auxiliary width (k = 3)

Shrinking the final auxiliary width from 2048 to 512 costs 0.2 points of top-5 on ImageNet (88.5% against 88.7%) 1812.11446 tab. 1. The reduced model has about 46M parameters, against about 102M 1812.11446 tab. 6.

Family B: unsupervised pre-training, then end-to-end fine-tuning

Source: arXiv:1601.02376, a card on ad click prediction. The task is click prediction on one-hot categorical fields, scored by AUC. The card's Paper vs code notes say the public code does not implement the hidden-layer pre-training described below for the FNN model.

Hidden layers pre-trained by RBM (FNN)

The hidden layers sit above a factorisation-machine embedding, and their initial weights come from layer-wise RBM pre-training with contrastive divergence. Fine-tuning then runs back-propagation over all weights 1601.02376 §3.1. The paper's FNN reaches 70.70% AUC on the pooled data, against 68.81% for logistic regression 1601.02376 tab. 1. The card records no run without this pre-training, so the gain cannot be attributed to it alone.

Sparse bottom layer pre-trained on sampled negatives (SNN)

The bottom layer is a sigmoid layer over all one-hot inputs. For pre-training, each field keeps its active unit and m sampled negatives, and the rest are ignored. Two samplers are used: an RBM (SNN-RBM) and a denoising autoencoder (SNN-DAE) 1601.02376 §3.2. The point of the sampling is cost: a dense first layer over about 1M inputs would need about 100 million weights at 100 units 1601.02376 §1. The paper reports m = 2 as best in most cases 1601.02376 §4.3. The SNN models reach 69.15% AUC on the pooled data 1601.02376 tab. 1.

When to use which

Do not treat the ImageNet numbers as a guarantee for other datasets or architectures, and do not compare them with the AUC figures from Family B. All figures are author-reported. No official code for the paper in Family A was found.

Implementation sketch

The common core of Family A is the same for every variant. Choices of head depth, downsampling and width are marked below the sketch.

# Greedy layer-wise training  [arxiv:1812.11446 alg. 1]
# Input:  labelled samples (x, y); number of blocks J
# Output: frozen blocks 0 .. J-1, each with a trained auxiliary head

def greedy_layerwise(samples, J):
    # Block inputs start as the raw data  [arxiv:1812.11446 eq. 1]
    inputs = samples.x
    blocks = []
    for j in range(J):
        # New block (3x3 conv, bias, ReLU) and its auxiliary head  [arxiv:1812.11446 §3.1]
        block, head = new_block(j), new_aux_head(j)
        # Fit only this block and its head; earlier blocks stay frozen  [arxiv:1812.11446 §3.2]
        fit(block, head, inputs, samples.y)
        # Keep the best iterate seen during fitting  [arxiv:1812.11446 §3.2]
        block.freeze()
        blocks.append(block)
        # Frozen block output is the next block's input  [arxiv:1812.11446 eq. 1]
        inputs = block.forward(inputs)
    return blocks

Where variants differ:

• Head depth: k = 1 pools and applies one linear layer. k = 2 or 3 adds k−1 constant-width convolutions before pooling 1812.11446 eq. 2.
• Downsampling: an invertible space-to-depth step, applied at chosen depths, is used in place of pooling 1812.11446 §3.1. The paper's ImageNet k = 1 model doubles channel width at those depths 1812.11446 tab. 5.
• Depths: the paper's ImageNet models use 8 blocks and downsample at blocks 2, 4 and 6 (k = 2, 3) or 2, 3, 4 and 6 (k = 1) 1812.11446 tab. 4 1812.11446 tab. 5.

Family B follows a different loop. For the sparse bottom layer, pre-train it on the sampled input (the active unit and m sampled negatives per field), then move on to the upper layers. Once all layers have been pre-trained, train the whole network with back-propagation of cross-entropy 1601.02376 §3.2 1601.02376 eq. 7. The stopping rule and learning rate are chosen on a validation set 1601.02376 §4.3.

Contradictions

No disagreement between papers is known. Within the papers:

• BoostResNet on CIFAR-10 is 82.0% in the text 1812.11446 §4.1 and 82.1% in the table 1812.11446 tab. 2.
• The introduction's single-block figures (61% on CIFAR-10, 12.3% on ImageNet) 1812.11446 §4 do not match the headline k = 1 results of 88.3% and 79.7% 1812.11446 tab. 2 1812.11446 tab. 3. The text does not say which configuration the introduction figures come from.
• Family B's text says FNN and SNN beat logistic regression and the factorisation machine on all tested datasets 1601.02376 §4.2. Table 1 does not support this for SNN: both SNN variants are below both baselines on advertisers 2259 and 3386 1601.02376 tab. 1.
• Family B's text says m = 2 is best 1601.02376 §4.3, and the card notes that the public code samples one negative per positive. This is a gap between paper and code, not between two papers; see the card's Paper vs code notes.

Papers

• Greedy Layerwise Learning Can Scale to ImageNet: trains CNN blocks with shallow auxiliary heads, without end-to-end backpropagation, and reports ImageNet and CIFAR-10 results against end-to-end and other layer-wise baselines.
• arXiv:1601.02376: on ad click data, pre-trains a sparse sigmoid layer with sampled negatives (RBM or denoising autoencoder) and initialises FNN hidden layers by RBM pre-training, then fine-tunes end-to-end on ad click data.