Progressive Neural Networks

In plain words

Training a network on a new task usually overwrites what it learned before. Progressive networks avoid this by never overwriting. For each new task they add a fresh network (a "column") next to the old ones and freeze the old ones, so they keep working exactly as before.

The new column is not isolated. At every layer it can read what the older columns computed, so it can reuse useful visual or motor features, ignore them, or build its own. The paper tests this on game-playing agents that learn from images. Transfer helps when the games share useful features, and because old columns are never overwritten, the approach avoids the damage that finetuning can do to earlier tasks.

The cost is size. Every new task adds a whole new network, so the model grows with the number of tasks, and at test time the caller has to say which column to use.

TL;DR

• A progressive network is a stack of columns, one per task. Earlier columns are frozen, and each new column receives their hidden activations through lateral connections §2 eq. 1.
• Forgetting is avoided by construction, because frozen columns are never updated §2.
• Experiments use asynchronous advantage actor-critic (A3C) on Pong variants, 12 Atari target games and 3D maze tasks. The transfer score is relative to training a single column from scratch §5.1.
• Author-reported: 2-column nets beat full finetuning on mean transfer score on Pong Soup and Labyrinth. On Atari the 2-column mean (132%) is slightly below full finetuning (133%), and the 3- and 4-column nets are only slightly above it tab. 1.

Problem

Finetuning a pretrained network on a new task is the usual transfer method. It only uses prior knowledge at initialisation, and it overwrites the old function, so the network cannot also solve the old task afterwards §1. Choosing a good initialisation for a long sequence of tasks needs knowledge of how similar the tasks are, which is often not available §1. Distillation needs stored data for all earlier tasks §1. The paper does not assume that tasks overlap. They may be orthogonal or even adversarial §2.

Method

Sections are numbered as in the arXiv v4 LaTeX source (§1 Introduction, §2 Progressive Networks, §3 Transfer Analysis, §4 Related Literature, §5 Experiments with subsections 5.1 to 5.4, §6 Conclusion). Appendix sections are cited by their order in the source (see Pitfalls).

Single column. A network with L layers and hidden activations h_i^{(1)} \in \mathbb{R}^{n_i}, trained on task 1 §2.

Adding a column. When switching to task k, all earlier parameters are frozen and a new column is added with random initialisation. Layer i of column k receives its own previous layer and the previous layer of every earlier column j < k through lateral weights U_i^{(k:j)} §2 eq. 1:

h_i^{(k)} = f\left( W_i^{(k)} h_{i-1}^{(k)} + \sum_{j<k} U_{i}^{(k:j)} h_{i-1}^{(j)} \right) eq. 1

• W_i^{(k)} \in \mathbb{R}^{n_i \times n_{i-1}} is the weight matrix of layer i of column k, and U_i^{(k:j)} \in \mathbb{R}^{n_i \times n_j} is the lateral matrix from layer i-1 of column j to layer i of column k §2.
• h_0 is the network input. f is ReLU for all intermediate layers. Biases are omitted for clarity §2.
• Lateral connections only feed a column from earlier columns, so the frozen columns are unaffected in the forward pass. Because the earlier parameters are constants for the optimiser, training column k cannot change tasks 1..k-1 §2.
• Widths may differ per column and layer; the paper notes this as a straightforward generalisation and does not evaluate it §2, footnote.

Reinforcement learning use. Each column is trained on one Markov decision process. The k-th column defines a policy \pi^{(k)}(a \mid s) by setting \pi^{(k)}(a \mid s) := h_L^{(k)}(s) §2. The paper does not state the output nonlinearity that makes this a probability distribution.

Adapters. In practice each lateral connection is replaced by an adapter §2 eq. 2:

h_i^{(k)} = \sigma \left( W_i^{(k)} h_{i-1}^{(k)} + U_{i}^{(k:j)} \sigma(V_{i}^{(k:j)} \alpha_{i-1}^{(<k)} h_{i-1}^{(<k)}) \right) eq. 2

• h_{i-1}^{(<k)} is the concatenation of the anterior columns' activations at layer i-1 §2.
• \alpha_{i-1}^{(<k)} is a learned scalar that scales the lateral inputs to balance their different magnitudes. It starts at a small random value §2 app. 3.
• V_i^{(k:j)} is a projection that reduces dimensionality. For convolutional layers the reduction is done with 1×1 convolutions §2.
• The purpose is better initial conditioning and fewer lateral parameters, so that as k grows the lateral parameter count stays of the same order as |\Theta^{(1)}| §2.

# Progressive network, column k, layer i, without adapters  [§2] [eq. 1]
# Input:  hidden_prev[j] = layer i-1 activations of column j (h_0 = input)
# Output: hidden[k] at layer i, shape (n_i,)

def progressive_layer(k, i, hidden_prev):
    # Own-column path through the trainable weights W_i^(k)  [§2] [eq. 1]
    own_input = weights[k][i] @ hidden_prev[k]          # (n_i,)
    # Lateral path from each frozen earlier column j < k  [§2] [eq. 1]
    lateral = zeros_like(own_input)
    for j in range(1, k):
        lateral = lateral + laterals[k][j][i] @ hidden_prev[j]
    # Element-wise ReLU on intermediate layers  [§2]
    return relu(own_input + lateral)                    # (n_i,)

# Adapter lateral path, column k, layer i  [§2] [eq. 2]
# Input:  hidden_prev[j] for j < k; alpha[i-1][k] a learned scalar
# Output: lateral contribution to layer i of column k, shape (n_i,)

def adapter_lateral(k, i, hidden_prev, alpha):
    # Concatenate the anterior features of all earlier columns  [§2]
    anterior = concat([hidden_prev[j] for j in range(1, k)])
    # Scale them by the learned scalar to balance their magnitudes  [§2]
    scaled = alpha[i - 1][k] * anterior
    # Project down to a lower-dimensional hidden layer  [§2] [eq. 2]
    projected = sigma(adapter_V[k][i] @ scaled)         # (n_{i-1},)
    # Map back to layer width; the outer sigma is applied to the sum  [eq. 2]
    return adapter_U[k][i] @ projected                  # (n_i,)

# Training schedule for K sequential tasks  [§2]
# Input:  tasks[1..K], each an MDP with its own environment
# Output: columns[1..K]; columns before k are frozen while k trains

def train_progressive(tasks):
    # Column 1 is an ordinary network trained on task 1  [§2]
    columns = [new_column()]
    train_a3c(columns[0], tasks[0])
    for k in range(2, len(tasks) + 1):
        # Freeze the parameters of all earlier columns  [§2]
        freeze(columns)
        # Random init for column k, plus lateral inputs from every earlier
        # column  [§2]
        columns.append(new_column_with_laterals(columns))
        # Only the new column's parameters receive gradients  [§2]
        train_a3c(columns[-1], tasks[k - 1])
    return columns

Transfer analysis (§3). Two measures estimate how much the final network's output relies on each column and feature in a given layer.

• Average Fisher Sensitivity (AFS). A diagonal Fisher matrix \hat{F} of the policy with respect to normalised activations \hat{h}, with expectation over the joint state-action distribution of the target-trained network §3. The normalised score is \text{AFS}(i,k,m) = \hat{F}_i^{(k)}(m,m) / \sum_k \hat{F}_i^{(k)}(m,m), and the per-layer score sums over features m §3. This display is unnumbered in the paper.
• Average Perturbation Sensitivity (APS). Gaussian noise is injected at a layer and the noise precision that causes a 50% performance drop is recorded. APS is that precision normalised across columns app. 1 eq. 3. The paper reports that APS and AFS give highly similar results §3 app. 1 fig. 9.

Pseudocode for AFS and APS is not given in the paper. Both are described in prose only.

Hyperparameters

Hyperparameters are sampled at random for each job. The paper averages the top 3 of 25 jobs per result §5.1. The values below are the sampling ranges and fixed settings from the appendix.

Results

Author-reported. Transfer score is the area under the learning curve of a column, relative to baseline 1, which is a single column trained from scratch on the target task §5.1. Source columns differ by domain:

• Pong Soup: the first column is trained on Pong, Noisy or H-flip, and the second on each other variant fig. 4.
• Atari: single-column sources are Pong, River Raid and Seaquest. Targets are 12 randomly selected Atari titles, listed in §5.3.
• Labyrinth: source is Maze Y, targets are 8 maze tasks app. 4 §5.4.

Findings the paper reports:

• Pong Soup §5.2 fig. 4: progressive nets beat baseline 3 on mean and median, and the mean gap is larger. The paper reads this as better exploitation of transfer when it is possible. Detailed analysis fig. 5 shows that switching Pong to H-flip reuses low- and mid-level vision but re-learns the fully connected layer. Pong to Zoom reuses one spatio-temporal filter with a strong temporal DC component. Pong to Noisy re-learns some first-layer filters.
• Atari §5.3 fig. 6: progressive nets show positive transfer on 8 of 12 targets, with 2 negative cases. Baseline 3 is positive on 5 of 12. Seaquest to Gopher is negative for baseline 3 but not for progressive nets §5.3. Positive transfer is associated with new visual features in the new column, a "sweet spot" between full reliance on source features and full reliance on new ones §5.3 fig. 7.
• Labyrinth §5.4 fig. 8: progressive nets are above all baselines. Transfer is smaller on Seek Track levels, which are easy. The mean (491%) is far above the median (115%) tab. 1.
• Compression app. 2 fig. 10: with more columns, a smaller share of each source column's feature maps has high AFS. The paper concludes that pruning or compression is plausible. It does not implement compression.

Paper vs code

No official code.

Searched: the arXiv metadata lists no project link and the LaTeX source contains no URLs. A GitHub repository search for "progressive neural networks" found only third-party reimplementations, which were not used. A search for "progressive networks" in the deepmind organisation returned no results, and a search for "progressive-neural-networks" in google-deepmind returned only an unrelated repository.

Pitfalls

• Adapter dimensions do not match. The text says the adapter's hidden layer projects onto an n_i-dimensional subspace §2, but in eq. 2 V_i^{(k:j)} \in \mathbb{R}^{n_{i-1} \times n_{i-1}^{(<k)}}, so the hidden width is n_{i-1}. The implementation cannot be resolved from the paper because no code is released.
• Adapter nonlinearity is not defined. eq. 2 uses \sigma twice, but the paper never says which function it is. The non-adapter layers use ReLU §2. Assume ReLU only if you have no other source.
• Scalar initialisation is ambiguous. The text says the scalar starts at "a random small value" §2. The sampling grid is {1, 1e-1, 1e-2} app. 3, and 1 is included. The paper does not say whether the scalar is per layer, per column, or per (layer, column) pair, although eq. 2 indexes it by layer and anterior columns.
• Policy and value outputs are underspecified. \pi^{(k)}(a \mid s) := h_L^{(k)}(s) is called a probability distribution §2, but the output nonlinearity (e.g. softmax) is not stated. The network has a value head §5.1 app. 3, but the paper does not say whether it gets lateral inputs.
• Feature-map count conflicts. app. 3 says every conv layer of the Atari model has 12 feature maps. The Pong analysis reports sensitivity scores for eight feature maps in the first conv layer of a single-column Pong network fig. 5. The Pong network may differ from the Atari network, but the paper does not say so.
• Kernel shapes are incomplete. The second conv layer is given as "kernel of size 4", without a stated second dimension, and the third is 3×4, which is not square. Read the shape literally; do not assume square kernels for layer 2 without a check.
• Optimiser details are missing. RMSProp decay, epsilon and any learning-rate schedule are not stated. Discount factor, n-step length and entropy/value loss weights are not stated either.
• Selection bias. Each result is the mean of the top 3 of 25 randomly sampled jobs §5.1. The paper does not measure how much this selection changes the comparison with baselines.
• Mean vs median. The mean is sensitive to outliers §5.2. Labyrinth's 2-column mean of 491% against a median of 115% tab. 1 shows this. Compare medians when the mean gap is large.
• Step units. Curves and results are in agent-perceived steps (action repeat 4), but the run limit is given in environment steps app. 3. Convert consistently when reproducing.
• AFS needs equal widths. The normalisation \sum_k \hat{F}_i^{(k)}(m,m) sums across columns for the same feature index m. This requires every column to have the same number of features in layer i. The paper's own footnote allows different widths §2, which would break this measure. The paper does not say how the expectation is estimated: number of samples, whether actions are sampled from \pi, and how \hat{h} is normalised beyond the variance scaling in app. 1.
• APS details are missing. The noise-precision search that finds the 50% drop point is not described. Each estimate averages 10 episodes app. 1.
• Labyrinth environment not named. The paper says only "a 3D maze environment" with rendered images and partial observability §5.4. The engine, action set details and frame settings are not stated beyond app. 5.
• Source order matters. Atari results depend on the order of source games (for example, Seaquest then River Raid then Pong §5.3 footnote). The paper does not study other orders.
• Growth and test-time routing. Parameters grow quadratically with the number of columns app. 2. At test time the caller must choose the column, which requires a task label §2. Training and inference compute for many columns are not reported.
• Name mismatch. The Pong variant is called "VH-flip" in the text §5.2 and "HV-flip" in the appendix figures app. 4. They are the same variant.
• Citations and numbering. Section, equation, figure and table numbers are taken from the arXiv v4 LaTeX source in order of appearance. Appendix numbers follow the source order, because the paper refers to "Section 3 of the Appendix" and "Appendix 2". A compiled PDF may label appendices with letters. The compiled PDF was not checked.

Lineage

• Builds on: finetuning and pretraining for transfer §1 §4, and distillation as an alternative for multitask learning §1.
• Compared against: finetuning baselines 2 and 3, and a two-column variant with a random frozen earlier column (baseline 4) §5.1 fig. 3. Also discussed: actor-mimic, cascade-correlation and block-modular architectures §4.
• Superseded by: not stated in this wiki.