Highway and Residual Networks learn Unrolled Iterative Estimation

In plain words

Deep networks are often pictured as a stack of layers, where each layer builds a more abstract description of the input than the one before. This paper says that picture does not fit the very deep networks with skip connections, called Highway and residual networks. If you delete one layer from a trained one of these networks, it barely gets worse, and shuffling its layers within a group also hurts only a little. That would be strange if every layer built something new.

The author's proposal is that the layers in one group, called a stage, work together on a single estimate of the same features. The first layer makes a rough guess and each later layer refines it without changing what the features mean. In this view a residual block adds a correction to its input. A Highway block blends its input with a new guess, using a learned gate that decides how much of each to keep.

The authors argue that both architectures follow from this one idea, and that it explains why layers can be removed or reordered. Their experiments are small and described as preliminary.

TL;DR

• Treats the blocks inside a stage of a Highway or residual network as refining one shared estimate, not computing a new representation at each layer §3.
• This explains why removing or shuffling layers in a trained network costs little §2 §4.4.
• Both architectures are derived from one requirement: each layer's output is an unbiased estimate of the same feature §3.2 §3.3.
• On ImageNet, a 50-layer Highway network and a residual network are close in top-5 error: 7.53% vs 7.17% with batch normalization, mean over 3 runs tab. 1b.
• On Penn Treebank, the gated Highway variants reach perplexity near 79, against 91.32 for the residual form and 92.60 for a plain layer tab. 1a. Results are author-reported.

Problem

The common view is that each layer of a deep network computes a new, more abstract representation §1. Under that view, deleting a layer from a trained Highway or residual network should disrupt the input of the next layer and badly hurt accuracy. Reported lesion studies instead show only small drops for most layers. Early layers and layers that change dimensionality hurt most, but the networks stay well above chance §2. In a trained 110-layer residual network, reshuffling layers within a stage increases error only smoothly, and many orders cost little §2. The paper treats these findings as the contradiction it needs to explain §2.

Method

Core claim. A stage is a run of blocks that keep the same dimensionality. In a stage, the blocks estimate and refine one level of representation. The level changes only at a dimensionality change, which is done by a projection §1 §3 fig. 1. The paper calls this unrolled iterative estimation.

Feature identity. Let a_i^k be unit i after block k, and A_i the feature that the stage converges to. The paper requires each a_i^k to be an unbiased estimator of A_i §3:

\mathbb{E}_{\mathbf{x}}[a_i^k - A_i] = 0 eq. 1

Consecutive blocks then have equal expected output: \mathbb{E}[a_i^k - a_i^{k-1}] = 0 eq. 7.

Residual networks. If the block output is a^k = a^{k-1} + F, the requirement in eq. 7 forces \mathbb{E}[F] = 0 eq. 9 §3.2. The paper says this is reasonable, especially with batch normalization §3.2. The ResNet formula is then:

y(\mathbf{x}) = F(\mathbf{x}) + \mathbf{x} eq. 5

Highway networks. For a new estimate H and the previous estimate a^{k-1}, the paper takes the unbiased linear mix with the least variance §3.3 app. A. Its weights, given error variances and covariance, are app. A:

q_1 = \frac{\sigma_B^2 - \sigma_{AB}}{\sigma_A^2 - 2\sigma_{AB} + \sigma_B^2} eq. 15

q_2 = \frac{\sigma_A^2 - \sigma_{AB}}{\sigma_A^2 - 2\sigma_{AB} + \sigma_B^2} eq. 16

Here A is the old estimate, B the new one, and \sigma_{AB} their covariance. The paper writes the covariance as \sigma^2_{AB} app. A. The paper then sets the gate T = \alpha_1 / (\alpha_1 + \alpha_2) and gets the Highway update §3.3 eq. 11.

Gated variants §3.3 §5.2:

• Full: y = H \cdot T + \mathbf{x} \cdot C eq. 3.
• Coupled: C = 1 - T, so the mixing weights sum to one eq. 4 §4.1.

Relation between the two. Residual networks are the special case where the gates are fixed at T = C = 1 §3.1. Setting F(\mathbf{x}) = T(\mathbf{x})[H(\mathbf{x}) - \mathbf{x}] turns eq. 5 into the coupled form eq. 4 §3.1.

Why a sigmoid gate. A sigmoid keeps both mixing weights positive, so the new variance is at most the old one §4.1. The paper says this requires the two error terms to share a sign, which holds when their covariance is zero §4.1. The pre-activation of T then estimates \log(\alpha_2 / \alpha_1), and a negative gate bias means the layer starts out contributing little §4.1.

# Coupled Highway layer  [eq. 4] [§3.3] [§4.1]
# Input:  x (B, D)
# Output: y (B, D)

def highway_layer(x):
    # New estimate H(x): an affine map and a nonlinearity  [§3.3]
    candidate = H(x)
    # Transform gate in (0, 1), a sigmoid of T's output  [§4.1]
    gate = sigmoid(T(x))
    # Coupled mix; the carry weight is 1 - gate  [eq. 4]
    return candidate * gate + x * (1 - gate)

# Residual block as one refinement step  [eq. 5] [§3.2]
# Input:  x (B, D)
# Output: y (B, D)

def residual_block(x):
    # Residual branch F(x); the derivation assumes E[F] = 0  [eq. 9]
    residual = F(x)
    # Identity skip keeps the previous estimate  [eq. 5]
    return residual + x

# Minimum-variance unbiased mix of old and new estimates  [app. A] [§3.3]
# Inputs: error variances var_old, var_new; error covariance cov
# Output: weight on the old estimate

def optimal_mix_weight(var_old, var_new, cov):
    # Term for the new estimate's error  [app. A]
    new_term = var_new - cov
    # Term for the old estimate's error  [app. A]
    old_term = var_old - cov
    # Weight on the old estimate  [eq. 15]
    return new_term / (old_term + new_term)

Architectures used in the experiments §5.1:

• Baseline: the 50-layer residual network of He et al. (2015), with batch normalization after every convolution.
• Highway: the new estimate H has the same layout as a residual block's residual branch, Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN.
• Highway-Full: as Highway, plus one extra ReLU.
• Transform branch T: Conv-BN-ReLU-Conv-BN-ReLU-Conv-BN-Sigmoid, with the same receptive fields and parameter count as H.

Stages and blocks. The paper describes residual networks as stages of blocks that share a dimensionality, with projections between stages and usually more blocks in later stages §4.3 fig. 1. It predicts that the first stages can be shorter and that the blocks of a stage refine the same kind of feature §4.3.

Hyperparameters

Results

Author-reported. Preliminary experiments. Setups are described in the Hyperparameters table.

ImageNet classification, 50-layer networks, top-5 error, mean ± std over 3 runs tab. 1b §5.1:

The residual network has the lowest validation error with BN. The paper calls the gap slight, and the final training loss of the residual network is lower §5.1 fig. 5. Without BN, the residual network has the lower validation error tab. 1b.

Penn Treebank language modelling, test perplexity tab. 1a §5.2:

The paper reads this as showing that gating the identity path matters for this task: Full, Coupled and C-Only are similar, and all beat T-Only and the residual form §5.2. The Full variant has more parameters than the others §5.2.

Claims tested §5.1: the paper says the first two claims made about Highway networks (that they are hard to train and need careful bias tuning) are contradicted by its runs. The third (that Highway networks need more parameters) is supported only weakly §5.1. Without BN, the Highway network fits the training data better than the residual network, which contradicts the third claim. Both networks overfit more and have higher validation error §5.1 fig. 5.

Paper vs code

No official code.

Searched: the arXiv abstract page (no code link; its comment reads "10 + 4 pages, accepted for ICLR 2017"); the GitHub repository search API for "unrolled iterative estimation", "highway residual greff" and "highway residual iterative" (no matching repositories); the profile of the first author, github.com/qwlouse (two pages of repositories, none about this paper); and the IDSIA organisation (the first 10 of 32 repositories, none about this paper). The PTB experiments reuse the code of Kim et al. (2015). That code belongs to a third party and is not part of this card.

Pitfalls

• Swapped α names. §3.3 defines \alpha_1 as the old estimate's error term minus the covariance and \alpha_2 as the new estimate's. app. A uses the opposite labels. Both give the same update, a^{k+1} = \frac{\alpha_2}{\alpha_1+\alpha_2}a^k + \frac{\alpha_1}{\alpha_1+\alpha_2}H eq. 10 (with \alpha as in §3.3), so implement the formula, not the labels.
• Index typo in the Highway formula. eq. 11 writes a^k on the left and a^{k-1} on the right. Read it as the update in eq. 10, which uses a^{k+1} on the left.
• Covariance notation. \sigma^2_{AB} in app. A means the covariance, not a squared standard deviation. Using it as a squared term gives the wrong weights.
• Unbiasedness is assumed. The Highway derivation assumes the new estimate H is unbiased for A §3.3. The residual derivation assumes \mathbb{E}[F] = 0 §3.2. The paper does not measure \mathbb{E}[H - A] for its own networks.
• Measured only on a pretrained network. The support for \mathbb{E}[F] = 0 comes from a 50-layer ResNet trained on ILSVRC-2015 (from He et al.), on 5000 validation images, averaged over spatial dimensions §4.2 fig. 3. Its mean error is close to zero in the first three stages but not in stage four, which the paper says underestimates the representation §4.2. The Highway runs are not part of this check.
• Gate bias. The transform bias of -1 is stated only for the ImageNet Highway runs §5.1. The paper's reading of the bias (a low value means the layer starts with little effect §4.1) is an interpretation.
• Positivity needs independence. The sigmoid bounds the mix to positive weights, which keeps the variance from growing. That holds when the old and new errors have zero covariance §4.1. The paper states this as an assumption, not a result.
• Parameter matching is approximate. Highway widths are reduced so the totals are "close", but the widths are not stated §5.1. The ImageNet comparison therefore cannot be reproduced exactly.
• Penn Treebank comparison is not parameter-matched. Full has more parameters §5.2, and the residual variant uses four layers while the others use two §5.2. The residual result depends on that choice.
• Small gaps, few runs. On ImageNet the gap between Highway and residual with BN is about 0.36 points of top-5 error, with standard deviations of 0.05 to 0.14 over three runs tab. 1b. The paper reports no significance test. The Penn Treebank values are single numbers with no spread reported.
• Training details are not in this paper. Optimiser, schedule, batch size and epochs for the ImageNet runs are defined by reference to He et al. (2015) §5.1. For PTB they are defined by Kim et al. (2015) §5.2. Neither is restated here.
• Figure 4 is a third-party image. The visual evidence figure reproduces a figure from Chu et al. (2017) with permission fig. 4. This card describes it in words only.
• Stage four. The paper notes that stage four, with three blocks, appears to underestimate the representation, and calls it a probable weak link §4.2. It does not test this further.

Lineage

• Builds on: Highway networks (Srivastava et al., 2015) and residual networks (He et al., 2015), both cited in §1 and §3.1. The paper gives no arXiv ids for them. It also cites the identity skip design of LSTM §3.1 (Hochreiter and Schmidhuber, 1997).
• Compared against: the representation view of deep networks §1. The ensemble view of residual networks (Veit et al., 2016; Huang et al., 2016) §2. Layer dropout (Huang et al., 2016) §2.
• Compared experimentally: Highway and residual variants on ILSVRC-2012 and Penn Treebank §5.
• Superseded by: not stated.