Highway Networks

In plain words

Very deep neural networks are hard to train. Past a certain number of layers, the learning signal fades or gets scrambled on its way through the stack, and the network stops improving. The usual fixes, such as careful starting values and training in stages, only go so far.

The authors give each layer a shortcut lane. Every layer has a learned gate that decides, for each value, how much of the layer's new computation to use and how much of the incoming value to pass straight through unchanged. The gate can start out mostly closed to new computation, so at the beginning the layer mostly passes its input along. Along the way the network learns where to open and close the gates. The authors call the open paths "information highways".

Training uses plain stochastic gradient descent with momentum and no extra stages. The authors report that stacks of 100 layers train well this way, and that a convolutional version trained directly matches or beats an earlier method that needed a teacher network and two training stages.

TL;DR

Each highway layer outputs a sigmoid-gated mix of a plain transform H(x) and the untouched input x, y = H(x)·T(x) + x·(1 − T(x)) §2, eq. 3. Gate biases initialised to negative values start the stack close to carrying its input §2.2. Author-reported: 100-layer highway nets train about as well as 10-layer plain nets with SGD §3.1, fig. 1. On CIFAR-10 (with random translations), a 19-layer convolutional highway net reaches 92.24% test accuracy, trained without a teacher tab. 1 §3.2.

Problem

Plain feedforward networks become harder to optimise as depth grows, and training with SGD stalls at the start unless the weights are initialised in a specific way §2.2. Even with a variance-preserving initialisation, very deep plain networks become difficult to optimise §3.1. Earlier deep CIFAR-10 results needed a two-stage procedure with a pretrained teacher to train beyond about five layers §3.2. The initialisation fix also depends on the exact form of the layer transform H, which is a limitation the authors want to avoid §2.2.

Method

The paper starts from a plain layer, then adds a gate that controls a blend between the transform and the input.

1. Plain layer. A layer applies a non-linear transform H with weights W_H to its input x, giving y = H(x, W_H) §2, eq. 1. H is usually an affine map followed by an activation §2.
2. Highway layer. Add a transform gate T(x, W_T) and a carry gate C(x, W_C), so that y = H(x)·T(x) + x·C(x) §2, eq. 2. For simplicity the paper sets C = 1 − T §2, eq. 3.
3. Gate behaviour. The output is exactly x when the gate is 0 and exactly H(x) when it is 1 §2, eq. 4. The gate uses a sigmoid, so it never reaches 0 or 1 exactly, and the two cases are limits §2.2.
4. Gate form. T(x) = σ(W_T^T x + b_T), where σ is the logistic sigmoid §2.2 §1.1.
5. Shape constraint. x, y, H(x) and T(x) must have the same dimension §2.1. To change width, the paper inserts a plain layer and then stacks highway layers; the alternatives of sub-sampling or zero-padding x are mentioned but not used §2.1.
6. Convolutional variant. H and T both use local receptive fields and shared weights. Zero-padding keeps the block state and gate maps the same size as the input §2.1.
7. Initialisation. Set each transform-gate bias b_T to a negative value, such as −1 or −3, so the network starts biased toward carry §2.2. This is independent of the form of H, which is the paper's main practical argument §2.2. The idea follows an LSTM gate-bias trick from Gers et al. §2.2.
8. Gradient. The paper gives the layer Jacobian only for the two limiting gate values: I when T = 0 and H′ when T = 1 §2, eq. 5. The general case is not written out.

# Highway layer  [§2, eq. 2-3]
# Input:  x (batch, d);  output y (batch, d)
# Learned: W_H, b_H (transform);  W_T, b_T (gate)

def highway_layer(inputs):
    # Plain transform H(x)  [§2, eq. 1]
    transformed = H(inputs, W_H)                 # (batch, d)
    # Transform gate T(x) = sigmoid(W_T^T x + b_T)  [§2.2]
    gate = sigmoid(inputs @ W_T + b_T)           # (batch, d), in (0, 1)
    # Carry gate fixed to 1 - T  [§2, eq. 3]
    carry = 1 - gate                             # (batch, d)
    # Mix transform and input, element-wise  [§2, eq. 2-3]
    outputs = transformed * gate + inputs * carry
    return outputs

# Negative gate-bias initialisation  [§2.2]
# Input:  layers (highway layers); bias_value < 0
# Output: the same layers, with gates biased toward carry

def init_gate_bias(layers, bias_value):
    # Bias every gate toward carrying the input at the start  [§2.2]
    for layer in layers:
        layer.gate_bias = bias_value             # b_T, e.g. -1 to -4
    return layers

Key equations, as numbered in the paper:

\vec{y} = H(\vec{x}, \vec{W_H}) \cdot T(\vec{x}, \vec{W_T}) + \vec{x} \cdot C(\vec{x}, \vec{W_C}) §2, eq. 2

\vec{y} = H(\vec{x}, \vec{W_H}) \cdot T(\vec{x}, \vec{W_T}) + \vec{x} \cdot (1 - T(\vec{x}, \vec{W_T})) §2, eq. 3

\frac{d\vec{y}}{d\vec{x}} = \begin{cases} \vec{I}, & T = \mathbf{0} \\ H'(\vec{x}, \vec{W_H}), & T = \mathbf{1} \end{cases} §2, eq. 5

Hyperparameters

Values not given by the paper are marked "not stated". The source is the 6-page workshop version; the extended version it points to (arXiv:1507.06228) was not read for this card.

Results

Author-reported. Setup: MNIST fully connected nets of 10 to 100 layers, measured on training cross-entropy to isolate optimisation §3.1. CIFAR-10 test accuracy for convolutional nets trained directly with backpropagation, compared with FitNet results reported by Romero et al. §3.2 tab. 1. The FitNet rows are quoted from Romero et al., not re-run by the authors tab. 1.

Optimisation (MNIST, training cross-entropy):

CIFAR-10 test accuracy (with random translations):

* Trained on 40K of the 50K training images tab. 1. The text says Highway 3 and 4 perform similar to the teacher §3.2.

The paper also reports a 900-layer highway net trained in preliminary experiments on MNIST, and a 900-layer net started on CIFAR-100 that was at 80 epochs when written §1 §3.1. No numbers are given for either.

Paper vs code

No official code.

Searched: the paper and its arXiv abstract page (no code link; comment reads "6 pages, 2 figures, Presented at ICML 2015 Deep Learning workshop"), the authors' project page cited in a third-party README (not reachable), GitHub repository search for "highway networks" and "highway networks in caffe", and an author-owner search (no hits). The only Caffe repository found, flukeskywalker/highway-networks, cites this paper but its owner is not verifiably an author, so it was not used.

Pitfalls

• Layer numbering in the paper disagrees with itself. The text says Highway 1 and Highway 4 are based on Fitnet 1 and Fitnet 4 §3.2, but the table names the Fitnet-4-based network "Highway 2 (Fitnet 4)" and uses "Highway 4" for a 32-layer net tab. 1. Use the table's labels.
• The product in eq. 2–3 is not defined. The paper writes both · and * and never says element-wise. The shape rules §2.1 and per-block gates §2 imply element-wise, which is what the pseudocode above uses (inferred, not stated).
• Gates never reach their limits. Since σ lies in (0, 1), the two cases in §2, eq. 4 cannot hold exactly §2.2. Implementations will always mix both terms.
• Gate bias range vs. learned behaviour. The search range is −1 to −10 §3.1. In the analysis nets, most gate biases decreased further during training rather than rising, and the strong negative bias seems to make the gates selective, not closed §4. Do not assume biases stay near their initial value.
• Jacobian is partial. Eq. 5 covers only the two limit cases §2, eq. 5. The general derivative of eq. 3 includes the gate's derivative, which the paper does not write out.
• Gate granularity in conv nets is not stated. The paper does not say whether the gate is per unit, per channel, or per position §2.1.
• Parameter matching is approximate. The 50 vs 71 units choice is meant to roughly match parameter counts §3.1; exact counts are not given for the MNIST nets.
• Comparisons are best-of-search. MNIST curves show the best of 40 random-search runs per depth §3.1 fig. 1; no variance across seeds is reported.
• The FitNet comparison is not like-for-like. FitNets used two-stage training with a teacher's soft targets; the highway nets were trained directly §3.2. Starred rows used 40K of 50K training images tab. 1.
• Unfinished results. The 900-layer CIFAR-100 run was still training when the paper was written §3.1; treat it as an observation, not a result.
• Abstract-only source. This is the 6-page workshop version. The source points to an extended version (arXiv:1507.06228) with more experiments and analysis; that version is not covered here.

Lineage

• Builds on: LSTM gating, with the gate-bias idea following Gers et al. (1999) §1 §2.2; initialisation from He et al. (2015), arXiv:1502.01852 §3.1; Glorot and Bengio (2010) §2.2.
• Compared against: FitNets, Romero et al. (2014), arXiv:1412.6550 §3.2 tab. 1.
• Extended version: arXiv:1507.06228, named in the source's note before §1, not read for this card.
• Superseded by: not stated in the wiki content.