Gated skip connection

Overview

A plain deep network passes its input through a stack of transforms. Past a certain depth, these networks become hard to optimise with gradient descent, and training stalls unless the weights are initialised in a specific way 1505.00387 §2.2. The shared idea of this concept is to let each layer pass part of its input forward untouched, with a learned gate deciding how much. The layer output is a weighted mix of a transform H(x) and the input x, with the weight set by a sigmoid gate T(x) computed from x. Input and output have the same shape 1505.00387 §2.1. The paper does not say the mix is element-wise; the card treats that as inferred from the per-block gates 1505.00387 §2, eq. 3.

The invariant every variant shares is that the layer can reduce to the identity when the gate closes, and can reduce to the plain transform when the gate opens. The sigmoid never reaches either limit exactly, so real layers always mix both terms 1505.00387 §2.2.

Variants

Only one variant has a card. Other gated or ungated skip designs are not covered here, because no card with reuse-licensed content exists for them yet.

Highway layer (fully connected or convolutional)

Each layer has two learned transforms: the plain transform H with weights W_H, and a transform gate T with weights W_T and bias b_T. The output is H(x)·T(x) + x·(1 − T(x)) 1505.00387 §2, eq. 3. The gate is a sigmoid of an affine map of x 1505.00387 §2.2. The paper fixes the carry gate to 1 − T rather than learning it separately 1505.00387 §2, eq. 3. In the convolutional form, H and T use local receptive fields with shared weights and zero-padding, so the gate maps match the input size 1505.00387 §2.1. Card: Highway Networks.

When to use which

The only comparison on the card is highway layers against plain layers and against one published teacher-based method. The table below states what the card supports for a given constraint.

Cost: each layer computes a second transform and holds W_T and b_T, so each layer has more parameters than a plain layer of the same width. The paper matches total parameter counts by using 50 highway units against 71 plain units on MNIST 1505.00387 §3.1. Compute cost per step is not on the card.

Implementation sketch

The core is one layer. Each gate bias is set negative at initialisation. The gate is computed per element, which is how the shape rules fit together 1505.00387 §2, eq. 3 1505.00387 §2.1. The gate's granularity (per unit, per channel or per position) is not stated for conv layers 1505.00387 §2.1.

# Highway layer  [arxiv:1505.00387 §2, eq. 2-3]
# Input:  x (batch, d);  output y (batch, d)
# Learned: W_H, b_H (transform);  W_T, b_T (gate)

def highway_layer(inputs):
    # Plain transform H(x)  [§2, eq. 1]
    transformed = H(inputs, W_H)                 # (batch, d)
    # Transform gate T(x) = sigmoid(W_T^T x + b_T)  [§2.2]
    gate = sigmoid(inputs @ W_T + b_T)           # (batch, d), in (0, 1)
    # Carry gate fixed to 1 - T  [§2, eq. 3]
    carry = 1 - gate                             # (batch, d)
    # Mix transform and input, element-wise  [§2, eq. 2-3]
    return transformed * gate + inputs * carry

# Negative gate-bias initialisation  [arxiv:1505.00387 §2.2]
# Input:  layers (highway layers); bias_value < 0
# Output: the same layers, with gates biased toward carry

def init_gate_bias(layers, bias_value):
    # Bias every gate toward carrying the input at the start  [§2.2]
    for layer in layers:
        layer.gate_bias = bias_value             # b_T, e.g. -1 to -4
    return layers

Gradient, for the two limit cases only. The paper does not write the general derivative of eq. 3 1505.00387 §2, eq. 5:

• Gate at 0: the Jacobian with respect to x is the identity.
• Gate at 1: the Jacobian is the derivative of H.

Gate-bias values seen in the sources:

Most gate biases decreased during training in the analysis nets, so they do not stay at their initial value 1505.00387 §4.

Contradictions

None between papers, since only one paper is covered. Within that paper, the text and the table name the same networks differently. The text says Highway 1 and Highway 4 follow FitNet 1 and FitNet 4 1505.00387 §3.2; the table names the FitNet-4-based net "Highway 2" and uses "Highway 4" for a 32-layer net 1505.00387 tab. 1. Use the table's labels when citing numbers.

Papers

• Highway Networks: introduces the sigmoid-gated highway layer and negative gate-bias initialisation; reports trained 100-layer MNIST nets and a 19-layer convolutional CIFAR-10 net at 92.24%.