--- kind: concept slug: gated-skip-connection title: Gated skip connection summary: >- A layer outputs a learned, sigmoid-weighted mix of its transform and its unchanged input, so signal can pass through many layers. Gate bias starts the stack near carry behaviour. One paper so far, with author-reported results on MNIST and CIFAR-10, and no comparison to ungated shortcuts on the card. tags: [deep-networks, gating, skip-connections, optimization, architecture] papers: ["1505.00387"] related: [deep-learning, convolutional-neural-network, recurrent-neural-network] status: draft updated_at: 2026-10-19 --- ## Overview A plain deep network passes its input through a stack of transforms. Past a certain depth, these networks become hard to optimise with gradient descent, and training stalls unless the weights are initialised in a specific way [arxiv:1505.00387 §2.2]. The shared idea of this concept is to let each layer pass part of its input forward untouched, with a learned gate deciding how much. The layer output is a weighted mix of a transform H(x) and the input x, with the weight set by a sigmoid gate T(x) computed from x. Input and output have the same shape [arxiv:1505.00387 §2.1]. The paper does not say the mix is element-wise; the card treats that as inferred from the per-block gates [arxiv:1505.00387 §2, eq. 3]. The invariant every variant shares is that the layer can reduce to the identity when the gate closes, and can reduce to the plain transform when the gate opens. The sigmoid never reaches either limit exactly, so real layers always mix both terms [arxiv:1505.00387 §2.2]. ## Variants Only one variant has a card. Other gated or ungated skip designs are not covered here, because no card with reuse-licensed content exists for them yet. ### Highway layer (fully connected or convolutional) Each layer has two learned transforms: the plain transform H with weights W_H, and a transform gate T with weights W_T and bias b_T. The output is H(x)·T(x) + x·(1 − T(x)) [arxiv:1505.00387 §2, eq. 3]. The gate is a sigmoid of an affine map of x [arxiv:1505.00387 §2.2]. The paper fixes the carry gate to 1 − T rather than learning it separately [arxiv:1505.00387 §2, eq. 3]. In the convolutional form, H and T use local receptive fields with shared weights and zero-padding, so the gate maps match the input size [arxiv:1505.00387 §2.1]. Card: [Highway Networks](/papers/1505.00387). ## When to use which The only comparison on the card is highway layers against plain layers and against one published teacher-based method. The table below states what the card supports for a given constraint. | Situation | Use | Why | |---|---|---| | Plain stack stops improving as depth grows, and you train with SGD | Highway layers with negative gate bias | Author-reported: 100-layer highway nets train about as well as 10-layer plain nets on MNIST [arxiv:1505.00387 §3.1] [fig. 1] | | Layer transform H varies and you do not want an init scheme tied to it | Negative gate-bias init | The paper's main practical claim is that the bias starts the stack near carry without depending on H [arxiv:1505.00387 §2.2] | | Output width differs from input width | A plain layer to change width, then highway layers | Gates need matching shapes; sub-sampling and zero-padding are mentioned but not used [arxiv:1505.00387 §2.1] | | You cannot afford a teacher network or two training stages | Train the highway net directly | Highway 2 reached 92.24% without a teacher, against FitNet 4 at 91.61% [arxiv:1505.00387 tab. 1]. The FitNet numbers are quoted from the original authors, not re-run | | You need a guarantee of identity at init | Check the gate bias value first | Negative bias gives near-carry behaviour at start. The card records no formal bound on the deviation [arxiv:1505.00387 §2.2] | Cost: each layer computes a second transform and holds W_T and b_T, so each layer has more parameters than a plain layer of the same width. The paper matches total parameter counts by using 50 highway units against 71 plain units on MNIST [arxiv:1505.00387 §3.1]. Compute cost per step is not on the card. ## Implementation sketch The core is one layer. Each gate bias is set negative at initialisation. The gate is computed per element, which is how the shape rules fit together [arxiv:1505.00387 §2, eq. 3] [arxiv:1505.00387 §2.1]. The gate's granularity (per unit, per channel or per position) is not stated for conv layers [arxiv:1505.00387 §2.1]. ```pseudocode # Highway layer [arxiv:1505.00387 §2, eq. 2-3] # Input: x (batch, d); output y (batch, d) # Learned: W_H, b_H (transform); W_T, b_T (gate) def highway_layer(inputs): # Plain transform H(x) [§2, eq. 1] transformed = H(inputs, W_H) # (batch, d) # Transform gate T(x) = sigmoid(W_T^T x + b_T) [§2.2] gate = sigmoid(inputs @ W_T + b_T) # (batch, d), in (0, 1) # Carry gate fixed to 1 - T [§2, eq. 3] carry = 1 - gate # (batch, d) # Mix transform and input, element-wise [§2, eq. 2-3] return transformed * gate + inputs * carry ``` ```pseudocode # Negative gate-bias initialisation [arxiv:1505.00387 §2.2] # Input: layers (highway layers); bias_value < 0 # Output: the same layers, with gates biased toward carry def init_gate_bias(layers, bias_value): # Bias every gate toward carrying the input at the start [§2.2] for layer in layers: layer.gate_bias = bias_value # b_T, e.g. -1 to -4 return layers ``` Gradient, for the two limit cases only. The paper does not write the general derivative of eq. 3 [arxiv:1505.00387 §2, eq. 5]: - Gate at 0: the Jacobian with respect to x is the identity. - Gate at 1: the Jacobian is the derivative of H. Gate-bias values seen in the sources: | Setting | Value | Source | |---|---|---| | Random-search range for b_T | −1 to −10 (highway nets only) | [arxiv:1505.00387 §3.1] | | Example values in the text | −1 or −3 | [arxiv:1505.00387 §2.2] | | Analysis nets, initial value | −2 (MNIST), −4 (CIFAR-100) | [arxiv:1505.00387 §4] [fig. 2] | Most gate biases decreased during training in the analysis nets, so they do not stay at their initial value [arxiv:1505.00387 §4]. ## Contradictions None between papers, since only one paper is covered. Within that paper, the text and the table name the same networks differently. The text says Highway 1 and Highway 4 follow FitNet 1 and FitNet 4 [arxiv:1505.00387 §3.2]; the table names the FitNet-4-based net "Highway 2" and uses "Highway 4" for a 32-layer net [arxiv:1505.00387 tab. 1]. Use the table's labels when citing numbers. ## Papers - [Highway Networks](/papers/1505.00387): introduces the sigmoid-gated highway layer and negative gate-bias initialisation; reports trained 100-layer MNIST nets and a 19-layer convolutional CIFAR-10 net at 92.24%.