Residual learning

Overview

A residual unit outputs y = h(x) + F(x). F is a learned block of layers and h is a shortcut that carries the input x forward. When h is the identity, the block only has to learn a correction to its input 1711.10684 §2.1.2. The same shape appears in an MLP block, where the shortcut is a linear map 2304.08424 §4. The skip can also span a whole network instead of one block: an MRI reconstruction network outputs a correction that is added to its zero-filled input 1709.00753 §III-E.

Variants differ in five places: what F is (convolutions or an MLP, and how many layers), where normalisation and activation sit relative to the addition, what the shortcut does when input and output shapes differ, how far the skip reaches, and how the branch and shortcut are combined. In spiking networks, adding two spike trains does not give a spike train, so the sum is replaced by a logical gate 2102.04159 §3.3. A ConvNet block can also scale its residual branch per channel before the addition 2201.03545 §2.6. The segmentation paper does not say how the shortcut is matched when shapes differ 1711.10684 §2.1.2, so that case is left to the implementer. An encrypted ResNet-20 does state it: a 1x1 convolution on the shortcut at each of its two stage changes 2106.07229 tab. 1. In offline RL, a residual MLP adds a bounded correction to each action sampled from a fixed flow policy 2601.10471 §3.2. In scene text recognition, the residual part is only the encoder backbone 1709.02054 §4.2.

The motivation given varies by paper. One paper argues that residual connections ease training of deep stacks when labels are scarce, but it does not compare against a non-residual network of the same size 1711.10684 §2.1.1. Another paper ablates its connections and finds that removing them lowers accuracy at some forecast horizons but not at the longest one 2304.08424 §5.4. In spiking networks, the authors argue that the ordinary residual block cannot pass its input through unchanged, and that its gradient product shrinks or grows with depth 2102.04159 §3.2.

This page rests on thirteen cards from different fields. Each reports results on its own benchmarks, so the page shows the design space, not a ranking. Two of the thirteen (point-cloud completion and scene text) use residual structure only in a weaker sense, and their entries say so.

Variants

Full pre-activation residual unit (segmentation)

The unit applies batch normalisation and ReLU before each of two 3x3 convolutions, and adds the identity shortcut 1711.10684 §2.1.2 fig. 1. Downsampling is a stride of 2 in the first convolution of a unit, not pooling, and channel counts change between levels 1711.10684 tab. I. The paper places this unit at every level of a 7-level encoder-decoder and removes U-Net's cropping step, so input and output have the same size 1711.10684 §2.1.3 1711.10684 §2.3. Whether an activation follows the addition is not stated 1711.10684 §2.1.2. Card: Road Extraction by Deep Residual U-Net.

Residual-on-residual with channel gating (denoising)

The network chains four enhancement attention modules. Each module adds its output to its input, and the body output is added to the extracted features 1904.07396 §3.2 1904.07396 eq. 6. Inside each module, a channel gate computed from globally pooled features rescales the channels 1904.07396 §3.2.1. The network's output is added to the noisy input 1904.07396 §3.2 1904.07396 eq. 3. In the ablation, the four check-mark columns (LSC, SSC, LC, FA) add to a baseline of 28.45 dB, reaching 28.96 dB with all of them on BSD68 1904.07396 tab. 1. This is a single run per row. Card: Real Image Denoising with Feature Attention.

Residual blocks after warping (video super-resolution)

BasicVSR refines a carried feature state with residual blocks after the state has been warped by estimated optical flow. Each branch uses 30 blocks of 64 channels 2012.02181 eq. 2 2012.02181 app. A. Here the residual blocks are a refinement stage, not the skip path that the other cards study. The card records no ablation of the residual blocks themselves; its ablations concern alignment and the IconVSR components 2012.02181 §3.1 2012.02181 tab. 3. Card: BasicVSR.

Residual MLP block with linear skip (forecasting)

Each block has a ReLU hidden layer, dropout on the output layer only, a fully linear skip, and layer norm on the sum 2304.08424 §4. The same block serves as the feature projection, inside the dense encoder, and in the temporal decoder 2304.08424 §4.1 2304.08424 §4.2. A global linear map from the look-back to the horizon is added to the output, so a purely linear model is a special case 2304.08424 §4.2. Card: Long-term Forecasting with TiDE.

Global residual over a zero-filled input (MRI reconstruction)

The generator is a convolutional autoencoder built from residual blocks. It takes the zero-filled image from an under-sampled k-space and predicts a residual, which is added back to that input, so the network learns the aliasing artefact rather than the image itself 1709.00753 §III-E. The paper stacks a second generator of the same design on the first, with its own weights 1709.00753 §III-G. Training adds a cyclic data-consistency term to the adversarial loss, with weights of 1.0 and 10.0 [arxiv:1709.00753 §III-D] 1709.00753 §III-F. Reported time is 0.061 s (ReconGAN) and 0.106 s (RefineGAN) per 256x256 brain image, against 8.57 s for CSCMRI 1709.00753 tab. 1. The hardware is not stated, and the sampling rate used for timing is not stated either. Card: Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss.

Bottleneck residual blocks in a detect-then-repaint pipeline (raindrop removal)

The detector is five convolutions with six bottleneck residual blocks, batch normalisation and ReLU after every convolution, and a stride of 2 in the second convolution 2609.37870 §4.1. Each bottleneck is a 1x1 convolution down to 64 channels, a 3x3 convolution, and a 1x1 convolution up to 256 channels 2609.37870 §4.1. The reconstruction network has the same design with eight residual blocks and a four-channel input 2609.37870 §4.2. A refine network with two residual blocks follows 2609.37870 §4.3. The card does not describe the shortcut in words, nor whether a block is trained to learn a correction to its input, so this entry records the block layout only. Card: Learning from synthetic photorealistic raindrop for single image raindrop removal.

Spike-element-wise residual block (spiking networks)

The branch A is a spiking residual mapping and S is the block input. The output is g(A, S), where g is ADD, AND or IAND 2102.04159 eq. 9 2102.04159 tab. 1. ADD is not binary, so k stacked blocks give outputs of at most k + 1. AND and IAND keep spikes binary 2102.04159 §3.3. Identity is reached by zeroing the weight and bias of the last BatchNorm in the branch for ADD and IAND, which makes A zero 2102.04159 §3.3. In an identity region with A held constant, the gradient through k blocks is exactly 1 2102.04159 eq. 11. The paper says deep directly trained Spiking ResNet-50, -101 and -152 do not converge without zero initialisation, but it does not say the same for SEW ResNet 2102.04159 §4.1. Card: Deep Residual Learning in Spiking Neural Networks.

Residual blocks in a position-map regressor (3D face)

One convolution is followed by 10 residual blocks that reduce a 256x256 face crop to an 8x8x512 feature map. A decoder of 17 transposed convolutions then writes a 256x256x3 map, where each cell holds the x, y and z position of one face point 1803.07835 §3.2 1803.07835 §3.1. The paper does not state the channel widths or strides of the residual blocks, so an implementer must choose them 1803.07835 §3.2. Card: Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network.

Inverted-bottleneck block with layer scale (ConvNet backbone)

The block is a 7x7 depthwise convolution, a LayerNorm, an expansion to four times the channels, a single GELU, a projection back, and a per-channel layer scale. The result is added to the block input, and stochastic depth is applied to the residual branch 2201.03545 §2.4 2201.03545 §2.5 2201.03545 §2.6. Layer scale starts at 1e-6 2201.03545 app. A.1. The paper reaches this block by changing a ResNet-50 one step at a time, and keeps each step only if it improves ImageNet-1K accuracy 2201.03545 §2 2201.03545 tab. 10. Card: A ConvNet for the 2020s.

Residual correction to a sampled action (offline RL)

A multi-step flow-matching policy is trained by flow matching alone and then kept fixed as a behaviour prior 2601.10471 §3.2. A four-layer MLP of width 512 adds a correction to each sampled action, and the executed action is the clipped sum of the base and the correction 2601.10471 app. B 2601.10471 alg. 1. The base action enters with its gradient stopped, so Q-learning updates only the correction 2601.10471 §3.2. The correction takes the sampled action as input, not only the state 2601.10471 §3.2 2601.10471 §5. A penalty on the mean squared correction keeps it near a per-task budget delta 2601.10471 §3.2 2601.10471 app. B.1. On cube-double task 1, the base flow alone scores 10 ± 2 and the same checkpoints with the correction score 62 ± 12, over 50 episodes 2601.10471 app. C.3. Card: DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction.

Additive skips in a grid encoder-decoder (point-cloud completion)

The encoder has four blocks, each a 4x4x4 3D convolution, batch normalisation, leaky ReLU and 2x2x2 max pooling, with 32, 64, 128 and 256 channels 2006.03761 §3.3. Encoder and decoder maps of the same size are added, in the manner of U-Net connections 2006.03761 §3.3. The card does not describe a block that learns a residual on a shortcut, and it does not explain the word "residual" in the title. This entry therefore records a skip sum only. The paper reports 2.723 against 4.016 for PCN on ShapeNet, in x10^-4 Chamfer 2006.03761 tab. 1. Card: GRNet: Gridding Residual Network for Dense Point Cloud Completion.

Residual encoder backbone with a focusing network (scene text)

The encoder is a 32-layer ResNet-style CNN, followed by a bidirectional LSTM 1709.02054 §4.2 1709.02054 tab. 1. The card's text does not describe the shortcut, so the residual part here is the backbone only. The focusing network is a separate loss on the attention, weighted by lambda; lambda of 0 disables it 1709.02054 §4.2 1709.02054 tab. 6. Part of the training data has character pixel labels, and the paper uses about 4 million synthetic word images for that subset 1709.02054 §4.2. Card: Focusing Attention: Towards Accurate Text Recognition in Natural Images.

Residual blocks under encryption (ResNet-20 on CKKS)

The network is a standard ResNet-20 with the same stages and blocks, trained in plaintext and then evaluated on an encrypted image 2106.07229 §4.1. Its shortcut uses a 1x1 convolution at each of the two stage changes [arxiv:2106.07229 tab. 1]. The residual addition itself is not written out in the card, so the card records the layer list and the shortcut projection only. Two changes make the block computable under encryption: ReLU becomes a composition of minimax polynomials, and bootstrapping is inserted in the middle and at the end of each ReLU 2106.07229 §3.4. The card reports a runtime of about 4 hours per image on a 112-core CPU 2106.07229 tab. 5. Whether the encrypted runs are real or simulated is not clear from the text 2106.07229 §4.1 2106.07229 §4.2. Card: Encrypted ResNet-20 with CKKS bootstrapping.

When to use which

Implementation sketch

The shared core is one line of arithmetic. The variants differ inside the block, in the shortcut, and in the combine rule.

# Common core: output = shortcut(x) + F(x)  [arxiv:1711.10684 §2.1.2]

def residual_unit(x, block, shortcut=identity):
    # Learned residual F(x): one or more layers  [arxiv:1711.10684 §2.1.2]
    residual = block(x)
    # Identity shortcut when shapes match; when they do not, a projection
    # or padding is needed. The segmentation paper does not say which
    # [arxiv:1711.10684 §2.1.2]; the encrypted ResNet-20 uses a 1x1 conv
    # [arxiv:2106.07229 tab. 1]
    return shortcut(x) + residual

# Variant: global residual over a zero-filled input  [arxiv:1709.00753 §III-E]

def refine_reconstruct(zero_filled, body):
    # The network predicts a correction, not the image  [arxiv:1709.00753 §III-E]
    correction = body(zero_filled)
    # The skip spans the whole network: input plus predicted residual
    # [arxiv:1709.00753 §III-E]
    return zero_filled + correction

# Variant: full pre-activation block, two BN-ReLU-conv stages
# [arxiv:1711.10684 §2.1.2] [arxiv:1711.10684 tab. I]

def preact_block(x, filters, stride):
    # Normalise and activate before each convolution  [arxiv:1711.10684 §2.1.2]
    h = conv3x3(relu(batch_norm(x)), filters, stride=stride)
    # Second stage keeps the spatial size  [arxiv:1711.10684 tab. I]
    return conv3x3(relu(batch_norm(h)), filters, stride=1)

# Variant: MLP block with linear skip and layer norm  [arxiv:2304.08424 §4]

def mlp_block(x):
    # ReLU hidden layer, dropout on the output layer only  [arxiv:2304.08424 §4]
    h = dropout(linear_out(relu(linear_hidden(x))))
    # Fully linear skip, then layer norm on the sum  [arxiv:2304.08424 §4]
    return layer_norm(h + linear_skip(x))

# Variant: spike-element-wise block, combine rule g  [arxiv:2102.04159 eq. 9]
# [arxiv:2102.04159 §3.3]

def sew_block(s, branch, rule="IAND"):
    # Spiking residual branch A = SN(F(S))  [arxiv:2102.04159 eq. 9]
    a = branch(s)
    # ADD is not binary; the output of k blocks is at most k + 1  [§3.3]
    if rule == "ADD":
        return a + s
    # IAND keeps spikes binary; for spikes it equals (1 - A) * S  [§3.3]
    if rule == "IAND":
        return (1 - a) * s
    raise ValueError(rule)

# Variant: residual correction on a fixed prior  [arxiv:2601.10471 §3.2] [alg. 1]

def residual_policy_act(s, prior_sample, residual, bounds):
    # Base action from the frozen flow prior; its gradient is stopped  [§3.2]
    base = stop_gradient(prior_sample)
    # Correction is an MLP of the state and the sampled action  [§3.2]
    shift = residual(s, base)
    # Executed action is the clipped sum  [alg. 1]
    return clip(base + shift, bounds)

Contradictions

No two papers report incompatible results for the same setting. Several cards show gaps between the paper text and the released code, and some text is unclear. None of these is resolved here.

• GRNet, the word "residual". The title names a residual network, but the body describes plain convolution blocks with additive U-Net-style skips 2006.03761 §3.3. The card does not explain the name. The paper's decoder stride cannot reach the stated 64 cubed grid, while the released code uses stride 2 and adds maps of matching size 2006.03761 §3.3. See the card's Paper vs code section.
• RIDNet, skip paths. The paper adds each module's output to its input and adds the extracted features to the body output 1904.07396 §3.2 1904.07396 eq. 6. The card's Paper vs code section records that the released model does not include these additions as written.
• TiDE, block count. The paper describes the encoder as multiple residual blocks and tunes the layer count over {1, 2, 3} 2304.08424 §4.1 2304.08424 tab. 7. The card records that the released code builds one fewer block than the layer count, so a count of one has no residual block, and the published Traffic, Weather and ETTm1 settings use one 2304.08424 tab. 8. See the card's Paper vs code section.
• RefineGAN, residual block size. The text says conv_i, which has stride 1, "reduces the dimension of the tensor by half". Read literally, a stride of 1 does not halve the spatial size, and the intended design is not stated 1709.00753 §III-E. See the card's Pitfalls.
• RIDNet, internal numbers. The text gives 3.82 dB and 4.14 dB over NC and NI 1904.07396 §4.3.4, while the table implies 3.80 dB and 4.12 dB 1904.07396 tab. 5. The card records the mismatch.
• SEW ResNet, block counts. The paper says there are 7, 15 and 15 SEW blocks in ResNet-18, -34 and -50 2102.04159 §4.1. The card's Paper vs code section records that the released code builds 8, 16 and 16, and the paper does not explain the difference.
• ConvNeXt, large kernels. Section 2.5 says the ResNet-200 regime also stops gaining at 7x7 2201.03545 §2.5. Appendix C says it saturates at 5x5 instead 2201.03545 app. C. Table 11 gives 82.32 for 5x5 and 82.30 for 7x7 2201.03545 tab. 11. The kernel sits inside the residual branch, not on the shortcut.

Papers

• Compressed Sensing MRI Reconstruction using a Generative Adversarial Network with a Cyclic Loss: a residual encoder-decoder generator that adds its output to the zero-filled image, trained adversarially with a cyclic data-consistency loss and chained with a second generator.
• Road Extraction by Deep Residual U-Net: full pre-activation residual units in a U-Net-style network with no cropping, reported to beat U-Net on one road dataset.
• Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network: a residual encoder-decoder that regresses a UV position map of 3D face points from one image, with a region-weighted loss and no 3D model fitting.
• Real Image Denoising with Feature Attention: residual-on-residual modules with channel feature attention for single-stage blind denoising of real photographs.
• BasicVSR: residual blocks refine warped features in a bidirectional recurrent video super-resolution baseline.
• Deep Residual Learning in Spiking Neural Networks: a spike-element-wise residual block that combines the branch and the input with ADD, AND or IAND, trained directly on ImageNet and DVS data.
• Encrypted ResNet-20 with CKKS bootstrapping: a ResNet-20 with bootstrapping and polynomial ReLU, evaluated on encrypted CIFAR-10 images with the plaintext weights.
• Long-term Forecasting with TiDE: residual MLP blocks with a linear skip, used as the building block of a long-horizon forecaster.
• A ConvNet for the 2020s: an inverted-bottleneck residual block with 7x7 depthwise convolution and layer scale, reached by modernising a ResNet-50 step by step.
• Learning from synthetic photorealistic raindrop for single image raindrop removal: bottleneck residual blocks in a three-stage detector, reconstructor and refiner, trained on a rendered raindrop dataset.
• Focusing Attention: Towards Accurate Text Recognition in Natural Images: a ResNet-style encoder with an attention decoder, plus a focusing network trained on part of the data with character pixel labels.
• GRNet: Gridding Residual Network for Dense Point Cloud Completion: a grid encoder-decoder with additive skips that completes partial point clouds through a gridding step.
• DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction: a fixed flow-matching prior with a small residual MLP that shifts each sampled action toward higher critic value.