Recurrent Back-Projection Network for Video Super-Resolution

In plain words

Video super-resolution means turning a blurry, low-resolution video into a sharp, high-resolution one. A single frame can only be sharpened by guessing the missing detail. Neighbouring frames often show the same scene from a slightly different position, so they contain detail the target frame lacks. The hard part is lining the frames up, because moving objects make that alignment imprecise.

RBPN takes the target frame and walks through its neighbours one at a time. For each neighbour it estimates how pixels moved between that neighbour and the target, and uses that motion as a hint. It keeps a running high-resolution estimate of the target and corrects it at each step, a bit like back-projection in classic multi-image super-resolution: it projects the estimate back down to low resolution, compares that with what was observed, and adds the difference. The final image is built from the estimates collected along the way.

The authors argue that treating every neighbour as its own source of information works better than stacking all frames into one input. They also report that the model is more accurate than earlier methods on the benchmarks they tested, with the largest margins on fast motion.

TL;DR

RBPN is a recurrent encoder-decoder for 4x video super-resolution. Each neighbouring frame, stacked with the target and a dense flow map, enters a loop that combines a single-image upscaler (DBPN) with a multi-frame residual path. The loop refines high-resolution features one neighbour at a time; a single conv layer reconstructs the target from their concatenation. Trained with per-pixel L1 on Vimeo-90k; author-reported gains over VSR-DUF, most on fast motion. Expect to pay for flow estimation and one DBPN call per neighbour.

Problem

Single-image super-resolution ignores the other frames of a video, which carry missing detail §1. Multi-image methods need precise, sub-pixel alignment of frames, and frame-to-frame alignment without temporal cues is hard §1. Recent video methods either concatenate frames as channels, which the authors say is hard to train, or feed frames to recurrent networks, which struggle to model both slow and quick changes §1. The authors also note that the Vid4 benchmark has little motion and only four sequences, which limits how well it separates methods §4.3.

Method

Overview. The input is n+1 LR frames: the target I_t and n context frames §3. The network runs in three stages: initial feature extraction, a chain of projection modules, and reconstruction. It is trained end-to-end §3.1.

Step 1, initial features. The target is mapped to LR features L_t. For each context frame I_{t-k}, a dense flow F_{t-k} between that frame and the target is concatenated with both frames. The resulting 8-channel stack is mapped to neighbour features M_{t-k} §3.1. The paper says the flow is precomputed §3.1; the official code computes it per sample (see Paper vs code).

Step 2, projection module. The projection module is shared across time steps. It takes LR features and neighbour features and outputs HR features H_{t-k} and the next LR features L_{t-k} §3.2. It is an encoder-decoder pair eq. 1 eq. 2. The encoder upscales the LR input with the SISR path and the neighbour features with the MISR path, takes the difference of the two, and returns the SISR path plus a learned residual of that difference eq. 3 eq. 4 eq. 5 eq. 6. The decoder maps the HR features back to LR features for the next step eq. 2.

Step 3, reconstruction. The HR features from all context steps are concatenated and passed to one convolution, which outputs the SR target frame §3.1. The paper describes this as the same style of reconstruction as its DBPN predecessor §3.1.

The paper's interpretation is that the encoder acts as SISR plus MISR, and the decoder "deciphers" the hidden state for the next step §3.3. If the context frames add no information, the MISR module is effectively ignored and the model falls back to recurrent SISR §3.3.

Algorithm. The paper has no algorithm box. The pseudocode below follows §3.1–§3.3 and eqs. 1–6. It is our summary, not the authors' code.

# RBPN forward pass for one target frame  (from §3.1-§3.3, eqs. 1-6)
# Input:  target LR frame (B, 3, H, W); neighbour LR frames and flows
# Output: SR frame (B, 3, sH, sW)

def rbpn_forward(target, neighbours, flows):
    # 1. Map the target to LR features L_t  [§3.1]
    lr_features = target_features(target)     # (B, 256, H, W)

    # 2. Neighbour features M from the 8-channel stack  [§3.1]
    #    stack = target (3) + neighbour (3) + flow (2)
    neighbour_features = []
    for neighbour, flow in zip(neighbours, flows):
        stacked = concat(target, neighbour, flow)
        neighbour_features.append(stacked_features(stacked))  # (B, 256, H, W)

    # 3. Projection chain, one step per neighbour  [§3.2]
    hr_features = []
    for m in neighbour_features:
        # SISR upscale of the LR features  [eq. 3]
        sisr_up = sisr_net(lr_features)       # (B, 64, sH, sW)
        # MISR upscale of the neighbour features  [eq. 4]
        misr_up = misr_net(m)                 # (B, 64, sH, sW)
        # Residual of the difference  [eq. 5]
        residual = residual_net(sisr_up - misr_up)
        # Refined HR features  [eq. 6]
        hr = sisr_up + residual               # (B, 64, sH, sW)
        hr_features.append(hr)
        # Decoder: HR features back to LR features  [eq. 2]
        lr_features = decoder_net(hr)         # (B, 256, H, W)

    # 4. Reconstruct from all HR features concatenated  [§3.1]
    stacked_hr = concat(hr_features)          # (B, n * 64, sH, sW)
    return reconstruction(stacked_hr)         # (B, 3, sH, sW)

Equations. Numbered as they appear in the source's align environments eq. 1-eq. 6:

H_{t-n} = \texttt{Net}_E(L_{t-n-1}, M_{t-n}; \theta_E) eq. 1

L_{t-n} = \texttt{Net}_D(H_{t-n}; \theta_D) eq. 2

H^l_{t-n-1} = \texttt{Net}_{sisr}(L_{t-n-1}) eq. 3

H^m_{t-n} = \texttt{Net}_{misr}(M_{t-n}) eq. 4

e_{t-n} = \texttt{Net}_{res}(H^l_{t-n-1} - H^m_{t-n}) eq. 5

H_{t-n} = H^l_{t-n-1} + e_{t-n} eq. 6

Context order. The paper uses three orders: past only (P), past and future (PF), and random past frames (PR) §4.2. The headline model is RBPN/6-PF, which uses 6 context frames §4.3.

Hyperparameters

All values are from the paper unless the Notes say otherwise. Scale factor is 4x throughout the main text §4.

Results

Author-reported. Setup: 4x SR, trained on Vimeo-90k, reported on Vid4, SPMCS-11 and Vimeo-90k test tiers §4.3. Baselines marked * in the source were copied from their papers.

Vimeo-90k, 4x, PSNR/SSIM tab. 9:

Clip counts are 1,616, 4,983 and 1,225 for slow, medium and fast, with mean flow magnitudes of 0.6, 2.5 and 8.3 tab. 9. The text reports RBPN/6-PF above VSR-DUF by 1.22, 1.44 and 2.54 dB §4.3.

SPMCS-11, 4x, average PSNR/SSIM tab. 8:

Vid4, 4x, average PSNR/SSIM tab. 7. The main row uses no border crop and drops the first and last two frames. A second value with * uses an 8-px crop:

The authors state that RBPN/6-PF is better than prior methods on Vid4 except VSR-DUF §4.3. They also argue that Vid4's ground truth contains artifacts that penalise sharper outputs §4.3.

Ablations on SPMCS-32, 4x, PSNR/SSIM (all from the main text):

The context-length study (fig. 6, RBPN/2 to RBPN/6) reports performance rising by roughly 0.2 dB as context grows §4.2.

Appendix. At 8x on Vimeo-90k (slow/medium/fast), RBPN/6 scores 28.94/0.809, 31.35/0.858 and 33.91/0.890, against 28.40/0.795, 30.28/0.837 and 32.74/0.872 for DBPN app. A.1. On SPMCS-32 at 8x, RBPN/6 scores 26.31/0.708 and DBPN 25.70/0.682. The appendix gives 2x numbers as well app. A.1. Model-size variants: RBPN/6-S 31.39/0.878 with 8,538k parameters, RBPN/6 31.64/0.883 with 12,771k, RBPN/6-L 31.58/0.882 with 17,619k app. A.2. Residual learning on the output does not help: 31.57/0.882 with, 31.64/0.883 without app. A.3. Runtime at 4x on a 120x160 input is 0.141 s for RBPN/6-PF and 0.058 s for RBPN/4-PF app. A.4.

Paper vs code

Official code: alterzero/RBPN-PyTorch, MIT license, copyright Muhammad Haris (GitHub user name matches the first author, and the README links the authors' project page). Commit 50ce38b1b17f3fe3b0b763a475027b9605c536c2. Its HEAD commit date (2020-04-10) is later than the paper's 2019 publication, so it may not match the paper's experiments exactly. Checked against the code at that commit.

• Flow is computed, not precomputed. The paper says the flow is precomputed §3.1. The loader calls get_flow for every neighbour on every sample, after augmentation dataset.py. The flow uses pyflow's coarse-to-fine solver with alpha 0.012, ratio 0.75, minimum width 20, 7 outer and 1 inner fixed-point iterations, and 30 SOR iterations dataset.py. The paper cites [liu2009beyond] for its flow §4.2; the code does not name that source.
• Flow direction. The code computes flow from the target to each neighbour dataset.py. The paper does not state a direction.
• LR schedule. The paper says the rate is divided by 10 "for half of total 150 epochs" §4.1. The code divides by 10 when (epoch+1) % 75 == 0, so after epochs 74 and 149 (1-indexed: 1e-4 for epochs 1-74, 1e-5 for 75-149, 1e-6 for epoch 150) main.py.
• Training colour space. The paper says measurements use only Y §4.1 and says nothing about training colour. The code builds three-channel RGB inputs and computes L1 on RGB main.py dataset.py.
• Augmentation. The code applies a vertical flip, a horizontal mirror and a 180-degree rotation, each with probability 0.5, to the target, neighbours and LR input together dataset.py. The paper lists rotation, flipping and random cropping §4.1.
• Net_res and Net_D have extra layers. The residual branch is five ResNet blocks followed by one 3x3 conv rbpn.py. The decoder is five ResNet blocks followed by the strided conv rbpn.py. The paper text describes only the blocks §4.1.
• Activations and norms. The paper does not state activations. The code uses PReLU and no normalisation rbpn.py. The ResnetBlock class defaults to batch norm base_networks.py, but RBPN passes norm=None.
• Initial feature extraction. The code maps the target with one 3x3 conv (256 channels, PReLU) and the 8-channel stack with another rbpn.py. The paper gives no kernel size for either.
• Recurrence starts from the target. The code starts the chain from the target's LR features and visits neighbours in order -3 to +3 (PF) rbpn.py dataset.py. The paper's indices (L_{t-n-1}, H_{t-n}) do not show where the chain starts.
• Evaluation protocol is not in the code. The PSNR call in eval.py is commented out, so the 8-px border crop and the frame removal rules §4.1 are not implemented there eval.py. The chop_forward path uses an 8-px overlap only for tiling large inputs eval.py.
• Pretraining. The code can load pretrained weights, but the flag defaults to off main.py. The paper does not mention pretraining.
• Residual option. The code keeps a residual flag (default off) main.py and adds bicubic output at test time when it is on eval.py. The paper's appendix says residual learning does not help app. A.3.
• Weights. The repository's weights/weights.txt points to an external download. No weights are in the repository.

Pitfalls

• Frame order matters. With PF, the context order is -3, -2, -1, +1, +2, +3 around the target dataset.py. Order changes PSNR by up to 0.17 dB in the train/test pairs of tab. 4 and by up to 0.05 dB in tab. 5.
• Compute grows with context length. The code runs the SISR module once per neighbour rbpn.py. The complexity table shows 0.141 s for RBPN/6-PF against 0.058 s for RBPN/4-PF app. A.4. The table does not name the hardware.
• Per-sample CPU flow. Computing pyflow for every neighbour in every sample adds CPU work to data loading. The paper's "precomputed" wording §3.1 would mislead a reimplementer. Precomputing flows on the same augmented LR patches would change the training conditions, so decide which one you want and say so.
• Patch geometry. Training crops 64x64 LR patches from 112x64 LR frames §4.1. The crop logic uses PIL sizes with the height and width variables swapped, so the crop offset varies only horizontally dataset.py. Confirm this if you rewrite the loader.
• Vid4 protocol differs. The main Vid4 row uses no border crop and drops the first and last two frames tab. 7. The 8-px crop and six/three-frame rule in §4.1 are different. Compare Vid4 numbers only under the same protocol.
• Complexity table labels. The complexity table lists RBPN/4-PF with PSNR 29.75 app. A.4. The SPMCS-11 table gives 29.75 for RBPN/4-P, the past-only order tab. 8. Treat the label as unclear.
• Uncertain figure numbers. Figure numbers used above follow source order in the LaTeX, not the compiled PDF. Recheck them against the arXiv PDF if they matter.
• Output range and colour. Measuring on Y while training on RGB (see Paper vs code) can change PSNR. Use the paper's Y protocol when comparing with the table numbers.
• Init and learning rate. The paper's "for half of total 150 epochs" is ambiguous. The code's double division by ten is the only concrete rule available.
• Activation and norm choices are not stated in the paper. The code uses PReLU and no normalisation, so use those if you want to match the released model.

Lineage

• Builds on: DBPN, "Deep back-projection networks for super-resolution" [haris2018deep], by the same authors. It provides Net_sisr and the back-projection idea. No card for it exists in this wiki.
• Compared against: DBPN, BRCN, VESPCN, B123+T, VSR-TOFLOW, DRDVSR, FRVSR and VSR-DUF §4.3.
• Superseded by: none recorded in this wiki.