BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment

In plain words

A video is a stack of frames, and the same scene appears in many of them, but each frame is shifted a little by motion. Video super-resolution takes a low-resolution video and produces a sharper, larger one by borrowing detail from neighbouring frames. The earlier BasicVSR model already did this by passing a running memory forward and backward through the clip. This paper keeps that design and makes two parts work harder.

The first change is in how the memory moves through time. It is passed back and forth several times, and each step looks at the two most recent frames instead of one, so detail that is hidden in one frame can be recovered from another. The second change is in how the memory is lined up with the current frame. Motion estimates give a rough alignment, and a learned correction refines it. The authors argue that this combination keeps fine detail where a plain shift would blur it, and that it stays fast.

TL;DR

• A video super-resolution (VSR) model built on BasicVSR. It changes propagation (second-order, in a grid of forward and backward sweeps) and alignment (flow-guided deformable convolution) §1 §3.
• The flow is the base of the alignment offsets, and a learned residue corrects it. The authors say this is easier to train than offsets learned from scratch §3.2.
• Author-reported: 32.39 dB on REDS4 (BI, x4) with 7.3M parameters and 77 ms, against 31.42 dB and 6.3M for BasicVSR tab. 1.
• The paper's 0.82 dB gain over BasicVSR is for a smaller variant, BasicVSR++ (S) tab. 2 §4.1.
• The official code is in the MMEditing repository (now MMagic). Several details are not in the paper; see Paper vs code.

Problem

Video SR must combine information from frames that are misaligned by motion. Two families of methods are common. Sliding-window methods restore each frame from a short window of neighbours, which limits how much distant information they use §1. Recurrent methods carry features across the whole clip, which gives more information at lower model size, but moving that information over long distances and aligning it across frames remain hard §1.

Within the recurrent family, BasicVSR uses optical-flow warping for alignment. The authors note that flow is not robust to occlusion, and that deformable alignment, which learns offsets directly, is more expressive but unstable to train, often because the offsets overflow §1 §3.2. The paper addresses both issues in the propagation and alignment stages §1 fig. 1.

Method

BasicVSR++ follows the BasicVSR layout: residual feature extraction per frame, recurrent propagation, aggregation and pixel-shuffle upsampling §3 fig. 2. Its two modifications are in propagation §3.1 and alignment §3.2.

1. Per-frame features. Each low-resolution frame passes through residual blocks. The appendix gives five blocks, 64 channels app. A.
2. Flows. A pre-trained SPyNet estimates optical flow between neighbouring frames. Its parameters and runtime are counted in the model size §4 tab. 1.
3. Grid propagation (§3.1). Four propagation branches run in order: backward, forward, backward, forward. The backward branch starts at the last frame. The second round refines features using the first, so information crosses the clip twice §3.1 §5. Going from one round to two gave a clear gain in the authors' ablation §5.
4. Second-order connection (§3.1). Each step uses the states of the two previous frames in the sweep, not just one. The two earlier states are aligned together by one module rather than separately, which the paper argues is cheaper and keeps complementary information §3.2 eq. 1.
5. Flow-guided deformable alignment (§3.2). For frame i, the two earlier states are warped onto frame i with their flows eq. 3. The warped states and the current feature feed a small convolutional network. It predicts, for each earlier frame, an offset residue and a modulation mask eq. 6. The offset is the flow plus that residue, so the flow is the base and the network only learns a deviation from it §3.2 eq. 6. A modulated deformable convolution then samples the unwarped earlier states with these offsets and masks eq. 5 eq. 7.
6. Residual update (§3.1). The aligned state is combined with the feature from the previous branch at the same frame. A stack of residual blocks produces the branch state eq. 2.
7. Reconstruction (§3.1). The spatial feature and the four branch states at each frame are concatenated, passed through convolution and pixel-shuffle upsampling, and added to the output. The output is four times the input size.

Equations (in the paper's order): eq. 1 alignment operator A; eq. 2 residual update; eq. 3 warp; eq. 4 first-order offset and mask; eq. 5 deformable convolution; eq. 6 second-order offset residue and mask; eq. 7 concatenated offsets and masks with the second-order convolution.

# BasicVSR++ forward pass  [§3.1] [fig. 2]
# Input:  lr_frames (T, 3, H, W)
# Output: hr_frames (T, 3, 4H, 4W)

def basicvsr_plusplus(lr_frames):
    # Residual features per LR frame, kept at LR size  [app. A]
    spatial = [extract(frame) for frame in lr_frames]  # (64, H, W)
    # One-step flows between neighbours, both directions  [§4]
    flows_bwd, flows_fwd = compute_flows(lr_frames)  # (T-1, 2, H, W)
    states = {"spatial": spatial}
    # Two rounds, each a backward then a forward sweep  [§3.1] [§5]
    for name in ["backward_1", "forward_1", "backward_2", "forward_2"]:
        states[name] = propagate(states, name, flows_bwd, flows_fwd)
    # Reconstruct each frame from the spatial feature and all branches  [§3.1]
    return [reconstruct(states, lr_frames, i) for i in range(T)]

# One second-order propagation branch  [§3.1] [eq. 1] [eq. 2]
# Input:  states (per-frame features and earlier branches), branch name
# Output: list of T branch states in time order

def propagate(states, name, flows_bwd, flows_fwd):
    # Visit frames in sweep order; backward starts at the last frame  [§3.1]
    order = frame_order(name, T)
    # Earlier branch the residual update builds on  [§3.1] [eq. 2]
    previous = states[previous_branch_name(name)]
    visited = []  # branch states so far, in visiting order
    result = {}   # branch state keyed by frame index
    for step, i in enumerate(order):
        if step == 0:
            # Nothing earlier yet, so the aligned state is zero  [§3.2]
            aligned = zeros(64, H, W)
        else:
            # Earlier states; zero before the sweep reaches them  [§3.2]
            prev1 = visited[-1]
            prev2 = visited[-2] if step > 1 else zeros(64, H, W)
            # Flows from frame i to one and two frames before  [§3.2]
            # The two-step flow formula is not stated in the paper
            flow1 = flow_between(i, order[step - 1])   # (2, H, W)
            flow2 = two_step_flow(i, order, step)      # (2, H, W)
            aligned = align(states["spatial"][i], prev1, prev2,
                            flow1, flow2)
        # Residual update with the earlier branch at the same frame  [eq. 2]
        result[i] = aligned + refine(concat(previous[i], aligned))
        visited.append(result[i])
    return [result[i] for i in range(T)]

# Flow-guided deformable alignment, second order  [§3.2] [eq. 3-7]
# Inputs: feat (64, H, W); prev1, prev2 (64, H, W); flow1, flow2 (2, H, W)
# Output: aligned feature (64, H, W)

def align(feat, prev1, prev2, flow1, flow2):
    # Warp both earlier states onto frame i with their flows  [eq. 3]
    warped1 = warp(prev1, flow1)  # (64, H, W)
    warped2 = warp(prev2, flow2)  # (64, H, W)
    # Offset and mask heads share their hidden layers  [tab. 5]
    # The paper's formula lists only features as input  [eq. 6]
    hidden = offset_mask_trunk(concat(feat, warped1, warped2))
    # Residue to each flow; the code bounds it with 10 * tanh  [eq. 6]
    residue1 = offset_head1(hidden)  # shape not stated in paper
    residue2 = offset_head2(hidden)  # shape not stated in paper
    offset1 = flow1 + residue1  # [eq. 6]
    offset2 = flow2 + residue2  # [eq. 6]
    masks = sigmoid(mask_head(hidden))  # [eq. 6]
    # Deformable conv on the unwarped states, with the combined offsets  [eq. 7]
    offsets = concat(offset1, offset2)  # [eq. 7]
    return modulated_deform_conv(concat(prev1, prev2), offsets, masks)

# Reconstruction of one output frame  [§3.1] [app. A]
# Input:  states (all branches), lr_frames, frame index i
# Output: hr frame (3, 4H, 4W)

def reconstruct(states, lr_frames, i):
    # Stack spatial feature and the four branch states at frame i  [§3.1]
    hr = concat(
        states["spatial"][i],
        states["backward_1"][i],
        states["forward_1"][i],
        states["backward_2"][i],
        states["forward_2"][i],
    )  # (320, H, W)
    hr = residual_blocks(hr)  # 5 blocks, count not stated in paper
    # Convolution and pixel shuffle to 4x; stages not stated in paper
    hr = pixel_shuffle_stage(hr)  # x2
    hr = pixel_shuffle_stage(hr)  # x2
    hr = conv_last(conv_hr(hr))
    # Add the upsampled input frame; not described in the paper
    return hr + bilinear_upsample(lr_frames[i], scale=4)

Hyperparameters

Results

Author-reported. Setup: x4 super-resolution; test sets REDS4, Vimeo-90K-T, Vid4 and UDM10; BI and BD degradations; PSNR/SSIM on the Y channel, except REDS4 on RGB; runtime at an LR size of 180 x 320; the GPU is not stated in the table caption tab. 1 §4.1. Values are PSNR (dB) / SSIM.

Other author-reported results:

• Smaller variant BasicVSR++ (S): 6.4M parameters, 69 ms, 32.24 dB on REDS4 (BI), against 31.42 dB for BasicVSR and 31.67 dB for IconVSR tab. 2.
• Ablation on the components, dataset not named in the caption. Baseline with none of the new parts 31.48 dB; flow-guided alignment 31.94 dB; plus second-order propagation 32.08 dB; full model 32.39 dB tab. 3.
• Alignment variants, dataset not named in the caption. No flow: 27.44 dB, with training that collapses. Offset-fidelity loss: 30.22 dB. Full model: 32.39 dB tab. 4 §5.
• NTIRE 2021: three champion and one runner-up placements across the video SR and compressed-video enhancement challenges. The paper gives no scores for these challenges §6 §1.

Paper vs code

Code: basicvsr_plusplus_net.py at the repository snapshot 0a560bba9b79ebe78574e1d4cbbdd0e798e63568. The paper links MMEditing in an abstract footnote. The repository is now called MMagic and is a later snapshot than the paper, so line numbers can move. The owner is OpenMMLab; the paper links it, but the authors' own commits are not verified.

• Residue bound. Paper Eq. 6 has no bound on the residue. Code applies 10 * tanh to it basicvsr_plusplus_net.py with the default at basicvsr_plusplus_net.py.
• Flows as estimator input. Paper Eq. 6 feeds only the features to the offset and mask estimator. tab. 5 lists 196 input channels, which equals 3 x 64 + 4, so the two flows are included. Code concatenates them basicvsr_plusplus_net.py and sizes the first layer for them basicvsr_plusplus_net.py.
• Flow channels for the offsets. Code swaps the two flow channels before adding them to the deformable offsets basicvsr_plusplus_net.py. The paper does not describe this.
• Two-step flow. The paper uses flow from frame i to frame i-2 but does not say how it is computed. Code composes it from one-step flows: the one-step flow plus the previous one-step flow warped by it basicvsr_plusplus_net.py.
• Residual update inputs. Paper Eq. 2 combines only the previous branch and the aligned feature. Code also concatenates the spatial feature and every earlier branch output at the same frame basicvsr_plusplus_net.py. Channel counts confirm this: 2, 3, 4 and 5 times 64 for the four branches basicvsr_plusplus_net.py.
• First step of each sweep. Code skips alignment at the first step, so no alignment is computed basicvsr_plusplus_net.py. Paper footnote 2 sets the earlier states and flows to zero, which implies an alignment on zeros. The two match only if the deformable-convolution bias is zero, which the paper does not state.
• Upsampling. Code uses five residual blocks and two x2 pixel-shuffle stages, then adds the bilinearly upsampled input frame basicvsr_plusplus_net.py and basicvsr_plusplus_net.py. The paper names neither the block count, the stage count nor a global skip.
• Vimeo-90K initialisation. Appendix B says the Vimeo model starts from REDS weights. Both Vimeo configs have the load_from line commented out basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py and basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bd.py.
• Flow freezing on Vimeo. Section 4 says the flow network is fixed for the first 5,000 iterations. The Vimeo configs set fix_iter=-1, which never freezes it basicvsr-pp_c64n7_4xb2-300k_vimeo90k-bi.py.
• Adam betas and Charbonnier epsilon. Not stated in the paper. Code uses betas (0.9, 0.99) and epsilon 1e-12 basicvsr-pp_c64n7_8xb1-600k_reds4.py and pixelwise_loss.py.
• Challenge configs. The compressed-video and NTIRE configs use 128 channels and 25 blocks basicvsr-pp_c128n25_600k_ntire-vsr.py. The paper's main configuration is 64 channels and 7 blocks, and its NTIRE section does not describe these settings.
• Mirror-extended inputs. Code checks whether the clip is an exact mirror of itself and, if so, reuses the backward flows as forward flows basicvsr_plusplus_net.py and basicvsr_plusplus_net.py. This is an efficiency shortcut the paper does not mention.
• REDS4 number. The repository README lists 32.3855 dB for the REDS4 config, which matches the paper's 32.39 dB README.md. The README also lists two Vimeo-90K-trained checkpoints under the REDS4 label (31.0126 dB and 29.2041 dB) README.md. The paper does not report these, and their setup is not described.

Pitfalls

• Headline gap. The abstract's 0.82 dB gain over BasicVSR is for BasicVSR++ (S), which has 6.4M parameters tab. 2. The full model is 0.97 dB above BasicVSR on REDS4 (BI) and 0.72 dB above IconVSR, from the table values tab. 1.
• Unnamed datasets. The ablation and alignment tables do not name their dataset. The full-model value 32.39 dB matches the REDS4 value in tab. 1, which suggests REDS4 but does not confirm it.
• Residue bound. The paper's offset equation has no bound, so an implementation that follows the paper alone differs from the released code. The paper does not report what the bound does to training.
• Two-step flow. The paper's two-step flow is undefined. Implementers who estimate it directly, rather than composing it, will get different results.
• Flow channel order. The code reverses the flow's two channels before adding them to the offsets. The paper does not mention this, and a mismatch would not raise an error.
• Vimeo initialisation. Following the paper and starting Vimeo training from scratch will differ from the released configs, because the code does not load REDS weights.
• Long clips. The backward branch needs the whole clip, so output for any frame depends on the entire input. Code moves cached features to CPU when a clip is longer than 100 frames basicvsr_plusplus_net.py. This saves GPU memory and slows inference. The paper does not report it.
• Cost scales with resolution. Features stay at LR size, and upsampling happens only at the end. The runtime in the table is for 180 x 320 LR frames; the paper does not give runtime for other sizes.
• Mirror check. The shortcut applies only to exact mirrors, which the code tests for exact equality. Any other clip computes flows in both directions.
• BD data. BD and BI inputs are prepared datasets in the repo. The Gaussian blur (sigma 1.6) is described in the paper but is not in this code, so it cannot be verified from the repository.
• Test window. Code evaluates REDS4 in 100-frame windows. The paper says full sequences. The behaviour on longer or shorter clips is not described.

Lineage

• Builds on: BasicVSR, which provides bidirectional recurrent propagation and flow-based feature alignment. This paper changes both: the propagation becomes second-order and runs in two rounds, and the alignment uses flow as the base of deformable offsets §1 §3.
• Relies on ideas cited in the paper but without cards in this wiki: deformable alignment from EDVR and TDAN; the higher-order connection cited as soltani2016higher §1.
• Compared against: EDVR, IconVSR, RBPN and others tab. 1.
• Superseded by: not known from wiki content.