BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond

In plain words

Video is a stack of frames, and the same scene shows up in many of them, though each frame is slightly shifted by motion. BasicVSR takes a low-resolution video and produces a sharper, larger version. It uses information from the whole clip, not a short window of frames: it sweeps through time once forward and once backward, carrying a running memory along each sweep, so the first and last frames both get to use the rest of the video.

Before the memory is combined with the current frame, it is shifted to line up with that frame using estimated motion. The shifting happens on internal feature maps rather than on raw pixels. The authors argue that this plain design, with few extra parts, is enough to beat more elaborate models while running faster.

A second model, IconVSR, adds two small extras. It refreshes the memory from a few selected frames, and it lets the two sweeps share what they have seen.

TL;DR

• A simple baseline for video super-resolution (VSR), built from four parts: bidirectional propagation, flow-based feature alignment, feature concatenation and pixel-shuffle upsampling §3.1.
• Author-reported: BasicVSR beats EDVR on REDS4 (BI), UDM10 and Vid4 (BD), but EDVR stays higher on Vimeo-90K-T and on Vid4 (BI) tab. 2.
• Author-reported runtime: 63 ms for BasicVSR against 378 ms for EDVR, at an LR size of 180×320; the GPU used for timing is not stated tab. 2.
• IconVSR adds information-refill and coupled propagation, for up to +0.31 dB over BasicVSR at 8.7M parameters §3.2 tab. 2.
• The official repository does not contain the model code (see Paper vs code).

Problem

Video SR has to combine information from neighbouring frames, but those frames are misaligned because of motion. The authors note that many published methods are complex: sliding-window designs use deformable alignment and attention (EDVR), and recurrent designs stack several projection modules (RBPN). The extra parts raise runtime and model size, and dissimilar designs make reimplementation and fair comparison hard §1 fig. 1.

Two propagation choices fail in known ways:

• Local windows discard distant frames. When the authors split test sequences into K independent segments, PSNR drops as the segments shrink, and the drop is largest at the segment ends §3.1 fig. 3.
• Unidirectional recurrence starves early frames. The first frame sees only itself, so early frames score lower. The authors measure a consistent drop of about 0.5 dB when only part of the sequence is used §3.1 fig. 4.

The authors also assume that long sequences are realistic inputs, which is the premise of the design §4.

Method

BasicVSR has four parts, one per functionality: propagation, alignment, aggregation and upsampling §3.1 fig. 2 tab. 1. The table below summarises how each method in the paper's comparison fills them.

Source: tab. 1.

Propagation (bidirectional). Two recurrent branches run in opposite time directions and do not exchange information. The backward state at frame i is computed from frame i and the backward state of frame i+1. The forward state at frame i is computed from frame i and the forward state of frame i-1 §3.1:

h_i^b = F_b(x_i, x_{i+1}, h_{i+1}^b), \qquad h_i^f = F_f(x_i, x_{i-1}, h_{i-1}^f) eq. 1

Alignment (flow-based, on features). For each branch, the flow between frame i and its sweep neighbour is estimated from the LR frames. The neighbour's state is warped with that flow, and residual blocks refine the result §3.1:

s_i^{b,f} = S(x_i, x_{i\pm1}), \qquad \bar{h}_i^{b,f} = W(h_{i\pm1}^{b,f}, s_i^{b,f}), \qquad h_i^{b,f} = R_{b,f}(x_i, \bar{h}_i^{b,f}) eq. 2

Here S is the flow estimator, W is the spatial warp, and R is a stack of residual blocks. Warping features rather than images is the authors' central alignment choice: image warping lost 0.17 dB in their test, and removing alignment entirely lost 1.19 dB §3.1.

Aggregation and upsampling. The two final states are combined and passed to an upsampler made of convolutions and pixel-shuffle layers §3.1:

y_i = U(h_i^f, h_i^b) eq. 3

The paper's own summary is that concatenation and pixel-shuffle are the commonly used choices and were sufficient here §3.1.

# BasicVSR forward pass  [§3.1] [eq. 1-3] [fig. 2]
# Input:  lr_frames (T, 3, H, W)
# Output: hr_frames (T, 3, 4H, 4W)

def basicvsr(lr_frames):
    # Backward sweep, last frame to first  [eq. 1]
    backward_states = run_branch(lr_frames, direction=-1)
    # Forward sweep, first frame to last  [eq. 1]
    forward_states = run_branch(lr_frames, direction=+1)
    hr_frames = []
    for i in range(T):
        # Upsample the two states of frame i  [eq. 3]
        hr_frames.append(upsample(forward_states[i], backward_states[i]))
    return hr_frames


def run_branch(lr_frames, direction):
    # Zero state at the start of the sweep  [fig. 4]
    state = zeros(C, H, W)                # C = 64 channels  [app. A]
    states = [None] * T
    for i in sweep_order(T, direction):
        # Flow to the sweep neighbour, undefined at the sequence end
        # (see Pitfalls)  [eq. 2]
        flow = flow_net(lr_frames[i], lr_frames[i + direction])
        # Move the carried state onto frame i  [eq. 2]
        aligned = warp(state, flow)
        # 30 residual blocks refine the aligned state  [app. A] [eq. 2]
        state = residual_blocks(lr_frames[i], aligned)
        states[i] = state
    return states

IconVSR: information-refill. Keyframes are chosen every five frames in all experiments app. A. At a keyframe, a separate extractor reads three consecutive LR frames and its output is fused with the aligned state by one convolution. Non-keyframes skip this step. The fused feature then goes through the residual blocks §3.2 fig. 5:

e_i = E(x_{i-1}, x_i, x_{i+1}), \qquad \hat{h}_i^{b,f} = \begin{cases} C(e_i, \bar{h}_i^{b,f}) & i \in I_{key} \\ \bar{h}_i^{b,f} & \text{otherwise} \end{cases} eq. 4

h_i^{b,f} = R_{b,f}(x_i, \hat{h}_i^{b,f}) eq. 5

In the paper, E is EDVR-M, a lightweight EDVR app. A. Unlike interval-based methods that process each keyframe interval on its own, the intervals in IconVSR stay connected through the propagation branches §3.2.

IconVSR: coupled propagation. The backward state of frame i is also fed to the forward branch at frame i. The output then uses only the forward state §3.2:

h_i^b = F_b(x_i, x_{i+1}, h_{i+1}^b), \qquad h_i^f = F_f(x_i, x_{i-1}, h_i^b, h_{i-1}^f), \qquad y_i = U(h_i^f) eq. 6

The authors say this adds no computational cost, because only the branch connections change §3.2.

# IconVSR changes to BasicVSR  [§3.2] [eq. 4-6] [fig. 5]
# Keyframes: every 5th frame  [app. A]

def icon_forward_step(i, state, backward_state, lr_frames):
    # Align the previous forward state to frame i  [eq. 2]
    flow = flow_net(lr_frames[i], lr_frames[i - 1])
    aligned = warp(state, flow)
    # Information-refill, keyframes only  [eq. 4]
    if i in keyframes:
        extra = extractor(lr_frames[i - 1], lr_frames[i], lr_frames[i + 1])
        aligned = fuse_conv(extra, aligned)
    # Coupled input from the backward branch; where it enters the
    # residual blocks is not stated  [eq. 5] [eq. 6]
    return residual_blocks(lr_frames[i], aligned, backward_state)

Hyperparameters

Values are as stated in the paper unless marked as from code. "Not stated" means the paper and its appendix do not give the value.

Charbonnier loss, from the appendix app. A:

\mathcal{L} = \frac{1}{N} \sum_{i=0}^{N} \rho(y_i - z_i), \qquad \rho(x) = \sqrt{x^2 + \epsilon^2} eq. 7

Here z_i is the ground-truth HR frame and N is the number of pixels.

Data splits §4 app. A:

• Training: REDS, minus the test and validation clips; Vimeo-90K.
• Test: REDS4 (clips 000, 011, 015, 020 of the REDS training set), Vid4, UDM10 and Vimeo-90K-T.
• Validation: REDSval4 (clips 000, 001, 006, 017 of the REDS validation set).

Model complexity tab. 4 app. A:

The paper's parameter counts include the flow estimator, so the comparison with other methods is said to be fair §4.

Results

Author-reported. Setup: 4× SR, trained on REDS and Vimeo-90K, tested on REDS4, Vid4, UDM10 and Vimeo-90K-T with BI and BD degradations. Metrics are PSNR/SSIM, computed on the Y channel except REDS4, which uses RGB. The runtime is measured at an LR size of 180×320 tab. 2 §4. The paper compares against 14 models.

Full quantitative comparison tab. 2. Format: PSNR/SSIM. Blank cells in the paper are "-".

The paper marks best results in red and second-best in blue. Those marks are lost in the table above.

Headline comparisons, as stated in the text and checked against tab. 2:

• BasicVSR vs RSDN: +0.61 dB on UDM10 (BD) at a similar parameter count (6.3M vs 6.2M) §4.
• BasicVSR vs EDVR: +0.33 dB on REDS4 (BI), with 6.3M vs 20.6M parameters and 63 ms vs 378 ms §4 tab. 2.
• Speed: "up to 24×" is the ratio of RBPN's runtime to BasicVSR's in tab. 2 (1507 ms vs 63 ms) §1.
• IconVSR vs BasicVSR: up to +0.31 dB (Vimeo-90K-T BD); the paper's gains are largest on Vimeo-90K-T and REDS4 §4 tab. 2.

Ablation of IconVSR components, PSNR on REDS4 / REDSval4 tab. 3 §5.1:

The paper says the two components add up to 0.28 dB over BasicVSR tab. 3. Without any keyframes, IconVSR behaves as a plain recurrent network and reaches 30.38 dB on REDSval4, which is 0.21 dB above BasicVSR §5.2.

Keyframe count at test time: fig. 10 plots PSNR on REDSval4 against the number of keyframes N for 100-frame clips with evenly spaced keyframes. The authors report that PSNR rises with N. The exact values are only in the figure.

Qualitative results are in fig. 6 (REDS4), fig. 7 (Vimeo-90K-T) and app. B (more datasets, plus an IconVSR ablation). The paper describes them in words: the outputs have sharper edges, and only BasicVSR and IconVSR recover some square and strip patterns §4.

Paper vs code

Repository: https://github.com/ckkelvinchan/BasicVSR-IconVSR, at commit 15dda03d77127ca54f08c53e30aff42b61c0a0fb, Apache-2.0. The owner is the first author's account, and the project page the paper's authors maintain links to it.

• The paper states that code "will be made publicly available" app. A. At this commit the repository holds only README.md, LICENSE and BD_degradation.m. It has no model, training, loss, or configuration code, so no architecture or training hyperparameter can be checked against code.
• The README points to OpenMMLab's MMEditing for BasicVSR and IconVSR. That is a third-party project, so we do not use it here and did not check it.
• The one code file, the BD degradation script, matches the paper's stated σ and subsampling. It adds a kernel size of 12, replicate boundary padding and a sampling offset of 2 pixels BD_degradation.m. None of these is stated in the paper.

Search log: the paper has no code link in its text or arXiv metadata. The project page links the same repository. The GitHub search for "BasicVSR_IconVSR" returned that repository first, owned by the first author.

Pitfalls

• Sequence ends are undefined. Eq. 1 uses x_{T+1} for the backward branch and x_0 for the forward branch, and eq. 4 uses x_{i+1} at the last keyframe. The paper says the bidirectional model uses zero feature initialisation fig. 4, but not how the flow, or the frames beyond the ends, are handled.
• Warp details are not stated. Interpolation method, flow sign convention, and the spatial resolution of the carried state are not given. The state is presumably at LR resolution, but the paper does not say so.
• Input channels are not stated. It is not clear which tensors are concatenated before the residual blocks R and before the upsampler U, nor how many convolutions U has or what upscaling each pixel-shuffle step applies.
• Output residual unclear. The paper does not say whether U predicts a residual on top of a bicubic-upsampled input or the HR frame directly.
• Coupled input placement. Eq. 6 feeds h_i^b into F_f, but does not say which layer of the forward branch receives it.
• Loss notation. In eq. 7, N is both the pixel count and the upper index of the frame sum. The pixel value range (0–1 or 0–255) is not stated, and ε = 10⁻⁸ scales with it.
• Vimeo-90K length. The compiled appendix says sequences are extended to 14 frames by time-flipping. A commented-out paragraph in the LaTeX source describes mirroring 7-frame clips, but it is not in the paper. The exact construction is therefore ambiguous.
• Mixed claims. The text says BasicVSR outperforms prior methods "including REDS4, UDM10, and Vid4". tab. 2 shows EDVR ahead on Vid4 (BI), 27.35 vs 27.24. Check the degradation column before quoting.
• Runtime is limited. The runtime is taken at a single LR size, 180×320, and the GPU is not named. The "24×" headline compares against RBPN, not against the fastest baseline.
• Keyframe mismatch. IconVSR is trained with a fixed interval of 5. The paper says fewer keyframes can be used at inference §5.2, but reports the quality effect only as a plot fig. 10.
• Whole-video inference. Test inputs are whole sequences app. A. Memory use for long videos is not reported.
• Results are author-reported. REDS4 is scored in RGB, the rest on Y. Numbers from other papers must use the same protocol before they are compared.
• Missing pieces. The Adam β and ε, the minimum learning rate of the cosine schedule and the weight initialisation are not stated.

Lineage

• Builds on: SPyNet for optical flow app. A; EDVR-M as the IconVSR feature extractor app. A; residual blocks and pixel-shuffle upsampling, as stated in §3.1.
• Compared against: the 14 methods in tab. 2 (VESPCN, SPMC, TOFlow, FRVSR, DUF, RBPN, EDVR-M, EDVR, MuCAN, PFNL, RLSP, TGA, RSDN, RRN).
• Superseded by: not known from wiki content.