Video frame interpolation

Overview

The task is to produce new frames between existing ones, so that a clip gains a factor k in frame rate. Between each pair of original frames the method must output k-1 frames 2012.08512 §3. The classic route estimates optical flow between neighbouring frames and warps pixels along it. The FLAVR paper names three costs of that route: errors in the flow estimate, weak handling of occlusions, and a warping step that is slow at inference 2012.08512 §1. Every variant has to learn the motion somehow, and every variant can build its training pairs from ordinary video by subsampling it, so no motion labels are needed 2012.08512 §3.

Variants

Flow-free 3D U-Net (FLAVR)

The network reads a clip of 2C neighbouring frames, stacked along time. A 3D residual encoder (R3D-18 without its classifier) with no temporal stride feeds a 3D U-Net decoder. Skip connections join encoder and decoder by channel concatenation. Time is folded into channels and fused by a 2D convolution, and a 7x7 convolution emits 3(k-1) channels, which become k-1 RGB frames in one pass. Spatio-temporal channel gating follows the (de)convolutions. The only loss in the main model is L1 2012.08512 §3 2012.08512 tab. 3. Card: FLAVR.

Author-reported results on the RGB-only comparison setting: 36.25 dB PSNR on Vimeo-90K at 2x, against 35.15 dB for QVI (RGB+flow) and 35.40 dB for AdaCoF (RGB) 2012.08512 tab. 1. At 8x on GoPro, 31.31 dB against 30.55 dB for QVI 2012.08512 tab. 2. Against QVI the paper reports runtime gains of 2.7x at 8x, 6.2x at 16x and 12.7x at 32x 2012.08512 §5. Timing used a 2080 Ti with 512 x 512 crops and forward passes only 2012.08512 § Benchmarking.

Flow-based methods

The paper compares against flow-based and flow-plus-depth methods such as QVI and DAIN 2012.08512 §4. No card in this wiki covers them, so their designs are not described here. See optical flow for the flow estimation side.

When to use which

Implementation sketch

The sketch follows the FLAVR design. Other variants may differ at every step.

Training sample for a k-times model. Indices are 0-based. Frame pair and frame pair + 1 of the subsampled video are the two anchors.

# Build one training sample for a k-times model  [arxiv:2012.08512 §3]
# video: list of frames, k: interpolation factor, C: neighbours per side
def make_sample(video, pair, k, C):
    # Keep every k-th frame to simulate the low frame rate  [arxiv:2012.08512 §3]
    low_rate = video[::k]
    # 2C low-rate frames centred on the two anchors  [arxiv:2012.08512 §3]
    context = low_rate[pair - C + 1 : pair + C + 1]
    # The k-1 original frames strictly between the anchors  [arxiv:2012.08512 §3]
    target = video[pair * k + 1 : (pair + 1) * k]
    return context, target

Inference for one set of k-1 in-between frames:

# One pass: 2C frames in, k-1 frames out  [arxiv:2012.08512 §3]
def interpolate(context, k):
    # Stack the frames along time: (3, 2C, H, W)  [arxiv:2012.08512 §3]
    clip = stack(context, axis=time)
    # Subtract a per-clip mean and add it back at the end  [arxiv:2012.08512 § Training details]
    mean = clip.mean(over time, height, width)
    feats = encoder(clip - mean)  # no temporal stride  [arxiv:2012.08512 §3]
    out = decoder(feats)  # skips joined by concatenation  [arxiv:2012.08512 tab. 3]
    out = conv2d(fold_time_into_channels(out))  # [arxiv:2012.08512 §3]
    # 7x7 conv to 3(k-1) channels, split into k-1 RGB frames  [arxiv:2012.08512 §3]
    frames = split(conv7x7(out), groups=k - 1)
    return [f + mean for f in frames]

Implementation notes. The official code differs from the paper in several settings, including batch size, the number of input frames used at 8x and the loss scale. The FLAVR card lists each difference with its location. Copy the paper's values only after checking them against that list.

Contradictions

• Text and table values for 2x. The text gives 33.33 dB on UCF101 and 27.44 dB on DAVIS. The table gives 33.31 and 27.43 2012.08512 §5 2012.08512 tab. 1. Both are reported here as printed; the page does not choose between them.
• Gating ablation. The text says removing spatio-temporal gating lowers PSNR from 36.3 to 36.1 dB. No row of the ablation table shows this 2012.08512 § Ablations 2012.08512 tab. 3.
• Input count at 8x. The text reports C=3 as best for 8x, but the main model's C is not stated 2012.08512 § Ablations. The paper does not say which C the main 8x model uses.

Papers

• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation: a flow-free 3D U-Net that predicts all k-1 frames in one pass, with author-reported speed and quality gains.