FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation

In plain words

Video frame interpolation means making new in-between frames for a video, for example to turn 30 frames per second into 240. Most methods first estimate how every pixel moves between two neighbouring frames (optical flow) and then warp the pixels. That step is slow, and it breaks down where objects overlap or move in curved paths. The task grows costly when many frames are needed between each pair, for example at 8x.

FLAVR skips the motion estimate. It looks at a short stack of neighbouring frames as one 3D block of video and passes it through a single convolutional network. The network learns the motion from the pixels themselves and writes out several in-between frames at once. It is trained on ordinary video: the training pairs are cut from the video itself, so no motion labels are needed.

The authors also show that a network trained this way learns features that help other video tasks, such as recognising actions or estimating flow, when it is fine-tuned on them. The claims are the authors' own.

TL;DR

FLAVR is a flow-free, end-to-end 3D U-Net for multi-frame video interpolation. It takes 2C neighbouring frames and outputs k-1 frames in one forward pass, trained with L1 loss on frames cut from raw video §3. Author-reported results beat the compared RGB-only methods on Vimeo-90K, UCF101, DAVIS, Adobe and GoPro, and are faster than QVI for multi-frame output §5. Use it if you need fast multi-frame interpolation without flow or CUDA warping kernels, and retrain per interpolation factor k §7.

Problem

Flow-based interpolation depends on the accuracy of a separate flow estimator. It handles occlusions poorly and needs a warping step that is slow at inference §1. Methods that model only linear or quadratic motion cannot capture complex trajectories without extra cost §1. Flow-based methods also tie their learned features to one task, so they transfer poorly to other tasks §1. The paper states that some recent methods take seconds for 8x interpolation and need custom CUDA kernels §1.

Method

Setting. The interpolation factor is k. A k-times model produces k-1 new frames between each pair of original frames §3. The input is 2C frames: C neighbours on each side of the pair being interpolated §3. The paper uses C=2 (4 inputs) for the main results §3 § Ablations.

1. Training pairs from raw video §3. Subsample the video by k to get a low-frame-rate sequence. For the pair A_i, A_(i+1), the input is a window of 2C frames of that sequence centred on the pair. The target is the k-1 original frames between A_i and A_(i+1) §3. The paper's 4x example uses frames {A1, A5, A9, A13} as input and {A6, A7, A8} as targets §3 fig. 2b.

# Build one training pair for k-times interpolation  [§3]
# Input:  video (list of frames), pair index i, factor k, context C
# Output: 2C input frames, k-1 target frames
def make_training_pair(video, i, k, C):
    # Subsample by k to get the low frame-rate video  [§3]
    low_rate = video[::k]
    # 2C low-rate frames around A_i and A_(i+1)  [§3]
    inputs = low_rate[i - C + 1 : i + C + 1]
    # Original frames strictly between A_i and A_(i+1)  [§3]
    targets = video[i * k + 1 : (i + 1) * k]
    return inputs, targets

2. Architecture §3 fig. 2a. The network is a 3D U-Net.

• Encoder: R3D-18 (ResNet-3D with 18 layers) with its classification layer removed, giving five stages conv1 to conv5. Each stage has two 3D convolutions and a skip connection §3. No temporal striding is used anywhere §3. Spatial stride 2 is used in conv1, conv3 and conv4 §3.
• Decoder: 3D convolutions and 3D transposed convolutions with stride 2 that upsample features §3. Skip connections join encoder and decoder features by channel concatenation, which the paper found best § Ablations tab. 3. A 3D convolution follows the transposed convolutions to reduce checkerboard artefacts §3.
• Temporal fusion: the time axis is moved into the channels and fused by a 2D convolution §3.
• Output: a 7x7 2D convolution produces 3(k-1) channels, which are split into k-1 RGB frames §3.
• Channel gating after every (de-)convolution layer §3 fig. 2a.

3. Spatio-temporal feature gating §3. For a feature map f of shape C x T x H x W, a gate is computed from the spatio-temporally pooled features, and each channel is scaled by its gate. The gate has a learned C x C weight and a bias §3.

# Spatio-temporal channel gating  [§3]
# features: (B, C, T, H, W)
def spatio_temporal_gate(features, weight, bias):
    # Global average over time, height and width  [§3]
    pooled = mean(features, over T, H, W)              # (B, C)
    # Sigmoid of a learned linear map of the pooled vector  [§3]
    gate = sigmoid(pooled @ weight.T + bias)           # (B, C)
    return features * gate[:, :, None, None, None]     # (B, C, T, H, W)

4. Forward pass §3 FLAVR_arch.py.

# One forward pass: 2C frames in, k-1 frames out  [§3]
# frames: list of 2C tensors, each (B, 3, H, W)
def flavr_forward(frames):
    # Stack the frames along a new time axis  # (B, 3, 2C, H, W)
    clip = stack(frames, time_axis)
    # Subtract one mean per sample and channel, added back at the end  [§3]
    clip_mean = mean(clip, over T, H, W)
    clip = clip - clip_mean
    # Encoder: five stages, no temporal stride  [§3]
    skip_0, skip_1, skip_2, skip_3, deepest = encoder(clip)
    # Decoder: each stage is joined with the matching encoder skip  [§3]
    x = relu_leaky(decoder_conv(deepest))
    x = join(x, skip_3)
    x = relu_leaky(upconv_1(x))
    x = join(x, skip_2)
    x = relu_leaky(upconv_2(x))
    x = join(x, skip_1)
    x = relu_leaky(decoder_conv(x))
    x = join(x, skip_0)
    x = relu_leaky(upconv_4(x))
    # Temporal fusion: move time into channels, then 2D convolution  [§3]
    x = relu_leaky(conv1x1(fuse_time_into_channels(x)))
    # 7x7 convolution to 3(k-1) channels, split into k-1 RGB frames  [§3]
    frames_out = split(conv7x7(x), into k-1 groups of 3 channels)
    # Add the clip mean back to every output frame  [§3]
    return [frame + clip_mean for frame in frames_out]

5. Loss §3. One L1 loss between the k-1 predicted frames and the k-1 ground-truth frames, averaged over the mini-batch §3. The paper states no other loss in the main model.

# Pixel loss over all predicted frames  [§3]
# predicted, targets: (B * (k-1), 3, H, W)
def training_loss(predicted, targets):
    # L1 distance between predicted and ground-truth frames  [§3]
    return mean(abs(predicted - targets))

6. Training and inference. Training uses Adam with learning rate 2e-4 and halves the rate when training plateaus app. Training details. The paper also gives batch size, epochs, crop size and augmentation app. Training details §4; these are in the Hyperparameters table. At test time, the 2C input frames give k-1 output frames in one forward pass; the paper reports timing for the whole set of frames app. Benchmarking.

Hyperparameters

Rows marked "app." come from the later, appendix-style sections of the arXiv source, which the paper labels by their titles. "Code" rows come from the official repository at code_commit. Values stated only in code are labelled.

Results

Author-reported. Setup: RGB-only comparison on the public benchmarks in §4, with baselines retrained by the authors on the same split (septuplets of Vimeo-90K for 2x; GoPro for 8x) §4. Numbers are PSNR (dB) and SSIM.

2x interpolation tab. 1 (± values as printed in the table):

Methods listed with RGB+flow or RGB+depth in the table are DAIN (RGB+depth+flow: 33.35 / 0.945 on Vimeo-90K) and QVI. The paper says FLAVR beats DAIN and QVI without flow or depth input §5.

8x interpolation tab. 2:

Speed §5 app. Benchmarking. Versus QVI, the paper reports runtime gains of 2.7x (8x), 6.2x (16x) and 12.7x (32x) interpolation §5. Timing used a 2080 Ti with 512 x 512 crops, averaged over 100 Adobe-240FPS samples, forward pass only app. Benchmarking.

Ablations tab. 3 (Vimeo-90K, 2x, PSNR / SSIM):

The text reports that removing spatio-temporal gating lowers PSNR from 36.3 to 36.1 § Ablations. No table row shows this. The text also reports that C=2 is best for 2x and C=3 for 8x § Ablations.

Downstream §6:

• Action recognition, pretrained encoder vs random initialisation: +13.08% on UCF101 and +4.48% on HMDB51 (text values) §6.
• Optical flow fine-tuning: lower EPE than random initialisation; the text gives no single number §6.
• Low-fps video object segmentation (label propagation with CRW on interpolated DAVIS): the paper reports gains; see the downstream tables §6.
• Motion magnification: SSIM rises from 0.732 to 0.801 on a held-out CoCo-Synth validation set when pretrained on FLAVR §6 § Motion magnification.

Robustness and user study. On SNU-Film, FLAVR is 1.28 dB ahead of CAIN in the hard setting and 1.62 dB ahead in the medium setting §5. In an AMT study on DAVIS videos at 8x, users preferred FLAVR over SuperSloMo by a wide margin, and chose FLAVR 35% of the time against QVI's 40%, with 20% no preference §5.

Paper vs code

Official code: https://github.com/tarun005/FLAVR (Apache-2.0), at commit ee17951c69ec27ce8dfe38254f4b7fdbe08d4fba. The repository owner is the first author, whose name matches the author list. The paper's author block links the project page tarun005.github.io/FLAVR, and the README links the same page README.md. The repository carries the interpolation model, training and test code only. The downstream experiments, the motion-magnification model and the supplementary baselines are not in it README.md.

• Data augmentation on GoPro. Paper: temporal reversal and horizontal flip on both datasets §4. Code: GoPro loader has no flip or reversal [code: dataset/GoPro.py#L26-L36]; Vimeo-90K loader has horizontal and vertical flip, colour jitter vimeo90k_septuplet.py and temporal reversal vimeo90k_septuplet.py.
• 8x input configuration. Paper: C=3 best for 8x § Ablations; the main model's C is not stated. Code: nbr_frame defaults to 4 for all factors config.py, so 8x GoPro training uses 4 inputs (C=2) spaced 8 frames apart GoPro.py.
• Batch size. Paper: 64 (Vimeo-90K) and 32 (GoPro) app. Training details. README command: 32 for Vimeo-90K README.md. Code default: 16 config.py.
• Epochs. Paper: 200 (2x) and 120 (4x, 8x) app. Training details. Code default: 200 config.py. The 4x and 8x value of 120 is not a code default.
• Loss scale. Paper equation: sum over pixels and frames, averaged over the batch §3. Code: nn.L1Loss averages over all elements, including channels, spatial positions and frames loss.py main.py. This is a constant scale difference (about 3·H·W·(k-1)); with Adam it matters little, but the loss values are not comparable.
• Decoder conv placement. Paper: a 3D convolution after the last transposed convolution §3. Code: the 3D convolution sits before the last transposed convolution; the last decoder stage is a transposed convolution with no convolution after it FLAVR_arch.py.
• Mean normalisation. Paper: one mean per mini-batch of inputs, chosen over a global mean or batch norm app. Training details. Code: one mean per sample and channel, over time and space FLAVR_arch.py FLAVR_arch.py.
• Learning-rate scheduler. Paper: halve on plateau, "cross-validated by the validation set" app. Training details. Code: ReduceLROnPlateau with patience 5, stepped on the loss of the loader named test main.py main.py. Check which split you use.
• Gating module. Paper: a learned C x C weight and bias with spatio-temporal pooling §3. Code matches this as a 1x1x1 convolution on pooled features, but the reduction argument (default 16) is never used [code: model/resnet_3D.py#L100-L116].
• Batch normalisation. Not mentioned in the paper's text as used. Code default: none in the interpolation model FLAVR_arch.py [code: main.py#L65].
• Middlebury and downstream code. The repository includes a Middlebury test script Middleburry_Test.py, but no action-recognition, flow or label-propagation training code.

No further discrepancies found in the frame-sampling rule, the encoder strides, the skip fusion, the 7x7 output layer or the L1 loss choice.

Pitfalls

• Table numbers are inferred. The LaTeX source does not show the compiled numbering. Table numbers here follow source order; check against the PDF.
• Abstract speed claim. The source abstract says "up to 6x" speed-up; the arXiv metadata abstract for version 3 says "3x". The main text gives 2.7x, 6.2x and 12.7x against QVI at 8x, 16x and 32x §5.
• Text and table disagree. The text gives UCF101 33.33 and DAVIS 27.44 §5; the table gives 33.31 and 27.43 tab. 1. The text gives Middlebury ranks of 2nd, 5th and 8th; a commented-out line in the source gives 1st, 4th and 7th.
• Printed SSIM. The SuperSloMo Adobe SSIM is printed as 0.391 tab. 2, which is probably a typo. Do not use it as a baseline value.
• Retrain per factor. A separate model is trained for each k. The paper states this limitation §7. Nothing in the paper shows one model for mixed k.
• Edge handling is not stated. For the first and last frames of a clip the paper does not say how the 2C window is padded. The code skips incomplete windows GoPro.py.
• Activations and kernels not stated. The paper does not give the decoder activation, the transposed-convolution kernel, or the temporal fusion kernel size. The code's values are listed in Hyperparameters.
• Baseline comparison is retrained. The authors retrained baselines on their own splits §4. Numbers from other papers for these baselines will differ.
• Released weights. The README links pretrained 2x, 4x and 8x models [code: README.md]. It says pretrained baseline models are no longer available, and the motion-magnification model is not released README.md.
• Cost of 3D convolutions. The paper argues that flow removal makes the method faster, but its runtime comparison uses a 2080 Ti at 512 x 512 [app. Benchmarking]. The text reports no memory figures.
• Generalisation. The authors state that learned models generalise poorly outside the training distribution §7.

Lineage

• Builds on: 3D ResNet-18 as encoder §3; U-Net §3; channel gating from squeeze-and-excitation style attention, following CAIN §2 §3.
• Compared against: DAIN, QVI, DVF, SepConv, SuperSloMo, CAIN, AdaCoF §4. Not compared, because training code was unavailable: SoftSplat, AAO and RRPN §4.
• Superseded by: not stated in this wiki.