---
kind: concept
slug: video-frame-interpolation
title: Video frame interpolation
summary: >-
  Making new in-between frames from neighbouring video frames, for example 30 fps to 240 fps. The one card so far, FLAVR, skips optical flow: one 3D U-Net predicts all k-1 frames in a single forward pass and trains on frames cut from ordinary video. Its gains and speed-ups are author-reported; a separate model is needed for each factor k.
tags: [video, frame-interpolation, motion, 3d-cnn, self-supervised-learning]
papers: ["2012.08512"]
related: [optical-flow, channel-attention, self-supervised-learning, video-generation, video-inpainting]
status: draft
updated_at: 2026-11-09
---

## Overview

The task is to produce new frames between existing ones, so that a clip gains
a factor k in frame rate. Between each pair of original frames the method must
output k-1 frames [arxiv:2012.08512 §3]. The classic route estimates optical
flow between neighbouring frames and warps pixels along it. The FLAVR paper
names three costs of that route: errors in the flow estimate, weak handling of
occlusions, and a warping step that is slow at inference [arxiv:2012.08512 §1].
Every variant has to learn the motion somehow, and every variant can build its
training pairs from ordinary video by subsampling it, so no motion labels are
needed [arxiv:2012.08512 §3].

## Variants

### Flow-free 3D U-Net (FLAVR)

The network reads a clip of 2C neighbouring frames, stacked along time. A
3D residual encoder (R3D-18 without its classifier) with no temporal stride
feeds a 3D U-Net decoder. Skip connections join encoder and decoder by channel
concatenation. Time is folded into channels and fused by a 2D convolution, and
a 7x7 convolution emits 3(k-1) channels, which become k-1 RGB frames in one
pass. Spatio-temporal channel gating follows the (de)convolutions. The only
loss in the main model is L1 [arxiv:2012.08512 §3] [arxiv:2012.08512 tab. 3].
Card: [FLAVR](/papers/2012.08512).

Author-reported results on the RGB-only comparison setting: 36.25 dB PSNR on
Vimeo-90K at 2x, against 35.15 dB for QVI (RGB+flow) and 35.40 dB for AdaCoF
(RGB) [arxiv:2012.08512 tab. 1]. At 8x on GoPro, 31.31 dB against 30.55 dB for
QVI [arxiv:2012.08512 tab. 2]. Against QVI the paper reports runtime gains of
2.7x at 8x, 6.2x at 16x and 12.7x at 32x [arxiv:2012.08512 §5]. Timing used a
2080 Ti with 512 x 512 crops and forward passes only [arxiv:2012.08512 § Benchmarking].

### Flow-based methods

The paper compares against flow-based and flow-plus-depth methods such as QVI
and DAIN [arxiv:2012.08512 §4]. No card in this wiki covers them, so their
designs are not described here. See [optical flow](/concepts/optical-flow) for
the flow estimation side.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Many in-between frames per pair, such as 8x, and speed matters | FLAVR | One forward pass gives all k-1 frames; the paper reports faster multi-frame output than QVI [arxiv:2012.08512 §5] |
| You can train one model per factor k | FLAVR | The paper trains a separate model per k and states this as a limitation [arxiv:2012.08512 §7] |
| One model must serve several factors | Not covered by a card | The paper shows no mixed-k model [arxiv:2012.08512 §7] |
| No labelled motion data, only raw video | FLAVR | Training pairs are cut from the video itself [arxiv:2012.08512 §3] |
| Content far from the training data | Be careful with FLAVR | The authors state that learned models generalise poorly outside their training distribution [arxiv:2012.08512 §7] |
| You want an encoder that later serves other video tasks | FLAVR | Author-reported gains on action recognition and a faster start for flow fine-tuning [arxiv:2012.08512 §6] |
| You need an explicit motion field from the interpolator | Not covered by a card | FLAVR has no flow output [arxiv:2012.08512 §1] |

## Implementation sketch

The sketch follows the FLAVR design. Other variants may differ at every step.

Training sample for a k-times model. Indices are 0-based. Frame `pair` and
frame `pair + 1` of the subsampled video are the two anchors.

```pseudocode
# Build one training sample for a k-times model  [arxiv:2012.08512 §3]
# video: list of frames, k: interpolation factor, C: neighbours per side
def make_sample(video, pair, k, C):
    # Keep every k-th frame to simulate the low frame rate  [arxiv:2012.08512 §3]
    low_rate = video[::k]
    # 2C low-rate frames centred on the two anchors  [arxiv:2012.08512 §3]
    context = low_rate[pair - C + 1 : pair + C + 1]
    # The k-1 original frames strictly between the anchors  [arxiv:2012.08512 §3]
    target = video[pair * k + 1 : (pair + 1) * k]
    return context, target
```

Inference for one set of k-1 in-between frames:

```pseudocode
# One pass: 2C frames in, k-1 frames out  [arxiv:2012.08512 §3]
def interpolate(context, k):
    # Stack the frames along time: (3, 2C, H, W)  [arxiv:2012.08512 §3]
    clip = stack(context, axis=time)
    # Subtract a per-clip mean and add it back at the end  [arxiv:2012.08512 § Training details]
    mean = clip.mean(over time, height, width)
    feats = encoder(clip - mean)  # no temporal stride  [arxiv:2012.08512 §3]
    out = decoder(feats)  # skips joined by concatenation  [arxiv:2012.08512 tab. 3]
    out = conv2d(fold_time_into_channels(out))  # [arxiv:2012.08512 §3]
    # 7x7 conv to 3(k-1) channels, split into k-1 RGB frames  [arxiv:2012.08512 §3]
    frames = split(conv7x7(out), groups=k - 1)
    return [f + mean for f in frames]
```

Implementation notes. The official code differs from the paper in several
settings, including batch size, the number of input frames used at 8x and the
loss scale. The [FLAVR card](/papers/2012.08512) lists each difference with
its location. Copy the paper's values only after checking them against that
list.

## Contradictions

- **Text and table values for 2x.** The text gives 33.33 dB on UCF101 and
  27.44 dB on DAVIS. The table gives 33.31 and 27.43 [arxiv:2012.08512 §5]
  [arxiv:2012.08512 tab. 1]. Both are reported here as printed; the page does
  not choose between them.
- **Gating ablation.** The text says removing spatio-temporal gating lowers
  PSNR from 36.3 to 36.1 dB. No row of the ablation table shows this
  [arxiv:2012.08512 § Ablations] [arxiv:2012.08512 tab. 3].
- **Input count at 8x.** The text reports C=3 as best for 8x, but the main
  model's C is not stated [arxiv:2012.08512 § Ablations]. The paper does not
  say which C the main 8x model uses.

## Papers

- [FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation](/papers/2012.08512): a flow-free 3D U-Net that predicts all k-1 frames in one pass, with author-reported speed and quality gains.