Towards Accurate Generative Models of Video: A New Metric & Challenges

In plain words

Video generators are hard to judge. The usual scores, PSNR and SSIM, compare one frame at a time against a known true video, so they cannot tell whether a generated video moves in a believable way, and they cannot be used when there is no true video to compare against. The paper proposes a score for a whole set of videos instead of single frames.

The score works like an image-quality score that is known as FID, but it looks at videos. A network that was trained to recognise human actions turns each clip into a list of numbers. Those numbers capture both how a clip looks and how it moves. The score then compares the real clips with the generated clips by their overall statistics (average and spread). A lower score means the generated set is closer to the real one.

The authors also built four benchmark scenarios from StarCraft 2 game videos, where current models struggle. In a human study, the new score agreed with people more often than PSNR and SSIM did.

TL;DR

• Proposes Fréchet Video Distance (FVD), a set-level metric for generated video, computed from I3D features §1 §2.
• Feature choice: Kinetics-400 I3D logits, chosen over the other embeddings tested §4.1.
• In a human study on BAIR, FVD agrees with raters more often than SSIM and PSNR in the paper's comparisons §4.3 tab. 1.
• Introduces SCV, four StarCraft 2 video benchmarks; the best of four baseline models varies by scenario §3 tab. 2.
• Author-reported. Official code computes FVD only, not the KVD variant, and only with Kinetics-400 weights.

Problem

Frame-level metrics (PSNR, SSIM) need a ground-truth sequence and score each frame on its own, so they miss temporal coherence and sample diversity §1 §2. Unconditional generators (for example GANs) have no ground truth, so these metrics cannot be applied to them at all §2. Pixel-level FID-style scores for images do not account for time, and the paper argues that FID applied frame by frame is a poor fit for video §2 §4.3. The paper also reports a gap between synthetic video benchmarks and real-world data, and a lack of benchmarks that isolate specific capabilities such as long-term memory §1 §3.

Method

Overview. FVD is FID applied to videos §2. A pretrained video network maps each clip to a feature vector. Real and generated clips are each summarised by a mean and a covariance matrix. Both are treated as multivariate Gaussians, and the squared Fréchet distance between the two Gaussians is the score §2 eq. 1 eq. 2. The paper states that a Gaussian is seldom an exact fit, but is a reasonable approximation in a good feature space §2.

Steps.

1. Take a set of real clips and a set of generated clips of the same size N §4.2.
2. Embed every clip with a pretrained I3D network. The reported choice is the Kinetics-400 logits layer §2 §4.1. The paper also evaluates the last pooling layer and a Kinetics-600 variant §4.1.
3. Compute the mean and covariance of each embedding set §2.
4. Evaluate the Fréchet distance in eq. 2.

Related metric. The paper also evaluates KVD: a kernel distance (squared maximum mean discrepancy) on the same I3D features, with the cubic polynomial kernel k(a, b) = (aᵀb + 1)³. KVD avoids the Gaussian assumption §2. It is unbiased and needs no matrix square root.

Pseudocode. The mapping from paper to code below is ours. Preprocessing is not stated in the paper; the official code does it, so it is included.

# Frechet Video Distance (FVD)  [§2] [eq. 2]
# Input:  real, fake: lists of clips, each (T, H, W, 3), pixels 0-255
# Output: scalar distance, lower is better

def fvd(real, fake, i3d):
    # Resize every frame to 224 x 224 and map pixels to [-1, 1]. Not
    # stated in the paper; the official code does this
    # [code: frechet_video_distance/frechet_video_distance.py#L49-L52]
    real_clips = preprocess(real, (224, 224))
    fake_clips = preprocess(fake, (224, 224))

    # Embed each clip with I3D, Kinetics-400 logits  [§2] [§4.1]
    # The code takes batches of 16 clips
    # [code: frechet_video_distance/frechet_video_distance.py#L83]
    real_feats = i3d(real_clips)  # (N, D)
    fake_feats = i3d(fake_clips)  # (M, D)

    # One Gaussian per set: mean and covariance  [§2]
    mu_real = mean(real_feats)
    mu_fake = mean(fake_feats)
    cov_real = covariance(real_feats)
    cov_fake = covariance(fake_feats)

    # Frechet distance between the two Gaussians  [eq. 2]
    # The square root is a matrix square root
    cov_cross = sqrt_matrix(cov_real @ cov_fake)
    mean_term = squared_norm(mu_real - mu_fake)
    trace_term = trace(cov_real + cov_fake - 2 * cov_cross)
    return mean_term + trace_term

Key equations.

Wasserstein-2 distance between the real and generated distributions, which FVD approximates eq. 1:

d(P_R, P_G) = \min_{X, Y} \mathrm{E}\,|X - Y|^2 eq. 1

Closed form for two Gaussians, the FVD formula eq. 2:

|\mu_R - \mu_G|^2 + \mathrm{Tr}\left(\Sigma_R + \Sigma_G - 2 (\Sigma_R \Sigma_G)^{\frac{1}{2}} \right) eq. 2

Unbiased squared MMD estimator used for KVD, with kernel k(a, b) = (aᵀb + 1)³ §2. The paper's third sum has an index typo; the standard form is shown:

\frac{1}{m(m-1)} \sum_{i \neq j} k(x_i, x_j) + \frac{1}{n(n-1)} \sum_{i \neq j} k(y_i, y_j) - \frac{2}{mn} \sum_{i} \sum_{j} k(x_i, y_j) §2

Noise study (metric validation). Four static distortions (black rectangle, Gaussian blur, Gaussian noise, salt-and-pepper) and four temporal distortions (local swap, global swap, interleaving, switching) are applied to real clips. Each distortion has its own intensity scale, with five or six levels §4.1 app. A tab. 3. The paper finds that every metric variant detects the distortions to some degree. The per-frame Inception baseline is weaker on temporal distortions. The Kinetics-400 logits variant is among the best in rank correlation with intensity §4.1 app. A fig. 3.

SCV benchmark. Four StarCraft 2 scenarios, rendered at 64 x 64 and 128 x 128 §3 app. B:

• Move Unit to Border (MUtB): one unit moves to the map edge, with random type, colour and direction §3 app. B.1.
• Collect Mineral Shards (CMS): two units collect randomly placed shards greedily §3 app. B.1.
• Brawl: two armies fight; the Terran side attacks the nearest enemy §3 app. B.1.
• Road Trip with Medivac (RTwM): a transport unit picks up units, visits beacons and unloads the same units at the last beacon §3 app. B.1.

Human study. Models are selected in two ways: "one metric equal" (models that tie on one metric) and "one metric spread" (models spread over that metric's range). Raters compare pairs of videos and choose the better one or report similar quality §4.3. Human agreement with each metric is reported in tab. 1.

Hyperparameters

Results

Author-reported. Setup: FVD on BAIR, KTH and the SCV scenarios for four video models trained with Tensor2Tensor §4.4 app. C. Lower FVD is better tab. 2. Hardware not stated; total compute "over 100 GPU-years" §1.

Human agreement tab. 1. Each cell is the percentage of human comparisons where the metric in that row agreed with the raters. The "eq." columns hold one metric fixed; "spr." columns use a wide range of values §4.3:

Baseline FVD on SCV and BAIR/KTH tab. 2. Left column of each SCV pair is 64 x 64; right column is 128 x 128.

Bold marks the best model in each column, as the paper does tab. 2.

Metric correlations §4.5. Over more than 20,000 models: SSIM and PSNR Pearson r = 0.730 (Kendall tau 0.648); SSIM and FVD r = -0.640 (tau -0.189); PSNR and FVD r = -0.278 (tau -0.007).

Sample-size and resolution findings. With identical distributions FVD is still above zero at small sample sizes, so FVD values are only comparable at the same sample size §4.2 fig. 4. In the resolution study, a difference of under 50 FVD points gave agreement close to chance; agreement rose sharply above 50 points §4.3.1 fig. 5.

Qualitative findings on SCV §4.4 app. D. MUtB is handled best at low resolution. CMS defeats most models, which learn that shards vanish but not the unit movement or order. Brawl produces blurry blobs. No model handles RTwM §4.4 app. D.

Paper vs code

The official code is the frechet_video_distance folder of google-research/google-research §2 footnote. It is a TensorFlow 1 module that computes FVD and nothing else. Repository checked at commit 8bedbbd982e7dcf5c5280e6ae2f0769295c3369e, Apache-2.0.

• Kinetics-600 is evaluated in the paper §4.1. The code loads only i3d-kinetics-400 frechet_video_distance.py.
• KVD (the kernel MMD variant) is defined and evaluated in the paper §2 §4.1. The code has no MMD function; its only metric is frechet_classifier_distance_from_activations frechet_video_distance.py.
• The paper compares the logits layer and the pooling layer §4.1. The code takes one tensor, under the name RGB/inception_i3d/Mean:0, which its comment calls the Kinetics-400 logits layer frechet_video_distance.py.
• Resizing to 224 x 224 with bilinear interpolation and scaling to [-1, 1] is done in code and not described in the paper frechet_video_distance.py.
• The code requires exactly 16 clips per call and asserts the pixel range frechet_video_distance.py frechet_video_distance.py frechet_video_distance.py. The example says set sizes must be divisible by 16 example.py.
• The Fréchet distance itself is delegated to the TF-GAN library, which is not in this repository, so the matrix square root method cannot be checked here frechet_video_distance.py.
• The code's example feeds constant zero and constant 255 clips of 15 frames at 64 x 64 and says FVD "should be around 131" example.py example.py. This is a sanity value from the code, not from the paper.
• The baseline models (CDNA, SV2P, SVP-FP, SAVP) are trained with Tensor2Tensor app. C. Their code is not in this repository, so the benchmark training recipe is unverified.
• The paper says the SCV generation code is released §3. No StarCraft or SCV code was found in the cloned repository. Unverified.

Pitfalls

• Sample size is part of the metric. FVD on the same models changes with N, and a nonzero floor appears even for identical distributions §4.2. Compare FVD values only at the same N. The paper uses 256 for BAIR and 1024 for KTH and SCV §4.4. Batch-size and sample-count choices are not stated for other datasets.
• Gaussian assumption. The paper calls the Gaussian a reasonable approximation, not an exact fit §2. Covariance estimates from the sample sizes used (256 to 1024 clips) are noisy §4.2 §4.4. KVD is the paper's alternative §2.
• Preprocessing is not in the paper. Resize and value range are only in the code frechet_video_distance.py. An implementation that does not match them will not reproduce published FVD values.
• Clip length. The paper uses different lengths per dataset: 16 or 20 frames for KTH and BAIR, full sequences from frame 0 for SCV §4.4. The code does not enforce length; the example uses 15 frames example.py.
• Matrix square root. Eq. 2 needs (Σ_R Σ_G)^(1/2). The paper does not say how it is computed, and the code delegates to TF-GAN. Small negative or complex parts from numerical error are not discussed in either source eq. 2 frechet_video_distance.py.
• Resolution threshold. The 50-point figure comes from one human study on one set of models §4.3.1. Treat it as a rough guide, not a standard.
• Human-study scope. The agreement numbers are from BAIR models only, with four model families and 10 models per condition §4.3. Ties in the resolution study count as disagreement with FVD §4.3.1 fig. 5.
• Unclear figure reference. The noise section says "Figure 1 of the Appendix", but the figure is in Appendix A §4.1 app. A. Cite it by section.
• Typo in the benchmark appendix. The VAE beta grid is written as "10^-6, 10^-5, 10^-6, 10^-3" app. C. The first and third values repeat, so the intended grid is not certain.
• Typo in the KVD equation. The sum over y in the MMD estimator reads j ≠ j §2. The standard unbiased form is in Key equations.
• FVD is a distribution metric. It scores sets, not a single sample against its ground truth, so it cannot replace per-sample error §2.
• Costs. Each video needs an I3D pass; the study used over 3,000 models for the human study and over 20,000 for the correlation study §1 §4.3 §4.5. Full reproduction is expensive.
• Data access. The paper gives a download link and a TensorFlow Datasets route for SCV §1. We did not check either. The SCV data licence is not stated in the paper; check before any reuse.

Lineage

• Builds on: Fréchet Inception Distance (FID; Heusel et al., 2017) for image generation §2, the I3D action-recognition network (Carreira et al., 2017) §2, and the kernel MMD estimator of Binkowski et al. for KVD §2. None are linked to an arXiv id on this card.
• Compared against: SSIM and PSNR §2 tab. 1; the video models CDNA, SV2P, SVP-FP and SAVP, which are baselines in the SCV benchmark §4.3 §4.4. Not linked to arXiv ids on this card.
• Built upon by: not stated in this card.