Generative model evaluation

Overview

A generative model produces samples that have no single correct answer, so a score has to say something about the whole output distribution, the individual sample, or both. The eight cards on this page cover several settings, and they lead to different families.

• Per-sample, reference-based scores. PSNR and SSIM compare one generated frame with one known true frame. They need a ground-truth sequence for every output and score each frame on its own, so they miss whether a sequence moves coherently or whether the set of outputs is diverse 1812.01717 §1 §2.
• Set-level, distribution-based scores. FVD compares the statistics of a set of generated clips with a set of real clips. It needs no paired ground truth, so it applies to unconditional generators as well 1812.01717 §2. A related kernel score, KVD, uses the same features but avoids the Gaussian assumption 1812.01717 §2. The image version of the same idea is FID, which a GAN survey reviews together with its variants 2103.09396 §2.1. Joint audio-video generators add a set-level audio distance, FAD 2212.09478 §4.3.
• Multi-criteria design scores. A design generator is judged on more than resemblance to its training data. The metric catalogue sorts scores into similarity, novelty and diversity, constraint satisfaction, performance and target achievement, and conditioning 2302.02913 §1 tab. 1. Many of these need a distance on the design space or on the performance space, and the paper leaves that choice to the practitioner 2302.02913 §3.2.
• Learned preference scores. A network is trained on human pairwise choices and then ranks new outputs. HPS does this for text-to-image images, and the same score also selects training data 2303.14420 §5 §6.
• Per-task checks and multi-metric benchmarks. A benchmark attaches a check to each task: a program, a vision or audio model answering a narrow question, or a reference comparison. MMMG does this across image, audio and interleaved generation 2505.17613 §3.2. EvalCrafter runs many automatic metrics and fits their weights to human ratings for text-to-video 2310.11440 §4.5.

Set-level scores depend on the sample count. FID is usually computed with more than 50K samples 2103.09396 §1.1, and FVD depends on N for identical distributions 1812.01717 §4.2. Several set-level and design scores also depend on a feature space or a distance choice. Scores are only comparable when those choices are fixed 1812.01717 §4.2 2302.02913 §3.2.

Variants

Frame-level reference metrics (PSNR, SSIM)

These are the baseline the video paper argues against for video. They need one ground-truth clip per output. Across more than 20,000 models, the paper reports a Pearson correlation of 0.730 between SSIM and PSNR, but of -0.640 between SSIM and FVD and -0.278 between PSNR and FVD 1812.01717 §4.5. The paper gives no separate definition for them that this page relies on.

Fréchet Video Distance (FVD)

The set-level variant. Real and generated clips are each embedded with a pretrained I3D action-recognition network. The paper uses the Kinetics-400 logits layer. Each embedding set is summarised by a mean and covariance, and the score is the Fréchet distance between the two Gaussians [arxiv:1812.01717 §2] §4.1. The GAN survey lists the same I3D network, pretrained on Kinetics-400 and Kinetics-600 2103.09396 §2.1.7. Card: Towards Accurate Generative Models of Video: A New Metric & Challenges.

In the paper's human study on BAIR, the FVD row agreed with raters in 74.9 % of comparisons where SSIM was held equal, 81.0 % where PSNR was held equal, and 63.0 % where KVD was held equal 1812.01717 tab. 1. These figures come from one human study on BAIR, with four model families 1812.01717 §4.3.

Generator papers report FVD as well, and their setups differ. MM-Diffusion reports FVD and KVD with the same I3D network, computed at 64×64 with 2,048 generated samples per model 2212.09478 §4.1 §4.3. Its ground-truth row is not zero: 17.83 on Landscape and 8.73 on AIST++ 2212.09478 tab. 1 tab. 2. PYoCo reports zero-shot UCF-101 FVD 355.19 2305.10474 tab. 2, and unconditional UCF-101 FVD 310 ± 13 from 2,048 videos [arxiv:2305.10474 §5.2] tab. 4. These values come from different datasets and protocols, so they are not directly comparable.

Kernel Video Distance (KVD)

The same I3D features scored with an unbiased squared maximum mean discrepancy under a cubic polynomial kernel, k(a, b) = (aᵀb + 1)³ 1812.01717 §2. It does not assume Gaussians and needs no matrix square root. Its human agreement is in the same table as FVD 1812.01717 tab. 1. MM-Diffusion also reports KVD beside FVD on both of its datasets 2212.09478 tab. 1 tab. 2. The paper does not specify the kernel parameters; its code uses an unparameterised polynomial kernel compute_fvd_kvd.py.

Image FID and its variants

FID scores a set of generated images against a set of real images, using Gaussians fitted to Inception features. The GAN survey warns that FID and IS are biased, need many samples, and can be raised by memorising the training data 2103.09396 §1.1 §4.1. The variants it reviews aim at these problems:

• Class-aware FID averages a Fréchet distance over K classes [arxiv:2103.09396 §2.1.2]. The survey prints its mean term as a norm, not a squared norm, and says to check the original paper before implementing it.
• FastFID is an optimisation of the FID computation. The survey reports it 25 to 500 times faster than the standard computation, depending on the number of generated samples, which makes FID usable as a training loss [arxiv:2103.09396 §2.1.3].
• Memorization-informed FID multiplies FID by a penalty when generated samples sit close to training samples. The survey does not state the threshold or the epsilon 2103.09396 §2.1.4.
• Bias-corrected FID and IS remove finite-sample bias by extrapolation 2103.09396 §2.1.5. Clean FID fixes the preprocessing, since the choice of resize library, interpolation and encoding can change FID substantially 2103.09396 §2.1.6.
• Precision and recall extensions include density and coverage, and α-precision, β-recall and authenticity 2103.09396 §2.7.

The survey runs no experiments of its own. Its figures are quoted from the works it cites, so check each one against its original source before use.

Text-to-video papers sometimes use a CLIP-based FID instead of the Inception one. PYoCo reports both on MSR-VTT, with CLIP-FID computed by a ViT-B/32 model and FID by Inception-V3 2305.10474 §5.2 tab. 3. Its final configuration reaches CLIP-FID 9.73 and FID 22.14 2305.10474 tab. 3. The two columns are not interchangeable.

Audio distance (FAD) and human studies for joint audio-video output

MM-Diffusion scores its audio with FAD, computed on AudioCLIP features and printed multiplied by 10⁴ 2212.09478 §4.3. On Landscape, the ground truth scores 7.51 and the complete-sampling model 10.72, while the DPM-Solver model scores 9.39 2212.09478 tab. 1. On AIST++ the ground truth scores 8.46 and the complete-sampling model 10.69 [arxiv:2212.09478 tab. 2]. Divide the printed value by 10⁴ before comparing it with unscaled FAD.

The same paper adds two human tests. Its Turing test asks viewers whether a clip is real. Ours is judged real for 84.9 % of Landscape clips and 49.6 % of AIST++ clips, against 92.5 % and 84.7 % for real clips 2212.09478 tab. 6. It used 10,000 votes, with 500 generated and 500 real pairs per dataset 2212.09478 §4.6 tab. 6. Its mean opinion scores rate video quality, audio quality and alignment on a 1 to 5 scale, with five users per task 2212.09478 tab. 5 §4.3. Both tests are reported by the authors only. For how to design such studies, see human evaluation.

Verifiable per-task checks and VLM judges (MMMG)

MMMG is a benchmark of 55 tasks and 1288 instructions in four modality combinations 2505.17613 §3.1 tab. 7. Each task has the check its authors found best matched human judgement 2505.17613 tab. 1. Checks include a vision-language model answering a rubric question, a reference similarity such as DreamSim or CLAP, and a program such as a colour-fill test or a pitch measurement 2505.17613 §3.2 app. C.3.

Human annotators judged the checks on 978 instructions and 2,596 questions. The authors report 94.4 % overall agreement, with 93.5 % for images, 92.3 % for audio, 96.6 % for interleaved image-text and 91.0 % for interleaved audio-text 2505.17613 §1 §4.1. Human validation covers 41 tasks only, so the overall figure should not be applied to the remaining tasks [arxiv:2505.17613 §1]. The judges are GPT-4o, Gemini 2.5 Pro and Qwen2.5-VL-72B-Instruct 2505.17613 app. C.1. The best image model, GPT Image, reaches 0.707 on this benchmark, against 0.654 for Gemini 2.5 Image 2505.17613 §4.2. Card: MMMG.

Each instruction is sampled four times, and temperature is 0 for the multimodal language models 2505.17613 app. C.2. The reported agreement and accuracies depend on the judge thresholds, which the paper does not give as numbers (see Paper vs code).

Latent-space and data-copying checks

These look inside the generator rather than only at its outputs.

• Perceptual path length (PPL) is the expected perceptual change, measured with a learned perceptual distance, along a short spherical step between two latent points. The step size is 1e-4. The survey does not state the number of pairs 2103.09396 §2.3.3.
• Linear separability trains linear SVMs from latent codes to attribute labels, using 200K generated images and 40 CelebA attributes, and keeps the 50 % most confident labels 2103.09396 §2.3.4. The survey does not say whether a higher or lower final score is better; it says only that low conditional entropy for an attribute suggests consistent latent directions.

Distribution matching for design generators

Used when the question is whether generated designs look like the training designs. The paper's set of scores is nearest datapoint (realism), nearest generated sample (coverage), precision-recall curve summaries (F-beta and area under the curve, after Sajjadi et al.), and maximum mean discrepancy 2302.02913 §4.1 §4.2 §4.3. In its 2-D experiment, the VAE and the GAN split the wins across these scores. The GAN wins nearest datapoint and MMD, while the VAE wins nearest generated sample and the precision-recall AUC 2302.02913 tab. 2. Card: Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design.

Novelty and diversity scores

These check whether a generated set differs from the data and from itself. The paper lists inter-sample distance, distance to the centroid, a DPP-based diversity score, and the convex hull of the set, with nearest datapoint used in the other direction for novelty 2302.02913 §5.1 §5.2 tab. 1. In the same 2-D experiment, the GAN has the larger convex hull and distance to centroid, while the VAE has the larger inter-sample distance and DPP diversity score 2302.02913 tab. 3.

Constraint satisfaction scores

For designs with hard rules, the paper's catalogue has constraint satisfaction and a constraint satisfaction rate, a signed distance to the constraint boundary, a predicted constraint satisfaction from a classifier trained on valid and invalid designs, and the distance to the nearest invalid design 2302.02913 tab. 1 §6.3. The classifier variant uses k-nearest neighbours with k = 5 in the authors' notebook 2302.02913 app. 14.3. In the 2-D experiment the GAN scores higher than the VAE on the constraint satisfaction, satisfaction rate and predicted satisfaction measures 2302.02913 tab. 4.

Performance and target achievement

For designs with functional goals. The paper uses hypervolume, a target achievement rate, a signed distance to target, and the design target achievement index (DTAI), a score for a vector of objectives that is normalised to [0, 1] 2302.02913 §7.1 §7.2 tab. 1. The paper's formula for DTAI is not reproduced on this page. On a bicycle-frame case study the paper reports DTAI-GAN as highest on all performance and target metrics 2302.02913 §9.4. The case-study numbers are in a figure, not a table, and were not reproduced here.

Conditioning adherence

For generators that take a condition, such as a target value. Adherence recomputes the condition from the generated design and measures its error against the request. The paper also scores the reconstruction of the condition and a conditional distribution score on the nearest 10 % of the data by condition value 2302.02913 §8.1 §8.4 tab. 6. In the 2-D tests, the conditional VAE has the lower adherence error 2302.02913 tab. 6.

Learned human-preference scores (HPS)

HPS fine-tunes CLIP ViT-L/14 so that a prompt's text embedding moves towards the embedding of the image people preferred among images made from that prompt. The score is 100 times the cosine similarity between the two embeddings. The paper gives the training objective in words only 2303.14420 §5. On 5,000 human choices, HPS reaches 43.5 % accuracy, against 32.9 % for CLIP ViT-L/14, 31.4 % for an aesthetic classifier and 26.1 % at random. Human participants reach 42.0 % 2303.14420 tab. 2. On 398 pairs of Stable Diffusion and DALL·E images, agreement with humans is 61.5 ± 1.1 for HPS, 56.8 ± 1.7 for CLIP, and 63.5 ± 4.3 between humans 2303.14420 tab. 3.

The same score selected training images for a rank-32 LoRA on Stable Diffusion v1.4. The adapter is preferred in a 100-prompt user study with 20 participants. The share of images with more than 10 positive votes rises from 22 % for the original model to 74 % for the adapted one 2303.14420 §6 §7.2. The paper's dataset is distributed under a non-commercial licence, and the card does not establish the licence of the classifier checkpoint. Card: Human Preference Score.

Multi-metric text-to-video benchmark (EvalCrafter)

EvalCrafter uses 700 prompts and 17 automatic metrics in four aspects: visual quality, text-video alignment, motion quality and temporal consistency 2310.11440 §3 §4. For each aspect, a linear model is fitted to 1-5 user ratings, and the four aspect scores are then combined. The paper reports 8,647 raw ratings, of which 1,024 were kept 2310.11440 §4.5. The released code sums the four aspects without weights eval_from_metrics.py.

The paper says the fitted score beats a plain average of its metrics on held-out Spearman correlation in all four aspects. In its table the margins are small for visual quality (55.4 against 55.0), temporal consistency (56.7 against 54.4) and alignment (32.3 against 31.9), and the motion average is a negative correlation (-38.2), so that row should be read by magnitude 2310.11440 tab. 3. Single metrics vary widely: CLIP-Score has a Spearman correlation of 6.3 with human ratings, against 26.7 for BLIP-BLEU 2310.11440 tab. 3. The full benchmark takes about two hours and at least 16 GB on one A100, without optimisation 2310.11440 app. D.6. Card: EvalCrafter.

When to use which

Implementation sketch

Our own pseudocode of the FVD core, after the video paper's definition. Preprocessing (resize, value range) is not stated in the paper and is taken from the authors' code 1812.01717 §2 frechet_video_distance.py.

# Set-level FVD, lower is better. N real and N generated clips.
# Embed both sets with a pretrained video network (Kinetics-400 logits).
real_feats = embed(real_clips)  # (N, D)
fake_feats = embed(fake_clips)  # (N, D)

# Fit one Gaussian per set: mean and covariance [arxiv:1812.01717 §2]
mu_r, cov_r = mean(real_feats), cov(real_feats)
mu_f, cov_f = mean(fake_feats), cov(fake_feats)

# Fréchet distance between the two Gaussians [arxiv:1812.01717 eq. 2]
score = |mu_r - mu_f|^2 + trace(cov_r + cov_f - 2 * sqrtm(cov_r @ cov_f))

Where variants differ: a KVD implementation replaces the last two steps with an unbiased MMD estimate on the same embeddings 1812.01717 §2. A frame-level implementation skips the embedding and compares each output with its own reference clip.

For design scores, the common core is a distance from one point set to another. Realism and novelty use nearest-neighbour distances, and coverage uses the reverse direction 2302.02913 §3.2 §4.2 §4.3. The authors' code uses Euclidean distance only evaluation.py. Distance choices are left to the practitioner in the paper 2302.02913 app. 14.2.

# Nearest-datapoint score for realism [arxiv:2302.02913 §4.2]
# Distance from each generated design to its closest training design
distances = pairwise_distance(samples, dataset)  # (num_samples, num_data)
per_sample = minimum(distances, axis=1)
score = mean(per_sample)  # lower is more realistic

Contradictions

• Generational distance direction (within 2302.02913). The text says the better model is closer to the reference front, which implies lower generational distance 2302.02913 §7.3. Table 1 marks it as minimise tab. 1. Table 5 bolds the higher value for the better model tab. 5. The paper does not resolve this, so check the value direction before comparing models.
• Entropy direction (within 2302.02913). Table 1 marks entropy as minimise tab. 1, while the paper treats it as a diversity score [arxiv:2302.02913 §5.2], which would usually be maximised. The paper does not resolve this.
• Averages and sign convention (within 2310.11440). The "Avg." rows in its correlation table are not plain averages of the rows above them, and the paper does not define them 2310.11440 tab. 3. Its "Ours" advantage is also not uniform: for text-video alignment the Kendall value is lower for the fitted score than for the average 2310.11440 tab. 3. The paper gives no sign convention for lower-is-better metrics, so read its correlations by magnitude 2310.11440 tab. 5.
• Sampler and the "complete sampling" claim (within 2212.09478). The text says complete DDPM sampling is better overall 2212.09478 §4.4. Its best FVD and KVD on both datasets, and its best AIST++ FAD, come from that row, but the best Landscape FAD comes from the DPM-Solver row [arxiv:2212.09478 tab. 1] tab. 2. The paper does not resolve this.
• Average human agreement (within 2505.17613). The abstract and the statistics give 94.4 % overall agreement 2505.17613 §1 tab. 7. The per-modality figures in the introduction average to about 93.4 %, and the human table's average row shows 0.935 for GPT-4o, while its caption gives 0.944 2505.17613 §1 tab. 8. For inter-annotator agreement, the text gives 97.1 % while the table's average row shows 0.980 2505.17613 §4.1 tab. 8.
• Number of models (within 2505.17613). The image table caption and the separability analysis say 10 models 2505.17613 tab. 9 app. D.3, while the confidence-interval caption and the leaderboard analysis say nine 2505.17613 tab. 13 §4.3. Check which set a number comes from.
• Whether FID-type scores track human choice (across papers). On BAIR video, the FVD row agrees with raters more often than SSIM and PSNR 1812.01717 tab. 1. On Discord text-to-image images, IS and FID show no meaningful difference between preferred and non-preferred images 2303.14420 tab. 1. The media, datasets and rater tasks differ, so these do not directly conflict, but neither result transfers to the other setting without a check.

Papers

• Towards Accurate Generative Models of Video: A New Metric & Challenges: proposes FVD and KVD as set-level video metrics, compares them with PSNR and SSIM in a human study, and adds the StarCraft 2 Videos benchmark.
• Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design: a review and practical guide to metrics for deep generative models in design, covering similarity, novelty and diversity, constraints, performance and target achievement, and conditioning, with 2-D experiments and two case studies.
• Pros and Cons of GAN Evaluation Measures: New Developments: a survey of GAN evaluation measures after 2019, covering FID variants, FVD, PPL, precision and recall extensions and human-judged scores. It runs no experiments of its own.
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference: fine-tunes CLIP on human pairwise choices to score text-to-image images, then uses the score to train a Stable Diffusion LoRA.
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models: a 700-prompt text-to-video benchmark with 17 automatic metrics in four aspects, whose weights are fitted to user ratings.
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation: a joint audio-video diffusion model that reports FVD, KVD and FAD, plus a Turing test and mean opinion scores against real clips.
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models: a text-to-video model whose evaluation reports IS, FVD, and both CLIP-FID and FID on MSR-VTT; the scores serve as a comparison, not as a metric study.
• MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation: a 55-task benchmark whose per-task checks are programs, VLM or audio judges with rubrics, or reference similarities, validated against human annotators on 41 tasks.