--- kind: concept slug: generative-model-evaluation title: Generative model evaluation summary: >- How to score a generative model's output beyond one frame or one sample. Covers frame-level reference metrics (PSNR, SSIM), set-level image, video and audio distances (FID, FVD, KVD, FAD), learned human-preference scores (HPS), design-oriented scores, per-task checks (MMMG) and multi-metric benchmarks (EvalCrafter). Set-level scores need a fixed sample count. All numbers are author-reported. tags: [evaluation, generative-models, video-generation, evaluation-metric, benchmarks, human-evaluation, human-preference, engineering-design, audio] papers: ["1812.01717", "2302.02913", "2103.09396", "2303.14420", "2310.11440", "2212.09478", "2305.10474", "2505.17613"] related: [human-evaluation, benchmark-validity, llm-as-a-judge, generative-adversarial-network, variational-autoencoder, diffusion-models, classifier-guidance, text-to-image-generation, video-generation, frechet-inception-distance, reward-modeling, llm-evaluation] status: draft updated_at: 2026-11-01 --- ## Overview A generative model produces samples that have no single correct answer, so a score has to say something about the whole output distribution, the individual sample, or both. The eight cards on this page cover several settings, and they lead to different families. - **Per-sample, reference-based scores.** PSNR and SSIM compare one generated frame with one known true frame. They need a ground-truth sequence for every output and score each frame on its own, so they miss whether a sequence moves coherently or whether the set of outputs is diverse [arxiv:1812.01717 §1] [§2]. - **Set-level, distribution-based scores.** FVD compares the statistics of a set of generated clips with a set of real clips. It needs no paired ground truth, so it applies to unconditional generators as well [arxiv:1812.01717 §2]. A related kernel score, KVD, uses the same features but avoids the Gaussian assumption [arxiv:1812.01717 §2]. The image version of the same idea is FID, which a GAN survey reviews together with its variants [arxiv:2103.09396 §2.1]. Joint audio-video generators add a set-level audio distance, FAD [arxiv:2212.09478 §4.3]. - **Multi-criteria design scores.** A design generator is judged on more than resemblance to its training data. The metric catalogue sorts scores into similarity, novelty and diversity, constraint satisfaction, performance and target achievement, and conditioning [arxiv:2302.02913 §1] [tab. 1]. Many of these need a distance on the design space or on the performance space, and the paper leaves that choice to the practitioner [arxiv:2302.02913 §3.2]. - **Learned preference scores.** A network is trained on human pairwise choices and then ranks new outputs. HPS does this for text-to-image images, and the same score also selects training data [arxiv:2303.14420 §5] [§6]. - **Per-task checks and multi-metric benchmarks.** A benchmark attaches a check to each task: a program, a vision or audio model answering a narrow question, or a reference comparison. MMMG does this across image, audio and interleaved generation [arxiv:2505.17613 §3.2]. EvalCrafter runs many automatic metrics and fits their weights to human ratings for text-to-video [arxiv:2310.11440 §4.5]. Set-level scores depend on the sample count. FID is usually computed with more than 50K samples [arxiv:2103.09396 §1.1], and FVD depends on N for identical distributions [arxiv:1812.01717 §4.2]. Several set-level and design scores also depend on a feature space or a distance choice. Scores are only comparable when those choices are fixed [arxiv:1812.01717 §4.2] [arxiv:2302.02913 §3.2]. ## Variants ### Frame-level reference metrics (PSNR, SSIM) These are the baseline the video paper argues against for video. They need one ground-truth clip per output. Across more than 20,000 models, the paper reports a Pearson correlation of 0.730 between SSIM and PSNR, but of -0.640 between SSIM and FVD and -0.278 between PSNR and FVD [arxiv:1812.01717 §4.5]. The paper gives no separate definition for them that this page relies on. ### Fréchet Video Distance (FVD) The set-level variant. Real and generated clips are each embedded with a pretrained I3D action-recognition network. The paper uses the Kinetics-400 logits layer. Each embedding set is summarised by a mean and covariance, and the score is the Fréchet distance between the two Gaussians [arxiv:1812.01717 §2] [§4.1]. The GAN survey lists the same I3D network, pretrained on Kinetics-400 and Kinetics-600 [arxiv:2103.09396 §2.1.7]. Card: [Towards Accurate Generative Models of Video: A New Metric & Challenges](/papers/1812.01717). In the paper's human study on BAIR, the FVD row agreed with raters in 74.9 % of comparisons where SSIM was held equal, 81.0 % where PSNR was held equal, and 63.0 % where KVD was held equal [arxiv:1812.01717 tab. 1]. These figures come from one human study on BAIR, with four model families [arxiv:1812.01717 §4.3]. Generator papers report FVD as well, and their setups differ. MM-Diffusion reports FVD and KVD with the same I3D network, computed at 64×64 with 2,048 generated samples per model [arxiv:2212.09478 §4.1] [§4.3]. Its ground-truth row is not zero: 17.83 on Landscape and 8.73 on AIST++ [arxiv:2212.09478 tab. 1] [tab. 2]. PYoCo reports zero-shot UCF-101 FVD 355.19 [arxiv:2305.10474 tab. 2], and unconditional UCF-101 FVD 310 ± 13 from 2,048 videos [arxiv:2305.10474 §5.2] [tab. 4]. These values come from different datasets and protocols, so they are not directly comparable. ### Kernel Video Distance (KVD) The same I3D features scored with an unbiased squared maximum mean discrepancy under a cubic polynomial kernel, k(a, b) = (aᵀb + 1)³ [arxiv:1812.01717 §2]. It does not assume Gaussians and needs no matrix square root. Its human agreement is in the same table as FVD [arxiv:1812.01717 tab. 1]. MM-Diffusion also reports KVD beside FVD on both of its datasets [arxiv:2212.09478 tab. 1] [tab. 2]. The paper does not specify the kernel parameters; its code uses an unparameterised polynomial kernel [code: evaluations/compute_fvd_kvd.py#L18-L34]. ### Image FID and its variants FID scores a set of generated images against a set of real images, using Gaussians fitted to Inception features. The GAN survey warns that FID and IS are biased, need many samples, and can be raised by memorising the training data [arxiv:2103.09396 §1.1] [§4.1]. The variants it reviews aim at these problems: - **Class-aware FID** averages a Fréchet distance over K classes [arxiv:2103.09396 §2.1.2]. The survey prints its mean term as a norm, not a squared norm, and says to check the original paper before implementing it. - **FastFID** is an optimisation of the FID computation. The survey reports it 25 to 500 times faster than the standard computation, depending on the number of generated samples, which makes FID usable as a training loss [arxiv:2103.09396 §2.1.3]. - **Memorization-informed FID** multiplies FID by a penalty when generated samples sit close to training samples. The survey does not state the threshold or the epsilon [arxiv:2103.09396 §2.1.4]. - **Bias-corrected FID and IS** remove finite-sample bias by extrapolation [arxiv:2103.09396 §2.1.5]. **Clean FID** fixes the preprocessing, since the choice of resize library, interpolation and encoding can change FID substantially [arxiv:2103.09396 §2.1.6]. - **Precision and recall extensions** include density and coverage, and α-precision, β-recall and authenticity [arxiv:2103.09396 §2.7]. The survey runs no experiments of its own. Its figures are quoted from the works it cites, so check each one against its original source before use. Text-to-video papers sometimes use a CLIP-based FID instead of the Inception one. PYoCo reports both on MSR-VTT, with CLIP-FID computed by a ViT-B/32 model and FID by Inception-V3 [arxiv:2305.10474 §5.2] [tab. 3]. Its final configuration reaches CLIP-FID 9.73 and FID 22.14 [arxiv:2305.10474 tab. 3]. The two columns are not interchangeable. ### Audio distance (FAD) and human studies for joint audio-video output MM-Diffusion scores its audio with FAD, computed on AudioCLIP features and printed multiplied by 10⁴ [arxiv:2212.09478 §4.3]. On Landscape, the ground truth scores 7.51 and the complete-sampling model 10.72, while the DPM-Solver model scores 9.39 [arxiv:2212.09478 tab. 1]. On AIST++ the ground truth scores 8.46 and the complete-sampling model 10.69 [arxiv:2212.09478 tab. 2]. Divide the printed value by 10⁴ before comparing it with unscaled FAD. The same paper adds two human tests. Its Turing test asks viewers whether a clip is real. Ours is judged real for 84.9 % of Landscape clips and 49.6 % of AIST++ clips, against 92.5 % and 84.7 % for real clips [arxiv:2212.09478 tab. 6]. It used 10,000 votes, with 500 generated and 500 real pairs per dataset [arxiv:2212.09478 §4.6] [tab. 6]. Its mean opinion scores rate video quality, audio quality and alignment on a 1 to 5 scale, with five users per task [arxiv:2212.09478 tab. 5] [§4.3]. Both tests are reported by the authors only. For how to design such studies, see [human evaluation](/concepts/human-evaluation). ### Verifiable per-task checks and VLM judges (MMMG) MMMG is a benchmark of 55 tasks and 1288 instructions in four modality combinations [arxiv:2505.17613 §3.1] [tab. 7]. Each task has the check its authors found best matched human judgement [arxiv:2505.17613 tab. 1]. Checks include a vision-language model answering a rubric question, a reference similarity such as DreamSim or CLAP, and a program such as a colour-fill test or a pitch measurement [arxiv:2505.17613 §3.2] [app. C.3]. Human annotators judged the checks on 978 instructions and 2,596 questions. The authors report 94.4 % overall agreement, with 93.5 % for images, 92.3 % for audio, 96.6 % for interleaved image-text and 91.0 % for interleaved audio-text [arxiv:2505.17613 §1] [§4.1]. Human validation covers 41 tasks only, so the overall figure should not be applied to the remaining tasks [arxiv:2505.17613 §1]. The judges are GPT-4o, Gemini 2.5 Pro and Qwen2.5-VL-72B-Instruct [arxiv:2505.17613 app. C.1]. The best image model, GPT Image, reaches 0.707 on this benchmark, against 0.654 for Gemini 2.5 Image [arxiv:2505.17613 §4.2]. Card: [MMMG](/papers/2505.17613). Each instruction is sampled four times, and temperature is 0 for the multimodal language models [arxiv:2505.17613 app. C.2]. The reported agreement and accuracies depend on the judge thresholds, which the paper does not give as numbers (see Paper vs code). ### Latent-space and data-copying checks These look inside the generator rather than only at its outputs. - **Perceptual path length (PPL)** is the expected perceptual change, measured with a learned perceptual distance, along a short spherical step between two latent points. The step size is 1e-4. The survey does not state the number of pairs [arxiv:2103.09396 §2.3.3]. - **Linear separability** trains linear SVMs from latent codes to attribute labels, using 200K generated images and 40 CelebA attributes, and keeps the 50 % most confident labels [arxiv:2103.09396 §2.3.4]. The survey does not say whether a higher or lower final score is better; it says only that low conditional entropy for an attribute suggests consistent latent directions. ### Distribution matching for design generators Used when the question is whether generated designs look like the training designs. The paper's set of scores is nearest datapoint (realism), nearest generated sample (coverage), precision-recall curve summaries (F-beta and area under the curve, after Sajjadi et al.), and maximum mean discrepancy [arxiv:2302.02913 §4.1] [§4.2] [§4.3]. In its 2-D experiment, the VAE and the GAN split the wins across these scores. The GAN wins nearest datapoint and MMD, while the VAE wins nearest generated sample and the precision-recall AUC [arxiv:2302.02913 tab. 2]. Card: [Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design](/papers/2302.02913). ### Novelty and diversity scores These check whether a generated set differs from the data and from itself. The paper lists inter-sample distance, distance to the centroid, a DPP-based diversity score, and the convex hull of the set, with nearest datapoint used in the other direction for novelty [arxiv:2302.02913 §5.1] [§5.2] [tab. 1]. In the same 2-D experiment, the GAN has the larger convex hull and distance to centroid, while the VAE has the larger inter-sample distance and DPP diversity score [arxiv:2302.02913 tab. 3]. ### Constraint satisfaction scores For designs with hard rules, the paper's catalogue has constraint satisfaction and a constraint satisfaction rate, a signed distance to the constraint boundary, a predicted constraint satisfaction from a classifier trained on valid and invalid designs, and the distance to the nearest invalid design [arxiv:2302.02913 tab. 1] [§6.3]. The classifier variant uses k-nearest neighbours with k = 5 in the authors' notebook [arxiv:2302.02913 app. 14.3]. In the 2-D experiment the GAN scores higher than the VAE on the constraint satisfaction, satisfaction rate and predicted satisfaction measures [arxiv:2302.02913 tab. 4]. ### Performance and target achievement For designs with functional goals. The paper uses hypervolume, a target achievement rate, a signed distance to target, and the design target achievement index (DTAI), a score for a vector of objectives that is normalised to [0, 1] [arxiv:2302.02913 §7.1] [§7.2] [tab. 1]. The paper's formula for DTAI is not reproduced on this page. On a bicycle-frame case study the paper reports DTAI-GAN as highest on all performance and target metrics [arxiv:2302.02913 §9.4]. The case-study numbers are in a figure, not a table, and were not reproduced here. ### Conditioning adherence For generators that take a condition, such as a target value. Adherence recomputes the condition from the generated design and measures its error against the request. The paper also scores the reconstruction of the condition and a conditional distribution score on the nearest 10 % of the data by condition value [arxiv:2302.02913 §8.1] [§8.4] [tab. 6]. In the 2-D tests, the conditional VAE has the lower adherence error [arxiv:2302.02913 tab. 6]. ### Learned human-preference scores (HPS) HPS fine-tunes CLIP ViT-L/14 so that a prompt's text embedding moves towards the embedding of the image people preferred among images made from that prompt. The score is 100 times the cosine similarity between the two embeddings. The paper gives the training objective in words only [arxiv:2303.14420 §5]. On 5,000 human choices, HPS reaches 43.5 % accuracy, against 32.9 % for CLIP ViT-L/14, 31.4 % for an aesthetic classifier and 26.1 % at random. Human participants reach 42.0 % [arxiv:2303.14420 tab. 2]. On 398 pairs of Stable Diffusion and DALL·E images, agreement with humans is 61.5 ± 1.1 for HPS, 56.8 ± 1.7 for CLIP, and 63.5 ± 4.3 between humans [arxiv:2303.14420 tab. 3]. The same score selected training images for a rank-32 LoRA on Stable Diffusion v1.4. The adapter is preferred in a 100-prompt user study with 20 participants. The share of images with more than 10 positive votes rises from 22 % for the original model to 74 % for the adapted one [arxiv:2303.14420 §6] [§7.2]. The paper's dataset is distributed under a non-commercial licence, and the card does not establish the licence of the classifier checkpoint. Card: [Human Preference Score](/papers/2303.14420). ### Multi-metric text-to-video benchmark (EvalCrafter) EvalCrafter uses 700 prompts and 17 automatic metrics in four aspects: visual quality, text-video alignment, motion quality and temporal consistency [arxiv:2310.11440 §3] [§4]. For each aspect, a linear model is fitted to 1-5 user ratings, and the four aspect scores are then combined. The paper reports 8,647 raw ratings, of which 1,024 were kept [arxiv:2310.11440 §4.5]. The released code sums the four aspects without weights [code: eval_from_metrics.py#L80]. The paper says the fitted score beats a plain average of its metrics on held-out Spearman correlation in all four aspects. In its table the margins are small for visual quality (55.4 against 55.0), temporal consistency (56.7 against 54.4) and alignment (32.3 against 31.9), and the motion average is a negative correlation (-38.2), so that row should be read by magnitude [arxiv:2310.11440 tab. 3]. Single metrics vary widely: CLIP-Score has a Spearman correlation of 6.3 with human ratings, against 26.7 for BLIP-BLEU [arxiv:2310.11440 tab. 3]. The full benchmark takes about two hours and at least 16 GB on one A100, without optimisation [arxiv:2310.11440 app. D.6]. Card: [EvalCrafter](/papers/2310.11440). ## When to use which | Situation | Use | Why | |---|---|---| | Each output has a ground-truth clip and per-frame fidelity is what you need | Frame-level PSNR or SSIM | They compare directly against that clip; the video paper treats them as the baseline, not as wrong [arxiv:1812.01717 §1] | | No paired ground truth, for example unconditional video generation | FVD or KVD | Set-level scores need only real and generated sets [arxiv:1812.01717 §2] | | You need a score that tracks human preference on video | FVD, with a human check on your own data | The paper's study favours FVD over PSNR and SSIM on BAIR only [arxiv:1812.01717 §4.3] | | Samples are few, or you want to compare against published numbers | Keep N fixed to the published value | FVD stays above zero for identical sets at small N, so values at different N cannot be compared [arxiv:1812.01717 §4.2]. Published video numbers also differ in dataset and sample count, as PYoCo's 2,048-video protocol shows [arxiv:2305.10474 §5.2] | | You want to avoid the Gaussian assumption | KVD | Kernel estimate, no fitted Gaussians [arxiv:1812.01717 §2] | | Comparing text-to-video results with another paper | Match the FID variant (CLIP-FID or Inception FID) and the sample count | PYoCo reports both, and the two columns differ [arxiv:2305.10474 §5.2] [tab. 3] | | Image sets with no paired data, such as GAN or diffusion samples | FID, with the sample count and preprocessing stated; FastFID if it must run inside training | The survey recommends more than 50K samples and reports FastFID speed-ups [arxiv:2103.09396 §1.1] [§2.1.3] | | You suspect the generator copies its training data | Memorization-informed FID or a data-copying check | Standard FID and IS can be raised by memorisation [arxiv:2103.09396 §4.1] [§2.1.4] | | You want to check the smoothness of a GAN latent space | Perceptual path length | It measures perceptual change along latent steps [arxiv:2103.09396 §2.3.3] | | Joint audio and video output, with a distribution score for each modality | FVD or KVD for video, FAD for audio, both on the same sample count | MM-Diffusion reports both on Landscape and AIST++ [arxiv:2212.09478 tab. 1] [tab. 2]. On Landscape its best FAD comes from the DPM-Solver row, not the complete-sampling row that is best on FVD, so read each column on its own [arxiv:2212.09478 tab. 1] | | Generated output must be ranked by subjective realism or quality | A human study with a clear question, plus an automatic score | MM-Diffusion's Turing test and mean opinion scores show the gap to real clips, and its two-stage baseline scores far lower on mean opinion [arxiv:2212.09478 tab. 5] [tab. 6] | | Generated designs must look like the training set, but the goal is more than that | Distribution-matching scores plus at least one scorer for the other goals | Similarity alone can rank models wrongly; the paper shows this on 2-D toy data [arxiv:2302.02913 §1] [§11.1] | | Designs have hard validity rules | Constraint satisfaction, with a learned validity classifier if invalid examples exist | The paper computes validity directly or via a classifier [arxiv:2302.02913 §6.3] | | Designs must reach performance targets | Target achievement or DTAI, with hypervolume for the front | The paper's bicycle study uses these together [arxiv:2302.02913 §7.2] [§9.4] | | The model must produce novel or varied designs | Novelty and diversity scores | Coverage and similarity do not measure this [arxiv:2302.02913 §5.1] | | The image-based similarity score depends on a network trained on natural images | Check the score on your domain first | Pretrained image networks may be biased toward their datasets, so use on structural designs needs a check [arxiv:2302.02913 §4] | | Output must satisfy precise, checkable instructions, such as counts, colours, text or tempo | Per-task checks: programs where the rule is exact, VLM or audio models with rubrics otherwise | MMMG pairs each task with the check that best matched humans [arxiv:2505.17613 tab. 1] [app. C.3]. Validate the judge on your own tasks, because the paper's human study covers 41 tasks only [arxiv:2505.17613 §1] | | Text-to-image output must be ranked by human preference, and you have pairwise choices | A learned preference scorer such as HPS | It beats CLIP and an aesthetic classifier on the paper's choice data [arxiv:2303.14420 tab. 2] [tab. 3]. Check the checkpoint's licence and score scale first, because the paper's code and its released scorer differ | | Text alignment is the only thing measured | Do not use CLIP-Score alone as a proxy for people | It has low correlation with human ratings in EvalCrafter [arxiv:2310.11440 tab. 3]; CLIP alone also trails HPS on preference [arxiv:2303.14420 tab. 2] | | A full text-to-video comparison across quality, alignment, motion and consistency | EvalCrafter's metrics, with your own fit to human ratings | Its weights are hard-coded in the released code [code: eval_from_metrics.py#L53-L68]. The fitting code is not in the repository, and the code has no LICENSE file, so cite it by path only | ## Implementation sketch Our own pseudocode of the FVD core, after the video paper's definition. Preprocessing (resize, value range) is not stated in the paper and is taken from the authors' code [arxiv:1812.01717 §2] [code: frechet_video_distance/frechet_video_distance.py#L49-L52]. ```pseudocode # Set-level FVD, lower is better. N real and N generated clips. # Embed both sets with a pretrained video network (Kinetics-400 logits). real_feats = embed(real_clips) # (N, D) fake_feats = embed(fake_clips) # (N, D) # Fit one Gaussian per set: mean and covariance [arxiv:1812.01717 §2] mu_r, cov_r = mean(real_feats), cov(real_feats) mu_f, cov_f = mean(fake_feats), cov(fake_feats) # Fréchet distance between the two Gaussians [arxiv:1812.01717 eq. 2] score = |mu_r - mu_f|^2 + trace(cov_r + cov_f - 2 * sqrtm(cov_r @ cov_f)) ``` Where variants differ: a KVD implementation replaces the last two steps with an unbiased MMD estimate on the same embeddings [arxiv:1812.01717 §2]. A frame-level implementation skips the embedding and compares each output with its own reference clip. For design scores, the common core is a distance from one point set to another. Realism and novelty use nearest-neighbour distances, and coverage uses the reverse direction [arxiv:2302.02913 §3.2] [§4.2] [§4.3]. The authors' code uses Euclidean distance only [code: evaluation.py#L53-L60]. Distance choices are left to the practitioner in the paper [arxiv:2302.02913 app. 14.2]. ```pseudocode # Nearest-datapoint score for realism [arxiv:2302.02913 §4.2] # Distance from each generated design to its closest training design distances = pairwise_distance(samples, dataset) # (num_samples, num_data) per_sample = minimum(distances, axis=1) score = mean(per_sample) # lower is more realistic ``` ## Contradictions - **Generational distance direction (within 2302.02913).** The text says the better model is closer to the reference front, which implies lower generational distance [arxiv:2302.02913 §7.3]. Table 1 marks it as minimise [tab. 1]. Table 5 bolds the higher value for the better model [tab. 5]. The paper does not resolve this, so check the value direction before comparing models. - **Entropy direction (within 2302.02913).** Table 1 marks entropy as minimise [tab. 1], while the paper treats it as a diversity score [arxiv:2302.02913 §5.2], which would usually be maximised. The paper does not resolve this. - **Averages and sign convention (within 2310.11440).** The "Avg." rows in its correlation table are not plain averages of the rows above them, and the paper does not define them [arxiv:2310.11440 tab. 3]. Its "Ours" advantage is also not uniform: for text-video alignment the Kendall value is lower for the fitted score than for the average [arxiv:2310.11440 tab. 3]. The paper gives no sign convention for lower-is-better metrics, so read its correlations by magnitude [arxiv:2310.11440 tab. 5]. - **Sampler and the "complete sampling" claim (within 2212.09478).** The text says complete DDPM sampling is better overall [arxiv:2212.09478 §4.4]. Its best FVD and KVD on both datasets, and its best AIST++ FAD, come from that row, but the best Landscape FAD comes from the DPM-Solver row [arxiv:2212.09478 tab. 1] [tab. 2]. The paper does not resolve this. - **Average human agreement (within 2505.17613).** The abstract and the statistics give 94.4 % overall agreement [arxiv:2505.17613 §1] [tab. 7]. The per-modality figures in the introduction average to about 93.4 %, and the human table's average row shows 0.935 for GPT-4o, while its caption gives 0.944 [arxiv:2505.17613 §1] [tab. 8]. For inter-annotator agreement, the text gives 97.1 % while the table's average row shows 0.980 [arxiv:2505.17613 §4.1] [tab. 8]. - **Number of models (within 2505.17613).** The image table caption and the separability analysis say 10 models [arxiv:2505.17613 tab. 9] [app. D.3], while the confidence-interval caption and the leaderboard analysis say nine [arxiv:2505.17613 tab. 13] [§4.3]. Check which set a number comes from. - **Whether FID-type scores track human choice (across papers).** On BAIR video, the FVD row agrees with raters more often than SSIM and PSNR [arxiv:1812.01717 tab. 1]. On Discord text-to-image images, IS and FID show no meaningful difference between preferred and non-preferred images [arxiv:2303.14420 tab. 1]. The media, datasets and rater tasks differ, so these do not directly conflict, but neither result transfers to the other setting without a check. ## Papers - [Towards Accurate Generative Models of Video: A New Metric & Challenges](/papers/1812.01717): proposes FVD and KVD as set-level video metrics, compares them with PSNR and SSIM in a human study, and adds the StarCraft 2 Videos benchmark. - [Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design](/papers/2302.02913): a review and practical guide to metrics for deep generative models in design, covering similarity, novelty and diversity, constraints, performance and target achievement, and conditioning, with 2-D experiments and two case studies. - [Pros and Cons of GAN Evaluation Measures: New Developments](/papers/2103.09396): a survey of GAN evaluation measures after 2019, covering FID variants, FVD, PPL, precision and recall extensions and human-judged scores. It runs no experiments of its own. - [Human Preference Score: Better Aligning Text-to-Image Models with Human Preference](/papers/2303.14420): fine-tunes CLIP on human pairwise choices to score text-to-image images, then uses the score to train a Stable Diffusion LoRA. - [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](/papers/2310.11440): a 700-prompt text-to-video benchmark with 17 automatic metrics in four aspects, whose weights are fitted to user ratings. - [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](/papers/2212.09478): a joint audio-video diffusion model that reports FVD, KVD and FAD, plus a Turing test and mean opinion scores against real clips. - [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](/papers/2305.10474): a text-to-video model whose evaluation reports IS, FVD, and both CLIP-FID and FID on MSR-VTT; the scores serve as a comparison, not as a metric study. - [MMMG: A Comprehensive and Reliable Benchmark for Multitask Multimodal Generation](/papers/2505.17613): a 55-task benchmark whose per-task checks are programs, VLM or audio judges with rubrics, or reference similarities, validated against human annotators on 41 tasks.