EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

In plain words

Text-to-video models turn a sentence into a short clip. Older papers judged them with one or two statistics that compare generated clips to real ones. Those numbers miss whether the clip matches the sentence, whether the motion is sensible, and whether things stay consistent from frame to frame.

EvalCrafter is a fixed test kit for this. It has a set of 700 prompts, a script that scores every generated clip with 17 automatic measurements, and a way to turn those measurements into one ranking that is tuned to what people said they liked. The measurements cover four things: how good the video looks, how well it matches the prompt, how it moves, and whether it stays consistent over time.

The authors ran eight video models through the kit and report which models did best on each part. The results are only as good as the automatic measurements, and several of them turn out to be fragile, as the pitfalls below explain.

TL;DR

• A 700-prompt benchmark in four content types (human, animal, object, landscape), with sub-types for styles and camera motion §3.
• Seventeen automatic metrics, grouped into four aspects: visual quality, text-video alignment, motion quality and temporal consistency §4.
• Per-aspect linear models are fitted to 1-5 user ratings (8,647 ratings in total, 1,024 kept), and the four aspect scores are summed into a final score §4.5.
• The code is public, but the repository has no LICENSE file. Several metrics differ from the paper's formulas (see Paper vs code).

Problem

Metrics such as FVD need real reference videos and measure distribution match, not prompt match §1 §4.1. Before this paper, text-to-video models were typically compared with FVD, IS or CLIP similarity, which the authors argue do not cover text alignment, motion or temporal consistency §1 §2. Video is also slow to generate; the authors estimate more than two minutes per sample for the models they tested, so per-sample evaluation during training is impractical app. D.6.

Method

Benchmark construction §3

1. Collect real-world prompts from Discord users of two text-to-video services (FullJourney and PikaLab): over 600k prompts, filtered to about 200k after removing repeats and meaningless entries §3.
2. Analyse prompt length and word classes. Most prompts have 3 to 40 words, and four meta-subject classes are used: human, animal, object and landscape §3 app. A.
3. Generate prompts with GPT-4 from each meta-class with sampled attributes. A self-check asks GPT-4 whether each generated description matches its metadata, and the authors then filter the prompts by hand §3.
4. Add prompts from real users and from two text-to-image benchmarks (DALL-Eval, DrawBench), with metadata produced by GPT-4 and checked for consistency §3.
5. Add 50 styles and 20 camera-motion prompts, placed randomly into 250 of the prompts. The average prompt has 12.3 words §3.

Metrics (17 in total; the aspect groupings follow §4 and the raw table tab. 4)

• Visual quality §4.1: DOVER aesthetic (VQA_A) and technical (VQA_T) scores, and Inception Score (IS).
• Text-video alignment §4.2: CLIP-Score (frame-wise, averaged); SD-Score, which compares video frames with five SDXL images per prompt eq. 1; BLIP-BLEU, which compares BLIP-2 captions of five frames with the prompt eq. 2; Detection-Score eq. 3, Count-Score eq. 4 and Color-Score, computed with SAM-Track on the video; OCR-Score, the mean of NED, CER and WER on text read by PaddleOCR; and Celebrity ID Score, a DeepFace distance to three reference images eq. 6.
• Motion quality §4.3: Action-Score (Kinetics-400 recognition with VideoMAE V2); Flow-Score (mean RAFT flow magnitude per frame pair); Motion AC-Score (whether the mean flow is above a threshold ρ = 5, compared with the amplitude named in the prompt).
• Temporal consistency §4.4: CLIP-Temp (cosine similarity of consecutive frames); Warping Error (flow-based warping difference between frames); Face Consistency (similarity of every frame to the first frame).

Human alignment §4.5

1. Generate videos for each model from the prompts (512 prompts in the first user study, later 700 app. D.9).
2. Collect 1-5 ratings from seven users per metric. The paper lists five aspects: video quality, text-video alignment, motion quality, temporal consistency and subjective likeness.
3. Pick a random 80% of the samples (from four methods) to fit a linear regression per aspect, and keep 20% for testing.
4. Combine the four aspect models into a total score.

Scores are reported on a 0-100 scale. The paper says the four aspect results are integrated into a total score §4.5; the code sums them unweighted eval_from_metrics.py.

# Human-aligned score for one model  [§4.5]
# Code: eval_from_metrics.py#L53-L80
# Input:  metrics: the 17 objective values for one model
# Output: four aspect scores and their sum (0-100 scale)

def final_score(metrics):
    # Visual quality: DOVER aesthetic, DOVER technical and IS  [§4.1]
    quality_inputs = [
        metrics["VQA_A"] / 100,
        metrics["VQA_T"] / 100,
        metrics["IS"] / 100,
    ]
    # Temporal: CLIP-Temp, 1 - warping error, face consistency  [§4.4]
    temporal_inputs = [
        metrics["clip_temp"],
        1 - metrics["warping_error"],
        metrics["face_consistency"],
    ]
    # Motion: Action, Motion AC, Flow / 100  [§4.3]
    motion_inputs = [
        metrics["action"],
        metrics["motion_ac"],
        metrics["flow"] / 100,
    ]
    # Text-video alignment: OCR and celebrity error are inverted  [§4.2]
    align_inputs = [
        metrics["clip_score"], metrics["blip_bleu"], metrics["sd_score"],
        metrics["detection"], metrics["color"], metrics["count"],
        1 - metrics["ocr"], 1 - metrics["celebrity_id"],
    ]
    # Each aspect: linear model, scaled to 0-100  [§4.5]
    # Coefficients are hard-coded in the repository
    # [code: eval_from_metrics.py#L53-L68]
    quality = 100 * (dot(QUALITY_W, quality_inputs) + QUALITY_B)
    temporal = 100 * (dot(TEMPORAL_W, temporal_inputs) + TEMPORAL_B)
    motion = 100 * (dot(MOTION_W, motion_inputs) + MOTION_B)
    align = 100 * (dot(ALIGN_W, align_inputs) + ALIGN_B)
    # Total: unweighted sum  [code: eval_from_metrics.py#L80]
    return quality + temporal + motion + align

Videos evaluated §5 tab. 1 app. B: eight models were run at their default settings, including classifier-free guidance: ModelScope, Floor33 Pictures, ZeroScope, Show-1, Hotshot-XL, VideoCrafter1, and two services, Gen2 and PikaLab. Their resolutions and clip lengths are in tab. 1. The user study §4.5 used five models: ModelScope, ZeroScope, Gen2, Floor33 Pictures and PikaLab. Gen2 and PikaLab were run with a 16:9 aspect ratio and with the PikaLab watermark added to every model, to make comparisons fair §4.5.

Hyperparameters

Results

Author-reported, from the paper's own runs. Setup: 700 prompts (the user study used the first 512), eight models run at default settings, scores on a 0-100 scale §5 app. D.9.

Human-aligned results tab. 2. Rank in brackets.

The table has no overall column. The paper calls Gen2 the best model overall [Finding 10, §5.2], but Show-1 is top on text-video alignment and PikaLab on temporal consistency tab. 2.

Correlation with human judgment tab. 3, Spearman's rho and Kendall's phi, in percent. The "Avg." row is a plain average of the metrics above it in the paper's text; "Ours" is the fitted score on held-out samples §5.1.

Raw metric values tab. 4. All values as printed; the caption says every metric is a percentage except Warping Error and Flow-Score. Arrows give the direction of "better" as printed in the table.

Whole correlation table tab. 5 (appendix B.2). This is the same analysis as tab. 3 with all eight text-video alignment metrics and the IS and Action-Score rows added.

Findings the authors draw §5.2:

• Rankings change across aspects, so one metric is not enough (Finding 1).
• Ranking by meta-type differs (Finding 2).
• Users rate visual appeal above text alignment (Finding 3).
• No tested model follows camera-motion instructions from text (Finding 4).
• Larger motion amplitude does not mean higher user preference (Finding 6).
• Text rendering is hard for all models (Finding 7).
• Some models sometimes generate badly broken videos (Finding 8).
• CLIP-Score has low correlation with humans: Spearman 6.3 against 26.7 for BLIP-BLEU (Finding 9).

Cost app. D.6: about two hours and at least 16 GB on an A100 for the full benchmark, without code optimisation. The authors say this is designed for offline evaluation, similar in role to FVD.

Paper vs code

Code: https://github.com/EvalCrafter/EvalCrafter at commit 1275bda05e74f295ad2585e3ebde8e95a81a15af (shallow clone; checked by reading). The project page links it as "Code" [project page]. The repository has no LICENSE file at the root, so the code licence is unknown and the code should only be cited by path and line. Vendored third-party tools (metrics/mmaction2, metrics/RAFT, metrics/DOVER, metrics/deepface, metrics/Segment-and-Track-Anything) each carry their own licence files; these were not checked in detail. The repository contains evaluation code only. It does not generate videos; the generated videos and the prompts are supplied by the user [README]. The human-feedback ratings are released separately as an external file [README].

Differences found by reading the code. Each is a place where an implementation following the paper would get a different number.

• Action-Score is not classification accuracy. The paper says it takes "the classification accuracy" §4.3. The code takes the top-5 Kinetics labels with their confidences, embeds the label names and the prompt's action with CLIP text, and returns the confidence-weighted sum of similarities action_score.py. Only prompts with an action attribute are scored action_score.py.
• BLIP-BLEU aggregates differently from eq. 2. Eq. 2 averages BLEU over the five captions eq. 2. The code takes the maximum over caption pairs for each BLEU order, then averages BLEU-1 to BLEU-4 Scores_with_CLIP.py Scores_with_CLIP.py. A CIDEr score is also computed but not used Scores_with_CLIP.py Scores_with_CLIP.py.
• Count-Score is clamped and counted differently. Eq. 4 has no clamp eq. 4. The code caps the mean relative error at 1, so the score cannot go below 0 object_attributes_eval.py. Counts are the number of SAM-tracked object IDs in each sampled frame, not a per-frame detector count object_attributes_eval.py.
• Color-Score uses a hue rule. The paper does not state how colours are read. The code averages the hue of the masked pixels and maps it to one of six colour names by fixed hue bins object_attributes_eval.py object_attributes_eval.py.
• Face Consistency is not face-specific. The paper describes "human identity" consistency §4.4. The code embeds whole frames with CLIP and compares each frame with the first frame; no face detector is used Scores_with_CLIP.py.
• CLIP inputs are not normalised. The frames are converted to float tensors with values from 0 to 255 and passed straight to get_image_features. No mean/std normalisation and no CLIP processor are applied Scores_with_CLIP.py Scores_with_CLIP.py Scores_with_CLIP.py Scores_with_CLIP.py. Frames are also resized to 224x224 without cropping Scores_with_CLIP.py Scores_with_CLIP.py.
• SD-Score images are generated without a seed. The code generates the SDXL images at 512x512 with 20 steps and no fixed seed, then caches them by video name Scores_with_CLIP.py. The output path is a hard-coded cluster path Scores_with_CLIP.py.
• Warping error is more involved than the paper says. The paper describes a pixel-wise difference §4.4. The code runs RAFT in both directions at half resolution, detects occlusions, and averages the squared error over non-occluded pixels per pair optical_flow_scores.py optical_flow_scores.py.
• OCR counts silent frames as perfect. The code gives frames with no OCR output an error of 0 ocr_score.py. The score is an error (lower is better), so a frame with no detected text looks like a perfect match.
• Celebrity ID uses DeepFace defaults. The paper gives no model or distance §4.2. The code calls DeepFace.verify without naming a model or distance metric, and takes the minimum over the reference images for each frame celebrity_id_score.py.
• Motion AC labels come from metadata. The paper does not state where the amplitude label comes from. The code reads it from metadata.json by the first four characters of the file name, and gives each video a binary score optical_flow_scores.py optical_flow_scores.py.
• IS uses every frame of each video. The code computes IS with 10 splits over all frames is.py is.py. The paper gives no split count or frame choice.
• DOVER output is squashed to 0-1. The code applies a sigmoid to both DOVER outputs and multiplies by 100 evaluate_a_set_of_videos.py. The paper says nothing about this step.
• The regression coefficients are only in the code. The paper says coefficients were fitted §4.5 but prints none. The code hard-codes them eval_from_metrics.py. The fitting code is not in the repository, and the training ratings are external [README].
• Intercept scaling is inconsistent. Three intercepts are multiplied by 5 but the alignment intercept is not eval_from_metrics.py eval_from_metrics.py eval_from_metrics.py eval_from_metrics.py. Whether this is intended is not stated.
• The final score uses transformed inputs. Temporal uses 1 minus warping error, and alignment uses 1 minus OCR and 1 minus celebrity error eval_from_metrics.py eval_from_metrics.py. The paper does not describe these transforms.
• Model ids and paths are repository-specific. The scripts expect checkpoints in checkpoints/ and read the prompt list from prompts/ Scores_with_CLIP.py. The Celebrity ID script reads reference images from ./celebrities/, which is not in the repository celebrity_id_score.py.

No other discrepancies were checked in detail, such as the VideoMAE V2 config variant or the Detection-Score keyword list.

Pitfalls

• The "Avg." rows are not plain averages of the rows shown. In tab. 3, the visual average of 42.1 and 53.6 is 47.9, not 55.0; the temporal average of 49.8 and 69.0 is 59.4, not 54.4; the motion average of -22.1 and -43.3 is -32.7, not -38.2. The paper never defines these averages. The rows also differ between tab. 3 and tab. 5: the visual average is 55.0 in one and 54.9 in the other, and the motion and temporal averages are identical even though tab. 5 adds metrics. Treat the "Ours" versus "Avg." comparison with care.
• The paper's claim that the fit beats the average is not uniform. The text says the fitted score correlates better than the average "consistently" app. B.2. In tab. 3 the Kendall value for text-video alignment is lower for "Ours" (22.5 against 22.7).
• Sign convention for "lower is better" metrics is not stated. Warping Error has a positive correlation (+69.0), while Celebrity ID (-26.0) and OCR (-8.3) have negative ones tab. 5. A positive value for warping error would mean that more warping goes with higher human scores, which contradicts its meaning. The paper gives no sign convention, so read these correlations by magnitude.
• The user study covers five models, but the results cover eight. §4.5 lists five models for the user study. tab. 2 reports human-aligned scores for eight, including VideoCrafter1, Show-1 and Hotshot-XL. The paper does not say how the other three were rated.
• Two prompt sets are mixed. The user study began with 512 prompts and was extended to 700; the authors say they reused the earlier user-study result instead of collecting new ratings app. D.9. The "2.5k videos" figure §4.5 corresponds to 5 models times 512 prompts, not 700.
• The stability claim has no table in the compiled paper. app. D.7 says score changes under prompt noise are below 0.2 points. A stability table exists in the source (tables/stability.tex), but it is not included in the paper, and its values include +0.63 for ModelScope on text-video alignment. Do not rely on the 0.2 claim.
• Sample sizes per metric are not stated. The number of prompts behind each metric is not given. Some metrics (Action-Score, Motion AC, Celebrity ID) only apply to prompts with the matching attribute.
• The fitting procedure is under-specified. The four methods used for fitting, the random seed and the regression objective beyond least squares are not given. The final score's unit and scale are not defined in the paper.
• Several metrics depend on hidden choices. Frame count, resize rule, CLIP normalisation, SDXL seed and BLEU aggregation all change the numbers (see Paper vs code). An implementer following only the paper will not reproduce the released scores.
• Some models are closed. Gen2 and PikaLab are web services, and Floor33 Pictures is not open source tab. 1. Their results cannot be regenerated from public weights.
• Check each metric's units before comparing. The caption of tab. 4 says all values are percentages except warping and flow, but IS values (about 14 to 18) and Flow-Score values (0.5 to 9.3) do not look like percentages, and the paper does not say what units they use.
• The benchmark is small and early. The paper calls the 700 prompts an initial version §3 and lists the number of prompts as a limitation §5.3. The model versions in tab. 1 date from 2023. The authors say the benchmark is meant to be updated periodically app. D.3.

Lineage

• Compared against: FVD and IS as the usual text-to-video measures §1 §4.1, and CLIP similarity §4.2.
• Prompt sources: FullJourney and PikaLab Discord prompts §3; DALL-Eval and DrawBench prompts §3.
• Metric sources, all cited in the paper: DOVER §4.1; RAFT §4.3, §4.4; VideoMAE V2 and MMAction2 §4.3; SAM-Track §4.2; DeepFace §4.2; PaddleOCR §4.2; BLIP-2 §4.2; SDXL §4.2.
• Models evaluated: ModelScope, ZeroScope, Floor33 Pictures, Show-1, Hotshot-XL, VideoCrafter1, Gen2 and PikaLab tab. 1 §5.