--- kind: concept slug: video-generation title: Video generation summary: >- Video from context frames, a GAN with continuous-time motion codes, conditioned video diffusion, a robot policy, joint audio and video diffusion, and image-model finetuning with correlated noise; plus a 17-metric text-to-video benchmark and a multiplayer world model. Covers FVD setups and the FVD results of six cards. All numbers are author-reported. tags: [video-generation, benchmarks, evaluation-metric, text-to-video, robot-manipulation, human-animation, autonomous-driving, novel-view-synthesis, multiplayer, world-model, audio, noise-prior] papers: ["1812.01717", "2112.14683", "2212.09478", "2305.10474", "2310.11440", "2311.16498", "2311.17918", "2403.12008", "2512.06963", "2608.06257"] related: [generative-model-evaluation, benchmark-validity, human-evaluation, diffusion-models, generative-adversarial-network, latent-diffusion, classifier-free-guidance, world-model, novel-view-synthesis, implicit-neural-representation, vision-language-action-model, action-chunking, text-to-image-generation, noise-schedule] status: draft updated_at: 2026-11-01 --- ## Overview A video generator produces frames that continue from a short context, such as two context frames on BAIR or ten on KTH [arxiv:1812.01717 §4.4]. A generated video has no single correct answer, so the score has to judge a set of videos rather than each frame against a reference [arxiv:1812.01717 §1] [arxiv:1812.01717 §2]. The first card on this page introduces that set-level score (FVD) and the benchmark setups it is applied to [arxiv:1812.01717 §4.4]. Its four model families are baselines from earlier work, not a new generator [arxiv:1812.01717 §4.3]. Card: [Towards Accurate Generative Models of Video: A New Metric & Challenges](/papers/1812.01717). The other five cards use video generators for four different jobs. A GAN can sample a whole clip from motion codes that are read at any frame time [arxiv:2112.14683 §3.1] ([StyleGAN-V](/papers/2112.14683)). A diffusion model can be steered by an input signal: a reference person and a pose sequence [arxiv:2311.16498 §3.3] ([MagicAnimate](/papers/2311.16498)), a 3D layout and ego actions for six driving cameras [arxiv:2311.17918 §3.3] ([Drive-WM](/papers/2311.17918)), or a camera orbit around one object [arxiv:2403.12008 §3] ([SV3D](/papers/2403.12008)). A fourth card uses the generator as the backbone of a robot policy [arxiv:2512.06963 §3.4] ([VideoVLA](/papers/2512.06963)). Two more cards widen the scope. EvalCrafter is a benchmark for text-to-video models that scores prompt match, motion and temporal consistency alongside visual quality, and does not rely on a set-level distribution score alone [arxiv:2310.11440 §1] ([EvalCrafter](/papers/2310.11440)). MASS is a multiplayer world model: it keeps one typed state of the shared world and renders each camera from that state, rather than keeping a separate video memory per player view [arxiv:2608.06257 §1] ([MASS](/papers/2608.06257)). Two more cards cover joint audio and video, and the finetuning of an image model. MM-Diffusion denoises a video and its soundtrack in one process, with two coupled networks that attend to each other through short windows [arxiv:2212.09478 §3.2] [arxiv:2212.09478 §3.3] ([MM-Diffusion](/papers/2212.09478)). PYoCo's main change is the starting noise rather than the network: the frames of a clip get correlated noise, so that finetuning an image model on video does not discard the frame-to-frame correlation it learned [arxiv:2305.10474 §4] ([PYoCo](/papers/2305.10474)). So the concept covers eight uses: a video model as the thing being measured by FVD, a text-to-video model as the thing being scored by a multi-metric suite, a GAN that generates clips, a conditioned diffusion model, a video model as the backbone of a policy, a rendered world model with shared state, a joint audio and video generator, and an image model adapted to video through its noise prior. Check which one you need before you pick a setup or a variant. ## Variants ### Benchmark subject: context-conditioned video models The first card compares four baseline models on the same benchmarks. It gives no architecture details for them, and none are linked to an arXiv id on the card [arxiv:1812.01717 tab. 2] [arxiv:1812.01717 app. C]. All four were trained with Tensor2Tensor [arxiv:1812.01717 app. C]. | Model | BAIR FVD | KTH FVD | SCV columns where best | |---|---|---|---| | CDNA | 296.5 | 150.8 | MUtB 128 (51.4) | | SV2P | 262.5 | 136.8 | RTwM 128 (1026.1) | | SVP-FP | 315.5 | 208.4 | MUtB 64 (276.7) | | SAVP | 116.4 | 78.0 | CMS 64 and 128, Brawl 64 and 128, RTwM 64, and the BAIR and KTH columns [arxiv:1812.01717 tab. 2] | Lower FVD is better. Each SCV scenario has a 64 x 64 and a 128 x 128 column [arxiv:1812.01717 tab. 2]. The card's human study uses BAIR models from CDNA, SV2P, SVP-FP and SAVP [arxiv:1812.01717 §4.3]. ### Evaluation suite: 17 automatic metrics fitted to human ratings EvalCrafter is a text-to-video benchmark with 700 prompts in four meta-classes: human, animal, object and landscape [arxiv:2310.11440 §3]. Some prompts come from real users of two text-to-video services, and others are generated with GPT-4 and then checked by hand [arxiv:2310.11440 §3]. Its 17 automatic metrics fall into four aspects: visual quality, text-video alignment, motion quality and temporal consistency [arxiv:2310.11440 §4]. For each aspect a linear model is fitted to 1-5 user ratings, and the four aspect scores are summed into one total on a 0-100 scale [arxiv:2310.11440 §4.5]. The paper prints no regression coefficients. The released repository has no LICENSE file, so its code can be read but not copied; see the [card](/papers/2310.11440) for the code's details. Author-reported human-aligned scores for eight models, with the rank in brackets, show that no model leads every aspect [arxiv:2310.11440 tab. 2]: | Model | Visual quality | Text-video alignment | Motion quality | Temporal consistency | |---|---|---|---|---| | Gen2 | 62.51 (1) | 60.98 (5) | 56.43 (1) | 64.41 (2) | | PikaLab | 60.77 (3) | 55.80 (6) | 55.77 (2) | 65.41 (1) | | VideoCrafter1 | 60.85 (2) | 61.95 (2) | 53.08 (5) | 55.89 (7) | | Show-1 | 52.19 (8) | 62.07 (1) | 53.74 (3) | 60.83 (3) | The table has no overall column. The paper calls Gen2 the best model overall [arxiv:2310.11440 §5.2]. The four rows above are the leaders on at least one aspect, not a full ranking; see [arxiv:2310.11440 tab. 2] for all eight. The fitted aspect scores correlate better with human ratings than the single metrics do. Spearman's rho on held-out samples is 55.4 for visual quality, 45.0 for motion amplitude, 56.7 for temporal consistency and 32.3 for text-video alignment [arxiv:2310.11440 tab. 3]. For comparison, CLIP-Score alone reaches 6.3 on text-video alignment, and BLIP-BLEU reaches 26.7 [arxiv:2310.11440 tab. 3]. These are author-reported. They rest on the 1,024 ratings kept from 8,647 collected, and the filter used is not stated [arxiv:2310.11440 §4.5]. The authors' full run takes about two hours and at least 16 GB on an A100, unoptimised, and they describe it as an offline benchmark, similar in role to FVD [arxiv:2310.11440 app. D.6]. ### Unconditional GAN generator: continuous-time motion codes StyleGAN-V keeps the StyleGAN2 image generator almost unchanged and adds a motion signal for each clip. A content code, shared by all frames, comes from the StyleGAN2 mapping network. Motion comes from a sparse grid of noise that two padding-free conv1d layers turn into codes, which can be read at real-valued times, so the frames of a clip are generated independently rather than in sequence [arxiv:2112.14683 §3.1] [arxiv:2112.14683 app. B]. The discriminator scores k frames of a clip together, with k = 3 as the default [arxiv:2112.14683 §3.2] [arxiv:2112.14683 tab. 3]. The paper's FVD16 and FVD128 numbers use the authors' own protocol, and the baselines were re-trained by the authors [arxiv:2112.14683 §4] [arxiv:2112.14683 app. C]. FVD16 results, with the GPU-day cost column from the paper: | Method | FaceForensics | SkyTimelapse | UCF101 | RainbowJelly | Cost (GPU-days) | |---|---|---|---|---|---| | MoCoGAN | 124.7 | 206.6 | 2886.9 | 1572.9 | 5 | | MoCoGAN + StyleGAN2 backbone | 55.62 | 85.88 | 1821.4 | 638.5 | 8 | | MoCoGAN-HD | 111.8 | 164.1 | 1729.6 | 579.1 | 7.5 + 9 | | VideoGPT (capped at 32 GPU-days) | 185.9 | 222.7 | 2880.6 | 136.0 | 16 + 16 | | DIGAN | 62.5 | 83.11 | 1630.2 | 436.6 | 16 | | StyleGAN-V | 47.41 | 79.52 | 1431.0 | 195.4 | 8 | All values are FVD16, lower is better [arxiv:2112.14683 tab. 1]. Two-stage methods are written "X + Y". The paper's text says the method "significantly outperforms" the others on almost all benchmarks [arxiv:2112.14683 §4.1]. The table supports the lowest FVD16 on FaceForensics, SkyTimelapse and UCF101, and VideoGPT is lower on RainbowJelly [arxiv:2112.14683 tab. 1]. Image quality is close to the image model it extends. On FaceForensics at 256 x 256 the authors report FID 9.44 for StyleGAN-V and 8.42 for StyleGAN2 [arxiv:2112.14683 §4.3]. Frames per clip matter: k = 2 gives a lower SkyTimelapse FVD16 (50.5 vs 79.52) but a higher FaceForensics FVD16 (60.41 vs 47.41) [arxiv:2112.14683 tab. 3]. ### Conditioned video diffusion: a reference, a layout or a camera steers the output The three cards below all start from a pretrained image or video diffusion model and add a conditioning input. They differ in what the input is and what the output is for. | Card | Conditioning input | Output | Main author-reported result | |---|---|---|---| | [MagicAnimate](/papers/2311.16498) | Reference photo and a DensePose sequence | Video of the person in that pose sequence | TikTok FVD 179.07 against 292.80 for DisCo [arxiv:2311.16498 tab. 1a] | | [Drive-WM](/papers/2311.17918) | Current six camera views, 3D layouts, text and ego actions | Six-camera driving video, used to score planned trajectories | nuScenes FVD 122.7 [arxiv:2311.17918 tab. 1] | | [SV3D](/papers/2403.12008) | One object image and an optional camera trajectory | 21-frame orbit, then a 3D mesh | GSO static orbit PSNR 21.26 against 18.34 for Stable Zero123 [arxiv:2403.12008 §3] | **Reference-driven animation (MagicAnimate).** The model inflates a 2D image UNet with temporal attention, so each spatial position attends across the frames of a clip [arxiv:2311.16498 §3.1]. A second trainable copy of the UNet reads the reference image and passes its features into the self-attention of the generator [arxiv:2311.16498 §3.2]. Pose enters as DensePose maps, which the paper prefers to sparse keypoints [arxiv:2311.16498 §3.3]. Long videos are split into overlapping segments whose noise predictions are averaged at each step [arxiv:2311.16498 §3.3]. On TikTok the temporal attention removes about 68 FVD points from the ablation, 247.30 to 179.07 [arxiv:2311.16498 tab. 2a]. Among the reference encoders tested, the paper's own gives the lowest FVD (179.07), ahead of IP-Adapter (590.99) and CLIP (724.96) [arxiv:2311.16498 tab. 2b]. The training details (learning rate, batch size, steps, sampling of frames) are not stated in the source the card used, and the released material is inference only [arxiv:2311.16498 §4]. **Layout and action conditioning (Drive-WM).** Drive-WM lifts a single-view image diffusion model to multiview video: it adds temporal layers, then multiview layers that attend across the six cameras, and freezes the spatial weights [arxiv:2311.17918 §3.1]. Its condition is a set of tokens, one for each kind of input, concatenated together [arxiv:2311.17918 §3.3] [arxiv:2311.17918 eq. 5]. The generator can build the six views in two groups, reference views first and stitched views second [arxiv:2311.17918 §3.2]. In the authors' table the factorized version has FVD 116.6 and KPM 94.4%, against 122.7 and 45.8% for the joint model, at a FID cost of 16.4 against 15.8 [arxiv:2311.17918 tab. 2]. For planning, the model renders the future for each candidate trajectory from VAD, and an image-based reward picks one [arxiv:2311.17918 §4.1] [arxiv:2311.17918 §4.2]. **Camera-controlled orbit (SV3D).** SV3D fine-tunes Stable Video Diffusion to produce a 21-frame orbit at 576 x 576 from one image [arxiv:2403.12008 §3]. Three models are trained: SV3D-u with no camera input, SV3D-c with a camera trajectory, and SV3D-p, which is first trained without camera input and then with it [arxiv:2403.12008 §3]. The authors use a triangle-wave classifier-free guidance weight, from 1.0 at the front view to 2.5 at the back view, which avoids over-sharpening the last frames [arxiv:2403.12008 §3]. A two-stage 3D pipeline fits a NeRF and then a mesh, and adds a masked score-distillation loss for parts that no generated view shows [arxiv:2403.12008 §4]. The mesh Chamfer distance is 0.024 for SV3D-p with masked SDS, against 0.039 for Stable Zero123 [arxiv:2403.12008 §4]. ### Policy backbone: video diffusion as a visuomotor model VideoVLA is a vision-language-action policy built on CogVideoX-5B, a video diffusion transformer [arxiv:2512.06963 §4.1]. Its design, in short: - **One joint denoiser.** The token sequence holds the text tokens, the current observation latent, the future video latents and the action tokens. Video and actions get Gaussian noise under the DDPM schedule and are denoised with the DDPM loss. The timestep enters through adaptive LayerNorm [arxiv:2512.06963 §3.4]. - **Action chunk.** The model outputs a chunk of 6 seven-dimensional actions. The first 3 are executed, then the model is queried again on the new observation [arxiv:2512.06963 §3.1] [arxiv:2512.06963 §4.1]. - **Pretrained video prior.** The backbone is initialised from pretrained CogVideoX [arxiv:2512.06963 §3.4]. Training from scratch on the same data drops the SIMPLER three-task Visual Matching average from 80.4 to 12.6. An OpenSora-1.1 backbone scores 50.2 [arxiv:2512.06963 tab. 7]. - **Video loss matters.** Removing the video loss drops the same average to 27.0, and an action-only model reaches 25.5 [arxiv:2512.06963 tab. 9]. Reported results, all author-reported and with small trial counts per task [arxiv:2512.06963 app. A]: | Setting | Best baseline (score) | VideoVLA (score) | Source | |---|---|---|---| | SIMPLER, 12 tasks, average | CogACT (62.6) | 63.0 | [arxiv:2512.06963 tab. 1] | | Novel objects, Google robot, 10 objects | CogACT (42.4) | 65.2 | [arxiv:2512.06963 tab. 2] | | New skills, Google robot, 7 skills | CogACT (20.4) | 48.6 | [arxiv:2512.06963 tab. 3] | | Real robot, in-domain, task average | CogACT (58.4) | 64.6 | [arxiv:2512.06963 tab. 4] | The paper's overall SIMPLER gap over CogACT is 0.4 points. At 24 trials per task one success is 4.2 points, so that gap is within noise [arxiv:2512.06963 app. A]. The larger gaps are on novel objects and new skills. Human-judged imagined videos score higher than the executed success rates: 84.0 vs 65.2 for novel objects and 63.4 vs 48.6 for new skills [arxiv:2512.06963 tab. 10]. ### Multiplayer world model: one typed state, many rendered views MASS keeps one authoritative state of a multiplayer world as a list of typed records: one per entity, plus a tick counter [arxiv:2608.06257 §3.1]. A Logic Engine, a decoder-only Transformer, predicts the next state from the current state and all players' actions [arxiv:2608.06257 §3.2]. A Rendering Engine, a residual U-Net, draws each requested camera from that same state [arxiv:2608.06257 §3.3]. Every view therefore reads one state, so views cannot disagree about what exists [arxiv:2608.06257 §4.5]. The schema, tokenizer and weights are per game, while the model design is shared [arxiv:2608.06257 §3.1]. On a matched two-player, two-view Snake benchmark, the authors report parser state recovery of 0.764 for MASS, 0.128 for B-PV (independent per-view video predictors) and 0.067 for MultiWorld [arxiv:2608.06257 tab. 2]. The matched test set has 8 held-out episodes [arxiv:2608.06257 app. B.1]. Rendering on held-out reference states at 256 x 256 gives PSNR 38.82 dB for Snake and 23.72 dB for Tron, the lowest of eight games [arxiv:2608.06257 tab. 3]. Rendering cost grows with the number of views: the authors measure a mean of 189.6 ms for one view and 842.4 ms for 1,024 views, on one H100 [arxiv:2608.06257 app. C.4]. ### Joint audio and video: two coupled denoisers MM-Diffusion denoises a short video and its audio together, under one joint training loss [arxiv:2212.09478 §3.2]. The video branch uses 2D spatial and 1D temporal layers, and the audio branch uses dilated 1D convolutions [arxiv:2212.09478 §3.3] [arxiv:2212.09478 app. A]. The two branches are linked by random-shift cross-modal attention at three of the four U-Net scales. The audio is cut into one segment per video frame, and each segment attends to a short window of frames whose start is shifted at random [arxiv:2212.09478 §3.3] [arxiv:2212.09478 tab. A.1]. The authors avoid full cross-attention, whose cost grows with the product of the video positions and the audio samples [arxiv:2212.09478 §3.3]. Clips are 16 frames at 64 x 64, paired with 25,600 audio samples at 16 kHz [arxiv:2212.09478 tab. A.1]. Results are author-reported. On Landscape, complete 1,000-step DDPM sampling gives FVD 117.20, against 305.36 for DIGAN and 600.30 for TATS-base; the default DPM-Solver row gives 229.08 [arxiv:2212.09478 tab. 1]. On AIST++ the same sampler gives 75.71, against 119.47 and 267.24 [arxiv:2212.09478 tab. 2]. All metrics are computed at 64 x 64, before a separate super-resolution model produces 256 x 256 output [arxiv:2212.09478 §4.1]. In the authors' Turing test, 84.9% of Landscape clips were judged real, against 92.5% for ground truth; on AIST++ the figures are 49.6% and 84.7% [arxiv:2212.09478 tab. 6]. On a mean opinion score, the two-stage baseline (audio first, then video) gets 1.61 for video quality on Landscape, against 3.75 for the joint model [arxiv:2212.09478 tab. 5]. The same trained model can generate audio from video, or video from audio, without extra training. The paper uses two methods: replacing the conditioning signal with a noised copy of the real one, and adding a gradient term that pulls the target toward the condition [arxiv:2212.09478 §3.4]. No table measures these conditional results. The paper shows them in figures and gives no value for the gradient weight [arxiv:2212.09478 §4.5] [arxiv:2212.09478 fig. 5]. ### Finetuning an image model: correlated noise prior PYoCo starts from eDiff-I, a text-to-image diffusion model, and finetunes it on images and videos together [arxiv:2305.10474 §4]. The authors argue that an image model learns frame-to-frame correlation from real video, and that independent noise per frame makes finetuning unlearn it. They measure an average cosine similarity of reversed noise of 0.206 ± 0.156 within a video, against 0.001 ± 0.009 across videos [arxiv:2305.10474 tab. 1]. Two constructions keep frames correlated, both controlled by one number α, where α = 0 gives independent noise. Mixed noise is a shared part plus a per-frame part [arxiv:2305.10474 eq. 3]. Progressive noise makes each frame perturb the previous one [arxiv:2305.10474 eq. 4]. The large-scale model uses progressive noise with α = 2 [arxiv:2305.10474 §5.3]. Results are author-reported. On UCF-101, finetuning from the image model with i.i.d. noise gives FVD 566.67, against 903.37 when trained from scratch. Adding mixed noise gives 337.40 and progressive noise 339.67 [arxiv:2305.10474 tab. 5]. That table does not state the model size. At large scale the authors report a zero-shot UCF-101 IS of 47.76 and FVD of 355.19 [arxiv:2305.10474 tab. 2], a zero-shot MSR-VTT CLIP-FID of 9.73 for the best ensemble configuration [arxiv:2305.10474 tab. 3], and an unconditional UCF-101 IS of 60.01 and FVD of 310 for the 253M model [arxiv:2305.10474 tab. 4]. Against VDM, they report about 10 times fewer parameters and about 14 times less training time for their unconditional model [arxiv:2305.10474 §5.2]. The α sweep was run only on the 69M UCF-101 model [arxiv:2305.10474 §5.3]. The large-scale training set is about 1.2 billion image-caption pairs and 22.5 million video-caption pairs, from public and proprietary sources, and its filter thresholds are not stated [arxiv:2305.10474 §5.1]. The card's search found no official code. ### Benchmark setups Each dataset fixes the number of context frames, the number of generated frames and the sample count used for FVD: - **BAIR:** 2 context frames and 14 generated frames, FVD over 256 samples [arxiv:1812.01717 §4.3] [arxiv:1812.01717 §4.4]. - **KTH:** 10 context frames and 10 generated frames, FVD over 1024 samples [arxiv:1812.01717 §4.4]. - **SCV (StarCraft 2 Videos):** four game scenarios, MUtB (one unit walks to the map edge), CMS (units gather shards), Brawl (two armies fight) and RTwM (a transport unit picks up and drops off units). Each is rendered at 64 x 64 and 128 x 128 [arxiv:1812.01717 §3] [arxiv:1812.01717 app. B.1]. The split is 10,000 train, 2,000 validation and 2,000 test clips per scenario. FVD uses 1024 samples [arxiv:1812.01717 §3] [arxiv:1812.01717 app. B] [arxiv:1812.01717 §4.4]. - **RTwM:** 2 context frames and up to 32 further frames, the longest setting in the card [arxiv:1812.01717 §4.4]. ## When to use which | Situation | Use | Why | |---|---|---| | Comparing against published FVD numbers on BAIR or KTH | The matching setup and sample count from the benchmark setups above | FVD values are only comparable at the same sample count [arxiv:1812.01717 §4.2] | | Testing on SCV | The resolution column that matches your renderer, 64 or 128 | Table 2 reports the two resolutions in separate columns [arxiv:1812.01717 tab. 2] | | Testing long rollouts | The RTwM setup, and expect little from the baselines | The card's qualitative review found no model that handles RTwM [arxiv:1812.01717 §4.4] [arxiv:1812.01717 app. D] | | You have fewer samples than the benchmark setup uses | Do not treat a single FVD number as settled; use the same N for every model you compare | With identical distributions, FVD is still above zero at small N [arxiv:1812.01717 §4.2] | | You need to judge one generated clip against its own ground truth | A per-frame reference metric, not FVD | FVD scores sets, not single samples [arxiv:1812.01717 §2]. See [generative-model-evaluation](/concepts/generative-model-evaluation) | | You need clips that run for many frames, from a GAN, on a budget near the 8 GPU-days in the cost column | The StyleGAN-V variant, with its motion codes | Motion codes can be read at any frame time, so clip length is not set by a frame list [arxiv:2112.14683 §3.1]. Its FVD16 lead holds on three of four datasets, not RainbowJelly [arxiv:2112.14683 tab. 1] | | You want a human animated from one photo and a pose sequence | The MagicAnimate variant | The authors report the lowest TikTok FVD of the methods compared [arxiv:2311.16498 tab. 1a]. Only inference code is released, so training has to be rebuilt from the paper, which leaves details open [arxiv:2311.16498 §4] | | You need a multi-camera world model that scores candidate trajectories | The Drive-WM variant | Planning gains are open loop on nuScenes [arxiv:2311.17918 §5.1] [arxiv:2311.17918 tab. 3]. The reward inherits the error of the perception models it runs on generated images [arxiv:2311.17918 §4.2] | | You need views of one object from chosen camera angles | The SV3D variant | The authors report higher PSNR and lower LPIPS than the single-image baselines on GSO static orbits [arxiv:2403.12008 §3]. Check the weight licence first (see below) | | You will ship commercially and need the released weights | Read each card's licence section before you choose | SV3D's weights are non-commercial, so the paper's CC BY 4.0 does not cover them [arxiv:2403.12008 §3] | | You want a robot policy that reuses a pretrained video model | A joint video-and-action denoiser as in VideoVLA, with the video branch kept in training | The video loss carries most of the gain: 80.4 vs 25.5 on three SIMPLER tasks [arxiv:2512.06963 tab. 9] | | Your control loop must run faster than about 3 Hz, or you cannot host a 5B backbone | Probably not this variant; the card gives no latency for other policies, so measure them yourself | VideoVLA reports about 1.1 s per chunk with 10 steps on an H100 [arxiv:2512.06963 app. D] | | You have no large pretrained video model to start from | Not this variant | Training from scratch scores 12.6 against 80.4 on the same three tasks [arxiv:2512.06963 tab. 7] | | You need the imagined videos to be right, not only the actions | Budget for human review | The card's imagination scores are human-judged [arxiv:2512.06963 tab. 10] [arxiv:2512.06963 §4.4] | | You want one number for text-to-video models that reflects prompt match and motion, not only distribution | The EvalCrafter suite, with its aspect scores | Its fitted aspect scores reach Spearman 32.3 to 56.7 against human ratings, against 6.3 for CLIP-Score alone on alignment [arxiv:2310.11440 tab. 3]. Expect about two hours and 16 GB on an A100 per run [arxiv:2310.11440 app. D.6] | | You want to reproduce EvalCrafter's published scores | Do not expect it from the paper alone | The paper prints no regression coefficients, and the CLIP frame resize is not stated [arxiv:2310.11440 §4.5] [arxiv:2310.11440 §4.2]. The repository has no LICENSE file, so code cannot be copied (see the card) | | You need a text-to-video number that is comparable to FVD in the literature | Keep FVD and the EvalCrafter scores apart | The two scores measure different things and use different scales [arxiv:2310.11440 §1] [arxiv:2310.11440 §4.5] | | Several views of one shared multiplayer world must agree, for example players in one game | The MASS design: one typed state, one rendering pass per view | Views decode from one predicted state, so cross-view disagreement is zero by construction [arxiv:2608.06257 tab. 2] [arxiv:2608.06257 §4.5] | | You need exact full-state prediction over long horizons | Not MASS as tested | Full-state exact match is 0.0% at 32 and 128 ticks on matched Snake [arxiv:2608.06257 app. B.2] [arxiv:2608.06257 tab. B.4] | | Many views at once on a tight latency budget | Measure it on your hardware before you commit | Rendering goes from 189.6 ms for one view to 842.4 ms for 1,024 views on one H100 [arxiv:2608.06257 app. C.4]. The typed model takes 45.55 ms per matched transition, against 3.07 to 3.94 ms for the dense baselines [arxiv:2608.06257 tab. B.4] | | You need one model that outputs a clip and its soundtrack together, with sound that follows the picture | The MM-Diffusion variant | It is the only card here that denoises both modalities in one process [arxiv:2212.09478 §3.2]. Budget for slow conditional sampling: the guided mode backpropagates through the U-Net at each step, and the shipped scripts use batch size 1 or 2 [code: ssh_scripts/video2audio_sample.sh#L13] | | You need audio steered by video (or video by audio) and a measured number for it | Not MM-Diffusion as published | The conditional results appear only in figures [arxiv:2212.09478 §4.5] [arxiv:2212.09478 fig. 5]. Run your own evaluation | | You have a pretrained text-to-image model but not a video-scale budget | The PYoCo variant, with correlated noise during finetuning | On UCF-101, correlated noise gives FVD 337.40 to 339.67 against 566.67 with i.i.d. noise [arxiv:2305.10474 tab. 5]. The large-scale data mix is not reproducible from the paper [arxiv:2305.10474 §5.1] | | You need PYoCo's large-scale numbers reproduced | Not from the paper alone | No official code was found. Training batch size, σ_max and guidance scale are not stated [arxiv:2305.10474 §5.1] [arxiv:2305.10474 app. A.3] | | You have no text-to-image model to start from | Not PYoCo | The method starts from eDiff-I, and training from scratch scores FVD 903.37 on UCF-101 [arxiv:2305.10474 §4] [arxiv:2305.10474 tab. 5] | Licences differ between code and weights across these cards. The cards record the code licence of each repository, and two of them show the gap: SV3D's code is MIT while its weights are non-commercial, and the StyleGAN-V and EvalCrafter repositories have no LICENSE file at all. Check the card before you copy anything from a repository or depend on a checkpoint. ## Implementation sketch The first loop is our own summary of the benchmark protocol. How a model rolls out from its context is model-specific and not stated on the card. ```pseudocode # Benchmark loop for a context-conditioned video model [arxiv:1812.01717 §4.4] # Keep n_samples fixed across every model you compare [arxiv:1812.01717 §4.2] def evaluate(model, dataset, n_context, n_gen, n_samples): # Cut clips of n_context + n_gen frames: BAIR 2 + 14, KTH 10 + 10 # [arxiv:1812.01717 §4.4] clips = sample_clips(dataset, n_samples, n_context + n_gen) context, target = split(clips, n_context) # Generate the continuation from the context frames [arxiv:1812.01717 §4.3] generated = model.generate(context, n_gen) # Score the whole set at once, not frame by frame [arxiv:1812.01717 §2] # FVD details are on generative-model-evaluation return fvd(target, generated) ``` The second loop is the policy use. It follows the paper's description of joint denoising, not its code. Names are ours. ```pseudocode # Policy from a joint video-and-action denoiser [arxiv:2512.06963 §3.4] # Executes a prefix of each chunk, then re-plans [arxiv:2512.06963 §4.1] def act(observation, instruction, denoiser): # Encode the instruction and the current observation only [arxiv:2512.06963 §3.3] text = encode_text(instruction) first_latent = encode_video(observation) # Start future latents and a 6 x 7 action chunk from Gaussian noise [arxiv:2512.06963 §3.1] video_latent = randn(future_latent_shape) action_chunk = randn(6, 7) # Denoise both in one sampling loop, 50 steps in the main setting [arxiv:2512.06963 §4.1] for t in timesteps(50): video_latent, action_chunk = denoiser.step( text, first_latent, video_latent, action_chunk, t ) # Execute the first 3 actions, then query again [arxiv:2512.06963 §4.1] return action_chunk[:3] ``` The conditioned-diffusion variants each have their own sampling loop, with their own reference or condition handling. See their cards rather than reusing either loop above. ## Contradictions Between cards: FVD values are not comparable across the cards. Each card uses its own dataset and protocol. EvalCrafter's 0-100 aspect scores are a different measure again, and are not comparable with any FVD value [arxiv:2310.11440 §4.5]. MASS reports matched-Snake metrics that are not on any of the other cards' scales [arxiv:2608.06257 tab. 2]. The benchmark card uses BAIR, KTH and SCV [arxiv:1812.01717 §4.4]. StyleGAN-V uses four 256 x 256 datasets with its own FVD protocol [arxiv:2112.14683 app. C]. MagicAnimate uses TikTok and TED-talks [arxiv:2311.16498 §4]. Drive-WM uses nuScenes with the TATS FVD code [arxiv:2311.17918 app. D.1]. The baseline numbers in the Drive-WM table (DriveGAN, DriveDreamer) do not state where they came from [arxiv:2311.17918 tab. 1]. MM-Diffusion computes FVD at 64 x 64 with an I3D classifier trained on Kinetics-400 [arxiv:2212.09478 §4.3]. PYoCo follows the zero-shot protocol of Make-A-Video for its UCF-101 and MSR-VTT numbers [arxiv:2305.10474 §5.2]. The benchmark card and the planning, animation and orbit cards measure different things, so their numbers are not compared with each other. Within a card: each of the following is a claim in the text that the paper's own tables do not support, or a point where text and table disagree. - **StyleGAN-V.** The text says the method "significantly outperforms" the others [arxiv:2112.14683 §4.1]. Table 1 shows the lowest FVD16 on three of four datasets only [arxiv:2112.14683 tab. 1]. The abstract's average improvement is not shown in the compiled sections [arxiv:2112.14683 abstract]. The text says removing the generator gives poor short- and long-term quality [arxiv:2112.14683 §4.2], but that variant has the lowest FaceForensics FVD128 in the ablation table, 41.77 against 89.34 for the default [arxiv:2112.14683 tab. 2]. - **MagicAnimate.** The text says joint training "consistently increases" quality [arxiv:2311.16498 §4.2]. In the joint-training table, FVD is lowest when spatial joint training is off and temporal is on (156.32), and the full model is at 179.07 [arxiv:2311.16498 tab. 2c]. Joint training improves FID and FID-VID in that table. - **SV3D.** The text says SV3D-c and SV3D-p outperform SV3D-u on static orbits [arxiv:2403.12008 §3]. The GSO static table gives SV3D-c PSNR 20.56 against 21.14 for SV3D-u, and only SV3D-p beats SV3D-u on PSNR [arxiv:2403.12008 §3]. - **Drive-WM.** The text says the combined reward outperforms each sub-reward [arxiv:2311.17918 §5.4]. The object reward alone gives the same average L2 (0.80 m), and the combined reward is only better on collision, 0.26% against 0.27% [arxiv:2311.17918 tab. 4]. The text also calls the 12.99 FID result single-view, while the table marks it multi-view [arxiv:2311.17918 tab. 1]. - **VideoVLA.** The paper gives two sampling settings that do not match. Section 4.1 describes 50 DDIM denoising steps, while the appendix timing of about 1.1 s per chunk uses 10 steps [arxiv:2512.06963 §4.1] [arxiv:2512.06963 app. D]. The card's repository check finds a 50-step DPM-Solver-style sampler as the default, and no 10-step setting [arxiv:2512.06963 §4.1]. Treat the latency figure as the 10-step setting until the authors confirm otherwise. - **EvalCrafter.** The text calls Gen2 the best model overall [arxiv:2310.11440 §5.2], but the table has no overall column, and Show-1 leads text-video alignment [arxiv:2310.11440 tab. 2]. The text says the fitted score beats the plain average "consistently" [arxiv:2310.11440 app. B.2]. In the alignment block, the fitted score's Kendall value (22.5) is lower than the average's (22.7) [arxiv:2310.11440 tab. 3]. The "Avg." rows are also not means of the rows shown: the visual rows 42.1 and 53.6 average 47.9, not the 55.0 printed [arxiv:2310.11440 tab. 3]. The visual average is 55.0 in one table and 54.9 in another [arxiv:2310.11440 tab. 5]. The user study is described with five models, but the human-aligned results cover eight [arxiv:2310.11440 §4.5] [arxiv:2310.11440 tab. 2]. The Warping Error correlation is printed as +69.0, and the paper does not give a sign convention [arxiv:2310.11440 tab. 3]. - **MASS.** The abstract and introduction claim 1,024 entities over 10,000 recurrent ticks [arxiv:2608.06257 §1] [arxiv:2608.06257 §4.5]. The appendix's stress runs use 2, 4 and 8 snakes up to 4,096 ticks [arxiv:2608.06257 app. C.3], and the compiled appendix has no table for the 1,024-entity, 10,000-tick case. The MultiWorld baseline has 5.60B trainable parameters, while the MASS predictor has 5.66M, so the comparison is not parameter-matched [arxiv:2608.06257 app. B.1]. Zero cross-view disagreement is a consequence of the design, not a test of the renderer [arxiv:2608.06257 §4.5]. - **MM-Diffusion.** The text says complete sampling is better overall [arxiv:2212.09478 §4.4], but its Landscape FAD is worse than the DPM-Solver row, 10.72 against 9.39 [arxiv:2212.09478 tab. 1]. The abstract claims zero-shot conditional generation that no table measures [arxiv:2212.09478 §4.5] [arxiv:2212.09478 fig. 5]. The paper's equations 10 and 12 use different prefactors, and the shipped code uses a third form [arxiv:2212.09478 eq. 10] [arxiv:2212.09478 eq. 12] [code: mm_diffusion/multimodal_gaussian_diffusion.py#L817]. - **PYoCo.** The text gives about 14 times less training time, from 75 and 925 GPU-days [arxiv:2305.10474 §5.2]. Those two figures divide to about 12.3, and the text does not say which model size the 75 GPU-days refers to. The α grid in §5.3 lists 1 twice, and the optimizer and sampler differ between the large-scale and UCF-101 runs [arxiv:2305.10474 app. A.3]. ## Papers - [Towards Accurate Generative Models of Video: A New Metric & Challenges](/papers/1812.01717): introduces FVD and the SCV benchmark, and reports FVD for four baseline video models on BAIR, KTH and SCV. - [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](/papers/2112.14683): a StyleGAN2-based video GAN whose motion codes can be read at any frame time, with author-reported FVD16 on four 256 x 256 datasets. - [EvalCrafter: Benchmarking and Evaluating Large Video Generation Models](/papers/2310.11440): a 700-prompt text-to-video benchmark with 17 automatic metrics in four aspects, fitted to 1-5 user ratings, with author-reported results for eight models. - [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](/papers/2311.16498): animates a reference person along DensePose motion with temporal attention and a reference encoder, with author-reported TikTok and TED-talks results. - [Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving](/papers/2311.17918): a six-camera driving video diffusion model conditioned on layouts and ego actions, used to score planned trajectories open loop on nuScenes. - [SV3D](/papers/2403.12008), arXiv:2403.12008: fine-tunes Stable Video Diffusion for camera-controlled orbits from one image, and adds a 3D pipeline with author-reported GSO results. - [VideoVLA: Video Generators Can Be Generalizable Robot Manipulators](/papers/2512.06963): adapts a pretrained video diffusion transformer into a policy that jointly denoises future video and a 6-step action chunk, with author-reported SIMPLER and real-robot results. - [MASS: Multiplayer World Models with Authoritative Shared State](/papers/2608.06257): a multiplayer world model with one typed shared state and a per-camera rendering engine, with author-reported matched-Snake results against video baselines and no official code found. - [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](/papers/2212.09478): two coupled diffusion U-Nets that generate video and audio together, with author-reported FVD on Landscape and AIST++ and an official MIT-licensed repository. - [Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models](/papers/2305.10474): finetunes a text-to-image model into a video model with correlated frame noise, with author-reported UCF-101 and MSR-VTT results and no official code found.