Video generation

Overview

A video generator produces frames that continue from a short context, such as two context frames on BAIR or ten on KTH 1812.01717 §4.4. A generated video has no single correct answer, so the score has to judge a set of videos rather than each frame against a reference 1812.01717 §1 1812.01717 §2. The first card on this page introduces that set-level score (FVD) and the benchmark setups it is applied to 1812.01717 §4.4. Its four model families are baselines from earlier work, not a new generator 1812.01717 §4.3. Card: Towards Accurate Generative Models of Video: A New Metric & Challenges.

The other five cards use video generators for four different jobs. A GAN can sample a whole clip from motion codes that are read at any frame time 2112.14683 §3.1 (StyleGAN-V). A diffusion model can be steered by an input signal: a reference person and a pose sequence 2311.16498 §3.3 (MagicAnimate), a 3D layout and ego actions for six driving cameras 2311.17918 §3.3 (Drive-WM), or a camera orbit around one object 2403.12008 §3 (SV3D). A fourth card uses the generator as the backbone of a robot policy 2512.06963 §3.4 (VideoVLA).

Two more cards widen the scope. EvalCrafter is a benchmark for text-to-video models that scores prompt match, motion and temporal consistency alongside visual quality, and does not rely on a set-level distribution score alone 2310.11440 §1 (EvalCrafter). MASS is a multiplayer world model: it keeps one typed state of the shared world and renders each camera from that state, rather than keeping a separate video memory per player view 2608.06257 §1 (MASS).

Two more cards cover joint audio and video, and the finetuning of an image model. MM-Diffusion denoises a video and its soundtrack in one process, with two coupled networks that attend to each other through short windows 2212.09478 §3.2 2212.09478 §3.3 (MM-Diffusion). PYoCo's main change is the starting noise rather than the network: the frames of a clip get correlated noise, so that finetuning an image model on video does not discard the frame-to-frame correlation it learned 2305.10474 §4 (PYoCo).

So the concept covers eight uses: a video model as the thing being measured by FVD, a text-to-video model as the thing being scored by a multi-metric suite, a GAN that generates clips, a conditioned diffusion model, a video model as the backbone of a policy, a rendered world model with shared state, a joint audio and video generator, and an image model adapted to video through its noise prior. Check which one you need before you pick a setup or a variant.

Variants

Benchmark subject: context-conditioned video models

The first card compares four baseline models on the same benchmarks. It gives no architecture details for them, and none are linked to an arXiv id on the card 1812.01717 tab. 2 1812.01717 app. C. All four were trained with Tensor2Tensor 1812.01717 app. C.

Lower FVD is better. Each SCV scenario has a 64 x 64 and a 128 x 128 column 1812.01717 tab. 2. The card's human study uses BAIR models from CDNA, SV2P, SVP-FP and SAVP 1812.01717 §4.3.

Evaluation suite: 17 automatic metrics fitted to human ratings

EvalCrafter is a text-to-video benchmark with 700 prompts in four meta-classes: human, animal, object and landscape 2310.11440 §3. Some prompts come from real users of two text-to-video services, and others are generated with GPT-4 and then checked by hand 2310.11440 §3. Its 17 automatic metrics fall into four aspects: visual quality, text-video alignment, motion quality and temporal consistency 2310.11440 §4. For each aspect a linear model is fitted to 1-5 user ratings, and the four aspect scores are summed into one total on a 0-100 scale 2310.11440 §4.5. The paper prints no regression coefficients. The released repository has no LICENSE file, so its code can be read but not copied; see the card for the code's details.

Author-reported human-aligned scores for eight models, with the rank in brackets, show that no model leads every aspect 2310.11440 tab. 2:

The table has no overall column. The paper calls Gen2 the best model overall 2310.11440 §5.2. The four rows above are the leaders on at least one aspect, not a full ranking; see 2310.11440 tab. 2 for all eight.

The fitted aspect scores correlate better with human ratings than the single metrics do. Spearman's rho on held-out samples is 55.4 for visual quality, 45.0 for motion amplitude, 56.7 for temporal consistency and 32.3 for text-video alignment 2310.11440 tab. 3. For comparison, CLIP-Score alone reaches 6.3 on text-video alignment, and BLIP-BLEU reaches 26.7 2310.11440 tab. 3. These are author-reported. They rest on the 1,024 ratings kept from 8,647 collected, and the filter used is not stated 2310.11440 §4.5.

The authors' full run takes about two hours and at least 16 GB on an A100, unoptimised, and they describe it as an offline benchmark, similar in role to FVD 2310.11440 app. D.6.

Unconditional GAN generator: continuous-time motion codes

StyleGAN-V keeps the StyleGAN2 image generator almost unchanged and adds a motion signal for each clip. A content code, shared by all frames, comes from the StyleGAN2 mapping network. Motion comes from a sparse grid of noise that two padding-free conv1d layers turn into codes, which can be read at real-valued times, so the frames of a clip are generated independently rather than in sequence 2112.14683 §3.1 2112.14683 app. B. The discriminator scores k frames of a clip together, with k = 3 as the default 2112.14683 §3.2 2112.14683 tab. 3.

The paper's FVD16 and FVD128 numbers use the authors' own protocol, and the baselines were re-trained by the authors 2112.14683 §4 2112.14683 app. C. FVD16 results, with the GPU-day cost column from the paper:

All values are FVD16, lower is better 2112.14683 tab. 1. Two-stage methods are written "X + Y". The paper's text says the method "significantly outperforms" the others on almost all benchmarks 2112.14683 §4.1. The table supports the lowest FVD16 on FaceForensics, SkyTimelapse and UCF101, and VideoGPT is lower on RainbowJelly 2112.14683 tab. 1.

Image quality is close to the image model it extends. On FaceForensics at 256 x 256 the authors report FID 9.44 for StyleGAN-V and 8.42 for StyleGAN2 2112.14683 §4.3. Frames per clip matter: k = 2 gives a lower SkyTimelapse FVD16 (50.5 vs 79.52) but a higher FaceForensics FVD16 (60.41 vs 47.41) 2112.14683 tab. 3.

Conditioned video diffusion: a reference, a layout or a camera steers the output

The three cards below all start from a pretrained image or video diffusion model and add a conditioning input. They differ in what the input is and what the output is for.

Reference-driven animation (MagicAnimate). The model inflates a 2D image UNet with temporal attention, so each spatial position attends across the frames of a clip 2311.16498 §3.1. A second trainable copy of the UNet reads the reference image and passes its features into the self-attention of the generator 2311.16498 §3.2. Pose enters as DensePose maps, which the paper prefers to sparse keypoints 2311.16498 §3.3. Long videos are split into overlapping segments whose noise predictions are averaged at each step 2311.16498 §3.3. On TikTok the temporal attention removes about 68 FVD points from the ablation, 247.30 to 179.07 2311.16498 tab. 2a. Among the reference encoders tested, the paper's own gives the lowest FVD (179.07), ahead of IP-Adapter (590.99) and CLIP (724.96) 2311.16498 tab. 2b. The training details (learning rate, batch size, steps, sampling of frames) are not stated in the source the card used, and the released material is inference only 2311.16498 §4.

Layout and action conditioning (Drive-WM). Drive-WM lifts a single-view image diffusion model to multiview video: it adds temporal layers, then multiview layers that attend across the six cameras, and freezes the spatial weights 2311.17918 §3.1. Its condition is a set of tokens, one for each kind of input, concatenated together 2311.17918 §3.3 2311.17918 eq. 5. The generator can build the six views in two groups, reference views first and stitched views second 2311.17918 §3.2. In the authors' table the factorized version has FVD 116.6 and KPM 94.4%, against 122.7 and 45.8% for the joint model, at a FID cost of 16.4 against 15.8 2311.17918 tab. 2. For planning, the model renders the future for each candidate trajectory from VAD, and an image-based reward picks one 2311.17918 §4.1 2311.17918 §4.2.

Camera-controlled orbit (SV3D). SV3D fine-tunes Stable Video Diffusion to produce a 21-frame orbit at 576 x 576 from one image 2403.12008 §3. Three models are trained: SV3D-u with no camera input, SV3D-c with a camera trajectory, and SV3D-p, which is first trained without camera input and then with it 2403.12008 §3. The authors use a triangle-wave classifier-free guidance weight, from 1.0 at the front view to 2.5 at the back view, which avoids over-sharpening the last frames 2403.12008 §3. A two-stage 3D pipeline fits a NeRF and then a mesh, and adds a masked score-distillation loss for parts that no generated view shows 2403.12008 §4. The mesh Chamfer distance is 0.024 for SV3D-p with masked SDS, against 0.039 for Stable Zero123 2403.12008 §4.

Policy backbone: video diffusion as a visuomotor model

VideoVLA is a vision-language-action policy built on CogVideoX-5B, a video diffusion transformer 2512.06963 §4.1. Its design, in short:

• One joint denoiser. The token sequence holds the text tokens, the current observation latent, the future video latents and the action tokens. Video and actions get Gaussian noise under the DDPM schedule and are denoised with the DDPM loss. The timestep enters through adaptive LayerNorm 2512.06963 §3.4.
• Action chunk. The model outputs a chunk of 6 seven-dimensional actions. The first 3 are executed, then the model is queried again on the new observation 2512.06963 §3.1 2512.06963 §4.1.
• Pretrained video prior. The backbone is initialised from pretrained CogVideoX 2512.06963 §3.4. Training from scratch on the same data drops the SIMPLER three-task Visual Matching average from 80.4 to 12.6. An OpenSora-1.1 backbone scores 50.2 2512.06963 tab. 7.
• Video loss matters. Removing the video loss drops the same average to 27.0, and an action-only model reaches 25.5 2512.06963 tab. 9.

Reported results, all author-reported and with small trial counts per task 2512.06963 app. A:

The paper's overall SIMPLER gap over CogACT is 0.4 points. At 24 trials per task one success is 4.2 points, so that gap is within noise 2512.06963 app. A. The larger gaps are on novel objects and new skills. Human-judged imagined videos score higher than the executed success rates: 84.0 vs 65.2 for novel objects and 63.4 vs 48.6 for new skills 2512.06963 tab. 10.

Multiplayer world model: one typed state, many rendered views

MASS keeps one authoritative state of a multiplayer world as a list of typed records: one per entity, plus a tick counter 2608.06257 §3.1. A Logic Engine, a decoder-only Transformer, predicts the next state from the current state and all players' actions 2608.06257 §3.2. A Rendering Engine, a residual U-Net, draws each requested camera from that same state 2608.06257 §3.3. Every view therefore reads one state, so views cannot disagree about what exists 2608.06257 §4.5. The schema, tokenizer and weights are per game, while the model design is shared 2608.06257 §3.1.

On a matched two-player, two-view Snake benchmark, the authors report parser state recovery of 0.764 for MASS, 0.128 for B-PV (independent per-view video predictors) and 0.067 for MultiWorld 2608.06257 tab. 2. The matched test set has 8 held-out episodes 2608.06257 app. B.1. Rendering on held-out reference states at 256 x 256 gives PSNR 38.82 dB for Snake and 23.72 dB for Tron, the lowest of eight games 2608.06257 tab. 3. Rendering cost grows with the number of views: the authors measure a mean of 189.6 ms for one view and 842.4 ms for 1,024 views, on one H100 2608.06257 app. C.4.

Joint audio and video: two coupled denoisers

MM-Diffusion denoises a short video and its audio together, under one joint training loss 2212.09478 §3.2. The video branch uses 2D spatial and 1D temporal layers, and the audio branch uses dilated 1D convolutions 2212.09478 §3.3 2212.09478 app. A. The two branches are linked by random-shift cross-modal attention at three of the four U-Net scales. The audio is cut into one segment per video frame, and each segment attends to a short window of frames whose start is shifted at random 2212.09478 §3.3 2212.09478 tab. A.1. The authors avoid full cross-attention, whose cost grows with the product of the video positions and the audio samples 2212.09478 §3.3. Clips are 16 frames at 64 x 64, paired with 25,600 audio samples at 16 kHz 2212.09478 tab. A.1.

Results are author-reported. On Landscape, complete 1,000-step DDPM sampling gives FVD 117.20, against 305.36 for DIGAN and 600.30 for TATS-base; the default DPM-Solver row gives 229.08 2212.09478 tab. 1. On AIST++ the same sampler gives 75.71, against 119.47 and 267.24 2212.09478 tab. 2. All metrics are computed at 64 x 64, before a separate super-resolution model produces 256 x 256 output 2212.09478 §4.1. In the authors' Turing test, 84.9% of Landscape clips were judged real, against 92.5% for ground truth; on AIST++ the figures are 49.6% and 84.7% 2212.09478 tab. 6. On a mean opinion score, the two-stage baseline (audio first, then video) gets 1.61 for video quality on Landscape, against 3.75 for the joint model 2212.09478 tab. 5.

The same trained model can generate audio from video, or video from audio, without extra training. The paper uses two methods: replacing the conditioning signal with a noised copy of the real one, and adding a gradient term that pulls the target toward the condition 2212.09478 §3.4. No table measures these conditional results. The paper shows them in figures and gives no value for the gradient weight 2212.09478 §4.5 2212.09478 fig. 5.

Finetuning an image model: correlated noise prior

PYoCo starts from eDiff-I, a text-to-image diffusion model, and finetunes it on images and videos together 2305.10474 §4. The authors argue that an image model learns frame-to-frame correlation from real video, and that independent noise per frame makes finetuning unlearn it. They measure an average cosine similarity of reversed noise of 0.206 ± 0.156 within a video, against 0.001 ± 0.009 across videos 2305.10474 tab. 1. Two constructions keep frames correlated, both controlled by one number α, where α = 0 gives independent noise. Mixed noise is a shared part plus a per-frame part 2305.10474 eq. 3. Progressive noise makes each frame perturb the previous one 2305.10474 eq. 4. The large-scale model uses progressive noise with α = 2 2305.10474 §5.3.

Results are author-reported. On UCF-101, finetuning from the image model with i.i.d. noise gives FVD 566.67, against 903.37 when trained from scratch. Adding mixed noise gives 337.40 and progressive noise 339.67 2305.10474 tab. 5. That table does not state the model size. At large scale the authors report a zero-shot UCF-101 IS of 47.76 and FVD of 355.19 2305.10474 tab. 2, a zero-shot MSR-VTT CLIP-FID of 9.73 for the best ensemble configuration 2305.10474 tab. 3, and an unconditional UCF-101 IS of 60.01 and FVD of 310 for the 253M model 2305.10474 tab. 4. Against VDM, they report about 10 times fewer parameters and about 14 times less training time for their unconditional model 2305.10474 §5.2.

The α sweep was run only on the 69M UCF-101 model 2305.10474 §5.3. The large-scale training set is about 1.2 billion image-caption pairs and 22.5 million video-caption pairs, from public and proprietary sources, and its filter thresholds are not stated 2305.10474 §5.1. The card's search found no official code.

Benchmark setups

Each dataset fixes the number of context frames, the number of generated frames and the sample count used for FVD:

• BAIR: 2 context frames and 14 generated frames, FVD over 256 samples 1812.01717 §4.3 1812.01717 §4.4.
• KTH: 10 context frames and 10 generated frames, FVD over 1024 samples 1812.01717 §4.4.
• SCV (StarCraft 2 Videos): four game scenarios, MUtB (one unit walks to the map edge), CMS (units gather shards), Brawl (two armies fight) and RTwM (a transport unit picks up and drops off units). Each is rendered at 64 x 64 and 128 x 128 1812.01717 §3 1812.01717 app. B.1. The split is 10,000 train, 2,000 validation and 2,000 test clips per scenario. FVD uses 1024 samples 1812.01717 §3 1812.01717 app. B 1812.01717 §4.4.
• RTwM: 2 context frames and up to 32 further frames, the longest setting in the card 1812.01717 §4.4.

When to use which

Licences differ between code and weights across these cards. The cards record the code licence of each repository, and two of them show the gap: SV3D's code is MIT while its weights are non-commercial, and the StyleGAN-V and EvalCrafter repositories have no LICENSE file at all. Check the card before you copy anything from a repository or depend on a checkpoint.

Implementation sketch

The first loop is our own summary of the benchmark protocol. How a model rolls out from its context is model-specific and not stated on the card.

# Benchmark loop for a context-conditioned video model  [arxiv:1812.01717 §4.4]
# Keep n_samples fixed across every model you compare  [arxiv:1812.01717 §4.2]

def evaluate(model, dataset, n_context, n_gen, n_samples):
    # Cut clips of n_context + n_gen frames: BAIR 2 + 14, KTH 10 + 10
    # [arxiv:1812.01717 §4.4]
    clips = sample_clips(dataset, n_samples, n_context + n_gen)
    context, target = split(clips, n_context)

    # Generate the continuation from the context frames  [arxiv:1812.01717 §4.3]
    generated = model.generate(context, n_gen)

    # Score the whole set at once, not frame by frame  [arxiv:1812.01717 §2]
    # FVD details are on generative-model-evaluation
    return fvd(target, generated)

The second loop is the policy use. It follows the paper's description of joint denoising, not its code. Names are ours.

# Policy from a joint video-and-action denoiser  [arxiv:2512.06963 §3.4]
# Executes a prefix of each chunk, then re-plans  [arxiv:2512.06963 §4.1]

def act(observation, instruction, denoiser):
    # Encode the instruction and the current observation only  [arxiv:2512.06963 §3.3]
    text = encode_text(instruction)
    first_latent = encode_video(observation)

    # Start future latents and a 6 x 7 action chunk from Gaussian noise  [arxiv:2512.06963 §3.1]
    video_latent = randn(future_latent_shape)
    action_chunk = randn(6, 7)

    # Denoise both in one sampling loop, 50 steps in the main setting  [arxiv:2512.06963 §4.1]
    for t in timesteps(50):
        video_latent, action_chunk = denoiser.step(
            text, first_latent, video_latent, action_chunk, t
        )

    # Execute the first 3 actions, then query again  [arxiv:2512.06963 §4.1]
    return action_chunk[:3]

The conditioned-diffusion variants each have their own sampling loop, with their own reference or condition handling. See their cards rather than reusing either loop above.

Contradictions

Between cards: FVD values are not comparable across the cards. Each card uses its own dataset and protocol. EvalCrafter's 0-100 aspect scores are a different measure again, and are not comparable with any FVD value 2310.11440 §4.5. MASS reports matched-Snake metrics that are not on any of the other cards' scales 2608.06257 tab. 2. The benchmark card uses BAIR, KTH and SCV 1812.01717 §4.4. StyleGAN-V uses four 256 x 256 datasets with its own FVD protocol 2112.14683 app. C. MagicAnimate uses TikTok and TED-talks 2311.16498 §4. Drive-WM uses nuScenes with the TATS FVD code 2311.17918 app. D.1. The baseline numbers in the Drive-WM table (DriveGAN, DriveDreamer) do not state where they came from 2311.17918 tab. 1. MM-Diffusion computes FVD at 64 x 64 with an I3D classifier trained on Kinetics-400 2212.09478 §4.3. PYoCo follows the zero-shot protocol of Make-A-Video for its UCF-101 and MSR-VTT numbers 2305.10474 §5.2. The benchmark card and the planning, animation and orbit cards measure different things, so their numbers are not compared with each other.

Within a card: each of the following is a claim in the text that the paper's own tables do not support, or a point where text and table disagree.

• StyleGAN-V. The text says the method "significantly outperforms" the others 2112.14683 §4.1. Table 1 shows the lowest FVD16 on three of four datasets only 2112.14683 tab. 1. The abstract's average improvement is not shown in the compiled sections 2112.14683 abstract. The text says removing the generator gives poor short- and long-term quality 2112.14683 §4.2, but that variant has the lowest FaceForensics FVD128 in the ablation table, 41.77 against 89.34 for the default 2112.14683 tab. 2.
• MagicAnimate. The text says joint training "consistently increases" quality 2311.16498 §4.2. In the joint-training table, FVD is lowest when spatial joint training is off and temporal is on (156.32), and the full model is at 179.07 2311.16498 tab. 2c. Joint training improves FID and FID-VID in that table.
• SV3D. The text says SV3D-c and SV3D-p outperform SV3D-u on static orbits 2403.12008 §3. The GSO static table gives SV3D-c PSNR 20.56 against 21.14 for SV3D-u, and only SV3D-p beats SV3D-u on PSNR 2403.12008 §3.
• Drive-WM. The text says the combined reward outperforms each sub-reward 2311.17918 §5.4. The object reward alone gives the same average L2 (0.80 m), and the combined reward is only better on collision, 0.26% against 0.27% 2311.17918 tab. 4. The text also calls the 12.99 FID result single-view, while the table marks it multi-view 2311.17918 tab. 1.
• VideoVLA. The paper gives two sampling settings that do not match. Section 4.1 describes 50 DDIM denoising steps, while the appendix timing of about 1.1 s per chunk uses 10 steps 2512.06963 §4.1 2512.06963 app. D. The card's repository check finds a 50-step DPM-Solver-style sampler as the default, and no 10-step setting 2512.06963 §4.1. Treat the latency figure as the 10-step setting until the authors confirm otherwise.
• EvalCrafter. The text calls Gen2 the best model overall 2310.11440 §5.2, but the table has no overall column, and Show-1 leads text-video alignment 2310.11440 tab. 2. The text says the fitted score beats the plain average "consistently" 2310.11440 app. B.2. In the alignment block, the fitted score's Kendall value (22.5) is lower than the average's (22.7) 2310.11440 tab. 3. The "Avg." rows are also not means of the rows shown: the visual rows 42.1 and 53.6 average 47.9, not the 55.0 printed 2310.11440 tab. 3. The visual average is 55.0 in one table and 54.9 in another 2310.11440 tab. 5. The user study is described with five models, but the human-aligned results cover eight 2310.11440 §4.5 2310.11440 tab. 2. The Warping Error correlation is printed as +69.0, and the paper does not give a sign convention 2310.11440 tab. 3.
• MASS. The abstract and introduction claim 1,024 entities over 10,000 recurrent ticks 2608.06257 §1 2608.06257 §4.5. The appendix's stress runs use 2, 4 and 8 snakes up to 4,096 ticks 2608.06257 app. C.3, and the compiled appendix has no table for the 1,024-entity, 10,000-tick case. The MultiWorld baseline has 5.60B trainable parameters, while the MASS predictor has 5.66M, so the comparison is not parameter-matched 2608.06257 app. B.1. Zero cross-view disagreement is a consequence of the design, not a test of the renderer 2608.06257 §4.5.
• MM-Diffusion. The text says complete sampling is better overall 2212.09478 §4.4, but its Landscape FAD is worse than the DPM-Solver row, 10.72 against 9.39 2212.09478 tab. 1. The abstract claims zero-shot conditional generation that no table measures 2212.09478 §4.5 2212.09478 fig. 5. The paper's equations 10 and 12 use different prefactors, and the shipped code uses a third form 2212.09478 eq. 10 2212.09478 eq. 12 multimodal_gaussian_diffusion.py.
• PYoCo. The text gives about 14 times less training time, from 75 and 925 GPU-days 2305.10474 §5.2. Those two figures divide to about 12.3, and the text does not say which model size the 75 GPU-days refers to. The α grid in §5.3 lists 1 twice, and the optimizer and sampler differ between the large-scale and UCF-101 runs 2305.10474 app. A.3.

Papers

• Towards Accurate Generative Models of Video: A New Metric & Challenges: introduces FVD and the SCV benchmark, and reports FVD for four baseline video models on BAIR, KTH and SCV.
• StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2: a StyleGAN2-based video GAN whose motion codes can be read at any frame time, with author-reported FVD16 on four 256 x 256 datasets.
• EvalCrafter: Benchmarking and Evaluating Large Video Generation Models: a 700-prompt text-to-video benchmark with 17 automatic metrics in four aspects, fitted to 1-5 user ratings, with author-reported results for eight models.
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model: animates a reference person along DensePose motion with temporal attention and a reference encoder, with author-reported TikTok and TED-talks results.
• Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving: a six-camera driving video diffusion model conditioned on layouts and ego actions, used to score planned trajectories open loop on nuScenes.
• SV3D, arXiv:2403.12008: fine-tunes Stable Video Diffusion for camera-controlled orbits from one image, and adds a 3D pipeline with author-reported GSO results.
• VideoVLA: Video Generators Can Be Generalizable Robot Manipulators: adapts a pretrained video diffusion transformer into a policy that jointly denoises future video and a 6-step action chunk, with author-reported SIMPLER and real-robot results.
• MASS: Multiplayer World Models with Authoritative Shared State: a multiplayer world model with one typed shared state and a per-camera rendering engine, with author-reported matched-Snake results against video baselines and no official code found.
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation: two coupled diffusion U-Nets that generate video and audio together, with author-reported FVD on Landscape and AIST++ and an official MIT-licensed repository.
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models: finetunes a text-to-image model into a video model with correlated frame noise, with author-reported UCF-101 and MSR-VTT results and no official code found.