Photorealistic Video Generation with Diffusion Models

In plain words

Most video generators either work one frame at a time or rely on a heavy convolutional network. This paper builds a video generator from a transformer that works on a compressed version of the pixels, called a latent. The same model also handles still images, which count as one-frame videos, so it can learn from image collections as well as from video. To keep the work manageable, each token attends only to nearby tokens: within one frame, or across a short stretch of frames. Longer videos are made by feeding earlier frames back in as conditioning. For text-to-video, three models are chained: a base generator and two upscalers that raise the resolution. The authors report strong scores on standard benchmarks without a common trick called classifier-free guidance.

TL;DR

W.A.L.T uses one transformer diffusion model for images and videos. A causal autoencoder compresses both into a shared, continuous latent space, and the transformer attends within windows, per frame and across frames, to save compute. The authors report state-of-the-art FVD on UCF-101 and Kinetics-600 and an ImageNet 256 FID of 2.56 without classifier-free guidance, which VDM++ beats at 2B parameters (2.40). A three-model cascade produces text-to-video at 512×896 and 8 frames per second. All numbers are author-reported.

Problem

• Full self-attention cost grows quadratically with sequence length, so video diffusion has mostly used U-Net backbones §1.
• Per-frame latents allow reuse of pretrained image autoencoders and image-text data, but they limit transformer backbones at high resolution. Spatiotemporal latents compress better but cannot use paired image-text data §1.
• Joint image and video training helps because labelled video data is scarce §4.1. This requires one latent space for both §4.1.

Method

Overview: an autoencoder maps images and videos to one latent space; a transformer denoiser, trained with diffusion, operates on latent patches using alternating window attention and conditions on text, timestep and, for frame prediction, past frames §1 fig. 2.

Background. Forward noising is x_t = sqrt(γ(t)) x_0 + sqrt(1 − γ(t)) ε, with γ decreasing from 1 to 0 §3 eq. 1. The denoiser is trained with a squared-error loss against a target y §3 eq. 2. The paper uses v-prediction, v = sqrt(1 − γ) ε − sqrt(γ) x_0 §3.

Step 1: Learning visual tokens §4.1.

1. A causal 3D CNN encoder-decoder, following the MAGVIT-v2 tokenizer, maps a video of 1 + T frames to a latent of 1 + t frames; spatial factor f_s and temporal factor f_t §4.1. The architecture and hyperparameters follow MAGVIT-v2 app. A.
2. The first frame is always encoded independently, so an image is a one-frame video §4.1.
3. The latent is real-valued with no codebook, unlike the original VQ-VAE §3.
4. The autoencoder is trained with reconstruction plus adversarial and perceptual losses §3. Loss weights are not stated.
5. The latent channel count c is set to 8 by default §5.2 tab. 3.
6. Latents are normalised before the transformer §5.2. The normaliser is not described.

Step 2: Patchify and window attention §4.2.

1. Each latent frame is cut into non-overlapping p × p patches, giving h_p = h/p and w_p = w/p tokens per side §4.2.
2. Each token gets a learned embedding, the sum of a space and a time embedding, added to the linear patch projection. An image gets only the time embedding of the first latent frame §4.2.
3. Blocks alternate two window types §1 fig. 2:
   • Spatial window (SW): all tokens of one latent frame, 1 × h_p × w_p §4.2.
   • Spatiotemporal window (STW): a 3D window spanning the 1 + t latent frames §4.2.
4. In STW layers an image's value embeddings pass through unchanged, through an identity attention mask §4.2.
5. Relative position embeddings are used in addition to absolute ones §4.2.

Step 3: Conditioning §4.3.

1. Text cross-attention: the input tokens (query) are concatenated with the text tokens (key, value) §4.3. Joint image-video training uses SW cross-attention only; video-only training uses the same window as the self-attention §4.3.
2. AdaLN-LoRA. A shared MLP produces six modulation vectors (γ1, γ2, β1, β2, α1, α2) of width d for the first block, from the condition plus timestep embedding c + t §4.3. Each later block i adds a rank-r correction eq. 3 eq. 4. The default is r = 2 §5.2. For a ViT-g with 1B parameters, the per-layer MLPs contribute 475M parameters; with r = 2 they contribute 12M §4.3.
3. Self-conditioning. With probability p_sc (default 0.9 §5.2), the model first predicts z̃_0 from z_t with an empty self-condition. It then runs again with that prediction, detached from gradients, concatenated to the input along channels. Otherwise it runs once §4.3.
4. The denoiser uses query-key normalisation §5.2.

Step 4: Autoregressive frame prediction §4.4.

1. During training, the model is conditioned on past frames with probability p_fp. The value of p_fp is not stated.
2. A binary mask m_fp marks the conditioning latent frames: one frame for image-to-video, two for video prediction. The input becomes concat(m_fp ∘ z_t, m_fp) along channels §4.4.
3. Inference uses classifier-free guidance with this conditioning §4.4.

Step 5: Cascade and super-resolution §4.5 §5.3 app. A.

1. The base model generates 128 × 128 video. Two super-resolution stages each upscale 2× with a depth-to-space convolution on the low-resolution latent §4.5.
2. Noise conditioning augmentation: a noise level t_sr is drawn uniformly up to a maximum, noise is added according to γ(t), and t_sr is fed to the AdaLN-LoRA layers §4.5. The maximum is not stated.
3. Aspect-ratio fine-tuning: the base is trained square and fine-tuned on a subset to 9:16 by interpolating position embeddings and scaling windows §4.5 app. A.
4. Long videos: the base produces the first 17 frames, the last 5 pixel frames are encoded into 2 latent frames, and these condition the next generation app. A.

# Joint step on a mixed batch of images and videos  [§4.1] [§4.2]
# An image is a one-frame video, so one model sees both kinds of batch.
def training_step(batch, text_tokens):
    # Encode into the shared latent space; first frame encoded alone  [§4.1]
    latents = causal_encoder(batch)              # (B, 1 + t, h, w, c)
    latents = normalise_latents(latents)         # method not stated  [§5.2]

    # Sample a timestep and noise, then noise the latents  [§3] [eq. 1]
    timestep = sample_uniform(0, 1)
    noise = sample_gaussian(latents.shape)
    gamma_t = noise_schedule(timestep)
    noisy = sqrt(gamma_t) * latents + sqrt(1 - gamma_t) * noise

    # Predict with self-conditioning  [§4.3]
    prediction = self_conditioned_denoise(noisy, text_tokens, timestep)

    # Compare with the v-prediction target  [§3] [eq. 2]
    target = sqrt(1 - gamma_t) * noise - sqrt(gamma_t) * latents
    return mean((target - prediction) ** 2)

# Self-conditioning  [§4.3]
# The first pass sees an all-zero self-condition, with probability p_sc.
def self_conditioned_denoise(noisy, text_tokens, timestep):
    self_condition = zeros_like(noisy)
    if random() < p_sc:
        # First pass: estimate z~0, detached from gradients  [§4.3]
        estimate = model(noisy, zeros_like(noisy), text_tokens, timestep)
        self_condition = stop_gradient(estimate)
    # Final pass joins the estimate to the input along channels  [§4.3]
    return model(noisy, self_condition, text_tokens, timestep)

# AdaLN-LoRA modulation  [§4.3] [eq. 3] [eq. 4]
# The first block uses the MLP; each later block adds a rank-r correction.
def modulation_params(cond_plus_time, mlp, lora_pairs):
    shared = mlp(cond_plus_time)                 # (B, 6 * d)
    params = [shared]
    for w_b, w_a in lora_pairs:
        # Row-vector form; the paper writes W_b W_a  [eq. 4]
        correction = (cond_plus_time @ w_b) @ w_a    # w_b: (d, r), w_a: (r, 6d)
        params.append(shared + correction)
    return params                                # gamma, beta, alpha per block

# Frame-prediction mask  [§4.4]
# Mask marks the conditioning latent frames: 1 for images, 2 for video.
def frame_condition(noisy, frame_mask):
    # Masked latent and mask are stacked along channels  [§4.4]
    return concat_channels(frame_mask * noisy, frame_mask)

# Noise conditioning augmentation for a super-resolution stage  [§4.5]
# The maximum noise level is not stated in the paper.
def augment_low_res(low_res, max_noise):
    # Upscale 2x by depth-to-space first  [§4.5]
    upscaled = depth_to_space(low_res, factor=2)
    level = sample_uniform(0, max_noise)         # t_sr  [§4.5]
    noise = sample_gaussian(upscaled.shape)
    gamma_t = noise_schedule(level)
    return sqrt(gamma_t) * upscaled + sqrt(1 - gamma_t) * noise, level

Hyperparameters

Table 8 tab. 8 lists the stage-wise values. Diffusion settings are shared. Values come from the table unless marked.

Results

Author-reported. Each row names its setup. Baselines are taken from the paper's tables.

The text reports a 44% FVD reduction from self-conditioning §5.2. The text says W.A.L.T-L outperforms prior work on ImageNet without CFG, except that VDM++ has a slightly better FID at 2B parameters §5.1. For text-to-video, the paper says it surpasses prior work on FVD and is competitive on IS, behind PYoCo, which uses CLIP and T5-XXL text encoders §5.3.

Paper vs code

No official code.

What was searched:

1. Paper links: the LaTeX source contains no code or repository link; the only link is the project page §5 (https://walt-video-diffusion.github.io/).
2. Project page: fetched; it links only the arXiv record, the PDF and a samples page, with no code release.
3. GitHub search: walt video diffusion returned one unrelated repository by a third party; walt org:google-research returned zero results; window attention latent transformer video returned zero results.

No discrepancies can be checked against code.

Pitfalls

• Patch size conflict. The text says both super-resolution stages use p = 2 §5.3. Table 8 gives patch size 4 for the 2×2× stage tab. 8.
• Base model size does not match its table row. The text gives 3B parameters for the base §5.3. Table 8 lists 52 layers at width 9216 tab. 8. A standard transformer block costs roughly 12 · d² parameters, which would put this row far above 3B. The table value is probably a typo or a different quantity; the card cannot resolve it.
• Default ablation numbers disagree. The default configuration scores 60.7 in the patch-size, AdaLN-LoRA r = 2 and "full method" rows tab. 3, but the self-conditioning row at p_sc = 0.9 shows 61.4 tab. 3. The 5 × 8 × 8 window row, which the text calls the default window, shows 59.6 tab. 3. Which run is the baseline is not stated.
• Latent-size claim vs table. The text calls c = 8 a sweet spot §5.2. In the same table, c = 16 has lower FVD (67.0 vs 75.4) at slightly lower IS tab. 3. The table caption says higher c is better, but c = 32 is worse on FVD tab. 3.
• QK-norm. Removing query-key normalisation gives FVD 59.0, better than the default 60.7 tab. 3, yet the paper keeps it for training stability §5.2.
• Bolding in Table 2. Table 2 bolds 2.56 for W.A.L.T, though the grey VDM++ row shows 2.40 tab. 2. The text acknowledges VDM++ has better FID §5.1.
• Self-conditioning target. The first pass is written as z̃_0, which reads like a clean-latent estimate, but the model predicts v §3 §4.3. The paper does not say which quantity is fed back, which input is used on the single-pass branch, or which pass carries the loss. The pseudocode above assumes the loss is on the final pass; this is not stated.
• AdaLN-LoRA shapes. As written, W_b ∈ R^{d×r} and W_a ∈ R^{r×6d} multiply to d × 6d, so the input must be applied as a row vector (see the AdaLN-LoRA pseudocode in Method). The paper does not say so. Also, the paper does not say whether the modulation uses γ or 1 + γ.
• Cross-attention concatenation. "Concatenate the input signal (query) with the conditioning signal (key, value)" §4.3 does not say whether tokens are joined along the sequence or elsewhere.
• Window formula typo. The spatiotemporal window is written as (1 + t) × h_p' × h_w' §4.2. The second term is presumably w_p'. Not confirmed.
• Hidden inputs. p_fp, the augmentation noise maximum, the guidance scale, the autoencoder learning rate and loss weights, warmup, weight decay and the EMA decay are not stated. The paper's text and tables do not give them.
• Frame-prediction mask. The mask formula concatenates m_fp ∘ z_t, but the paper does not say how past frames enter z_t §4.4.
• Data and compute. The text-to-video training set is partly internal and cannot be reproduced §5.3. Hardware and total compute are not stated.
• Speed claim. The "up to 2×" speed-up for local window attention is measured in steps per second on UCF-101 tab. 3 §5.2.
• Different baselines. The text-to-video comparison uses different text encoders and protocols across rows, including adapted prompts for UCF-101 app. A. IS and FVD across rows are not like-for-like.
• Sampling budgets. Table 8 lists guidance for text-to-video but none for the class-conditional models tab. 8. Table 1 compares 50 DDIM steps with baseline budgets from 1 to 1024 steps tab. 1, so speed comparisons are not like-for-like.

Lineage

• Builds on: latent diffusion (Rombach et al., 2022) §1; MAGVIT-v2 tokenizer, arXiv:2310.05737 §4.1; ViT patchify §4.2; AdaLN-LoRA, inspired by LoRA §4.3; self-conditioning after Chen et al., 2022 §4.3; cascaded diffusion for super-resolution §4.5; zero terminal SNR, arXiv:2305.08891 §5.2.
• Compared against: MAGVIT, MAGVIT-v2, Video Diffusion, RIN, TATS, Phenaki tab. 1; DiT, ADM, MDT, MaskDiT, RIN, VDM++ tab. 2; CogVideo, MagicVideo, Make-A-Video, Video LDM, PYoCo tab. 5.
• Superseded by: not known from the content of this wiki.