Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

In plain words

The model here is trained on short clips of 16 frames. If you simply keep generating past that length, the picture usually starts to fall apart. This paper looks at why, and finds that the cause sits in the first step of the pipeline: the part that squeezes video into a short list of discrete codes. Zero-filling at the edges of each clip makes the codes near the clip border look different from the codes in the middle, so the second-stage model never sees the kind of sequences it has to continue.

The fix is to pad the edges by copying the nearest real frame instead of filling with zeros. Copying is cheap, and it keeps the codes consistent wherever a clip starts. A second stage is added to keep the story on track: a model that first places sparse "anchor" frames across the whole video, and a second model that fills in the frames between each pair of anchors. The authors report videos of 1024 frames on several benchmarks, and they also propose ways to measure quality and consistency over time, because long-video evaluation had not been well studied.

TL;DR

TATS is a two-stage model for video generation. A 3D VQGAN compresses clips into discrete tokens, and its replicate padding makes the tokens position-independent so a sliding-window transformer can keep generating. A hierarchical transformer (sparse anchors, then interpolation) improves coherence over long spans. The author-reported results cover UCF-101, Sky Time-lapse, Taichi-HD, AudioSet-Drum and MUGEN, with 16-frame short-video scores and 1024-frame long-video analyses.

Problem

Transformers over discrete video tokens can in principle generate arbitrarily long sequences by sliding a window, but this fails in practice. Tokens produced by a VQGAN encoder depend on the zero padding at the clip boundaries, so the tokens near the boundary differ from the tokens of real long videos, and the window sees sequences it never trained on §1 §2.2 app. A.2. Generating longer videos also requires modelling long-range structure such as a consistent theme and a storyline §2.3. GAN-based long-video methods are reported to need very large training resources §1, and autoregressive sampling errors accumulate over time §2.3.

Method

Overview §2 fig. 3: a time-agnostic video VQGAN encodes 16-frame clips into a grid of discrete tokens (temporal compression 4, spatial 8). A decoder-only transformer models those tokens. For long videos, the transformer uses a sliding window at inference, and a hierarchical extension adds an autoregressive model for sparse anchor slices and an interpolation model for the frames in between. Conditioning on text or audio prepends extra tokens to the video tokens.

2.1 Vanilla video VQGAN

The VQGAN of [esser2021taming] is extended from 2D to 3D convolutions: a 3D encoder maps a clip to latents, each latent is replaced by its nearest codebook entry, and a decoder maps the quantized latents back to frames §2.1. The codebook is updated with an exponential moving average, the quantization step uses a straight-through gradient, and the commitment term has weight 0.25 §2.1. Training adds an L1 reconstruction loss, a spatial discriminator on one random frame, a temporal discriminator on the whole clip, and feature-matching losses for the discriminators and a VGG perceptual term §2.1. The paper's objective weights reconstruction and matching by λ_rec and λ_match, and the discriminator by λ_disc §2.1. Plain GAN losses on VideoGPT's architecture collapsed, so the paper uses a convolution-only encoder and decoder, doubles channels when resolution halves, clips gradients to norm 1, and uses synced batch normalization with gradient accumulation app. A.1. The paper's formulas (equations are unnumbered in the source; cited by section):

VQVAE loss, with sg the stop-gradient operator and β = 0.25 §2.1:

\mathcal{L}_\text{vqvae} = \|x - \hat{x}\|_1 + \|\mathrm{sg}[f_\mathcal{E}(x)] - c_z\|_2^2 + \beta \|\mathrm{sg}[c_z] - f_\mathcal{E}(x)\|_2^2

Discriminator loss, applied to the spatial and temporal discriminators §2.1:

\mathcal{L}_\text{disc} = \log f_\mathcal{D}(x) + \log(1 - f_\mathcal{D}(\hat{x}))

Feature matching, with p_i a learned constant for VGG and the reciprocal of the element count for discriminator layers §2.1:

\mathcal{L}_\text{match} = \sum_i p_i \left\| f^{(i)}(\hat{x}) - f^{(i)}(x) \right\|_1

Overall VQGAN objective §2.1:

\min_{f_\mathcal{E}, f_\mathcal{G}, \mathcal{C}} \left( \max_{f_{\mathcal{D}_s}, f_{\mathcal{D}_t}} \lambda_\text{disc} \mathcal{L}_\text{disc} \right) + \min_{f_\mathcal{E}, f_\mathcal{G}, \mathcal{C}} \left( \lambda_\text{match} \mathcal{L}_\text{match} + \lambda_\text{rec} \mathcal{L}_\text{rec} + \mathcal{L}_\text{codebook} + \beta \mathcal{L}_\text{commit} \right)

# Video VQGAN training step  [§2.1] [app. A.1]
# Input:  clip (B, T, H, W, 3), T = 16 for training
# Output: generator loss and discriminator loss

def vqgan_step(clip):
    # 1. Encode, quantize and decode the clip  [§2.1]
    latents = encoder(clip)                        # (B, c, t, h, w)
    tokens, quantized = nearest_codebook_entry(latents)
    recon = decoder(quantized)                     # (B, T, H, W, 3)

    # 2. Reconstruction with L1, weight lambda_rec  [§2.1] [app. B.2]
    recon_loss = lambda_rec * l1(recon, clip)

    # 3. Commitment pulls latents to their codes, weight 0.25  [§2.1]
    commit_loss = 0.25 * mse(latents, stop_grad(quantized))

    # 4. Discriminators: one random frame and the whole clip  [§2.1]
    frame = random_frame(recon)
    adv_loss = gan_loss(spatial_disc(frame)) + gan_loss(temporal_disc(recon))

    # 5. Feature matching with a VGG term and discriminator features  [§2.1]
    match_loss = feature_matching(recon, clip)

    # Codebook entries move by EMA, not by a gradient  [§2.1]
    return recon_loss + commit_loss + adv_loss + match_loss

2.2 Time-agnostic VQGAN

The paper argues that the encoder is not temporally shift-equivariant: the same frames give different tokens depending on where they sit in the clip, so real data rarely matches the token prefixes that a sliding window needs §2.2 app. A.2. Removing all temporal padding and padding with real frames would make it exact, but it is expensive for large compression rates and deep networks, and it discards frames near the ends of short videos §2.2. The paper instead pads by replicating the boundary frames, which assumes the video is frozen beyond its ends (the "boring videos" assumption) §2.2. In the paper's experiments, replicate padding is used with no real frames added §2.2. Its equivariance score is 0.75 against 1.0 for real-frame padding; the reflect and circular modes score lower app. A.3.

The autoregressive prior models the flattened token sequence in row-major order, frame by frame, with a start token §2.2. The sliding window is the training length: to generate slice t, the model conditions on the t−1 slices before it §2.2. The prior and its training loss §2.2:

p(\mathbf{z}) = p(\mathbf{z}_0) \prod_{i=0}^{t h w - 1} p(\mathbf{z}_{i+1} \mid \mathbf{z}_{0:i}), \qquad \mathcal{L}_\text{transformer} = \mathbb{E}_{\mathbf{z} \sim p(\mathbf{z}_\text{data})}[-\log p(\mathbf{z})]

The paper's equivariance score, which measures how many tokens agree when the same frames sit at the start or at the end of a clip app. A.3:

\sum_{i=1}^{t} \sum_{j=0}^{h-1} \sum_{k=0}^{w-1} \frac{\mathbb{1}\left([f_\mathcal{E}(\mathbf{x}_{0:T})]_{i,j,k}, [f_\mathcal{E}(\mathbf{x}_{d:T+d})]_{i-1,j,k}\right)}{t \times h \times w}

# Autoregressive prior over video tokens  [§2.2]
# Input:  tokens (B, N) with N = t * h * w, row-major, frame by frame
# Output: scalar negative log-likelihood

def prior_loss(tokens):
    # 1. Shift right and prepend the start token  [§2.2]
    inputs = concat(start_token, tokens[:, :-1])   # (B, N)

    # 2. Predict each token from the tokens before it  [§2.2]
    logits = transformer(inputs)                   # (B, N, K)

    # 3. Negative log-likelihood over the training tokens  [§2.2]
    return cross_entropy(logits, tokens)

# Long video by sliding window  [§2.2]
# Input:  number of latent slices to generate, training window length W
# Output: token video (n_slices, h, w), decoded to frames afterwards

def sample_long_video(n_slices, window_len):
    # 1. Start from the class or condition token  [§2.2]
    slices = []
    for t in range(n_slices):
        # 2. Keep the most recent slices so the context matches training  [§2.2]
        context = last_slices(slices, window_len - 1)
        # 3. Sample the tokens of slice t, top-k 2048 and top-p 0.80  [app. B.2]
        slices.append(sample_slice(context, top_k=2048, top_p=0.80))
    return slices

2.3 Time-sensitive transformer

Two additions give longer-range structure §2.3. First, a hierarchical model: an autoregressive transformer generates a sparse set of token slices, four times sparser than the full video, and an interpolation transformer fills in the missing slices between each pair of adjacent sparse slices §2.3. The code fills three slices per gap sample_vqgan_transformer_hierarchical.py. Each interpolated token attends to the tokens generated so far in a causal way and to the sparse anchor slices at both ends §2.3 app. A.4. The anchor slice at the far end of the frames being generated must not be visible, because a multi-layer attention then leaks the middle frames into themselves and training collapses app. A.4. Second, conditioning: text or audio is encoded into discrete tokens (CLIP's BPE tokenizer for text; a second VQGAN on STFT data for audio) which are prepended to the video tokens, and the start token is removed. At inference the sliding window is applied to both the condition tokens and the video tokens §2.3 §3.1.

# Hierarchical long video  [§2.3] [app. A.4]
# Input:  number of latent slices to generate
# Output: token video with every slice filled in

def sample_hierarchical(n_slices):
    # 1. Sparse anchor slices: one per four slices  [§2.3]
    anchors = sample_sparse_autoregressive(n_slices // 4 + 1)

    # 2. Fill the three missing slices between each pair of anchors  [§2.3]
    video = empty_video(n_slices)
    for a, b in adjacent_pairs(anchors):
        # Causal attention to earlier tokens and to both anchors  [app. A.4]
        middle = sample_interpolation(earlier=video, left=a, right=b)
        video = write_between(video, a, b, middle)
    return video

# Conditioning prefix  [§2.3]
# Input:  condition tokens (text or audio), video tokens
# Output: prefix for the transformer

def conditioned_sequence(condition_tokens, video_tokens):
    # The start token is dropped; condition tokens come first  [§2.3]
    return concat(condition_tokens, video_tokens)

Hyperparameters

Values are from the paper unless marked "code" (taken from the official repository, not stated in the paper) or "derived" (computed from stated values). The repository's training commands are the closest the authors give to their setup. Stated values are in app. B.2 unless a different location is given.

Results

Author-reported. Setup: 16-frame short-video generation §3.2 and 1024-frame long-video generation §3.3, compared against the authors' reported numbers for the listed baselines. Full table: tab. 1. Rows marked * in the source table were trained on the full UCF-101 dataset, not its train split.

Long videos §3.3 §3.4: FVD measured on consecutive 16-frame clips degrades more slowly for TATS-base than for the vanilla VQGAN, MoCoGAN-HD and CCVS, and TATS-hierarchical further improves on TATS-base (DIGAN degrades severely on UCF-101 and Taichi-HD §3.3); the paper reports no numbers for these curves in the text fig. 5. The coherence scores compare each 16-frame clip at time t with the first 16 frames, using a C3D classifier p(y|·) §3.3:

\mathrm{CCS}_t = \frac{1}{N} \sum_i \mathbb{1}\left(\arg\max p(y \mid x_i^{(0:15)}) = \arg\max p(y \mid x_i^{(t:t+15)})\right)

\mathrm{ICS}_t = \sum_i p(y \mid x_i^{(t:t+15)}) \log \frac{p(y \mid x_i^{(t:t+15)})}{p(y \mid x_i^{(0:15)})}

On UCF-101, more than half of the 1024-frame videos from TATS-hierarchical stay in the same predicted class in the last 16 frames §3.3 fig. 6. Sky videos generated by TATS show smooth transitions between weather and times of day. The paper measures the first and the last frame of each long video: generated videos change more than real ones (LPIPS 0.3461 ± 0.1184 for generated against 0.1839 ± 0.0683 for real; colour histogram correlation 0.0797 against 0.7330) tab. 2 §3.4. The paper reads this as evidence that the transitions are more prominent in generated videos than in the training data §3.4.

Other results: a human preference study compares TATS with DIGAN on fewer artifacts, with TATS chosen 355 times and DIGAN 175 times app. C.3. The paper's own computation time for one 1024-frame video is 30 min for TATS-base and 7.5 min plus 23 s for the hierarchical model on one Quadro P6000 GPU app. B.2; across methods, DIGAN takes 4.2 s and CCVS 22 min app. B.3.

Paper vs code

Code: official repository songweige/TATS (MIT) at commit 03bbce1daa3da7e9bff8221e9ec433df9484817f, linked from the project page. The repository implements the VQGAN, both transformers, the sampling scripts and FVD/KVD. It does not contain the IS (C3D) or the CCS/ICS coherence code.

Consistent with the paper: replicate padding is the default [code: tats/tats_vqgan.py#L250]; commitment weight 0.25 [code: tats/modules/codebook.py#L64]; VQGAN Adam betas (0.5, 0.9) [code: tats/tats_vqgan.py#L206]; no random restart in the training command [code: README.md#L113-L118]; gradient clip 1.0 README.md; λ_rec = 4 tats_vqgan.py; 24 layers, 16 heads, embedding 1024 [code: README.md#L129]; top-k 2048 and top-p 0.8 for short videos [code: README.md#L53-L55].

Discrepancies:

• Codebook size: the paper uses 16,384 app. B.2. The class default is 2,048 tats_vqgan.py; the README training command sets 16,384 README.md.
• VQGAN learning rate: the paper states 3e-5 app. B.2. The training script rescales the base rate to accumulate × (GPUs/8) × (batch/4) × base train_vqgan.py. With the README command (6 accumulation, 8 GPUs, batch 2) that gives 9e-5. The class default is 3e-4 tats_vqgan.py.
• Training length: the paper states 30K steps for the VQGAN app. B.2; the README command sets 2,000,000 maximum steps README.md. For transformers the paper states 500K steps, or 1.35M for TATS-base on UCF-101 app. B.2; the README sets 2,000,000 README.md.
• Discriminator loss: the paper writes the log-form GAN objective §2.1. The code defaults to hinge loss tats_vqgan.py, and the generator term is the negative mean of discriminator logits tats_vqgan.py.
• Perceptual and feature-matching weights: the paper uses both with λ_match = 4 app. B.2. The class defaults are 0 for both tats_vqgan.py; the README command sets both to 4 README.md.
• GAN start: the paper uses 10K steps app. B.2; the class default is 50,000 tats_vqgan.py; the README uses 10,000 README.md.
• Normalization: the paper replaces group norm with synced batch norm app. A.1. The class default is group norm tats_vqgan.py; the README uses batch norm README.md.
• Long-video context: the paper conditions on the t−1 earlier slices, the full training window §2.2. The long-video script calls sample_long_fast, which conditions only on --temporal_sample_pos earlier slices (default 1) sample_vqgan_transformer_long_videos.py sample_vqgan_transformer_long_videos.py. The full-window function sample_long is defined sample_vqgan_transformer_long_videos.py but no call site uses it.
• Top-p in long videos: the paper's general setting is 0.80 app. B.2. The long-video script defaults to 0.92 sample_vqgan_transformer_long_videos.py, and the README's long-video command does not override it README.md.
• Interpolation mask: the paper describes a mask that exposes both anchors and blocks attention to the last anchor app. A.4. The code builds the mask from n_unmasked with +1 offsets and zeroes a block of rows against middle columns gpt.py. The README uses n_unmasked 256 for the interpolation run README.md. I did not run the code, so I cannot confirm the mask matches the paper's figure exactly.

Pitfalls

• Padding is the whole idea. Replicate padding assumes the video is frozen beyond its ends, which the paper calls the "boring videos" assumption; reflect padding models reversed motion. The paper's equivariance score is 0.75 for replicate padding against 1.0 for real-frame padding, which costs more compute app. A.3.
• The problem was hidden in image VQGAN, because the generated token sits in the spatial centre, far from the zero-padded border of a large image. With 16-frame training clips, almost every token is near a temporal border, so the issue shows up for long videos app. A.2.
• VQGAN with GAN losses collapses easily. The paper lists the changes that fixed it: convolution-only encoder, channels doubled when resolution halves, gradient clipping to norm 1, and synced batch norm with gradient accumulation to avoid blob artifacts app. A.1.
• The interpolation transformer must not attend to the far anchor of the frames it generates. The paper reports that doing so creates a shortcut that makes training collapse app. A.4.
• Errors accumulate with sliding-window generation. Failure cases show repeated tokens and quality loss after a few hundred frames, more often with large motion or a clear end to the action app. C.2.
• Sampling is expensive: about 30 minutes for one 1024-frame TATS-base video on one P6000 GPU app. B.2.
• Frame sampling changes the data: TATS-base trains on every fourth Taichi-HD frame to match prior work, while the hierarchical model does not, because skipping would drop 60% of videos app. B.1.
• Two rows in Table 1 that were trained on the full UCF-101 split have lower FVD than TATS-base's train-split result (CCVS* with StyleGAN 386, with real frames 389, against 420) tab. 1. The short-video claim for FVD on UCF-101 applies to the train-split rows only.
• Class labels change UCF-101 scores a lot (IS 57.63 to 79.28). Compare only models under the same conditioning §3.2.
• FVD and KVD use an I3D model trained on Kinetics-400. The paper suggests this may explain why Sky Time-lapse results look unusual for DIGAN §3.3. The coherence metrics CCS and ICS use a C3D model, and that code is not in the repository.
• The human study reports 530 votes, but the description says about five raters on each of 100 pairs, which would give about 500 app. C.3. Treat the exact counts with care.
• Long-video quality curves are shown only as figures, so there are no numeric values for the degradation claim in the text fig. 5.

Lineage

• Builds on: VQGAN (Esser et al., 2021) for the latent quantization and GAN training §2.1; VideoGPT (VideoGPT), whose 3D VQ-VAE with axial attention the paper replaces with a convolution-only design after GAN training collapsed app. A.1; sliding-window generation as used for long text (Brown et al., 2020) §2.2.
• Compared against: MoCoGAN-HD, DIGAN, CCVS, VideoGPT, StyleGAN-V, CogVideo, Video Diffusion and TGAN variants on UCF-101 tab. 1; Sound2Sight and CCVS on AudioSet-Drum tab. 1; DIGAN in a human study app. C.3.
• Superseded by: not stated in the content this wiki holds.