StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

In plain words

Most video generators build a clip as a fixed list of frames. That makes long or high-frame-rate videos expensive, and clips tend to loop. StyleGAN-V treats a video as a signal that can be read at any moment in time. It keeps the image generator StyleGAN2 almost unchanged. The only new input is a short motion signal for each clip, which sets how the scene moves over time. The judge that separates real from fake clips looks at only a few random frames from each clip and joins their features together, which makes training cheaper. The authors report image quality close to the StyleGAN2 image model it is built on, and training directly at 1024 by 1024 resolution.

TL;DR

StyleGAN-V is a video GAN built on StyleGAN2. A content code sets the look of a clip, and a motion code gives wave-based time embeddings that are acyclic and can be queried at real-valued times, so frames are generated without autoregression. The discriminator sees three random frames per clip (the default k) together with their time gaps. In the authors' Table 1 it has the lowest FVD16 on three of four 256 by 256 benchmarks; VideoGPT is lower on RainbowJelly. All numbers are author-reported.

Problem

Video generators built on conv3d blocks or on recurrent (LSTM) motion models are expensive at high resolution, and recurrent models become unstable when unrolled far §1 §4.1. Standard sine positional embeddings are periodic, so a generated video repeats itself §3.1. Discriminators that judge images and videos separately add cost §3.2. Training on all frames is costly, so the paper also asks how few frames per clip can still learn the video distribution §3.3.

Method

Videos are modelled as functions x(t) of a continuous time t ≥ 0, and training uses sparse subsamples of them §3. The model is a GAN with three parts: a generator that produces any frame at any time, a discriminator that judges groups of sparsely sampled frames, and a sampler that chooses those frames.

Generator

1. Content code. Sample z^c from a standard normal. Map it to a 512-d latent w with the StyleGAN2 mapping network. w is shared by all frames of a clip §3.1.
2. Motion grid. Sample trajectory noise z^m on a grid of times t_0, t_1, … with spacing δ^z. The grid length n is chosen so the grid covers the largest query time §3.1. Default δ^z is 16; SkyTimelapse uses 256 §4.1.
3. Motion mapping. Pass the grid through two padding-less conv1d layers with kernel size 11, stride 1 and hidden size 512, and no dilation. The output is a sequence u_t0, …, u_tn that is 20 codes longer than the grid app. B. Padding is avoided so the model stays stable when unrolled.
4. Acyclic time embedding. For a query time t, take the two grid codes u_ℓ and u_r that bracket t, and compute a 512-d motion code v_t from them §3.1 eq. 1 to eq. 5. The wave parameters come from u_ℓ. A stitching step makes the embedding start and end at the grid points, and alignment vectors restore the expressive power that stitching removes §3.1 eq. 4.
5. Synthesis. Tile v_t into the constant input of the StyleGAN2 synthesis network, which produces the frame x_t. This is the only change to the synthesis network §3.1 fig. 5. Frames are produced independently, so generation is non-autoregressive §3.1.

# Generate frames of one clip at arbitrary times  [§3.1]
# Shapes: content (B, 512), noise (B, L, 512), times (B, F) in frames
def generate_frames(content_noise, motion_noise, times):
    # Content latent, shared by all frames of the clip  [§3.1]
    w = stylegan2_mapping(content_noise)  # (B, 512)
    # Two padding-less conv1d layers, kernel 11, hidden 512  [app. B]
    # Output is the grid length plus 20 codes  [app. B]
    trajs = motion_conv1d(motion_noise)  # (B, n + 20, 512)
    # Grid codes that bracket each query time  [§3.1]
    left = floor(times / delta_z)  # (B, F), delta_z = grid spacing
    u_left = gather(trajs, left)  # (B, F, 512)
    u_right = gather(trajs, left + 1)  # (B, F, 512)
    # Acyclic motion code for each frame  [§3.1] [eq. 4]
    v = motion_code(times, u_left, u_right, delta_z)  # (B, F, 512)
    # Tile v into the constant input and synthesize  [§3.1] [fig. 5]
    frames = synthesis(w, v)  # (B, F, 3, H, W)
    return frames

# Acyclic motion code v_t for one frame at time t  [§3.1] [eq. 2] to [eq. 5]
# Paper form with amplitudes; the code drops alpha  [code: src/training/motion.py#L205]
def motion_code(t, u_left, u_right, delta_z):
    # Wave parameters come from the left grid code only  [§3.1] [eq. 3]
    alpha = W_alpha @ u_left  # amplitudes (512,)
    rho = W_rho @ u_left  # phases (512,)
    # Periods are bounded to (0, 2) times a fixed scale sigma  [eq. 5] [app. B]
    omega = (tanh(W_omega @ u_left) + 1) * sigma  # (512,)
    # Grid points on both sides of t  [§3.1]
    t_left = t - (t % delta_z)  # (F,)
    t_right = t_left + delta_z  # (F,)
    # Raw waves at t and at both grid points  [eq. 2]
    raw = alpha * sin(omega * t + rho)  # (512,)
    raw_left = alpha * sin(omega * t_left + rho)  # (512,)
    raw_right = alpha * sin(omega * t_right + rho)  # (512,)
    # Stitch: remove the interpolated raw values, add back aligners  [eq. 4]
    frac = (t - t_left) / delta_z  # interpolation weight in [0, 1)
    removed = (1 - frac) * raw_left + frac * raw_right
    aligners = (1 - frac) * (W_a @ u_left) + frac * (W_a @ u_right)
    return raw - removed + aligners  # (512,)

Discriminator

The discriminator takes k frames of a clip, extracts features from each frame independently with a StyleGAN2-style backbone, concatenates the features channel-wise at 16 by 16 resolution, and outputs one real/fake logit §3.2 fig. 6. Time gaps δ_i = t_{i+1} − t_i condition the output. The paper encodes them with positional encoding, a two-layer MLP and a projection discriminator dot product §3.2. The official code differs; see Paper vs code.

Sampling of training frames

The paper samples k frames per clip with a total span of at most 32 frames §3.3. The clip offset is uniform over the time range that fits the maximum time T = 1024. Intermediate frames are placed at random positions without repetition §3.3. Real clips are cut as a random window of up to 1024 frames from a randomly chosen video app. B.

# Pick k frame times from one clip window of length L  [§3.3]
# Paper text: gap uniform in [k - 1, 32]. Algorithm 1 (uncompiled): log-uniform
def sample_frame_times(length, k, max_gap=32):
    # Candidate gaps: powers of two (1, 2, ..., 2048), clipped to the window
    # [code: configs/sampling/random.yaml#L4] [code: src/training/layers.py#L387]
    gaps = [d for d in powers_of_two if k - 1 <= d <= min(length - 1, max_gap)]
    gap = random_choice(gaps)
    # Integer start offset for real clips  [code: src/training/dataset.py#L441]
    start = random_int(0, length - gap - 1)
    # Remaining k - 2 frames: distinct random positions inside the gap  [§3.3]
    inner = random_sample(range(1, gap), k - 2)
    return sorted([start, start + gap] + [start + i for i in inner])

Losses and augmentation

The generator and discriminator use the non-saturating logistic loss with R1 regularization on real frames app. B. Adaptive differentiable augmentation from StyleGAN2-ADA is applied with the same random parameters to every frame of a clip ("video-consistent"). The paper says quality drops if this is turned off app. B.

# Losses for one batch of real and fake clips  [app. B]
# Logits are per clip; penalty is per clip  [code: src/training/loss.py#L90]
def gan_losses(real_logits, fake_logits, real_grad_sq, r1_gamma):
    # Generator: non-saturating loss, minimise softplus(-logit)  [app. B]
    generator_loss = mean(softplus(-fake_logits))
    # Discriminator: real up, fake down  [app. B]
    real_term = softplus(-real_logits)  # (B,)
    fake_term = softplus(fake_logits)  # (B,)
    # R1 on real inputs; code scales by r1_gamma / 2  [code: src/training/loss.py#L166]
    r1_term = (r1_gamma / 2) * real_grad_sq  # (B,)
    discriminator_loss = mean(real_term + fake_term + r1_term)
    return generator_loss, discriminator_loss

Projection and editing (appendix)

To embed a real photo, the authors optimise the W+ latent with LPIPS reconstruction loss for 1000 Adam steps, starting from learning rate 0.1. The motion codes are kept fixed during this step app. B. CLIP editing follows StyleCLIP, treating the image as the frame at t = 0. Motion codes are resampled at each editing step app. B. Masking is used only for one sky example app. B.

Hyperparameters

Results

Author-reported. Setup: four 256² datasets in Table 1 (the paper's RainbowJelly benchmark is the authors' own and cannot be released app. E). Each method is trained from scratch by the authors on four V100 GPUs, except VideoGPT, which ran on four A6000 GPUs with batch size 4. Metrics use the authors' FVD protocol app. C. Dataset sizes are 4.04 h (FaceForensics), 12.99 h (SkyTimelapse), 0.51 h (UCF101), 7.99 h (RainbowJelly) app. E.

tab. 1. FF = FaceForensics, Sky = SkyTimelapse, RJ = RainbowJelly. Costs are GPU-days. Two-stage methods are written "X + Y". MoCoGAN-HD's cost was halved by the authors to account for a slower codebase §4.1. Bold in the paper marks the best value per column; VideoGPT is best on RJ FVD16 and DIGAN on Sky FVD128.

• The paper's text says the method "significantly outperforms" the others on almost all benchmarks §4.1. The table supports lowest FVD16 on FaceForensics, SkyTimelapse and UCF101 only.
• Image quality: on FaceForensics 256², the authors report FID 9.44 in 8 GPU-days for StyleGAN-V against FID 8.42 in 7.72 GPU-days for StyleGAN2, about 10% worse §4.3.
• MEAD 1024², 7 days on 4× V100: FID 24.12 and FVD16 156.1 §4.3. The paper reports that MoCoGAN-HD does not keep the speaker's identity at this scale §4.3.
• The abstract claims about 30% average improvement over the closest runner-up [abstract]. The compiled sections read here do not show how that average is computed.

Ablations tab. 2

Frames per clip k tab. 3

Properties §4.3

• The generator produces arbitrarily long clips; the paper shows hour-long samples §1 fig. 1 and MEAD samples at 1024² §4.3 fig. 8, and higher frame rates in the supplement §4.3.
• Content and motion are separated: changing the motion codes does not change the appearance, and resampling the content code does not change the motion §4.3.
• Latent editing of images and of the first frame of a clip works as in StyleGAN2 §4.3 fig. 2.

Paper vs code

Official code: github.com/universome/stylegan-v at commit 3fecd69c602e1cda204357201461c0fb0a634909, linked from the paper's appendix app. B. The owner matches the project page in the abstract (universome.github.io). There is no LICENSE file, and the README says the code builds on INR-GAN, which is built on NVIDIA's StyleGAN2-ADA under the NVIDIA license README.md. Code is cited by path and line only, never quoted.

1. Period spacing. Paper says periods are linearly spaced in the period length app. B, eq. 6. The code spaces log2 of the period length evenly, which gives geometric spacing motion.py.
2. Minimum period. Paper says ω_min = 2³ frames for most datasets app. B. The code default is 16 stylegan-v.yaml, and the limitations section says SkyTimelapse was raised "from 16 to 256" app. A. The README command sets 256 for SkyTimelapse README.md.
3. Amplitudes and sin/cos. The paper's embedding has amplitude α and a single sine eq. 2. The code has no amplitude. It concatenates sin and cos of 256 frequencies, which gives 512 dims motion.py motion.py.
4. Phases. Paper uses ρ = W_ρ u eq. 3. The code multiplies the phase by a per-frequency scale, max_period / period_length motion.py motion.py.
5. Periods. Eq. 3 and eq. 5 both use W_ω. The code only has the tanh form, tanh(W_ω u) + 1, and no separate linear ω motion.py.
6. Discriminator width. Paper says each frame gives 512 × 16 × 16 features §3.2. The code halves channels at 16 × 16 (num_frames_div_factor = 2), so each frame gives 256 channels there and the concatenation is 256k channels, with the default channel cap of 512 networks.py networks.py networks.py.
7. Time conditioning in D. Paper uses a positional encoding, a two-layer MLP and a projection dot product §3.2. The code uses a learned embedding over the integer gap, 1024 entries of 256 dims, concatenated with log-spaced Fourier features of the gap. It feeds that vector to a mapping network whose output is the projection condition layers.py networks.py networks.py.
8. Phase dropout. Config key phase_dropout_std: 1.0 exists [code: configs/model/stylegan-v.yaml#L45], but nothing in src/ reads it. The paper's phase-dropout paragraph is commented out in the source and not in the compiled paper.
9. Sampling distribution. Paper text says the gap is uniform on [k − 1, 32] §3.3. Algorithm 1 (not compiled into the paper, since algorithms/sampling.tex is never included) draws log-uniform powers of two app. B. The code draws from powers of two, which matches Algorithm 1 layers.py. Algorithm 1 places intermediate frames at equal spacing; the text and code place them at random.
10. StyleGAN2 loss. Paper says the optimisation is "exactly the same" as StyleGAN2, including the loss §4.1. The video config sets style mixing and path-length regularisation to 0.0 stylegan-v.yaml. The StyleGAN2 base config uses 0.9 and 2.0 base.yaml.
11. R1 coefficient. Paper states 0.2 for all experiments app. B. The code computes r1_gamma = 0.0002 × res² / batch train.py. That is 0.2048 at 256² with batch 64, but about 13.1 at 1024² with batch 16 on 4 GPUs (derived from these formulas; the MEAD run's actual value is not stated).
12. Motion grid length. The code sizes the noise grid for the maximum clip length, not for each query time: ceil((max_num_frames − 1) / δ^z) + 2 + 20 codes motion.py motion.py. With defaults (1024 frames, δ^z = 16) that is 86 codes (derived).

Pitfalls

• Frequency scale is ambiguous. The paper's linear-spacing formula (eq. 6) does not match the code's geometric spacing (see Paper vs code, item 1). The appendix's ω_min of 2³ conflicts with the code default and with the limitations section. Implement the code's behaviour (16 to 1024) and say so in the card you write.
• Eq. 3 and eq. 5 both define ω. Use the tanh form, which the code uses and which the text says was found "useful in practice" §3.1.
• Algorithm 1 is not compiled. The paper's text, the compiled body and Algorithm 1 disagree about the gap distribution. The code follows Algorithm 1 (log-uniform). Use the code when in doubt.
• Real and generated times differ. Real frames use integer offsets [code: src/training/dataset.py#L441]. Generated frames use fractional offsets layers.py stylegan-v.yaml. The discriminator's learned embedding rounds the gap to an integer [code: src/training/layers.py#L292], but the Fourier features use the exact gap. Fake gaps are therefore rounded in one branch only.
• Video-consistent augmentation is needed. Per-frame augmentation makes the discriminator underperform and lowers quality app. B. Apply the same random augmentation to every frame of a clip loss.py.
• R1 does not transfer by default. Using the paper's 0.2 at 1024² is a different setting from the code's heuristic (13.1 under the derived batch of 16). Fix r1_gamma explicitly.
• Loss differs from the paper's "same as StyleGAN2". Style mixing and path-length regularisation are off in the video config (Paper vs code, item 10). Copying the StyleGAN2 defaults changes the training.
• Ablation text vs numbers. The text says removing the generator gives "poor short-term and long-term" quality §4.2, but "w/o our G" has the lowest FaceForensics FVD128 in Table 2 (41.77 vs 89.34). The appendix says FVD does not capture motion collapse, and that LSTM codes match FVD128 on SkyTimelapse (165.8 and 196.1 vs 197.0) app. C.
• k = 2 is not worse on every dataset. Table 3 has k = 2 with lower SkyTimelapse FVD16 (50.5 vs 79.52) and higher FaceForensics FVD16 (60.41 vs 47.41). The default is k = 3 base.yaml.
• FVD numbers are not comparable across papers. The authors re-trained all baselines and used their own FVD protocol §4 app. C. They also changed DIGAN's sampling to uniform, capped VideoGPT at 32 GPU-days with batch size 4, and halved MoCoGAN-HD's cost §4.1.
• Checkpoints are chosen on the reported metric. Each method reports its checkpoint with the lowest FVD16 §4.1, which is also the headline number.
• RainbowJelly cannot be reproduced from the release. The authors cannot release the source video for copyright reasons. They release a processing script instead app. E.
• Limitations the authors list. Periodic motion still appears sometimes, new content that appears late in a clip is poorly handled, and the model is sensitive to the minimum period length app. A.
• Cost. Training is about 5% more expensive than StyleGAN2 at the same resolution §1. The authors' StyleGAN2 cost is 7.72 GPU-days in their environment §4.1.

Lineage

• Builds on: StyleGAN2 for the generator, discriminator and training procedure §3 app. B; MoCoGAN's content and motion split §3.
• Considered and not used: INR-GAN, because the authors found it did not tolerate mixed precision well app. B.
• Compared against: MoCoGAN, MoCoGAN with a StyleGAN2 backbone, VideoGPT, MoCoGAN-HD and DIGAN §4.1.
• Superseded by: not stated in the compiled content.