Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models

In plain words

A generative model learns from examples so that it can make new things that look like them. Over the last decade several different recipes for this have grown up in parallel. Some learn to undo a gradual noising process, some pit two networks against each other, some predict one piece at a time, some learn an invertible mapping from simple noise to data, and some score configurations with a single energy function.

Each recipe makes different trade-offs. Some are slow to sample from, some are hard to train stably, some give exact probabilities and some do not, and some need a very large model before they look good. This paper is a survey that puts these families side by side. It explains the basic idea behind each one, shows how they relate, and gives a rough comparison table so a reader can see which trade-offs go with which family.

It is useful as a map. It does not declare a winner, and it does not run any new experiments.

TL;DR

A survey of five deep generative model families (energy-based, variational autoencoders, GANs, autoregressive, normalizing flows) plus hybrids §1. It gives the core equations, training recipes and design trade-offs of each, and a coarse star-rated comparison on CIFAR-10 tab. 1. It is useful for orientation, but its numbers are compiled from other papers and rated by the authors, so they are not directly comparable.

Problem

Generative modelling aims to produce samples from the same distribution as the training data. Each family handles this with a different compromise. Energy-based models are flexible but need Markov chain sampling. Latent-variable models with a tractable bound are cheap to sample from but tend to blur. Adversarial models give sharp samples but train unstably and lack a likelihood. Autoregressive models give exact likelihoods but sample one variable at a time. Flows give exact likelihoods and invertibility at the cost of architectural restrictions §1 §2 §3 §4 §5 §6.

Existing surveys each cover one family in depth, or one application area, so a single comparison across families is what the authors say they add §1.

The paper also notes that higher likelihood does not reliably mean better samples, so the choice of objective affects sample quality §1 §2.5 §7.

Method

This is a review, so there is no method to reimplement. The card below records the core formulas and training recipes of each family as the paper presents them. Numbers in brackets are the paper's own section, equation, table or figure numbers.

Energy-based models §2

An energy function E_\theta(\mathbf{x}) defines a density proportional to e^{-E_\theta(\mathbf{x})} after normalisation, with the normalising integral intractable for most models eq. 1 §2.

Training with contrastive divergence lowers the energy of data and raises the energy of model samples. The gradient of the negative log-likelihood has a data term and a model-sample term eq. 2. Model samples come from Langevin dynamics: start from a simple initial distribution, then repeat a gradient step on the energy plus Gaussian noise eq. 4 §2.2.

The Boltzmann and restricted Boltzmann machine are early energy models; restricting connectivity in the RBM makes the hidden units exactly computable eq. 3 §2.1.

Score matching avoids the normalising constant by fitting the gradient of the log-density. Denoising score matching fits the score on noise-corrupted data eq. 6 eq. 7 §2.3. The paper groups diffusion models here, since their training is closely related to denoising score matching. Forward noising uses a fixed Markov chain with a schedule \beta_t eq. 8a eq. 8b §2.3.

Hybrids correct an implicit generator's samples with an energy term, via exponential tilting eq. 9 §2.4.1. Noise-contrastive estimation turns energy training into classification against a noise distribution eq. 10 §2.4.2.

Variational autoencoders §3

A latent-variable model p_\theta(\mathbf{x}\mid\mathbf{z}) with prior p_\theta(\mathbf{z}) is trained through a tractable lower bound on the log-likelihood. An encoder gives an approximate posterior q_\phi(\mathbf{z}\mid\mathbf{x}) that is shared across data points (amortised inference) §3.

The bound decomposes into a KL term and an expected reconstruction term eq. 11 eq. 12. The paper gives the closed form for a diagonal Gaussian posterior against a standard normal prior, together with a Monte Carlo estimate of the reconstruction term using L reparameterised samples \mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon} eq. 13 fig. 2 §3.

The paper says blurry samples have been linked both to the likelihood objective and to the restrictiveness of the Gaussian posterior, and it reports evidence for the second cause §3.

Variants covered: a \beta weight on the KL term for disentanglement §3; a two-term objective that matches the aggregate posterior to the prior eq. 14 §3.1; hierarchical VAEs with latent levels conditioned on earlier levels eq. 15a eq. 15b eq. 16 fig. 3 §3.1.1; regularised autoencoders with a discrete codebook, as in VQ-VAE, using the straight-through estimator §3.2; autoregressive decoders, with posterior collapse mitigations §3.3 §3.3.1; and methods that bridge amortised and stochastic inference, such as semi-amortised VAEs and VAEBMs §3.4.

Generative adversarial networks §4

A discriminator D and generator G play a minimax game with value function V(G,D) eq. 17 fig. 4 §4.

With G fixed, the optimal discriminator is p_d / (p_d + p_g). The paper then rewrites the generator objective as a Jensen-Shannon divergence plus a constant it calls C eq. 18 §4. The paper does not state the value of C.

Losses used in the paper's comparison are in the table below fig. 5a. The generator loss plot is in fig. 5b.

Wasserstein distance is given in primal form and in its 1-Lipschitz dual form eq. 19a eq. 19b §4.1.1. Spectral normalisation divides each weight matrix by its estimated spectral norm eq. 20 §4.1.2. Its estimate uses a power iteration eq. 21 §4.1.2.

Augmenting every discriminator input with a random transform T is described for a WGAN objective eq. 22a eq. 22b §4.1.3. Discriminator-driven MCMC refinement and the duality gap are covered in §4.1.4 and §4.1.5 eq. 23.

Architectures: LapGAN, StackGAN and progressive growing for high resolution §4.2; DCGAN's fully convolutional design; BigGAN's large batches and spectral normalisation; skip-connected multi-scale designs §4.2. Training speed: small-batch and adversarial-robustness variants, depthwise convolutions and a lightweight GAN §4.3.

Autoregressive models §5

The joint distribution factors by the chain rule into conditionals, and training minimises the negative log-likelihood eq. 24 eq. 25 §5.

Architectures: masked MLPs (NADE, RNADE, MADE) with masks that enforce the ordering eq. 26a eq. 26b §5.1.1; RNNs (LSTM, GRU) §5.1.2; causal and dilated convolutions §5.1.3; self-attention with the standard scaled dot-product form eq. 27 §5.1.4; and linear attention in a kernel form with feature map \phi eq. 28 §5.1.4. Multi-scale designs reduce the cost of high-resolution images §5.1.5.

Output modelling choices (mu-law companding, bit-splitting, mixture of logistics, raster versus zig-zag order) are listed in §5.2.

Normalizing flows §6

An invertible map f transforms a base density. The change-of-variables rule gives the density of the output, and a chain of maps gives the log-density as a sum of log-determinants eq. 29 eq. 30a eq. 30b §6.

Coupling layers update half the inputs using a function of the other half eq. 31a eq. 31b §6.1. Affine coupling uses an elementwise scale and shift eq. 32 §6.1.1. Monotone elementwise maps (integrals of positive functions, polynomials, mixture-of-logits CDFs, splines) are covered in eq. 33 §6.1.2. Autoregressive flows (MAF, IAF) are given in eq. 34 §6.1.3. Density distillation trains an IAF student against an autoregressive teacher by minimising a KL divergence eq. 35 §6.1.4.

Convolutional flows (1x1 convolutions, emerging convolutions) are in §6.2. Residual flows: the planar flow eq. 36 eq. 37 §6.3.1 and Lipschitz-constrained residual blocks with fixed-point inversion, including i-ResNet §6.3.2 tab. 3.

Surjective and stochastic layers eq. 39 §6.4; discrete flows eq. 40 §6.5; continuous-time flows with a neural ODE, its adjoint gradient and FFJORD eq. 41 eq. 42 eq. 43 eq. 44 §6.6; trajectory regularisation toward straight paths §6.6.1.

The layer table below is the paper's tab. 3, with forms condensed. The paper marks the layer with a Low Rank / Coupling / Convolutions / Lipschitz Residual grouping.

Sources: tab. 3.

Pseudocode

The first block is contrastive-divergence training with Langevin negatives eq. 2 eq. 4 §2.2:

# Contrastive divergence step with Langevin negatives  [§2.2] [eq. 2] [eq. 4]
# Input:  energy_net, data batch x_pos (B, D), step size alpha, steps K
# Output: scalar loss whose gradient in theta is the eq. 2 estimate

def contrastive_divergence_loss(energy_net, x_pos, alpha, K):
    # Start negative chains from a simple initial distribution  [§2.2]
    x_neg = sample_initial_like(x_pos)                  # (B, D)
    # Repeat a gradient step on the energy plus Gaussian noise  [eq. 4]
    for step in range(K):
        grad = energy_gradient(energy_net, x_neg)       # (B, D)
        noise = sqrt(alpha) * standard_normal_like(x_neg)
        x_neg = x_neg - (alpha / 2) * grad + noise
    # Lower energy on data, raise it on samples; negatives are constants
    # so only the energy parameters receive the eq. 2 gradient  [eq. 2]
    loss = mean(energy_net(x_pos)) - mean(energy_net(x_neg))
    return loss

The second block is the reparameterised VAE objective eq. 13 §3:

# Negative ELBO with reparameterised samples  [§3] [eq. 13]
# Input:  images x (B, D); encoder, decoder; L samples
# Output: scalar loss to minimise

def vae_loss(encoder, decoder, x, num_samples):
    # Amortised posterior: a mean and std per latent unit  [§3]
    mu, sigma = encoder(x)                              # (B, J), (B, J)
    # Closed-form KL to a standard normal  [eq. 13]
    kl = -0.5 * sum(1 + log(sigma**2) - mu**2 - sigma**2, axis=J)
    # Monte Carlo estimate of the reconstruction term  [eq. 13]
    recon = zeros(B)                                    # (B,)
    for sample in range(num_samples):
        # Reparameterise: z = mu + sigma * eps, eps ~ N(0, I)  [§3]
        eps = standard_normal_like(mu)                  # (B, J)
        z = mu + sigma * eps                            # (B, J)
        recon = recon + log_likelihood(decoder(z), x)   # (B,)
    recon = recon / num_samples
    # The paper writes the bound to maximise, so negate it  [eq. 13]
    return mean(kl - recon)

The third block is the augmented WGAN losses as printed eq. 22a eq. 22b §4.1.3:

# Augmented WGAN losses as printed in the paper  [§4.1.3] [eq. 22]
# Input:  real batch x, latents z, augmentation T, critic D, generator G
# Output: critic value (maximised) and generator loss (minimised)

def augmented_wgan_losses(critic, generator, x, z):
    # Generated batch  [§4.1.3]
    fake = generator(z)                                 # (B, ...)
    # Critic value; the critic maximises this  [eq. 22a]
    critic_value = (mean(critic(T(x))) - mean(critic(T(fake))))
    # Generator loss; the generator minimises this  [eq. 22b]
    generator_loss = mean(critic(T(fake)))
    return critic_value, generator_loss

Hyperparameters

The paper trains no models, so there are no training hyperparameters to copy. The table lists the numeric settings that do appear in the text and that matter when using it.

Results

Author-reported. This is a survey, so the paper runs no experiments of its own. Setup: CIFAR-10, with FID and negative log-likelihood in bits per dimension tab. 1. The paper does not state the training budget, model size or source for each row. The numbers come from the cited works. Some NLL entries are upper bounds (≤), which are variational bounds, not exact likelihoods tab. 1.

Columns are the paper's star ratings (higher is better, 1 to 5), the free-form Jacobian and exact density flags, and the FID and NLL values.

Star rules used for the rating columns tab. 2:

Conclusion, author's view §9: GANs have led on sample quality for some time, but the gap to other families is closing. Hybrid models offer a middle ground at the cost of extra complexity. The authors expect attention and implicit representations to matter for scaling.

Paper vs code

No official code. The paper has no code link in its text or arXiv metadata. A GitHub repository search for the title's key words and for the authors' surname returned no repositories. This paper is a survey and has no code of its own.

Pitfalls

• NLL columns mix kinds of number. Entries marked ≤ are variational upper bounds, while plain entries are exact or author-reported values tab. 1. Do not rank a VAE bound against an exact autoregressive likelihood.
• Table 1 is not like-for-like. FID and NLL come from different papers with different training budgets, which the paper does not state per row tab. 1. The star ratings are the authors' coarse, subjective mapping to tab. 2. The text itself calls the training-days rating anecdotal §1.
• Denoising diffusion appears under energy-based models §2.3 tab. 1. Readers looking for a separate family will not find one here.
• GAN loss table signs are mixed. Some rows are written as losses to minimise (NSGAN), and the WGAN discriminator row is written as a quantity to maximise fig. 5a. The generator form of NSGAN is the non-saturating one, which is not the minimax value in eq. 17.
• Constant in the GAN reformulation is undefined. The paper writes C without a value eq. 18.
• Spectral-norm power iteration is printed without normalisation. eq. 21 does not normalise \mathbf{u} and \mathbf{v} between iterations, and the paper does not say whether that is intended §4.1.2.
• Augmentation is described for WGAN only. eq. 22a eq. 22b show the WGAN form, with the augmentation distribution and strength not stated §4.1.3.
• Langevin step and noise are coupled in one formula. eq. 4 uses drift \alpha/2 and noise variance \alpha, but the text says they are often tuned separately, and the step count is not stated §2.2.
• Linear attention and causality. eq. 28 shows the reordering that caches \phi(\mathbf{K})^T\mathbf{V} for queries. The paper does not say how this works under a causal mask, which autoregressive use requires §5.1.4.
• Flow log-determinant signs depend on direction. tab. 3 gives opposite signs for MAF and IAF, and does not say which direction each map is defined in. eq. 30b uses base-to-data composition, so check direction before implementing either.
• i-ResNet log-determinant is truncated. tab. 3 gives the power series, but the number of terms and the trace estimator are not stated §6.3.2.
• Continuous-flow notation is inconsistent. eq. 42 integrates over \mathbf{z}(t) while the left side uses \mathbf{x}(t), and eq. 43 integrates from t_1 to t_0. Solver and tolerance are not stated §6.6.
• Wording about posteriors in continuous flows. The text says the true posterior "can be recovered" as flow length grows §6.6. The paper does not define this sense of "posterior", so treat the claim as loose.
• Scope is images. The paper states its focus is on image models §1, and its applications section is brief §8.

Lineage

• Builds on: none; this is a survey of the families in §2 to §6.
• Compared against: itself. tab. 1 compares its own list of methods on CIFAR-10.
• Surveys cited in §1 (names only, no cards here): flows, GANs, energy-based models, and a general 2018 survey. Their content is not used in this card.
• Superseded by: not known from content already in this wiki.