An Introduction to Deep Generative Modeling

In plain words

A generative model learns to make new examples that look like the examples it was trained on. This tutorial starts from one idea: take a simple random input, a standard Gaussian, and run it through a neural network that turns it into something realistic. The hard part is training that network without knowing which random input should produce which real example.

The paper compares three ways to train such a network. Flows use a network that can be run backwards, so every real example can be traced back to its random input and its probability computed exactly. Variational autoencoders add a second network that guesses the random input for each example, and train both with a score that rewards good reconstructions. Generative adversarial networks skip the guessing altogether: a second network learns to tell real examples from fakes, and the generator tries to fool it.

Each approach has a cost. Flows need the input and output to have the same size. The VAE samples in the paper look blurry. Adversarial training is a balancing act that is hard to tune. The paper uses two small test problems to show these trade-offs.

TL;DR

• Tutorial and derivation paper: one generator-from-Gaussian framework, three training families (normalizing flows, VAEs, GANs), with optimal transport as the link between them §1 §2.1.
• Two testbeds: 2-D moons and MNIST, both small enough to rerun §2.2. The paper states no benchmark scores; comparisons are visual plus one energy-distance test fig. 11.
• The authors' MIT-licensed code reproduces the run settings (runAll.sh) closely, but the VAE's sampling and entropy terms use different scale conventions, and the RealNVP log-determinant includes an unused term vae.py, realNVP.py.

Problem

Generative training needs a distribution to match, but the data distribution is known only through samples, and the latent variable behind each sample is unknown §1 §2.3. Three obstacles follow from that: the training problem is ill-posed, so results depend heavily on hyperparameters and design choices §1, item 1; comparing a generated distribution with the data requires either inverting the generator or a two-sample test, and both are hard §1, item 2 §2.3; and the latent dimension is usually unknown, so an overestimate makes the generator non-injective §1, item 3. Flows assume the latent and data dimensions are equal, which is rarely true §6. The paper's setting is g_θ : R^q → R^n with a Gaussian latent Z §2.1 eq. 1.

Assumption: the paper treats the latent Z as a univariate Gaussian in R^p in one sentence and as R^q everywhere else §2.1. The card uses q.

Method

The paper derives three approaches to fit g_θ. Equation numbers were counted from the LaTeX source in document order; the PDF was not read, so cite section numbers where the count is uncertain.

Shared setup §2.1–§2.3. Sample z from the latent prior, push it through the generator, and read off a new sample eq. 1. The exact likelihood p_X(x) is an intractable integral over the latent eq. 2; the paper uses a Gaussian (width σ) or Bernoulli likelihood as the conditional p_g(x|z) eq. 3 eq. 4. The MNIST generator is a three-layer CNN with batch normalisation, ReLU and a sigmoid output eq. 5 §2.2.

# Sample from a trained generator  [§2.1] [eq. 1]
# Input:  generator g(theta, .) : R^q -> R^n, count
# Output: count new samples in R^n

def sample_new(generator, count, q):
    # Draw latents from the prior, a standard Gaussian in R^q  [§2.1]
    z = normal(mean=0, std=1, shape=(count, q))
    # Push each latent through the generator  [eq. 1]
    return generator(z)  # (count, n)

Normalizing flows §3.1–§3.2. Make the generator a diffeomorphism with q = n. The change-of-variables formula then gives the density exactly eq. 6. Training minimises the negative log-likelihood, a Monte Carlo average over the batch eq. 7 eq. 8. For a composition of K layers, the inverse and the log-determinant are sums over layers eq. 9 eq. 10.

# Negative log-likelihood of a flow  [§3.1] [eq. 8] [eq. 10]
# Input:  batch x (s, n); layers f_1..f_K, each with an inverse
# Output: scalar loss

def flow_nll(x, layers):
    # Run the inverses from the data back to the latent  [eq. 9]
    y = x
    logdet = zeros(s)
    for layer in reversed(layers):
        y, layer_logdet = layer.inverse(y)
        logdet = logdet + layer_logdet  # sum of log det of inverses  [eq. 10]
    z = y
    # Gaussian term minus log-determinant, averaged over the batch  [eq. 8]
    return mean(0.5 * sum(z**2, axis=1) - logdet) + n / 2 * log(2 * pi)

Real NVP coupling layer §3.2. The layer keeps one coordinate unchanged and scales and shifts the other using two networks that take the kept coordinate as input eq. 11. Its Jacobian is triangular, so the log-determinant is the scale output eq. 12, and the inverse is closed form eq. 13. The paper alternates which coordinate is kept from layer to layer §3.2.

# Affine coupling layer (real NVP)  [§3.2] [eq. 11] [eq. 12] [eq. 13]
# Input:  y (2,); index of the kept coordinate
# Output: forward image and log|det| for one example

def coupling_forward(y, keep, s_net, t_net):
    # The kept coordinate passes through unchanged  [eq. 11]
    kept = y[keep]
    scale = s_net(kept)            # scalar, s_j(y_1)  [§3.2]
    shift = t_net(kept)            # scalar, t_j(y_1)  [§3.2]
    moved = y[1 - keep] * exp(scale) + shift  # [eq. 11]
    out = assemble(kept, moved, keep)
    # Jacobian determinant is exp(scale)  [eq. 12]
    return out, scale

Continuous normalizing flows §3.3. Define the generator as the end state of an ODE with a learned velocity v_θ eq. 14. The inverse integrates backward eq. 15, and the log-determinant is the integral of the negative divergence of v_θ along the path eq. 16. The regularised objective adds a transport cost to the likelihood, weighted by a scalar α eq. 19. The paper's OT-Flow example uses the gradient of a scalar potential as the velocity and penalises violations of the Hamilton–Jacobi–Bellman conditions §3.3.

# Regularised continuous-flow objective  [§3.3] [eq. 16] [eq. 19]
# Input:  batch x (s, n); velocity v(y, t); weight alpha; horizon T
# Output: scalar objective

def cnf_objective(x, v, alpha, T):
    # Integrate the characteristics backward to the latent  [eq. 15]
    z, logdet_inv = integrate_backward(x, v, T)  # logdet from [eq. 16]
    # Transport cost: integral of 0.5 * |v|^2 along the path  [eq. 19]
    cost = integrate_path_cost(x, v, T)
    # Likelihood term under the Gaussian prior  [eq. 19]
    nll = mean(0.5 * sum(z**2, axis=1) - logdet_inv)
    return cost + alpha * nll

Variational autoencoder §4. The generator is not invertible, so the exact likelihood is replaced by a variational lower bound with an encoder e_ψ(z|x) that approximates the posterior eq. 20 eq. 21. Training minimises the negative bound eq. 22 eq. 23. The encoder is a Gaussian with mean μ and a log-covariance Σ eq. 24, and sampling uses the reparameterisation z = μ + exp(Σ) ε eq. 25. The paper's MNIST example uses one Monte Carlo sample per image §4.2.

# VAE loss with one Monte Carlo sample  [§4.1] [§4.2] [eq. 22] [eq. 25]
# Input:  batch x (s, 28, 28); encoder e_psi; decoder g_theta; latent dim q
# Output: scalar loss

def vae_loss(x, encoder, decoder, q):
    # Encoder gives a Gaussian per image; log-covariance  [eq. 24]
    mu, log_cov = encoder(x)
    eps = normal(mean=0, std=1, shape=mu.shape)
    # Reparameterised sample  [eq. 25]; see Pitfalls on the scale
    z = mu + exp(log_cov) * eps
    # Bernoulli log-likelihood of the pixels  [§4.2] [eq. 4]
    recon = bernoulli_log_likelihood(decoder(z), x)
    # Log-prior of z under the standard Gaussian  [§4.1]
    log_prior = -0.5 * sum(z**2, axis=1) - q / 2 * log(2 * pi)
    # Log-density of the approximate posterior  [eq. 24]
    log_post = -0.5 * sum(eps**2, axis=1) - 0.5 * sum(log_cov, axis=1)
    log_post = log_post - q / 2 * log(2 * pi)
    return mean(-recon - log_prior + log_post)  # [eq. 22]

Generative adversarial networks §5. A discriminator d_φ is trained to separate real samples from generated ones, and the generator minimises the same cross-entropy eq. 29 eq. 30. In alternating stochastic steps, the discriminator ascends its objective eq. 31 and the generator descends on log(1 − d) eq. 32. The Wasserstein variant replaces the discriminator with a 1-Lipschitz critic in the Kantorovich–Rubinstein dual of the Wasserstein-1 distance eq. 34 eq. 35.

# One alternating GAN step  [§5.1] [eq. 29] [eq. 31] [eq. 32]
# Input:  real batch x, latent batch z, discriminator d, generator g
# Output: updated phi (discriminator) and theta (generator)

def gan_step(x, z, d, g, lr_d, lr_g):
    # Discriminator ascends log d(x) + log(1 - d(g(z)))  [eq. 31]
    j_gan = mean(log(d(x))) + mean(log(1 - d(g(z))))
    phi = phi + lr_d * grad(j_gan, phi)
    # Generator descends on log(1 - d(g(z))), with the updated phi  [eq. 32]
    j_gen = mean(log(1 - d(g(z))))
    theta = theta - lr_g * grad(j_gen, theta)

Wasserstein critic §5.2 (the paper gives no pseudocode): constrain the critic's weights to a bound, then run RMSprop steps on the dual objective §5.2 eq. 35.

Hyperparameters

Values marked code are from the authors' repository at code_commit, used by runAll.sh; they are not in the paper. Paths are relative to that repository.

Results

Author-reported. The paper has no benchmark table. Its results are figures on the two testbeds, with qualitative statements; the only quantitative comparison is an energy-distance test curve. Setup: MNIST at the sizes in §2.2; the two baselines compared are the VAE and DCGAN, with the WGAN as a third, all with random or VAE-initialised generators §5.

Paper vs code

The paper links the repository in §1 and §2.2 §1 §2.2. The repository is EmoryMLIP/DeepGenerativeModelingIntro at 7e2d5c0a7883ea00064370f0e2c068a1e84f4400. The owner is an Emory group, and the MIT copyright line names the first author, Lars Ruthotto LICENSE. Code paths below are relative to that repository. The card follows arXiv v2, the version whose source was fetched.

• Run settings match the paper for DCGAN, VAE, RealNVP and OT-Flow. runAll.sh uses batch 64, 50 epochs and width 32 for the VAE runAll.sh, 64 and 50,000 steps for the GANs runAll.sh, 256 and 20,000 for RealNVP runAll.sh. The argument defaults differ (VAE defaults are batch 128, width 4, 2 epochs trainVAEmnist.py), so use the run script values.
• VAE sampling and entropy use different scale conventions. The paper defines the posterior as N(μ, exp(Σ)) eq. 24, which reads exp(Σ) as a covariance, but samples with z = μ + exp(Σ) ε eq. 25, which reads it as a standard deviation. The code samples with std = exp(logvar) vae.py, matching eq. 25, but computes the posterior's log-density with −0.5 Σ logvar vae.py, matching the covariance reading of eq. 24. The two cannot hold together, so the bound is not the one the paper derives. Pick one convention and state it.
• The VAE log-prior constant uses the channel count. n = x.shape[1] is the channel dimension (1 for MNIST) rather than the latent dimension vae.py vae.py. This changes logged ELBO values by a constant and does not change gradients.
• RealNVP parity is the opposite of the paper's text. The paper says that for even j the first coordinate is kept §3.2. The code's first layer (k = 0) keeps the first coordinate trainRealNVP.py realNVP.py; with 1-indexed j, this is the odd case. The model family is the same, but layer indices in a comparison will be off by one.
• RealNVP log-determinant includes an unused term (code reading, not run). Each layer's scale network outputs two values, but only the one for the transformed coordinate is applied realNVP.py. The log-determinant sums both outputs realNVP.py realNVP.py, so the unused output enters the likelihood. The paper's s_j, t_j : R → R §3.2 is scalar, so the code's two-output design is also a departure. Check ∫ p(x) dx ≈ 1 on a grid before trusting likelihood values.
• RealNVP networks end in Tanh. The paper describes three affine layers and does not mention an output activation §3.2. The code adds Tanh to both s and t trainRealNVP.py, which bounds the scale and shift.
• OT-Flow training needs an external package. trainOTFlow.py imports OTFlowProblem from an OT-Flow checkout at a hard-coded local path and raises if it is missing trainOTFlow.py. The README names the external repository as a dependency README.md. The repository's own OTFlow.py has its own integrator OTFlow.py but not the training loss.
• Moons noise is a code default. The paper gives no noise level for the moons data §2.2; the code uses 0.05 trainRealNVP.py.
• WGAN iteration count differs. The paper says "50 iterations of RMSprop" §5.2. The run script trains 50,000 generator steps, each preceded by five critic updates runAll.sh trainWGANmnist.py trainWGANmnist.py. Learning rate 5e-5 trainWGANmnist.py and clip ±0.01 trainWGANmnist.py are not in the paper.
• Discriminator input normalisation is not in the paper. The discriminator maps inputs from [0,1] to [−1,1] before its first layer modelMNIST.py.
• DCGAN generator loss. The code uses the saturating loss log(1 − d(g(z))), matching eq. 32 trainDCGANmnist.py; the non-saturating alternative is commented out trainDCGANmnist.py. The generator step draws a new z trainDCGANmnist.py, but the paper uses the same z^(i) symbol in both steps §5.1.
• ε-test logging. In the WGAN script the logged ε-test is summed over the plotting interval, not averaged trainWGANmnist.py trainWGANmnist.py. Compare curves with the same plot interval.
• Code is MIT. LICENSE

Pitfalls

• Latent dimension. The paper uses q = 2 for MNIST so the latent can be plotted, and says a larger latent may improve image quality §2.2. Choosing q is left open by the paper §1, item 3.
• Flow invertibility. Flows need q = n and an invertible generator §3. The paper says these assumptions are rarely satisfied in practice §6.
• The VAE scale convention (see Paper vs code). An implementer who copies eq. 24 and eq. 25 literally will get a bound that does not match the code. The paper does not resolve it.
• Sampling mismatch in VAEs. Training draws latents from the approximate posterior, but sampling draws from the prior, and the paper finds the posterior is not Gaussian §4.2 fig. 8. The paper says this may cause undesired effects when the generator sees inputs it was not trained on §6.
• GAN balance. Training the discriminator to optimality stops the generator from learning §5.1. The paper reports that changing the batch size can cause training to fail completely, and that results depend on learning rates and architecture §5.1 §6.
• Saddle-point instability. Saddle points are unstable, so a slightly suboptimal generator can let the discriminator win a lot §5.1. Monitoring the GAN objective does not show quality §5.1. The paper's evaluation is visual plus the energy-distance test fig. 11.
• Energy-distance test details. ε and the sample count are not in the paper §5.1. The code uses ε = 0.1 on one batch of 64 generated and 64 real images per step epsTest.py trainDCGANmnist.py.
• Continuous-flow ODE solvers. Backward integration can drift with simple schemes; the paper recommends monitoring inverse errors ‖g⁻¹(g(z)) − z‖ and ‖g(g⁻¹(x)) − x‖ §3.3. The number of solver steps is not stated; the code uses 4 for training and 8 for validation trainOTFlow.py.
• OT-Flow penalty weights. α and the HJB penalty weights are not in the paper §3.3 eq. 19. Use the code defaults only as a starting point; the paper calls the penalty an empirical improvement §3.3.
• Notation. The latent is R^p in one sentence §2.1 and R^q elsewhere.
• Hidden costs. The authors' run script trains each GAN for 50,000 steps at batch 64, and the WGAN run also does five critic updates per step runAll.sh trainWGANmnist.py. The VAE run is 50 epochs runAll.sh. The paper does not report run times.

Lineage

• Builds on: the VAE of Kingma & Welling §4 and Rezende et al. §4, the GAN of Goodfellow et al. §5.1, the Wasserstein GAN of Arjovsky et al. §5.2, the real NVP flow of Dinh et al. §3.2, and the OT-Flow CNF of Onken et al. §3.3. The paper names these by author and year only; this card gives no arXiv ids for them.
• Compared against: the paper compares its three families against each other on the same MNIST generator §5 and against the VAE-initialised GAN §5.1 §5.2.
• Related directions the paper names but does not test: normalizing flows in the VAE's latent space §6, and variational-inequality training for GANs §6.