Normalizing flows

Overview

A normalizing flow maps data to a simple base distribution, usually a standard normal, through a chain of invertible steps 2008.02401 §4. Each step's log-determinant term is tracked, so the density of any point is exact 2008.02401 eq. 4. Running the chain backwards turns base samples into new points 2008.02401 §6.1.

The shared invariant is that every step can be inverted and its effect on the density is known. The four method cards use it differently. StyleFlow trains its flow for likelihood on latent codes. ArtFlow uses only the invertibility: its training objective is a content and style loss, not a likelihood 2103.16877 §5.1. CLIP-Forge trains its flow for likelihood on shape embeddings, conditioned on CLIP features of rendered views 2110.02624 §3.2. CFLOW-AD uses the flow on local patch features from a frozen encoder, one flow per feature scale, and scores each location by its likelihood 2107.12571 §4.2.

There are two families. Discrete flows stack a fixed number of invertible layers, such as actnorm, a 1×1 convolution and a coupling step 2103.16877 §4.2. CLIP-Forge's discrete flow uses RealNVP affine coupling, with the condition in every layer 2110.02624 §3.2. CFLOW-AD also uses discrete coupling blocks, eight per scale, each conditioned on a 2D position code 2107.12571 §4.2 2107.12571 §5.3. Continuous flows define the map as the solution of an ODE, and the change in log-density becomes an integral of a trace 2008.02401 eq. 7 2008.02401 eq. 8.

The survey 2103.04922 §6 groups the wider family in more detail. It adds coupling layers that update half the inputs from the other half, the autoregressive forms MAF and IAF, and residual flows whose inverse is found by fixed-point iteration 2103.04922 §6.1 2103.04922 §6.1.3 2103.04922 §6.3.2.

Variants

Continuous conditional flow on a GAN's latent space (StyleFlow)

StyleFlow fits one flow between StyleGAN's W latent and a standard normal. The flow is conditioned on a 17-entry attribute vector, and all attributes are trained jointly 2008.02401 §3 2008.02401 §5.1. The authors report that training on one attribute at a time gave entangled vector fields, and that joint training keeps identity and unedited attributes better 2008.02401 §7.6 2008.02401 fig. 12. Likelihood training uses an ODE solver with the adjoint method and a Hutchinson estimate of the trace 2008.02401 §5.2 2008.02401 §5.3.

Editing has three steps. The image is projected to W+, its base code is found under the source attributes, and the code is decoded under the target attributes. Only the W+ rows of the edit are replaced; the others stay from the source 2008.02401 §6.2 2008.02401 §6.2.3. For sequential edits, the paper's fast mode skips re-projection and can produce sudden jumps, while the accurate mode re-projects all 18 vectors after each edit 2008.02401 §6.2.

Author-reported: on sequential edits of generated images, FID is 53.15 against 64.69 for GANSpace 2008.02401 tab. 1, and the face-recognition accuracy for identity is 0.950 against 0.550 for GANSpace 2008.02401 tab. 2. Image2StyleGAN and InterFaceGAN were retrained or re-implemented by the authors; GANSpace used its official code 2008.02401 §7.3. Inference takes 0.61 s with four CNF blocks and 0.21 s with six, on the authors' hardware. The paper calls six blocks a faster option with a slight loss of disentanglement 2008.02401 tab. 4 2008.02401 §5.3. Card: StyleFlow. The paper is CC BY 4.0, but its released code is non-commercial, so this page uses only results the paper reports.

Discrete coupling flow as an invertible feature transform (ArtFlow)

ArtFlow replaces the encoder and decoder of a style transfer model with one reversible network, the PFN 2103.16877 §4.1. It has two blocks of eight flows 2103.16877 §5.1 2103.16877 fig. 6. Each flow is an actnorm, a 1×1 convolution and an additive coupling step. The authors chose additive coupling over affine coupling for efficiency and training stability 2103.16877 §4.2. The network follows the Glow model 2103.16877 §4.2.

The pipeline projects the content and style images forward, applies AdaIN or WCT to the latent features, and runs the reverse pass to get the output 2103.16877 §4.1. The authors call the forward pass lossless. In a 20-round test where each output is fed back in as content, ArtFlow with AdaIN or WCT shows no visible content leak, while plain AdaIN and WCT lose content 2103.16877 §5.3 2103.16877 fig. 10.

Author-reported at 512×512: ArtFlow+AdaIN reaches SSIM 0.45 against 0.29 for AdaIN, at 0.408 s against 0.064 s 2103.16877 tab. 1. ArtFlow+WCT takes 0.428 s against 0.997 s for WCT 2103.16877 tab. 1. ArtFlow is not the best on Gram loss; OptimalWCT is 2103.16877 tab. 1. Among four flow layouts, the two-block, eight-flow layout gives the authors' preferred trade-off between SSIM and Gram loss 2103.16877 app. E 2103.16877 tab. 3. Card: ArtFlow.

Conditional coupling flow as a prior over shape embeddings (CLIP-Forge)

CLIP-Forge does not use the flow on pixels or on GAN codes. A voxel autoencoder first maps each shape to a 128-d embedding, and a conditional RealNVP flow is then fitted between a unit Gaussian and those embeddings 2110.02624 §3.1 2110.02624 §3.2. The condition is a frozen CLIP image feature of a rendered view. It enters the scale and translation networks of every coupling layer 2110.02624 eq. 2 2110.02624 eq. 3. The paper's main setting uses the dimension-wise mask rather than the checkerboard mask 2110.02624 §4 2110.02624 tab. 4.

Training uses image features only. At inference, the CLIP text feature replaces the image feature, and the flow is inverted on Gaussian noise to give shape embeddings, which a decoder turns into occupancy values 2110.02624 §3.3. No per-prompt optimisation is needed 2110.02624 §1.

Author-reported: among the stage 2 prior designs, the dimension-wise mask gave FID 2591.87, against 5227.32 when the condition went to the prior network and 6052.62 for MAF 2110.02624 tab. 4. These are means over three seeds. The headline FID of 2425.25 comes from the best seed 2110.02624 app. A 2110.02624 tab. 1. Card: CLIP-Forge.

Conditional discrete flow for anomaly localization (CFLOW-AD)

CFLOW-AD is trained on anomaly-free images only. A frozen ImageNet-pretrained Wide-ResNet-50 gives features at three scales 2107.12571 §4.1 2107.12571 §5.3. Each scale has its own flow of eight coupling blocks, conditioned on a 128-d sin/cos position code, so the flow can model features that depend on where they sit in the image 2107.12571 §4.2 2107.12571 §5.3.

At test time each location gets a log-likelihood per scale. These are turned into probabilities, upsampled to the input size and summed over scales. The anomaly score is the maximum of the summed map minus the map, so high values mean anomalous 2107.12571 §4.2 2107.12571 eq. 8.

The paper's ablations isolate the flow design. With WRN50 at 512 input, the conditional flow averages 97.36 localization AUROC against 96.86 for an unconditional flow 2107.12571 tab. 2. Eight coupling blocks beat four at two scales, 96.46 against 96.31 2107.12571 tab. 2, and three scales beat two, 97.87 against 96.46 at 256 input 2107.12571 tab. 2.

Author-reported on MVTec AD, averaged over 15 classes: image AUROC 98.26, pixel AUROC 98.62 and AUPRO 94.60. The strongest prior model in that table, PaDiM, reports 97.5 pixel AUROC and 92.1 AUPRO 2107.12571 tab. 4. Speed is 27 frames per second at 256 input with WRN50 and 9 at 512 2107.12571 tab. 6. The paper compares these GPU timings with SPADE and PaDiM timings taken from other hardware, so the speed gap is not a clean comparison 2107.12571 §5.5. Card: CFLOW-AD.

Residual and continuous flows (survey view)

None of the four method cards builds these two families. The survey describes residual flows as a map of the form x + f(x), where f is Lipschitz below one. Its inverse is a fixed-point iteration, and its log-determinant is a power series whose number of terms the survey does not state 2103.04922 tab. 3 2103.04922 §6.3.2. Continuous flows define the map with a neural ODE, get gradients through the adjoint method, and use FFJORD as the example 2103.04922 §6.6 2103.04922 eq. 41.

The survey's CIFAR-10 table lists bits per dimension for several flows, each copied from its cited work: GLOW 3.35, RealNVP 3.49, FFJORD 3.40 and Residual Flow 3.28. It marks GLOW and RealNVP as exact-density, and FFJORD and Residual Flow as approximate. The survey does not state training budgets per row, so these values are not a fair ranking 2103.04922 tab. 1. Card: Deep Generative Modelling survey.

When to use which

Implementation sketch

Discrete and continuous flows share the same likelihood: a base term minus the accumulated log-determinant 2008.02401 eq. 4 2008.02401 eq. 8.

# Discrete flow: exact log-likelihood by change of variables [eq. 4]
# Input: x (B, D); layers, a list of invertible steps
# Output: log p(x), shape (B,)

def discrete_log_likelihood(x, layers):
    # Pass x through each invertible step, adding its log-det [eq. 4]
    z, log_det = x, zeros(B)
    for layer in layers:
        z, step_log_det = layer.forward(z)
        log_det = log_det + step_log_det
    # Score the base point under N(0, I), minus the accumulated log-det [eq. 4]
    return standard_normal_logpdf(z) - log_det

# Continuous flow: same quantity, integrated over time [eq. 7] [eq. 8]
# Trace estimated with Hutchinson's method [§5.2]

def continuous_log_likelihood(x, velocity, t0, t1):
    # Integrate the state and the trace of the Jacobian together [eq. 8]
    z, trace_integral = odeint_with_trace(velocity, x, t0, t1)
    # Same base term as the discrete case, minus the trace integral [eq. 8]
    return standard_normal_logpdf(z) - trace_integral

A conditional discrete flow such as CLIP-Forge's adds the condition to the input of each scale and translation network. The log-determinant is still the sum of the log-scales 2110.02624 §3.2 2110.02624 eq. 3. CFLOW-AD uses the same per-location log-likelihood as its anomaly score, with one flow per scale 2107.12571 eq. 8 2107.12571 §4.2.

# Additive coupling step, forward and exact inverse [§4.2] [arxiv:2103.16877]
# Input: x (B, C, H, W) with C even

def coupling_forward(x, net):
    # Keep the first half of the channels unchanged [§4.2]
    x_a, x_b = split_channels(x)
    # Shift the second half by a function of the first [§4.2]
    return concat_channels(x_a, x_b + net(x_a))

def coupling_reverse(y, net):
    # Subtract the same shift; net need not be invertible [§4.2]
    y_a, y_b = split_channels(y)
    return concat_channels(y_a, y_b - net(y_a))

Autoregressive flows need a direction check before implementation. The survey's table gives MAF and IAF log-determinants with opposite signs, and it does not say which way each map runs 2103.04922 tab. 3 2103.04922 §6.1.3.

Contradictions

None known between the papers. They use different tasks, metrics and baselines, so their numbers do not compare. Within ArtFlow, a single-block layout has higher SSIM but worse Gram loss than the selected layout 2103.16877 tab. 3. The authors chose the trade-off; it is not a conflict between sources.

Within CLIP-Forge, the RealNVP-D row of the prior ablation gives FID 2591.87 2110.02624 tab. 4, while the headline result for the same design gives 2425.25 2110.02624 tab. 1. The paper attributes the gap to seed selection: ablations average three seeds, and the main section reports the best seed 2110.02624 app. A.

CFLOW-AD's ablation and headline numbers are not directly comparable. The ablation reports 97.87 for WRN50 at 256 input with three scales 2107.12571 tab. 2. The headline 98.62 uses the best input size for each class, chosen from that ablation 2107.12571 tab. 4 2107.12571 §5.2. The released code keeps the best epoch on the test set, so the headline may be optimistic; see the card.

The survey's CIFAR-10 table does not conflict with the four cards. It covers other tasks and other works, and it is not like-for-like across rows 2103.04922 tab. 1 2103.04922 tab. 2. Its star ratings are the authors' subjective mapping, so they are not measurements either.

Papers

• StyleFlow: a conditional continuous flow on StyleGAN's W latent, used for attribute sampling and editing.
• ArtFlow: a Glow-style coupling flow as the lossless feature transform for style transfer.
• CLIP-Forge: a conditional RealNVP prior over shape embeddings, conditioned on frozen CLIP features, for zero-shot text-to-shape.
• CFLOW-AD: one conditional discrete flow per feature scale on a frozen encoder, scoring anomaly locations from normal images only.
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models: a survey that places flows beside other generative families and tabulates their layer forms.