---
kind: concept
slug: normalizing-flows
title: Normalizing flows
summary: >-
  Normalizing flows are invertible maps with a tractable Jacobian, so density and inverse are both exact. Five cards: a continuous flow that edits StyleGAN latents, a coupling flow for style transfer, a conditional RealNVP prior for text-to-shape, a conditional flow for anomaly maps, and a survey of the families. Trade-off: exact invertibility against compute per step.
tags: [generative-models, invertible-networks, density-estimation, image-editing, style-transfer, 3d-generation, anomaly-detection, latency]
papers: ["2008.02401", "2103.16877", "2110.02624", "2107.12571", "2103.04922"]
related: [latent-space-editing, neural-style-transfer, adaptive-instance-normalization, generative-adversarial-network, diffusion-models, image-to-image-translation, neural-implicit-surface, vision-language-pretraining, anomaly-detection, positional-encoding, variational-autoencoder, autoregressive-image-generation, flow-matching]
status: draft
updated_at: 2026-10-31
---

## Overview

A normalizing flow maps data to a simple base distribution, usually a standard
normal, through a chain of invertible steps [arxiv:2008.02401 §4]. Each step's
log-determinant term is tracked, so the density of any point is exact
[arxiv:2008.02401 eq. 4]. Running the chain backwards turns base samples into
new points [arxiv:2008.02401 §6.1].

The shared invariant is that every step can be inverted and its effect on the
density is known. The four method cards use it differently. StyleFlow trains its flow
for likelihood on latent codes. ArtFlow uses only the invertibility: its
training objective is a content and style loss, not a likelihood
[arxiv:2103.16877 §5.1]. CLIP-Forge trains its flow for likelihood on shape
embeddings, conditioned on CLIP features of rendered views
[arxiv:2110.02624 §3.2]. CFLOW-AD uses the flow on local patch features from a
frozen encoder, one flow per feature scale, and scores each location by its
likelihood [arxiv:2107.12571 §4.2].

There are two families. Discrete flows stack a fixed number of invertible
layers, such as actnorm, a 1×1 convolution and a coupling step
[arxiv:2103.16877 §4.2]. CLIP-Forge's discrete flow uses RealNVP affine
coupling, with the condition in every layer [arxiv:2110.02624 §3.2]. CFLOW-AD
also uses discrete coupling blocks, eight per scale, each conditioned on a
2D position code [arxiv:2107.12571 §4.2] [arxiv:2107.12571 §5.3].
Continuous flows define the map as the solution of an
ODE, and the change in log-density becomes an integral of a trace
[arxiv:2008.02401 eq. 7] [arxiv:2008.02401 eq. 8].

The survey [arxiv:2103.04922 §6] groups the wider family in more detail. It
adds coupling layers that update half the inputs from the other half, the
autoregressive forms MAF and IAF, and residual flows whose inverse is found by
fixed-point iteration [arxiv:2103.04922 §6.1] [arxiv:2103.04922 §6.1.3]
[arxiv:2103.04922 §6.3.2].

## Variants

### Continuous conditional flow on a GAN's latent space (StyleFlow)

StyleFlow fits one flow between StyleGAN's W latent and a standard normal. The
flow is conditioned on a 17-entry attribute vector, and all attributes are
trained jointly [arxiv:2008.02401 §3] [arxiv:2008.02401 §5.1]. The authors
report that training on one attribute at a time gave entangled vector fields,
and that joint training keeps identity and unedited attributes better
[arxiv:2008.02401 §7.6] [arxiv:2008.02401 fig. 12]. Likelihood training uses an
ODE solver with the adjoint method and a Hutchinson estimate of the trace
[arxiv:2008.02401 §5.2] [arxiv:2008.02401 §5.3].

Editing has three steps. The image is projected to W+, its base code is found
under the source attributes, and the code is decoded under the target
attributes. Only the W+ rows of the edit are replaced; the others stay from the
source [arxiv:2008.02401 §6.2] [arxiv:2008.02401 §6.2.3]. For sequential edits,
the paper's fast mode skips re-projection and can produce sudden jumps, while
the accurate mode re-projects all 18 vectors after each edit
[arxiv:2008.02401 §6.2].

Author-reported: on sequential edits of generated images, FID is 53.15 against
64.69 for GANSpace [arxiv:2008.02401 tab. 1], and the face-recognition accuracy
for identity is 0.950 against 0.550 for GANSpace [arxiv:2008.02401 tab. 2].
Image2StyleGAN and InterFaceGAN were retrained or re-implemented by the
authors; GANSpace used its official code [arxiv:2008.02401 §7.3]. Inference
takes 0.61 s with four CNF blocks and 0.21 s with six, on the authors'
hardware. The paper calls six blocks a faster option with a slight loss of
disentanglement [arxiv:2008.02401 tab. 4] [arxiv:2008.02401 §5.3]. Card:
[StyleFlow](/papers/2008.02401).
The paper is CC BY 4.0, but its released code is non-commercial, so this page
uses only results the paper reports.

### Discrete coupling flow as an invertible feature transform (ArtFlow)

ArtFlow replaces the encoder and decoder of a style transfer model with one
reversible network, the PFN [arxiv:2103.16877 §4.1]. It has two blocks of
eight flows [arxiv:2103.16877 §5.1] [arxiv:2103.16877 fig. 6]. Each flow is an
actnorm, a 1×1 convolution and an additive coupling step. The authors chose
additive coupling over affine coupling for efficiency and training stability
[arxiv:2103.16877 §4.2]. The network follows the Glow model
[arxiv:2103.16877 §4.2].

The pipeline projects the content and style images forward, applies AdaIN or
WCT to the latent features, and runs the reverse pass to get the output
[arxiv:2103.16877 §4.1]. The authors call the forward pass lossless. In a
20-round test where each output is fed back in as content, ArtFlow with AdaIN
or WCT shows no visible content leak, while plain AdaIN and WCT lose content
[arxiv:2103.16877 §5.3] [arxiv:2103.16877 fig. 10].

Author-reported at 512×512: ArtFlow+AdaIN reaches SSIM 0.45 against 0.29 for
AdaIN, at 0.408 s against 0.064 s [arxiv:2103.16877 tab. 1]. ArtFlow+WCT takes
0.428 s against 0.997 s for WCT [arxiv:2103.16877 tab. 1]. ArtFlow is not the
best on Gram loss; OptimalWCT is [arxiv:2103.16877 tab. 1]. Among four flow
layouts, the two-block, eight-flow layout gives the authors' preferred
trade-off between SSIM and Gram loss [arxiv:2103.16877 app. E]
[arxiv:2103.16877 tab. 3]. Card:
[ArtFlow](/papers/2103.16877).

### Conditional coupling flow as a prior over shape embeddings (CLIP-Forge)

CLIP-Forge does not use the flow on pixels or on GAN codes. A voxel autoencoder
first maps each shape to a 128-d embedding, and a conditional RealNVP flow is
then fitted between a unit Gaussian and those embeddings [arxiv:2110.02624 §3.1]
[arxiv:2110.02624 §3.2]. The condition is a frozen CLIP image feature of a
rendered view. It enters the scale and translation networks of every coupling
layer [arxiv:2110.02624 eq. 2] [arxiv:2110.02624 eq. 3]. The paper's main
setting uses the dimension-wise mask rather than the checkerboard mask
[arxiv:2110.02624 §4] [arxiv:2110.02624 tab. 4].

Training uses image features only. At inference, the CLIP text feature replaces
the image feature, and the flow is inverted on Gaussian noise to give shape
embeddings, which a decoder turns into occupancy values [arxiv:2110.02624 §3.3].
No per-prompt optimisation is needed [arxiv:2110.02624 §1].

Author-reported: among the stage 2 prior designs, the dimension-wise mask gave
FID 2591.87, against 5227.32 when the condition went to the prior network and
6052.62 for MAF [arxiv:2110.02624 tab. 4]. These are means over three seeds.
The headline FID of 2425.25 comes from the best seed [arxiv:2110.02624 app. A]
[arxiv:2110.02624 tab. 1]. Card:
[CLIP-Forge](/papers/2110.02624).

### Conditional discrete flow for anomaly localization (CFLOW-AD)

CFLOW-AD is trained on anomaly-free images only. A frozen ImageNet-pretrained
Wide-ResNet-50 gives features at three scales [arxiv:2107.12571 §4.1]
[arxiv:2107.12571 §5.3]. Each scale has its own flow of eight coupling blocks,
conditioned on a 128-d sin/cos position code, so the flow can model features
that depend on where they sit in the image [arxiv:2107.12571 §4.2]
[arxiv:2107.12571 §5.3].

At test time each location gets a log-likelihood per scale. These are turned
into probabilities, upsampled to the input size and summed over scales. The
anomaly score is the maximum of the summed map minus the map, so high values
mean anomalous [arxiv:2107.12571 §4.2] [arxiv:2107.12571 eq. 8].

The paper's ablations isolate the flow design. With WRN50 at 512 input, the
conditional flow averages 97.36 localization AUROC against 96.86 for an
unconditional flow [arxiv:2107.12571 tab. 2]. Eight coupling blocks beat four
at two scales, 96.46 against 96.31 [arxiv:2107.12571 tab. 2], and three
scales beat two, 97.87 against 96.46 at 256 input [arxiv:2107.12571 tab. 2].

Author-reported on MVTec AD, averaged over 15 classes: image AUROC 98.26,
pixel AUROC 98.62 and AUPRO 94.60. The strongest prior model in that table,
PaDiM, reports 97.5 pixel AUROC and 92.1 AUPRO [arxiv:2107.12571 tab. 4]. Speed is 27 frames per
second at 256 input with WRN50 and 9 at 512 [arxiv:2107.12571 tab. 6]. The
paper compares these GPU timings with SPADE and PaDiM timings taken from other
hardware, so the speed gap is not a clean comparison [arxiv:2107.12571 §5.5].
Card: [CFLOW-AD](/papers/2107.12571).

### Residual and continuous flows (survey view)

None of the four method cards builds these two families. The survey describes
residual flows as a map of the form x + f(x), where f is Lipschitz below one.
Its inverse is a fixed-point iteration, and its log-determinant is a power
series whose number of terms the survey does not state
[arxiv:2103.04922 tab. 3] [arxiv:2103.04922 §6.3.2]. Continuous flows define
the map with a neural ODE, get gradients through the adjoint method, and use
FFJORD as the example [arxiv:2103.04922 §6.6] [arxiv:2103.04922 eq. 41].

The survey's CIFAR-10 table lists bits per dimension for several flows, each
copied from its cited work: GLOW 3.35, RealNVP 3.49, FFJORD 3.40 and Residual
Flow 3.28. It marks GLOW and RealNVP as exact-density, and FFJORD and Residual
Flow as approximate. The survey does not state training budgets per row, so
these values are not a fair ranking [arxiv:2103.04922 tab. 1]. Card:
[Deep Generative Modelling survey](/papers/2103.04922).

## When to use which

| Situation | Use | Why |
|---|---|---|
| Exact likelihood over a continuous latent, with conditions, and ODE solves at sampling time are acceptable | [Continuous conditional flow](#continuous-conditional-flow-on-a-gans-latent-space-styleflow) | Trained for exact likelihood with a trace term [arxiv:2008.02401 §5.2]; 0.61 s inference with four blocks [arxiv:2008.02401 tab. 4] |
| Editing a pretrained GAN without retraining its generator, and attribute labels can be produced | [Continuous conditional flow](#continuous-conditional-flow-on-a-gans-latent-space-styleflow) | The flow is trained on W codes [arxiv:2008.02401 §3]. Labels came from a face-attribute service and a lighting estimator [arxiv:2008.02401 §5.1] |
| An exactly invertible feature transform inside a stylization pipeline, with no learned decoder | [Discrete coupling flow](#discrete-coupling-flow-as-an-invertible-feature-transform-artflow) | Reverse pass is exact [arxiv:2103.16877 §4.1]; no visible leak over 20 rounds [arxiv:2103.16877 §5.3]; 0.408 s against 0.064 s for AdaIN [arxiv:2103.16877 tab. 1] |
| Latency dominates and repeated transfer is not needed | [Adaptive instance normalization](/concepts/adaptive-instance-normalization) with a learned decoder | 0.064 s and SSIM 0.29 [arxiv:2103.16877 tab. 1]; the paper reports content leak under repeated transfer [arxiv:2103.16877 §5.3] |
| A conditional density over a learned embedding, sampled in one feed-forward pass, with a frozen encoder supplying the condition, and paired (shape, text) data are scarce | [Conditional coupling flow](#conditional-coupling-flow-as-a-prior-over-shape-embeddings-clip-forge) | Trained on image features only and run with text features at inference [arxiv:2110.02624 §3.3]; RealNVP-D beat MAF in the paper's prior ablation, 2591.87 against 6052.62 FID [arxiv:2110.02624 tab. 4]. The paper does not measure the text-image gap |
| Sequential edits must not drift from the source | StyleFlow accurate mode | Fast mode can jump; accurate mode re-projects after every edit [arxiv:2008.02401 §6.2]. The paper does not state its cost |
| Per-location anomaly scores and heat maps, trained on normal images only, with a frozen pretrained encoder and a GPU at test time | [CFLOW-AD](#conditional-discrete-flow-for-anomaly-localization-cflow-ad) | Trained without anomalies and scored per location [arxiv:2107.12571 §1] [arxiv:2107.12571 §4.2]; 98.62 pixel AUROC on MVTec AD, author-reported [arxiv:2107.12571 tab. 4]; 27 fps at 256 input with WRN50 [arxiv:2107.12571 tab. 6]. The released code normalises by the test-set maximum, so scores change with the test set; see [the card](/papers/2107.12571) |
| Choosing a flow family before reading papers, or comparing flows with other generative families | [Residual and continuous flows (survey view)](#residual-and-continuous-flows-survey-view) and the [survey card](/papers/2103.04922) | The survey tabulates layer forms and their log-determinants [arxiv:2103.04922 tab. 3] and marks which families have exact density [arxiv:2103.04922 tab. 1]. Its numbers are compiled from cited works, not re-measured |
| Limited memory at test time, where a stored gallery or per-position covariance is too large | CFLOW-AD with a smaller encoder | The flow is the only learned state beyond the encoder [arxiv:2107.12571 tab. 1]. The table lists 25 MB for MobileNetV3-L and 1,400 MB for SPADE with WRN50 on MVTec [arxiv:2107.12571 tab. 6]. The MobileNetV3-L row is 98.16 average AUROC at 512 input, above WRN50's 97.36 at the same input [arxiv:2107.12571 tab. 2] |

## Implementation sketch

Discrete and continuous flows share the same likelihood: a base term minus the
accumulated log-determinant [arxiv:2008.02401 eq. 4] [arxiv:2008.02401 eq. 8].

```pseudocode
# Discrete flow: exact log-likelihood by change of variables [eq. 4]
# Input: x (B, D); layers, a list of invertible steps
# Output: log p(x), shape (B,)

def discrete_log_likelihood(x, layers):
    # Pass x through each invertible step, adding its log-det [eq. 4]
    z, log_det = x, zeros(B)
    for layer in layers:
        z, step_log_det = layer.forward(z)
        log_det = log_det + step_log_det
    # Score the base point under N(0, I), minus the accumulated log-det [eq. 4]
    return standard_normal_logpdf(z) - log_det
```

```pseudocode
# Continuous flow: same quantity, integrated over time [eq. 7] [eq. 8]
# Trace estimated with Hutchinson's method [§5.2]

def continuous_log_likelihood(x, velocity, t0, t1):
    # Integrate the state and the trace of the Jacobian together [eq. 8]
    z, trace_integral = odeint_with_trace(velocity, x, t0, t1)
    # Same base term as the discrete case, minus the trace integral [eq. 8]
    return standard_normal_logpdf(z) - trace_integral
```

A conditional discrete flow such as CLIP-Forge's adds the condition to the
input of each scale and translation network. The log-determinant is still the
sum of the log-scales [arxiv:2110.02624 §3.2] [arxiv:2110.02624 eq. 3].
CFLOW-AD uses the same per-location log-likelihood as its anomaly score, with
one flow per scale [arxiv:2107.12571 eq. 8] [arxiv:2107.12571 §4.2].

```pseudocode
# Additive coupling step, forward and exact inverse [§4.2] [arxiv:2103.16877]
# Input: x (B, C, H, W) with C even

def coupling_forward(x, net):
    # Keep the first half of the channels unchanged [§4.2]
    x_a, x_b = split_channels(x)
    # Shift the second half by a function of the first [§4.2]
    return concat_channels(x_a, x_b + net(x_a))

def coupling_reverse(y, net):
    # Subtract the same shift; net need not be invertible [§4.2]
    y_a, y_b = split_channels(y)
    return concat_channels(y_a, y_b - net(y_a))
```

Autoregressive flows need a direction check before implementation. The
survey's table gives MAF and IAF log-determinants with opposite signs, and it
does not say which way each map runs [arxiv:2103.04922 tab. 3]
[arxiv:2103.04922 §6.1.3].

## Contradictions

None known between the papers. They use different tasks, metrics and
baselines, so their numbers do not compare. Within ArtFlow, a single-block
layout has higher SSIM but worse Gram loss than the selected layout
[arxiv:2103.16877 tab. 3]. The authors chose the trade-off; it is not a
conflict between sources.

Within CLIP-Forge, the RealNVP-D row of the prior ablation gives FID 2591.87
[arxiv:2110.02624 tab. 4], while the headline result for the same design gives
2425.25 [arxiv:2110.02624 tab. 1]. The paper attributes the gap to seed
selection: ablations average three seeds, and the main section reports the best
seed [arxiv:2110.02624 app. A].

CFLOW-AD's ablation and headline numbers are not directly comparable. The
ablation reports 97.87 for WRN50 at 256 input with three scales
[arxiv:2107.12571 tab. 2]. The headline 98.62 uses the best input size
for each class, chosen from that ablation [arxiv:2107.12571 tab. 4]
[arxiv:2107.12571 §5.2]. The released code keeps the best epoch on the test
set, so the headline may be optimistic; see [the card](/papers/2107.12571).

The survey's CIFAR-10 table does not conflict with the four cards. It covers
other tasks and other works, and it is not like-for-like across rows
[arxiv:2103.04922 tab. 1] [arxiv:2103.04922 tab. 2]. Its star ratings are the
authors' subjective mapping, so they are not measurements either.

## Papers

- [StyleFlow](/papers/2008.02401): a conditional continuous flow on StyleGAN's W latent, used for attribute sampling and editing.
- [ArtFlow](/papers/2103.16877): a Glow-style coupling flow as the lossless feature transform for style transfer.
- [CLIP-Forge](/papers/2110.02624): a conditional RealNVP prior over shape embeddings, conditioned on frozen CLIP features, for zero-shot text-to-shape.
- [CFLOW-AD](/papers/2107.12571): one conditional discrete flow per feature scale on a frozen encoder, scoring anomaly locations from normal images only.
- [Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models](/papers/2103.04922): a survey that places flows beside other generative families and tabulates their layer forms.