ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows

In plain words

Style transfer repaints a photo in the look of a painting. Many fast methods do it by encoding the image with a fixed network, changing its features to match the style, and decoding them back to pixels. The decoder never inverts the encoder perfectly, so if you feed the output back in again and again, the picture slowly loses its content. The paper calls this content leak.

ArtFlow replaces the encoder-decoder pair with one network built from steps that can be undone exactly. Running it forward turns an image into features; running the same network backwards turns features into an image, with nothing lost on the way. The style is applied to the features in the middle, and the backward pass produces the stylized picture.

The point is that the content survives repeated stylization, because the only step that changes the picture is the style step. The results on the card are the authors' own measurements.

TL;DR

Universal style transfer (one trained model for arbitrary style images) whose outputs lose content when stylization is repeated; the authors call this content leak §3.1 fig. 1. ArtFlow replaces the usual encoder-decoder with a reversible flow network (PFN) in a projection-transfer-reversion pipeline, and applies AdaIN or WCT to the latent features §4.1. Author-reported: SSIM 0.45 for ArtFlow+AdaIN against 0.29 for AdaIN, and the most user-study votes (314 of 799) tab. 1 tab. 2. The official code departs from the text in batch size, iteration count, the loss distance and the WCT path, so read Paper vs code before implementing.

Problem

The universal methods AdaIN, WCT and Avatar-Net share three parts: a fixed VGG19 encoder, a learned decoder that maps features back to pixels, and a feature transfer module §3.1 §1. The authors attribute content leak to three causes §3.2:

• Reconstruction error. The decoder cannot invert the VGG features exactly, because pooling discards spatial information. Errors accumulate over rounds §3.2.
• Biased decoder training. AdaIN's decoder is trained with a style loss on top of the content loss, so it trades reconstruction accuracy for artistic effects. Repeated auto-encoding of an image shows the artefacts build up over rounds §3.2 fig. 3 fig. 4.
• Biased transfer module. Avatar-Net's style decorator replaces patches in a way that cannot be undone, so its output is biased towards the style §3.2 fig. 5.

The failure is shown by applying each method 20 times in a row, feeding each output back as the new content §3.1 fig. 1. The authors say the same reasoning implies the problem also affects other state-of-the-art methods §3.1; they do not test those.

Method

ArtFlow runs in three steps, which the authors call projection, transfer and reversion. They replace the encoder-transfer-decoder scheme of earlier methods §4.1 fig. 6 alg. 1:

1. Projection. Content and style images go through the forward pass of the PFN. The authors state this pass is lossless §4.1.
2. Transfer. AdaIN or WCT combines the content and style features in the latent space §4.1 §4.3.
3. Reversion. The reverse pass of the PFN maps the stylized features back to an image §4.1.

The PFN follows the Glow model: a chain of reversible operators §1 §4.2. It has two flow blocks with eight flows each §5.1 fig. 6. Each flow chains three reversible operators: Actnorm, a 1×1 convolution that mixes channels, and a coupling step that shifts half the channels using the other half §4.2. The authors use additive coupling because it is more efficient and more stable in training than affine coupling §4.2. A flow with additive coupling alone is sufficient in their experiments §4.2.

The three operators, in the paper's notation eq. 3 eq. 4 §4.2:

\mathbf y_{i,j} = W\,\mathbf x_{i,j} eq. 3

\mathbf y_{i,j} = \mathbf w \odot \mathbf x_{i,j} + \mathbf b eq. 4

\mathbf y_b = \mathrm{NN}(\mathbf x_a) + \mathbf x_b, \qquad \mathbf y = \mathrm{concat}(\mathbf x_a, \mathbf y_b) §4.2

Here W is a learned c \times c matrix, \mathbf w and \mathbf b are the per-channel scale and bias, and the split \mathbf x_a, \mathbf x_b halves the channels. \mathrm{NN} may be any network that keeps the shape §4.2. The paper says the inverse can be derived easily and does not write it out §4.2.

Transfer. AdaIN normalises each channel of the content features with its own mean and standard deviation, then rescales it with the style's statistics app. B glow_adain.py. For centred features the output is the content divided by its standard deviation, times the style standard deviation app. B. WCT whitens the content features and then colours them with the style covariance §4.3 app. C.

Unbiased transfer. The authors model features as a bilinear product of a content factor and a style factor §4.3. A transfer is unbiased if the content factor of the output equals that of the content input, and the style factor equals that of the style input §4.3. AdaIN is proven unbiased under this definition §4.3 app. B, and so is WCT §4.3 app. C. The Avatar-Net decorator does not fit the bilinear model, and the authors argue it is biased §4.3 §3.2.

Training. The objective uses two losses computed on VGG19 features of the output image §3.2 §5.1:

\mathcal L_c = \lVert F(G(t)) - t \rVert_2 eq. 1

\mathcal L_s = \sum_i \lVert \mu(\phi_i(G(t))) - \mu(\phi_i(s)) \rVert_2 + \sum_i \lVert \sigma(\phi_i(G(t))) - \sigma(\phi_i(s)) \rVert_2 eq. 2

In §3.2, F and G denote the encoder and decoder of the auto-encoder, t is the AdaIN output and \phi_i is a VGG19 layer. The experiments use these two losses as ArtFlow's training objective, with the content weight 0.1 and the style weight 1 §5.1. The text does not redefine t or G for ArtFlow, so see Pitfalls.

# ArtFlow inference: projection, transfer, reversion  [app. A] [alg. 1]
# Input:  content, style: (B, 3, H, W); H and W divisible by 4
#         [code: test.py#L28-L29]
# Output: stylized image (B, 3, H, W)
# Latent: (B, 48, H/4, W/4) after two squeezes
#         [code: glow_adain.py#L308-L316]

def artflow_transfer(content, style, pfn, transfer):
    # Projection: lossless forward pass of the PFN on both images  [§4.1]
    content_latent = pfn.forward(content)
    style_latent = pfn.forward(style)
    # Transfer: AdaIN or WCT on the latents  [§4.1] [§4.3]
    stylized_latent = transfer(content_latent, style_latent)
    # Reversion: reverse PFN pass maps the stylized latent to pixels  [§4.1]
    return pfn.reverse(stylized_latent)

# One PFN flow step, forward  [§4.2] [code: glow_adain.py#L274-L279]
# The code applies Actnorm, then the 1x1 convolution, then coupling.
def flow_forward(x):
    # Actnorm: per-channel scale and shift  [eq. 4]
    # Code form is scale * (x + loc)  [code: glow_adain.py#L105-L112]
    x = actnorm_scale * (x + actnorm_loc)
    # Invertible 1x1 convolution: a c x c matrix over channels  [eq. 3]
    x = conv_1x1(x, weight)
    # Additive coupling: first half kept, second half shifted  [§4.2]
    # NN: conv3x3 to 512 ch, ReLU, conv1x1 512, ReLU, zero-init conv3x3
    # [code: glow_adain.py#L210-L216] [code: glow_adain.py#L236-L238]
    x_a, x_b = split_channels(x)
    x_b = x_b + coupling_net(x_a)
    return concat_channels(x_a, x_b)


# Exact inverse of the same step  [§4.2] [code: glow_adain.py#L281-L287]
def flow_reverse(y):
    y_a, y_b = split_channels(y)
    y_b = y_b - coupling_net(y_a)
    y = conv_1x1(concat_channels(y_a, y_b), inverse(weight))
    return y / actnorm_scale - actnorm_loc

# One training step  [§5.1] [code: train.py#L186-L201]
# Losses use fixed VGG19 features of the output image.
def training_step(content, style, pfn, vgg, optimizer):
    # Forward pass of both batches through the PFN  [§4.2]
    content_latent = pfn.forward(content)
    style_latent = pfn.forward(style)
    # Stylize with AdaIN in latent space, then reverse  [§4.3]
    # [code: train.py#L189]
    stylized = pfn.reverse(adain(content_latent, style_latent))
    # Content loss: relu4_1 features of the stylized image against
    # those of the content image; eq. 1 uses an L2 norm, the code MSE
    # [eq. 1] [code: net.py#L141] [code: net.py#L126]
    loss_c = mse(vgg.relu4_1(stylized), vgg.relu4_1(content))
    # Style loss over relu1_1 to relu4_1: match channel mean and std
    # [eq. 2] [code: net.py#L142-L144] [code: net.py#L128-L134]
    loss_s = 0
    for stylized_f, style_f in zip(vgg_layers(stylized), vgg_layers(style)):
        loss_s += mse(mean(stylized_f), mean(style_f))
        loss_s += mse(std(stylized_f), std(style_f))
    # Weights 0.1 (content) and 1 (style)  [§5.1] [code: train.py#L79-L80]
    loss = 0.1 * loss_c + 1.0 * loss_s
    # Adam with lr = 1e-4 / (1 + 5e-5 * iteration)  [§5.1]
    # [code: train.py#L53-L57] [code: train.py#L164]
    # Gradient-norm clip at 5 (code only)  [code: train.py#L200]
    adam_step(loss, learning_rate(iteration), clip_norm=5)

Hyperparameters

Values marked "code only" are not in the paper and come from the released code.

Results

Author-reported throughout. The comparison set is StyleSwap, AdaIN, WCT, LinearWCT, OptimalWCT, Avatar-Net and Self-Contained §5.2 §5.3. Training uses MS-COCO content and WikiArt style images §5.1. The test set for the metrics is not stated §5.2. SSIM and the content loss follow Yoo et al. §5.2, and the Gram loss is the mean squared error between Gram matrices §5.2. Which layers each metric uses is not stated. Timing is at 512×512 tab. 1.

Quantitative comparison tab. 1:

The authors read the table as follows. ArtFlow has the highest SSIM, which they take as better preservation of content §5.2. StyleSwap has the lowest content loss tab. 1. ArtFlow does not have the lowest Gram loss, which they explain by its focus on content leak §5.2. ArtFlow+AdaIN has a lower Gram loss than AdaIN tab. 1. ArtFlow+AdaIN is slower than AdaIN because the PFN has no pooling, and ArtFlow+WCT is faster than WCT because it needs no multi-level stylization §5.2.

User study tab. 2: the 43 content and 27 style images give 1161 content-style pairs. For each pair, users saw every method's result and chose the most preferable one §6. The paper reports 799 effective votes §6:

Flow architecture ablation app. E tab. 3: the PFN (Flow8-Block2: 8 flows per block, 2 blocks) is compared with three other layouts. The authors conclude it gives the best trade-off between SSIM and Gram loss app. E:

Qualitative results. Repeated transfer over 20 rounds: vanilla AdaIN, WCT, Avatar-Net and Self-Contained lose content; with the PFN in place of the auto-encoder, AdaIN and WCT show no leak. With Avatar-Net the leak is still reduced, though the decorator remains biased §5.3 fig. 10 app. H. Content-style separation: the inverted content factor of ArtFlow+AdaIN carries less style colour than that of AdaIN or WCT §5.4 fig. 11 app. I. Reverse style transfer, which the authors claim only ArtFlow can do, recovers the original content app. E.1 fig. 14. A portrait model (FFHQ content, MetFaces style) is shown qualitatively only §5.2 fig. 9 app. G fig. 20 fig. 21.

Paper vs code

Official code: the paper's title-page footnote links https://github.com/pkuanjie/ArtFlow p. 1. The README calls it the official PyTorch implementation Readme.md. The clone is at commit bbc33ecb745989efc9ef937118c60cbcee4153b2. The repository has no LICENSE file, so its code license is unknown. Nothing is quoted from it; the card cites paths and line numbers only.

Discrepancies, each with both citations:

• Batch size. Paper: 2 §5.1. Code default and both training scripts: 4 train.py train_adain.sh train_wct.sh.
• Iterations. Paper: 60,000 §5.1. Code default: 160,000, and the scripts do not override it train.py. Checkpoints are written every 5,000 iterations and at the end train.py train.py. Which checkpoint the paper reports is not stated.
• Content-loss target. Eq. 1 uses t, the AdaIN output, as the target eq. 1 §3.2. The code compares the stylized image's relu4_1 features with those of the original content image net.py train.py.
• Distance. Eqs. 1 and 2 use L2 norms eq. 1 eq. 2. The code uses mean squared error net.py net.py.
• Style layers. Paper: "a layer" of VGG19 §3.2. Code: four layers, relu1_1 to relu4_1 net.py net.py.
• Operator order. The text lists coupling, the 1×1 convolution, then Actnorm §4.2. The code applies Actnorm, the 1×1 convolution, then coupling glow_adain.py. The code's order is the one that runs; the text list may not be an ordering.
• 1×1 convolution. Paper: a plain learned W eq. 3. Code: LU-decomposed by default glow_adain.py train.py.
• Squeeze placement. Paper: squeezes inserted into "certain parts" of the PFN §4.2. Code: one squeeze at the start of each block, so two in total glow_adain.py glow_adain.py.
• Learning-rate schedule. Paper: decay 5e-5 §5.1. Code: lr / (1 + decay × iteration) train.py.
• Optimizer settings. Paper: Adam §5.1. Code: Adam with the PyTorch default betas and eps train.py.
• Gradient clipping. Paper: not stated. Code: norm clip at 5 train.py.
• WCT batch handling. Paper: whitening and colouring per image §4.3 app. C. Code: the batch and spatial dimensions are flattened into one matrix glow_wct.py glow_wct.py. The WCT training script uses batch 4 train_wct.sh.
• WCT regularisation. The identity term in the content covariance is commented out glow_wct.py. Eigenvalues below 1e-5 are dropped glow_wct.py glow_wct.py.
• Avatar-Net variant. The code has a decorator operator glow_decorator.py. The README documents checkpoints for AdaIN, WCT and portrait only Readme.md. The paper's result for the PFN with the Avatar-Net decorator is qualitative only §5.3 fig. 10, so no number exists to compare against, and no checkpoint is released for it.
• Weights and data. Pretrained models and the VGG19 weights are on Google Drive, not in the repository Readme.md Readme.md. Training data is not included.

Pitfalls

• Batch size and iterations do not match between paper and code (see Paper vs code). Pick one source and state it. The paper's 60,000 iterations and batch 2 are not what the released scripts run.
• Loss distance. Eqs. 1 and 2 use L2 norms, but the code uses mean squared error, which scales the terms differently. The weights 0.1 and 1 apply to whichever form is used, so changing the form changes the balance between content and style. The paper does not discuss this.
• Content target for the ArtFlow loss. Eq. 1 defines the target as t, the AdaIN output, with F and G taken from the auto-encoder of §3.2 §3.2. The text then uses eqs. 1 and 2 as ArtFlow's objective without redefining them §5.1. The code's choice is the content image's features train.py.
• --affine is a trap. The flag is declared with type=bool train.py. bool("False") is True, so --affine False on the command line turns on affine coupling. The default is additive. Use the default.
• --no_lu does not reach the last block. The last block is always built with LU-decomposed 1×1 convolutions, because the call does not pass conv_lu glow_adain.py train.py. With the default two blocks, --no_lu changes only the first block.
• Actnorm is initialised from data. The first forward pass sets loc and scale from that batch glow_adain.py glow_adain.py. Training spends its first iteration on this pass and takes no optimizer step train.py. The initialised flag is a saved buffer, so resuming does not repeat it glow_adain.py.
• Missing checkpoint runs silently. If the decoder file is absent, test.py prints a message and runs with random weights test.py.
• Output size can differ from input. Test sides are rounded down to a multiple of 4, because the two squeezes need that test.py. The stylized output has the rounded size, not the input size.
• Training distorts aspect ratio. Images are resized to a square before cropping train.py. The paper says 512×512 too §5.1, so this matches the text, but it is a choice worth noting for non-square data.
• "Unbiased" is a property of the transfer module. The proofs use the bilinear model in §4.3 for AdaIN app. B and WCT app. C. The content-leak evidence is qualitative, from 20-round repeated transfer §5.3. It is not a quantitative metric.
• Metric details are missing. The dataset for Table 1, the layers used for SSIM, content loss and Gram loss, and the timing hardware are not stated in the text §5.2 tab. 1.
• Timing cost. ArtFlow+AdaIN takes 0.408 s against 0.064 s for AdaIN at 512×512 (author-reported) tab. 1. The authors attribute this to the absence of pooling §5.2.
• Gram loss. ArtFlow is not best on Gram loss. OptimalWCT is best tab. 1. Do not read Table 1 as an overall ranking.
• User study. The paper reports 799 effective votes but does not define what makes a vote effective, so the exclusion rule is unknown §6.
• ZeroConv2d pads with 1, not 0. The zero-initialised final convolution pads its input with ones glow_adain.py. This has no effect at initialisation, but it matters once the weights are non-zero.
• The LU 1×1 convolution inverts a matrix on every reverse call. The inverse is computed each time glow_adain.py. Expect a cost per reverse pass, and consider caching it at inference.
• WCT drops small eigenvalues. Eigenvalues below 1e-5 are discarded, and the cut-off is a numerically sensitive step glow_wct.py.
• Figure and table numbers come from source order. The card counts them in the LaTeX source, not from the PDF. Check against the published PDF before citing numbers to readers.

Lineage

• Builds on: the Glow model, which the paper cites as the basis of the PFN §4.2 §1. NICE is arXiv:1410.8516 §1 [dinh2014nice]. RealNVP §1 and the bilinear model of Tenenbaum and Freeman §4.3 have no arXiv id in the paper's bibliography. AdaIN and WCT are the transfer modules used §4.3. Neither has a card here.
• Compared against: StyleSwap arXiv:1612.04337 [chen2016fast], LinearWCT [li2018learning], OptimalWCT [lu2019optimal], Avatar-Net [sheng2018avatar], and Self-Contained [chen2020self] tab. 1 §5.3. None has a card here.
• Superseded by: not known from content in this wiki.