Style Transfer by Relaxed Optimal Transport and Self-Similarity

In plain words

Style transfer takes a content picture, such as a photo, and redraws it with the look of a style picture, such as a painting. This paper does that by repeatedly nudging a new image until it scores well on two checks.

The style check asks that the new image's small visual patches, taken together, match the patches of the style picture. Each patch in the new image should find a close partner in the style picture, and the style picture's patches should be covered too. Computing this exactly is expensive, so the paper uses a cheaper bound. The style check also compares overall average and spread of the patch features, and colour palettes.

The content check does not compare pixels. It asks that the pattern of which spots look alike to each other stays the same as in the content picture. The layout and shapes survive, but the pixel values can change a lot.

A user can also pin a region of the content to a region of the style, so that, for example, the hair takes its texture from the clouds. Each image is its own optimisation, which takes tens of seconds to a few minutes.

TL;DR

• Optimisation-based style transfer. The output is refined with RMSprop against a fixed VGG16 feature extractor §2 §2.1 §2.5.
• Style is a relaxed earth mover's distance (REMD) over VGG16 hypercolumn features, with a cosine ground metric. Moment and palette terms are added §2.2 eq. 7 eq. 9.
• Content is the normalised pairwise cosine-distance pattern of features, which constrains layout but not pixels §2.3 eq. 11.
• Region-to-region and point-to-point guidance are supported. The released entry point supports region masks only §2.4 styleTransfer.py.
• The 662-worker study reports higher style scores than prior methods at each content score. The text gives no numeric table §4.1 fig. 8.

Problem

Style and content are hard to define. The paper frames style as a distribution over deep features and compares distributions with an earth mover's distance, and frames content as self-similarity §1.

The exact earth mover's distance between two feature sets of sizes m and n costs O(max(m, n)^3) per evaluation. The paper says this is too costly for gradient descent, which needs it at every step §2.2.

The authors argue that pixel-based content definitions from prior work are harder to satisfy than their self-similarity definition, which constrains only relative similarity within the image §1. Prior methods are compared in the human study §4.1.

Method

Overview. The output X is optimised to minimise the combined loss of eq. 1. The content weight α trades content preservation against stylisation §2. The loss has a content term ℓ_C, a style term made of ℓ_r (REMD), ℓ_m (moments) and ℓ_p (palette), and it is optimised coarse to fine on a Laplacian pyramid §2.5.

L(X,I_C,I_S)= \frac{\alpha\ell_C + \ell_m + \ell_r + \frac{1}{\alpha}\ell_p}{2+\alpha+\frac{1}{\alpha}} eq. 1

Step 1. Features. Features are hypercolumns: the activations of several VGG16 layers, each bilinearly upsampled to the image size and concatenated along the channel axis. This gives one feature vector per pixel. The paper uses all convolutional layers except layers 9, 10, 12 and 13, which it leaves unnumbered §2.1. The code's selection is listed in Paper vs code. Features are sampled at a subset of locations, described in Step 7.

Step 2. Relaxed EMD (style). Exact transport eq. 2 is replaced by a relaxation that keeps one of the two marginal constraints at a time eq. 5 eq. 6. The relaxed value is the larger of the two one-sided costs, each the average over points of the cheapest match eq. 7 eq. 8.

\ell_r = \max\!\left(\tfrac{1}{n}\sum_i \min_j C_{ij},\; \tfrac{1}{m}\sum_j \min_i C_{ij}\right) eq. 8

The ground metric is cosine distance, C_ij = 1 − cos(A_i, B_j) eq. 9. The paper says a Euclidean ground metric gave significantly worse results §2.2.

Step 3. Moment matching. Cosine distance ignores feature magnitude, which causes over- and under-saturation. The moment term matches the mean and the covariance of the two feature sets with L1 differences §2.2 eq. 10.

Step 4. Palette. A REMD is computed between pixel colours of X and of the style image, in a decorrelated colour space whose first channel is the mean colour. The paper uses a Euclidean ground metric, weights this term by 1/α, and says palette shifts conflict with content preservation §2.2.

Step 5. Content (self-similarity). Let D^X be the pairwise cosine-distance matrix of the output features, and D^C the same for the content image. Each matrix is column-normalised by its column sums. The content loss is the mean absolute difference of the two normalised matrices over all n² entries, where n is the number of sampled points §2.3 eq. 11.

\mathcal{L}_{content}(X,C) = \frac{1}{n^2} \sum_{i,j} \left|\frac{D^X_{ij}}{\sum_i D^X_{ij}}- \frac{D^{I_C}_{ij}}{\sum_i D^{I_C}_{ij}}\right| eq. 11

Step 6. User control. A user gives paired sets of locations in the output and the style image. In the cost matrix, a pair that matches a user constraint has cost β·D_cos with β = 5. An output location in a constrained set paired with a style location outside its partner set has cost ∞. All other pairs keep D_cos. For point constraints, eight extra point constraints are generated around each click, described as a uniform 9 × 9 grid with 20 px spacing at 512 px §2.4. Figure 5 shows guidance in use fig. 5.

Step 7. Coarse-to-fine optimisation. The content and style images are scaled to a long side of 64 px. The loss is optimised at four resolutions, 64, 128, 256 and 512 px, with the output of each scale bilinearly upsampled to start the next §2.5. The content weight α starts at 16.0 and is halved at each scale, so it reaches 1.0 at the last scale in the paper's description §2.5. The code does not match this, see Paper vs code.

# Coarse-to-fine optimisation over a Laplacian pyramid  [§2.5]
# Input:  content C, style S, alpha0 = 16.0
# Output: stylised image at 512 px long side

def stylize(content, style, alpha0=16.0):
    # Content weight starts at alpha0 and is halved after each scale  [§2.5]
    alpha = alpha0
    previous = None
    for scale in range(4):
        # Long side doubles from 64 px to 512 px  [§2.5]
        size = 64 * 2 ** scale
        content_s = resize(content, size)
        style_s = resize(style, size)
        if scale == 0:
            # Start from the style mean colour plus the content's
            # high-frequency band  [§2.5]
            init = mean_colour(style_s) + high_band(content_s)
        else:
            # Upsample the previous result  [§2.5]
            init = upsample(previous, size)
            # Code also adds the band here, except at the last scale.
            # The paper does not say this  [code: styleTransfer.py#L50-L55]
            if scale < 3:
                init = init + high_band(content_s)
        # Optimise the pyramid entries rather than pixels  [§2.5]
        pyramid = laplacian_pyramid(init, levels=5)
        # Learning rate 2e-3, and 1e-3 at the last scale  [§2.5]
        lr = 1e-3 if scale == 3 else 2e-3
        previous = optimise(pyramid, lr, alpha, content_s, style_s, steps=200)
        # The paper says 200 steps per scale  [§2.5]
        alpha = alpha / 2
    return previous

Step 8. Sampling. Pairwise terms need too many features to use every location. The content and output terms use 1024 locations drawn from a grid over the image with a random offset, and the style term uses 1024 random style locations. The paper resamples the content-side locations after each optimiser step §2.5.

# Objective for one optimisation step, using sampled features  [eq. 1]
# out_feats, con_feats: sampled output and content features (n, d)
# sty_feats: style features (m, d), rgb_*: colour samples (n, 3)
def objective(out_feats, con_feats, sty_feats, rgb_out, rgb_sty, alpha):
    # Content term: self-similarity of the sampled features  [eq. 11]
    loss_content = content_loss(out_feats, con_feats)
    # Style term: relaxed EMD with cosine ground metric  [eq. 7] [eq. 9]
    loss_remd = relaxed_emd(out_feats, sty_feats, cosine_cost)
    # Moment matching of mean and covariance  [eq. 10]
    loss_moment = moment_loss(out_feats, sty_feats)
    # Palette: relaxed EMD on colours, weighted by 1 / alpha  [§2.2]
    loss_palette = relaxed_emd(rgb_out, rgb_sty, euclidean_cost)
    numerator = (alpha * loss_content + loss_moment + loss_remd
                 + loss_palette / alpha)
    # Normalisation constant from the paper  [eq. 1]
    denominator = 2 + alpha + 1 / alpha
    return numerator / denominator

# Relaxed EMD between two sets of features  [eq. 7] [eq. 8]
# queries: (n, d), keys: (m, d); cost(queries, keys) -> (n, m)
def relaxed_emd(queries, keys, cost):
    # Pairwise ground-metric matrix  [eq. 9]
    costs = cost(queries, keys)
    # Mean over queries of the cheapest key  [eq. 8]
    query_term = mean(min_over_keys(costs))
    # Mean over keys of the cheapest query  [eq. 8]
    key_term = mean(min_over_queries(costs))
    # Relaxed EMD takes the larger of the two  [eq. 7]
    return max(query_term, key_term)

Hyperparameters

Results

Author-reported. Setup: a user study on Amazon Mechanical Turk with 662 workers. Three regimes (paired, unpaired, texture) with 30 pairs each, 90 in total. Each pairing had an average of 4.98 votes, minimum 4 and maximum 5. The baselines were Gatys et al. 2016, Gu et al. 2018, Li and Wand 2016 and Mechrez et al. 2018. Each baseline ran at its default content weight, at a quarter and at double it. Gatys et al. was also run at 1/100 and 100 times the default, and Gu et al. only at its default. The method was run at 4 times its default content weight. Scores are over about 1580 pairings per point on average §4.1.

The full timing table is below. Hardware: an Intel i5-7600 CPU and an NVIDIA GTX 1080 §4.4.

Timings in seconds tab. 1. X marks out-of-memory. * is projected from the reshuffle paper's 512 px timing, using the speed-up the authors measured for the CNNMRF method between their hardware and this one §4.4. The ablations are qualitative figures §4.2 fig. 9 fig. 11.

Paper vs code

Code: https://github.com/nkolkin13/STROTSS at commit 9a1543a36901cf744d3afbba6d8576a64de73f17. The owner is an author. The repository has no LICENSE file, so no code is quoted, and every citation is by path and line.

• Iterations. Paper: 200 per scale §2.5. Code: 250 st_helper.py.
• Final content weight. Paper: 1.0 at the last scale §2.5. Code: 16 halved after each of the first three scales gives 2.0 at the last scale styleTransfer.py styleTransfer.py.
• Content scale. Paper: 1/n² eq. 11. Code: the mean is multiplied by the number of sampled points n, which is 1024, so the content term is n times the paper's value contextual_loss.py. This changes the effective α.
• Denominator. Paper: 2 + α + 1/α eq. 1. Code: α + 2 [code: stylize_objectives.py#L74-L76]. The difference is a single scalar that scales the whole loss. RMSprop is largely insensitive to that.
• Colour weight. Paper: 1/α eq. 1. Code: 1/max(α, 1) [code: stylize_objectives.py#L69]. Identical while α ≥ 1, which holds for the default schedule.
• Palette metric. Paper: Euclidean §2.2. Code: cosine plus Euclidean, the latter divided by the channel count and then square-rooted [code: contextual_loss.py#L16] contextual_loss.py.
• Style sampling. Paper: 1024 random style locations §2.5. Code: 1000 random locations per draw, five draws per image, from every third file that matches the style path prefix. That gives about 5000 style features per image, fixed for each scale st_helper.py st_helper.py [code: utils.py#L300-L344].
• Content grid. Paper: 1024 locations on a uniform grid with a random offset, resampled after each step §2.5. Code: the grid's offset is drawn once per scale. init_inds appends a new grid on every call, but get_feature_inds and shuffle_feature_inds read index 0 only. Each step draws a new random 1024 from the same grid stylize_objectives.py [code: stylize_objectives.py#L217-L223] st_helper.py. Strides use floor for x and ceiling for y stylize_objectives.py.
• Feature sampling. Paper: not stated. Code: content and output features are bilinearly interpolated stylize_objectives.py, while style features are truncated to integer positions vgg_pt.py.
• Hypercolumn layers. Paper: "all conv layers except 9, 10, 12, 13" §2.1. Code: the post-ReLU outputs at VGG feature indices 1, 3, 6, 8, 11, 13, 15, 22 and 29, which are conv1_1 to conv3_3 (seven layers), conv4_3 and conv5_3, plus the raw RGB input, for 2179 channels vgg_pt.py [code: stylize_objectives.py#L49]. The paper's numbering is not defined, so the sets may differ.
• Intermediate band. Paper: each output is upsampled to start the next scale, with no mention of the band §2.5. Code: the band is added at scales 2 and 3 but not at the last scale styleTransfer.py.
• Normalisation. Paper: not stated. Code: ImageNet mean and standard deviation before VGG vgg_pt.py vgg_pt.py. The README says a normalisation bug was fixed in 2020, which "dramatically" improved results. The paper predates that fix README.md.
• Region guidance. Paper: β = 5 and an ∞ cost for mismatched pairs §2.4. Code: a separate REMD for each region pair, averaged over pairs, with no β stylize_objectives.py. Region masks restrict the sampling stylize_objectives.py.
• Point guidance. The released entry point sets use_guidance_points = False, so point guidance is never reached styleTransfer.py. The point-grid builder is in build_guidance, which nothing calls. Its grid is 3 × 3 with spacing 1% of the long side, not the paper's 20 px utils.py utils.py.
• Noise. A noise term is computed and never used styleTransfer.py.

Pitfalls

• Content term scale. Taking the paper's α schedule and the code's content term at face value gives different relative weights. The code's content term is n times larger contextual_loss.py. Decide which one you reproduce before tuning α.
• Final α. The paper's α = 1 at the last scale does not follow from its own halving schedule. Four scales with a halving between each pair give 2 §2.5.
• Lower bound. The REMD is a lower bound on the exact EMD, with a ratio of about 0.6 §4.3. The paper says a better approximation may give better results and leaves it to future work §4.3 §5.
• Point-constraint count. The text gives eight extra points, but also a uniform 9 × 9 grid. The two are not reconciled §2.4.
• Style set size. The style features in each step are a fixed set of several thousand, not 1024 st_helper.py. The timing table does not state the style set size.
• Randomness. Sampling and the random offset make each run stochastic. No seeds are stated.
• Cosine ground metric. It ignores magnitude, so the moment term is needed to stop saturation §4.2.
• Single-image style path. style_path + '*' is a prefix match, so a single file path can match more than one file st_helper.py.
• Guidance. Use the released code's region masks, not the paper's β and ∞ costs. The point-guidance path is dead code in the entry point [code: styleTransfer.py#L81].
• Output resolution. The final output long side is 512 px under the default of five scales. Larger outputs need more memory, and the paper reports the other methods running out of memory at 1024 px tab. 1.
• Duplicate label. The ablation figure label mturk_ab_fig is defined twice in the source. This only affects the cross-reference §4.2.

Lineage

• Builds on: Gatys et al. 2016, whose two-input gradient-descent setup this method follows §2. Hypercolumn features from Mostajabi et al. 2015 and Hariharan et al. 2015 §2.1. Relaxed EMD from Kusner et al. 2015 §1 §2.2. Self-similarity descriptors from Shechtman and Irani 2007 §2.3. VGG16 from Simonyan and Zisserman 2014 §2.1.
• Compared against: Gatys et al. 2016, Gu et al. 2018 (Reshuffle), Li and Wand 2016 (CNNMRF), Mechrez et al. 2018 (Contextual) §4.1 §4.4.
• Superseded by: not stated in the paper.