Splicing ViT Features for Semantic Appearance Transfer

In plain words

Given two photos, a "structure" photo and an "appearance" photo, the method produces a new picture with the layout of the first and the look of the second. Objects in the first photo take on the appearance of their semantically related counterparts in the second.

Earlier approaches either work on whole-image styles, need large collections of pictures from each side, or need extra inputs such as segmentation or correspondences. This method needs none of these. It trains a small image generator on just the one pair, and judges its own output with a frozen image model that already captures objects and their parts. The judgement has two parts: the output must match the look of the appearance photo, and its layout must match the structure photo.

TL;DR

• Single-pair, no-adversarial method: a U-Net generator is trained from scratch for each image pair. §3
• Losses are computed in the feature space of a frozen, self-supervised DINO ViT-B/8, with no segmentation or correspondences. §3.2 §3.3
• Author-reported: in 2AFC user studies the output is preferred over SA, STROTSS and WCT² on most collections, e.g. 79.0% against STROTSS and 83.1% against WCT² on Wild-Pairs. §4.1 tab. 1
• Limits: depends on DINO's semantic matches; it fails when the two objects are semantically unrelated or one image is far from natural. §4.3 fig. 10

Problem

Transfer appearance between two natural images so that objects in the structure image take on the look of semantically related objects in the appearance image §1 fig. 1. Earlier options fail here for three reasons:

• GAN image-to-image translation needs image collections from both domains, or a domain-specific model §2.
• Neural style transfer is global and not region-based across objects §2.
• Semantic style transfer is mostly restricted to colour, or needs extra inputs such as segmentation or correspondences §2.

The paper's assumption is that a pre-trained, self-supervised ViT provides enough semantic knowledge to replace those extra inputs §1 §2.

Method

The generator G_θ maps the structure image I_s to an output I_o. Training uses only a single pair {I_s, I_t} (structure, appearance). A DINO ViT-B/8 model is frozen and used only to compute losses §3 fig. 2 app. A.2.

Representations used for the losses §3.1 §3.2:

• Appearance: the CLS token of the deepest (12th) transformer layer. The CLS token is spatially flexible and carries global appearance such as texture and object parts §3.2 fig. 3 fig. 4.
• Structure: the cosine self-similarity matrix of the deepest-layer keys. The matrix has size (n+1) × (n+1) for n patches plus the CLS token. Keys are used because they keep layout and semantics and discard most texture §3.2 eq. 2 fig. 5.
• Identity: the deepest-layer keys of the appearance image, which should be preserved when the generator is fed the appearance image §3.3 eq. 7.

Objective §3.3 eq. 4:

\mathcal{L}_{\mathsf{splice}} = \mathcal{L}_{\mathsf{app}} + \alpha\,\mathcal{L}_{\mathsf{structure}} + \beta\,\mathcal{L}_{\mathsf{id}}

with α = β = 0.1 for all experiments §3.3. The appearance term is the driving loss and has an implicit weight of 1 in the equation.

Appearance loss eq. 5:

\mathcal{L}_{\mathsf{app}} = \left\| t^{L}_{\mathsf{cls}}(I_t) - t^{L}_{\mathsf{cls}}(I_o) \right\|_2

Structure loss eq. 6:

\mathcal{L}_{\mathsf{structure}} = \left\| S^{L}(I_s) - S^{L}(I_o) \right\|_F

Identity loss eq. 7:

\mathcal{L}_{\mathsf{id}} = \left\| K^{L}(I_t) - K^{L}(G_\theta(I_t)) \right\|_F

Self-similarity eq. 2:

S^{L}(I)_{ij} = \mathrm{cos\text{-}sim}\big(k^{L}_i(I),\, k^{L}_j(I)\big)

Training uses internal examples: random crops and augmentations of both images, so the generator learns a mapping rather than a single-image optimisation §3.3 app. A.4. Each training batch holds one augmented structure image and one augmented appearance image. Every 75 iterations the unaugmented pair is also added to the batch app. A.3.

# One training step of the Splice objective  [eq. 4]
# Input:  structure image I_s, appearance image I_t (single pair)
# Output: scalar loss for one update of G_theta

def splice_loss(G, dino, I_s, I_t):
    # Augment both images; I_s also gets colour and blur  [app. A.4]
    I_s_aug = augment_structure(I_s)
    I_t_aug = augment_appearance(I_t)

    # Generate the output from the augmented structure image  [§3.3]
    I_o = G(I_s_aug)

    # Appearance: CLS token of the deepest layer  [§3.3] [eq. 5]
    loss_app = l2_norm(cls_token(dino, I_t_aug) - cls_token(dino, I_o))

    # Structure: self-similarity of deepest-layer keys  [§3.2] [eq. 6]
    loss_struct = frobenius(selfsim(dino, I_s_aug) - selfsim(dino, I_o))

    # Identity: the generator keeps the keys of the appearance image  [eq. 7]
    loss_id = frobenius(keys(dino, I_t_aug) - keys(dino, G(I_t_aug)))

    # Weighted sum with alpha = beta = 0.1  [eq. 4] [§3.3]
    return loss_app + 0.1 * loss_struct + 0.1 * loss_id

Generator app. A.1: a U-Net with five levels and channel widths 16, 32, 64, 128, 128 on the encoder side, and a mirrored decoder. Each level has convolutions, batch normalisation and LeakyReLU. Each encoder level has a 1×1 convolution that is concatenated into the decoder at the same level. The output goes through a 1×1 convolution and a sigmoid to RGB.

Data augmentation for training app. A.4:

• Structure image: a square crop whose side is 95–100% of the image height, horizontal flip with p = 0.5, colour jitter with p = 0.5, and 3×3 Gaussian blur with p = 0.5 and σ drawn from [0.1, 2.0].
• Appearance image: the same crop, and horizontal flip with p = 0.5. No colour jitter or blur.

Feature extraction for the losses resizes both images to 224 px on the shorter side with the aspect ratio kept, using bicubic interpolation app. A.3.

Resolution: the paper reports results up to HD §1 and a 512 × 512 training pair app. A.3. The Wild-Pairs collection has 25 pairs at 512–2000 px §4. A generator is trained per pair, so inference on a new pair means training again.

Hyperparameters

Results

Author-reported. Setup: AMT two-alternative forced-choice survey on 65 images from AFHQ, Mountains and Wild-Pairs, 7000 judgments in total. Each cell is the percentage of judgments in favour of Splice (mean ± std) §4.1.2 tab. 1.

Structure preservation: mean IoU of segmentation masks between the structure image and the output, using Mask R-CNN for Wild-Pairs and a semantic segmentation model of Zhou et al. for Mountains §4.1.2 tab. 2.

The paper states that WCT² has the highest IoU because it only changes colours §4.1.2. The Mountains WCT² std is printed as ±0.82 in the source table, which is probably a typo; it is copied here as printed.

Qualitative comparisons with SA, STROTSS and WCT² are in fig. 7, the SinCUT comparison in fig. 8, and the loss ablation in fig. 9 §4.1 §4.2.

Paper vs code

The official repository is omerbt/Splice, linked from the authors' project page. Its owner is an author, so it qualifies as official. The commit used is 12e014670fed33e75ad723a7c8ed0b094d4516ea. The repository has no LICENSE file and no license statement in its README, so code_license is recorded as "none". No code is quoted in this card, only referenced by path.

Discrepancies found by comparing the paper with the code:

• Loss weights. The paper sets α = β = 0.1 with an appearance weight of 1 eq. 4 app. A.3. The code uses λ_cls = 10, λ_ssim = 1, λ_id = 1 config.yaml. The ratios match, but the code's absolute scale is 10 times the paper's.
• Norms. The paper uses L2 for appearance and Frobenius norms for structure and identity eq. 5 eq. 6 eq. 7. The code uses F.mse_loss (mean squared error) for all three losses.py losses.py losses.py.
• Entire-image pass. The paper adds the unaugmented structure and appearance images to the batch every 75 iterations app. A.3. The code runs the unaugmented structure image through the generator every 75 steps, with a structure self-similarity loss against that image and a CLS loss against an augmented appearance crop. The unaugmented appearance image is never passed through the generator model.py losses.py.
• Warm-up. The code disables the structure and identity losses until step cls_warmup = 1, so the first step uses the appearance loss only config.yaml losses.py. The paper states no warm-up.
• Training length. The paper trains for 2000 iterations app. A.3. The config sets n_epochs to 10000, and each epoch is one generator update on one augmented pair config.yaml train.py.
• Optimizer. The paper says "Adam" with no betas app. A.3. The code uses β1 = 0.0 and β2 = 0.99 config.yaml util.py. The learning rate, 0.002 with no scheduler, matches config.yaml.
• Weight init. The paper does not state an init. The code uses Xavier normal with gain 0.02 config.yaml networks.py.
• Blur probability. The paper states p = 0.5 for blur app. A.4. The code uses p = 0.2 transforms.py. The code's colour jitter values (brightness 0.4, contrast 0.4, saturation 0.2, hue 0.1) are not stated in the paper transforms.py.
• ViT resize. The paper says bicubic app. A.3. The code uses Resize(224, max_size=480) with the default bilinear filter, so the longer side is capped at 480 px. The code also applies ImageNet mean and std normalisation, which the paper does not mention losses.py.
• Self-similarity heads. The paper defines the self-similarity over keys without mentioning heads eq. 2. The code concatenates all heads' keys for each token before taking the cosine similarity extractor.py.
• Generator kernels. The paper says all layers use 3 × 3 convolutions app. A.1. The code uses 1 × 1 skip convolutions with 4 channels per level and bilinear upsampling after stride-2 downsampling skip.py skip.py.
• Keys layer index. The code uses zero-based layer 11 (the last of 12 blocks) for keys and the last block's CLS token losses.py losses.py. This matches "deepest layer" in the paper.

Pitfalls

• Per-pair training. A new generator is trained for every pair, which takes about 20 minutes on one GPU at 512 × 512 app. A.3. The paper gives no inference-time budget.
• Frozen but backpropagated. DINO is frozen, but its losses are backpropagated into the generator. Gradients flow through the ViT on every step.
• Scale mismatch. Implementing the paper's weights gives a total loss about 10 times smaller than the code's. The paper does not say whether the absolute scale matters. Our inference, not stated in the paper: with Adam the absolute scale mainly interacts with the ε term, so the effect is probably small, but the code was not run to check.
• Unstated Adam betas and init. Reproducing the paper's numbers from the paper alone requires guessing β and init. The code's values are the only record.
• Colour jitter strengths and blur sigma. The paper does not give colour jitter strengths. The code's values are listed above; the code's blur σ comes from torchvision defaults and is not stated.
• Resize filter. Bicubic and bilinear interpolation give different features for DINO. The ImageNet normalisation and the 480 px cap also change the input the ViT sees.
• Non-realistic or unrelated inputs. The method fails when the two images are semantically unrelated or one is far from natural, because DINO has no semantic match to offer §4.3 fig. 10.
• Metric limits. The main quality evidence is an AMT preference study with 65 images and std up to ±18 points. The IoU table is a proxy for layout preservation, not for appearance quality §4.1.2 tab. 2.
• Equation numbers. Equation numbers were counted from the LaTeX source, where unnumbered equation* blocks are skipped. If the compiled PDF numbers differ, the section and label should be used instead.

Lineage

• Builds on DINO, a self-supervised ViT trained by self-distillation §3 [dino], and on the Deep ViT feature study by Amir et al. [/papers/2112.05814] that motivates using keys and their self-similarity §3.2.
• Uses the feature-inversion idea with a Deep Image Prior generator to study ViT features; the Deep Image Prior paper is cited as [UlyanovVL17] §3.2.
• Compared against: Swapping Autoencoders (SA) [park2020swapping], STROTSS [kolkin2019style], WCT² [Yoo_2019_ICCV] and SinCUT [park2020contrastive] §4.1.
• Superseded by: not known from content already in this wiki.