Unsupervised Cross-Domain Image Generation

In plain words

Suppose you want to turn a photo of a person into a cartoon avatar that still looks like them, but nobody has ever drawn matching avatar/photo pairs for you to learn from. All you have is a big pile of photos, a separate pile of avatars, and a pretrained network that describes a face as a list of numbers (a "face fingerprint").

This paper's Domain Transfer Network learns the translation anyway. Its generator takes the fingerprint of a photo and paints an image. Three pressures shape what it paints: a critic network must be unable to tell its output from real avatars; the fingerprint of the generated avatar must match the fingerprint of the original photo; and when the generator is given an avatar as input, it should return that same avatar unchanged.

The same recipe turns street-number photos (SVHN) into handwritten-looking MNIST digits that an MNIST classifier reads correctly about nine times out of ten, and the translated digits can serve as a simple way to adapt a classifier to a new domain without labels. Generated emoji also turned out far easier to match back to the right person than emoji made by human annotators.

TL;DR

DTN learns a map G: S → T from unpaired samples of two domains plus a fixed, given feature function f, such that f(x) ≈ f(G(x)) §1 §3. The generator is built as G = g ∘ f, so g only decodes f's representation §4. Training combines a three-class GAN (generated-from-S, generated-from-T, real T), an MSE f-constancy loss on source samples, an MSE identity loss on target samples and a total-variation smoothness term eq. 3–7. Author-reported: 90.66% of SVHN test digits transferred to MNIST are classified correctly by an MNIST classifier, versus 13.71% for a plain GAN+constancy baseline tab. 1; on photo→emoji the generated emoji give median rank 16 for identity retrieval among 100,001 faces, versus 16,311 for human-made emoji tab. 4. Relevant if you want unpaired translation that preserves a specific pretrained representation; there is no official code, and the face data and f (DeepFace) are not public.

Problem

Given unlabeled sets s ⊂ S and t ⊂ T and a multivariate function f defined on S ∪ T, learn G: S → T such that f(x) ∼ f(G(x)) §1. No paired examples are available §1. The paper argues that style transfer only matches the style of one or a few images rather than a target distribution, and that sketch-generation methods need matched pairs §2. A direct formulation (binary GAN plus f-constancy, G acting on raw pixels) "does not produce desirable results" §3; it scores 13.71% on SVHN→MNIST tab. 1.

Method

Baseline formulation §3. Minimise R = R_GAN + α R_CONST over G with

R_{\text{GAN}} = \max_D \mathbb{E}_{x \sim \mathcal{D}_S} \log [1-D(G(x))] + \mathbb{E}_{x \sim \mathcal{D}_T} \log [D(x)] eq. 1

R_{\text{CONST}} = \mathbb{E}_{x \sim \mathcal{D}_S}\, d(f(x),f(G(x))) eq. 2

where D is a binary classifier on T and d is a metric §3. The authors tried MSE, cosine distance, hinge metric-learning and triplet losses for d and report results were mostly unchanged, so they use MSE §3.

Domain Transfer Network §4. Two changes on top of the baseline:

1. Compose the generator as G = g ∘ f with a learned g, so G works from f's representation §4. The authors argue g, trained on both domains, adapts f's representation to T, where f is usually less accurate §4 §6.
2. Also feed target samples x ∈ t through G during training: the discriminator becomes three-class and a new identity term L_TID asks G to leave target samples unchanged §4.

The losses are L_D (minimised over D) and L_G = L_GANG + α L_CONST + β L_TID + γ L_TV (minimised over g) §4:

L_{\text{D}} = -\mathbb{E}_{x \in \mathbf s} \log D_1(g(f(x))) - \mathbb{E}_{x \in \mathbf t} \log D_2(g(f(x))) - \mathbb{E}_{x \in \mathbf t} \log D_3(x) eq. 3

L_{\text{GANG}} = -\mathbb{E}_{x \in \mathbf s} \log D_3(g(f(x))) - \mathbb{E}_{x \in \mathbf t} \log D_3(g(f(x))) eq. 4

L_{\text{CONST}} = \sum_{x \in \mathbf s} d(f(x),f(g(f(x)))) eq. 5

L_{\text{TID}} = \sum_{x \in \mathbf t} d_2(x,G(x)) eq. 6

L_{TV}(z) = \sum_{i,j} \left( \left(z_{i,j+1} - z_{ij}\right)^2 + \left(z_{i+1,j} - z_{ij}\right)^2 \right)^{\frac{B}{2}}, \quad B = 1 eq. 7

D_i(x) is the probability D assigns to class i ∈ {1, 2, 3}: class 1 = generated from a source sample, class 2 = generated from a target sample, class 3 = real target sample §4 eq. 3. The generator tries to get both kinds of generated sample classified as class 3 eq. 4. Both d and d₂ are MSE §4. L_TV is computed on the generated image z = G(x) for slight smoothing §4. Optimiser: Adam §4. The authors also tried replacing d₂ with a second GAN and saw no noticeable gain §4. Fig. 1 shows the data flow: f feeds g, losses are attached to the outputs, and only G is used after training fig. 1.

# DTN training step (alternating updates; schedule not stated)  [§4]
# Input:  source batch xs from s, target batch xt from t
# f: fixed pretrained feature function; g, D: trainable networks

def dtn_step(xs, xt, alpha, beta, gamma):
    # Encode both domains with the fixed f  [§4]
    fs = f(xs)
    ft = f(xt)
    # Decode into the target domain: G = g o f  [§4]
    gen_s = g(fs)
    gen_t = g(ft)

    # Discriminator: 3-way cross-entropy  [eq. 3]
    # class 1 = G(source), class 2 = G(target), class 3 = real target
    loss_d = (
        mean(-log(D(gen_s)[1]))
        + mean(-log(D(gen_t)[2]))
        + mean(-log(D(xt)[3]))
    )
    update(D, loss_d)

    # Generator adversarial term: both outputs should look real  [eq. 4]
    loss_gan_g = mean(-log(D(gen_s)[3])) + mean(-log(D(gen_t)[3]))
    # f-constancy on source samples, d = MSE  [eq. 5] [§4]
    loss_const = sum(mse(fs, f(gen_s)))
    # Target identity, d2 = MSE  [eq. 6] [§4]
    loss_tid = sum(mse(xt, gen_t))
    # Total variation, B = 1; which outputs it covers is not stated  [eq. 7]
    loss_tv = mean(tv(z) for z in concat(gen_s, gen_t))

    # Combined generator loss, minimised over g only  [§4]
    loss_g = (
        loss_gan_g
        + alpha * loss_const
        + beta * loss_tid
        + gamma * loss_tv
    )
    update(g, loss_g)

# Total variation of one image z (H, W), exponent B / 2 with B = 1  [eq. 7]
def tv(z):
    dx = z[:, 1:] - z[:, :-1]
    dy = z[1:, :] - z[:-1, :]
    return sum(sqrt(dx[:-1, :] ** 2 + dy[:, :-1] ** 2))

Per-person selection §5.2. With several photos X_p of one person, pick the one whose emoji best preserves f:

\arg\min_{x \in X_p} \| f(x)-f(G(x))\| eq. 8

Domain adaptation via transfer §2 §5.1. Transfer the labeled source set to T with G, keep the source labels, and fit a classifier there; the paper uses a nearest-neighbour classifier on G(s) §5.1.

Architectures.

• Digits, f: four conv layers with 64, 128, 256, 128 filters, each followed by max pooling and ReLU, trained on SVHN extra (4.95% SVHN test error) §5.1. Inside DTN, f maps a 32×32 RGB image to the last conv layer activations, 128×1×1, taken after a 4×4 max-pool and before the ReLU §5.1. MNIST images are fed to f by repeating the grayscale channel three times §5.1.
• Digits, g: DCGAN-inspired; four blocks of deconvolution + batch norm + ReLU, tanh output, mapping 128-D to 32×32 grayscale §5.1.
• Digits, D: four batch-normalised conv layers with ReLU §5.1.
• Faces, f: the 256-D representation layer of DeepFace, trained on four million labeled images disjoint from s §5.2.
• Faces, g: five blocks of upscaling convolution + batch norm + ReLU from 256-D to 64×64 RGB; adding a 1×1 conv to each block lowered L_CONST training error and makes g 9 layers deep §5.2.
• Faces, D: input 152×152 RGB (natural or scaled-up emoji); six blocks of stride-2 conv + batch norm + leaky ReLU (slope 0.2) §5.2.
• Faces, post-processing: the 64×64 output is upscaled with a published super-resolution network, not retrained §5.2 app. B.

Hyperparameters

Results

Author-reported. Digits: SVHN→MNIST, evaluated by an MNIST classifier with the same architecture as f trained on MNIST train (99.4% on MNIST test), applied to G(SVHN test) and scored against the true SVHN labels §5.1. For reference, f itself reaches 76.08% on t §5.1.

Unsupervised domain adaptation SVHN→MNIST, nearest-neighbour classifier on transferred, labeled source samples §5.1:

Unseen class: accuracy on SVHN-test digits of class 3 after transfer §5.1.1:

Removing the class from t hurts far more than removing it from s; removing it from both beats removing it from t only, because otherwise G learns to map source 3s to other target classes §5.1.1 fig. 3.

Faces: for 118 random CelebA images, professional annotators built emoji by hand on bitmoji.com §5.2. Each probe (manual emoji or G(x)) retrieves against a gallery of a second photo of the same person plus 100,000 random faces not in s, using the VGG face descriptor (chosen to avoid bias from evaluating with f) §5.2:

Qualitative results: SVHN→MNIST and Facescrub→emoji samples fig. 2, all 80 Facescrub identities fig. 6 app. A, with and without super-resolution fig. 7 app. B, g applied to the 256 standard basis vectors of the face representation, showing much variation and so no strong mode effect fig. 8 app. C, and the reverse directions MNIST→SVHN and emoji→face, which look less appealing fig. 9 app. D §6. Gatys-style neural style transfer with an emoji as style image does not produce something in the emoji domain, whereas the DTN output does and even fills in missing hair fig. 5 §5.3. A DTN trained with CelebA photos as s and their Johnson-style stylised versions as t reproduces the style transfer §5.3 fig. 5.

Paper vs code

No official code.

Pitfalls

• f is frozen. L_G is minimised over g only and L_D over D §4; f is never updated. The "adaptation" of f to T happens through g and the downstream D, not by fine-tuning f §4 §6.
• Three-class D, not binary. Generated-from-target samples get their own class (2), separate from generated-from-source (1); the generator pushes both to class 3 eq. 3 eq. 4. Collapsing to a binary D drops this structure.
• Sums vs means. L_CONST and L_TID are written as sums over the training sets eq. 5 eq. 6 while the GAN terms are expectations eq. 3 eq. 4. How they are normalised per batch is not stated, so the weights α, β, γ (15/15/0 digits, 100/1/0.05 faces) may not transfer unless you match the reduction.
• The text says "identity matrix" for L_TID §4; it means the identity mapping on T, as eq. 6 shows.
• Resolution mismatch on faces. g outputs 64×64 §5.2, emoji in t are 152×152 §5.2, D takes 152×152 inputs including "scaled-up emoji" §5.2, and DeepFace f must also consume G(x). The resizing used before D, f and the L_TID MSE against the 152×152 target is not stated.
• Channel handling on digits. g emits 32×32 grayscale with tanh §5.1 while f expects RGB; the paper only states that MNIST images are replicated to three channels for f §5.1, so presumably G's output is too. Input normalisation ranges are not stated.
• Which f layer. For digits, the representation is the last conv layer after the 4×4 max-pool and before the ReLU §5.1; taking it after the ReLU changes L_CONST.
• TV with B = 1 takes a square root of summed squared differences eq. 7; its gradient is undefined where both differences are zero, and the paper does not say how this is handled. The paper calls it anisotropic §4; for B = 1 the form in eq. 7 couples the two directions under one root. Edge pixels lacking an i+1 or j+1 neighbour are not addressed.
• Unstated training details: learning rate, Adam betas, batch size, iterations, D/G update ratio, kernel sizes and widths §4 §5. Expect to tune them.
• Reproducibility of the face experiment depends on non-public parts: DeepFace f, the one-million-face set s, and scraped bitmoji emoji §5.2. The digit setup uses only public SVHN and MNIST.
• Evaluation leakage risk. The digit target set t is the MNIST test set §5.1; the evaluation classifier is trained on MNIST train and applied to transferred SVHN test images, so no MNIST test image is scored directly, but a reimplementation should not reuse MNIST test for model selection.
• Asymmetry. f is typically trained on one domain only; reversing S and T (MNIST→SVHN, emoji→face) gives worse results §6 app. D. The method relies on f being informative on S.
• Quality vs constancy trade-off: increasing α improves f-constancy but introduces noise and distortions on faces §5.2.
• Small text errors: tab. 1's caption says "SHVN", tab. 4 says "DTS network", and §5.1.1 calls the numeric tab. 3 "qualitative" tab. 1 tab. 4 §5.1.1.

Lineage

• Builds on: GANs (Goodfellow et al., arXiv:1406.2661) with a DCGAN-style generator (Radford et al., arXiv:1511.06434) §2; ternary discriminators from conditional GAN work §2; feature-plus-pixel losses as in Dosovitskiy & Brox's DeePSiM §2. DTN differs by applying the feature loss on S and the pixel loss on T without paired data §2.
• Compared against: a binary-GAN + constancy baseline §3 tab. 1; domain adaptation methods SA and DANN tab. 2; neural style transfer of Gatys et al. §5.3 fig. 5; human annotators tab. 4.
• Superseded by: not covered in this wiki yet.