Image-to-image translation

Overview

Image-to-image translation takes an image from a source domain and produces an image in a target domain (another style, another imaging modality, another rendering, another outfit, another sensor's view of the same city) that still reflects the content of the input. The ten reuse-licensed cards agree on the goal but differ in what ties the output to the input. The unpaired transfer network keeps a fixed feature extractor's output unchanged 1611.02200 §4. The paired medical generator penalises pixel and gradient differences to a ground-truth target 1612.05362 §2.1. The multi-domain generator uses a cycle term to keep the input's structure while a style code sets the target look 1912.01865 §2.2. The training-free method copies internal features from a reconstruction run into the translation run 2211.12572 §4. The clothing generator constrains its output with a coarse map of the input's body layout 1710.07346 §3.2. The segmentation method reads only a structure code of the input, so the translated image keeps the input's layout 1903.12212 §3.2. The style method makes the transfer invertible, so the input's content can be recovered 2103.16877 §4.1. The single-pair method trains a new generator per image pair, and its losses come from a frozen ViT 2201.00424 §3.3. The aerial method translates labelled source images so that a segmenter can be fine-tuned on them [arxiv:1905.03198 §4.2]. The anomaly method is not a translation goal in itself: it uses a label-to-image generator as a probe 1904.07595 §3.

Most of the trained GAN variants (unpaired, paired, multi-domain, clothing and aerial) combine an adversarial term that pushes outputs toward the target distribution with a term that ties each output to its input. Without the second term, the output can look realistic while losing the layout of the input. The GAN-only baseline in the unpaired paper shows this: it reaches 13.71% on the digit transfer task 1611.02200 tab. 1. Three variants have no adversarial term. The ArtFlow training uses only feature losses 2103.16877 §5.1, the single-pair method uses frozen-ViT feature losses and no discriminator [arxiv:2201.00424 §3.3], and the training-free method has no training at all [arxiv:2211.12572 §4].

Variants

Unpaired transfer with a frozen feature extractor (Domain Transfer Network)

No paired examples are needed. The network learns G = g ∘ f from unlabelled samples of two domains, where f is a fixed function. Only g and the discriminator are trained, and the constraints are an f-constancy term on source samples and an identity term on target samples 1611.02200 §4 eq. 5 eq. 6. On the SVHN-to-MNIST digit task, an MNIST classifier reads 90.66% of the transferred digits correctly 1611.02200 tab. 1. These are author-reported numbers. The digit data are public, but the face experiment relies on a non-public feature network and face set 1611.02200 §5.2. Card: Unsupervised Cross-Domain Image Generation.

One generator for many domains with style codes (StarGAN v2)

This variant is for one model that translates between several domains, where each input should yield several outputs. Separate models would need one generator per ordered pair of domains, K(K-1) for K domains [arxiv:1912.01865 §1]. A single generator takes a style code. The code comes from a mapping network on a random Gaussian latent, or from a style encoder on a reference image, and it is injected by AdaIN 1912.01865 §2.1 app. E. Training adds a style-reconstruction term, a diversity term and a cycle term to the adversarial loss 1912.01865 §2.2 eq. 5. The diversity weight decays linearly to zero over the 100K training iterations 1912.01865 app. B. Author-reported latent-guided FID is 13.7 on CelebA-HQ, against 33.1 for MSGAN, and 16.2 on AFHQ, against 61.4 for MSGAN 1912.01865 tab. 2. With reference images, FID is 23.8 and 19.8, against 39.6 and 69.8 for MSGAN 1912.01865 tab. 3. The user study has 76 valid workers and no significance test 1912.01865 §3.2 tab. 4. The official code is licensed CC BY-NC 4.0, so it cannot be reused commercially. Card: StarGAN v2: Diverse Image Synthesis for Multiple Domains.

Paired 3D GAN with an auto-context stack (MR to CT)

This variant needs paired, voxel-aligned MR and CT volumes of the same subject 1612.05362 §3. A 3D fully convolutional generator maps a 32³ MR patch to a 16³ CT patch, trained with an adversarial loss, an L2 loss and a 3D gradient-difference loss, with weights 0.5, 1 and 1 1612.05362 §2.1 §4. Several such GANs are chained by the Auto-Context Model, so that later stages see the previous CT estimate as a second input channel; two iterations are used 1612.05362 §2.2 §4.2. Author-reported PSNR on 16 brain subjects is 27.6, against 26.3 for the structured random forest baseline 1612.05362 tab. 1. On 22 pelvis subjects it is 34.1, against 32.1 1612.05362 tab. 2. The gain from the GAN over a plain FCN is 24.7 to 25.9 PSNR in an ablation whose dataset is not stated 1612.05362 §4.1. The gains from more auto-context iterations are only plotted 1612.05362 §4.2 fig. 5. The released repository does not implement the auto-context stage, so the paper's pipeline must be rebuilt from the text 1612.05362 §2.2. Card: Medical Image Synthesis with Context-Aware Generative Adversarial Networks.

Training-free editing of a frozen diffusion model (Plug-and-Play)

No training is done. The guidance image is inverted with the empty prompt to a noise latent x_T. The translation is then sampled from the same x_T with the target prompt, while a decoder feature and the self-attention queries and keys are replaced by those of the guidance run during the early steps 2211.12572 §4 alg. 1. The defaults are 50 sampling steps, with the feature injected for 40 of them and attention for 25 2211.12572 app. C. On the real-image benchmark, the structure distance rises from 0.058 to 0.090 without feature injection and to 0.097 without self-attention injection 2211.12572 tab. 1. The main comparison with SDEdit, Prompt-to-Prompt, DiffuseIT and VQGAN-CLIP is plotted, with no numbers given [arxiv:2211.12572 §5.1] fig. 9. Card: Plug-and-Play.

Two-stage text-conditioned clothing synthesis (Be Your Own Prada)

The input is a photo of a person and a text description of a new outfit. The method needs no pairs of one person in two outfits; it trains on one photo and one sentence per person 1710.07346 §3.1. The photo is reduced to a seven-label segmentation map, then merged to four classes and down-sampled to 8x8. This keeps the body shape and pose, while the old garment's outline is lost 1710.07346 §3.2. Stage 1 generates a new label map from noise, that constraint and a design code made of attributes and the text embedding 1710.07346 §3.2 eq. 2. Stage 2 renders one channel per label and takes each pixel from the channel of its label 1710.07346 §3.3 eq. 4. The two GANs are trained separately, because the argmax between them is not differentiable 1710.07346 §3.4. Author-reported attribute mAP is 82.6, against 71.8 to 74.7 for the baselines and 86.8 for real photos 1710.07346 tab. 1. In a 50-volunteer ranking, the method has mean rank 1.544, against 2.286 for the best non-GAN baseline 1710.07346 tab. 2. Output is 128x128. Card: Be Your Own Prada.

Structure-texture split with cross-domain swap for segmentation (DISE)

This variant is a domain adaptation method for semantic segmentation, and translation is its mechanism. Each image is encoded into a domain-invariant structure code and a domain-specific texture code 1903.12212 §3.1. The segmenter reads only the structure code 1903.12212 §3.2. Swapping textures between a labelled source image and an unlabelled target image gives translated images that inherit the source labels 1903.12212 §3.1 §3.2. Target labels are never used in the loss 1903.12212 §3.2. The paper does not state the loss weights 1903.12212 §3.2 eq. 1. The author-reported result on GTA5 to Cityscapes is 45.4 mIoU, with a DeepLab-v2 ResNet-101 base, against 42.4 for the output-space adaptation of Tsai et al. 1903.12212 tab. 2. The ablation gives 39.8 for source-only training, 44.1 without the label-transfer term and 45.4 with it 1903.12212 tab. 4. Card: All about Structure.

Reversible-flow style transfer (ArtFlow)

Universal style transfer normally uses a fixed encoder, a style step and a learned decoder. The authors argue that the decoder does not invert the encoder exactly, so repeated stylization loses content, which they call content leak 2103.16877 §3.2. ArtFlow replaces the encoder and decoder with one invertible network, a Glow-style flow of two blocks with eight flows each 2103.16877 §5.1. AdaIN or WCT acts on the latent features, and the reverse pass returns pixels 2103.16877 §4.1. Training uses a content loss and a feature-statistics style loss on VGG19 features, weighted 0.1 and 1 2103.16877 §5.1. The content-leak evidence comes from 20 rounds of repeated transfer, shown qualitatively 2103.16877 §5.3. Author-reported SSIM is 0.45 for ArtFlow with AdaIN, against 0.29 for AdaIN alone 2103.16877 tab. 1. At 512×512, its time is 0.408 s, against 0.064 s for AdaIN 2103.16877 tab. 1. Card: ArtFlow.

Single-pair generator with frozen ViT feature losses (Splice)

The inputs are one structure image and one appearance image. No collection of either domain is needed. A small U-Net is trained from scratch on that pair 2201.00424 §3 app. A.1. A frozen DINO ViT-B/8 is used only to compute losses, and there is no discriminator 2201.00424 §3 app. A.2. The appearance term matches the deepest CLS token of the output to that of the appearance image 2201.00424 eq. 5. The structure term matches the self-similarity of the deepest-layer keys of the output to that of the structure image 2201.00424 eq. 2 eq. 6. An identity term keeps the keys of the appearance image when the generator is fed that image 2201.00424 eq. 7. The structure and identity weights are both 0.1 2201.00424 §3.3. Training runs 2000 iterations at a constant learning rate of 2 × 10⁻³, about 20 minutes for a 512 × 512 pair on one GPU 2201.00424 app. A.3. Author-reported two-alternative forced-choice preference on Wild-Pairs is 79.0% against STROTSS and 83.1% against WCT² 2201.00424 tab. 1. Because a generator is trained per pair, a new pair needs new training 2201.00424 §3. The method fails when the two images have no semantic match the ViT can use [arxiv:2201.00424 §4.3] fig. 10. Card: Splicing ViT Features for Semantic Appearance Transfer.

Translation of labelled source images for aerial segmentation (CycleGAN-style)

No target labels are used to train the translator or the segmenter. An unpaired translator with two generators and two discriminators is trained on the source (Potsdam) and target (Vaihingen) image sets 1905.03198 §4.1. The source-to-target generator restyles every labelled source image, and each translated image keeps its source label 1905.03198 §4.2. A segmenter trained on the source is then fine-tuned on the restyled images, starting from its source weights 1905.03198 §4.3 eq. 9. The translator loss has an adversarial term for each direction and a reconstruction term that asks each generator to undo the other 1905.03198 eq. 3 eq. 8. The paper states no weight for the reconstruction term 1905.03198 eq. 8. Author-reported average accuracy on Vaihingen rises from 0.35 to 0.52 1905.03198 tab. 3. The largest gains are for Building, from 0.23 to 0.71, and Tree, from 0.06 to 0.51. Low vegetation falls from 0.38 to 0.27 1905.03198 tab. 4. Fine-tuning reports the best epoch on the target test set, so these numbers are optimistic 1905.03198 §5.2.4. All results come from one source and target pair and one segmenter 1905.03198 §6. Card: Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images.

Resynthesis as a probe for unexpected objects (discrepancy network)

This variant does not aim at a new image. A segmenter labels a street scene, a pix2pixHD generator repaints the scene from those labels alone, and a discrepancy network marks the pixels where the original and the repainting disagree 1904.07595 §3 fig. 2. The discrepancy network reads the original image, the repainting and the label map. Image features come from a pre-trained VGG16 with shared weights 1904.07595 §3.1 fig. 3. It is trained on repaintings of scenes in which some known object instances were relabelled at random. Its target is the set of pixels whose label changed, so it never needs unknown classes 1904.07595 §3.2 fig. 4. The paper does not state what fraction of instances is changed 1904.07595 §3.2. On Lost and Found, the author-reported AUROC is 0.82 for all pixels except the ego-vehicle and 0.92 to 0.93 for the road-only region. The RBM baseline reaches 0.74 and 0.86 1904.07595 §4.2 fig. 5. On a 60-image road anomaly set, the AUROC is 0.79 to 0.83, against 0.59 for the RBM baseline fig. 5. The same repainting also feeds an attack detector. It takes the L2 distance between HOG descriptors of the input and of the repainting, and a logistic regression reads that distance 1904.07595 §3.3 tab. 1. Each frame needs one segmenter pass, one generator pass and two VGG16 passes, and the paper gives no timing 1904.07595 §3.1. Card: Detecting the Unexpected via Image Resynthesis.

When to use which

Implementation sketch

The adversarial core of the trained GAN variants (unpaired, paired, clothing and aerial), in simplified form. The unpaired paper uses a three-class discriminator and the paired paper uses a binary one, so this sketch follows the paired paper's binary form. The flow, single-pair, resynthesis and training-free variants do not use this loop.

# Shared core of trained translation  [arxiv:1612.05362 eq. 3] [eq. 5]
# Input:  source batch x, target batch y, tie_loss (variant-specific)
# Output: generator G

def translation_step(G, D, x, y, tie_loss):
    # Discriminator: real targets get label 1, generated outputs get 0
    fake = G(x)
    loss_d = bce(D(y), 1) + bce(D(fake), 0)
    update(D, loss_d)
    # Generator: fool D and stay tied to the input  [arxiv:1612.05362 eq. 5]
    loss_g = adv_weight * bce(D(fake), 1) + tie_loss(x, fake, y)
    update(G, loss_g)

Where the variants differ:

• Unpaired transfer. The tie term compares features of the source input and the output under a frozen extractor, and adds an identity term on target samples 1611.02200 eq. 5 eq. 6.
• Multi-domain style-code GAN. The tie is a cycle term that maps the output back with the input's own estimated style. Two more terms, style reconstruction and a diversity term, are added to the same objective 1912.01865 eq. 2 eq. 3 eq. 4.
• Paired 3D GAN. The tie term is an L2 loss to the paired CT patch plus a gradient-difference loss 1612.05362 eq. 5 eq. 6 eq. 7.
• Clothing, two stages. The tie is the coarse shape-map constraint. Each stage is a conditional GAN, trained on its own tuple, and the second stage composes pixels from per-label channels 1710.07346 §3.2 §3.3 §3.4.
• Structure-texture swap. The tie is a texture swap with reconstruction losses. A label-transfer term gives translated images the source labels 1903.12212 §3.2.
• Aerial translator. Two generators and two discriminators replace the single pair. Each generator's output must map back to its input under the other generator, and the reconstruction term is weighted 1905.03198 eq. 6 eq. 7 eq. 8. The source labels are then reused for the translated images, and the segmenter is fine-tuned on them 1905.03198 §4.2 §4.3.
• Single-pair ViT features. No discriminator. The tie is a sum of feature losses from a frozen ViT: a CLS-token term for appearance, a self-similarity term for structure and a key identity term 2201.00424 eq. 4.
• Reversible flow. No adversarial term. The tie is a feature-statistics loss on the reversed output, and the style step runs in latent space 2103.16877 §4.1 §5.1.
• Resynthesis probe. Not a translation loop. A label-to-image generator repaints the scene, and a discrepancy network scores each pixel from the original, the repainting and the labels 1904.07595 §3 §3.1.
• Training-free diffusion. There is no training step. The tie comes from injecting the guidance run's features and attention during sampling, not from a loss 2211.12572 alg. 1.

Contradictions

No two papers report incompatible numbers for the same quantity. The ten papers measure different things: classifier accuracy, PSNR, structure distance, SSIM, attribute mAP, mIoU, AUROC, average accuracy, FID and a preference rate. They also use different data. Their numbers should not be placed side by side as if they measured the same quantity 1611.02200 tab. 1 1612.05362 tab. 1 2211.12572 tab. 1 1710.07346 tab. 1 1903.12212 tab. 2 2103.16877 tab. 1 2201.00424 tab. 1 1905.03198 tab. 3 1904.07595 §4.2 1912.01865 tab. 2.

One paper is internally inconsistent. The DISE GTA5 table gives the method of Tsai et al. 42.4 mIoU, while the ablation calls the same method "seg-map adaptation" at 42.6 1903.12212 tab. 2 tab. 4 §4.3. Use the table value only with that caveat.

The aerial paper also gives different "before" figures. The abstract and its results table give 0.35 for the source model on Vaihingen, while the fine-tuning text gives 34% and 39% to 52% 1905.03198 tab. 3 §5.2.4. The gain is quoted both as 17 points and as 5% to 17% 1905.03198 §5.2.4 §6. Its discussion says unaffected classes stay the same, but the per-class table shows Low vegetation falling 1905.03198 §6 tab. 4. Use the table values only with those caveats.

Papers

• Unsupervised Cross-Domain Image Generation (Taigman et al.): unpaired transfer that keeps a frozen extractor's representation, shown on digits and faces.
• Medical Image Synthesis with Context-Aware Generative Adversarial Networks (Nie et al.): paired 3D GAN with auto-context stages for MR-to-CT synthesis.
• StarGAN v2: Diverse Image Synthesis for Multiple Domains (Choi et al.): one generator for several domains, with style codes from a mapping network or a reference image.
• Plug-and-Play (Tumanyan et al.): training-free translation with a frozen Stable Diffusion, using feature and attention injection.
• Be Your Own Prada: Fashion Synthesis with Structural Coherence (Zhu et al.): two-stage conditional GAN that redresses a person from a text description, through a segmentation map.
• All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation (Chang et al.): structure and texture split with cross-domain swap for unsupervised segmentation adaptation.
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows (An et al.): universal style transfer with an invertible flow that reduces content leak under repeated stylization.
• Splicing ViT Features for Semantic Appearance Transfer (Tumanyan et al.): single-pair generator trained with frozen DINO ViT losses, with no adversarial training.
• Unsupervised Domain Adaptation using Generative Adversarial Networks for Semantic Segmentation of Aerial Images (Benjdira et al.): CycleGAN-style translation of labelled source patches to the target sensor look, followed by segmenter fine-tuning.
• Detecting the Unexpected via Image Resynthesis (Lis et al.): resynthesis from a segmenter's label map, compared with the image by a discrepancy network, for unexpected-object detection.