Paper · 2018-03-12 · cs.CV

An Introduction to Image Synthesis with Generative Adversarial Nets

He Huang, Philip S. Yu, Changhu Wang

A 2018 survey of GAN image synthesis. It sorts methods into direct, hierarchical and iterative designs, reviews text-to-image and image-to-image translation models with their losses, and lists evaluation metrics (Inception Score, FID). It argues the discriminator acts as a learned loss. No new experiments and no official code.

Generative adversarial network (GAN)Text-to-image generation Image-to-image translation Generative model evaluationsurveyimage-synthesisgenerative-models

arXiv PDF Markdown9 min read

AI-compiled by claude-haiku-5-5 on 2026-11-08, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Generative adversarial networks (GANs) learn to make images by pitting two networks against each other. A generator turns random noise into a picture, and a discriminator tries to tell those pictures apart from real ones. Each side improves because the other does, and the generator ends up producing images that look real.

This paper is a guide to using GANs for making images. It covers how to steer what gets generated, using a text description or a label, and how to turn one kind of picture into another, such as black-and-white into colour or one face attribute into another. It groups the methods by how many generators and discriminators they use, shows the training losses behind the main ones, and lists how generated images are scored. Its central point is that the discriminator learns a loss function, so the designer does not have to write one by hand.

TL;DR

This is a 2018 survey, not a new method. It groups GAN image synthesis into direct, hierarchical and iterative designs §3, reviews text-to-image §4 and image-to-image translation §5 models, lists metrics such as Inception Score and FID §6, and argues that the discriminator acts as a learned loss §7. It runs no experiments of its own and has no code. Use it as a map of the literature, not as an implementation reference.

Problem

GAN training is unstable, and the generator can collapse onto a few modes, for example one good sample per digit or one style per class §2.5. Hand-written losses are hard to define for image synthesis, because many outputs are valid for one input and the training set cannot cover them all §7. The survey limits itself to image synthesis and mentions other GAN uses only briefly §1. It assumes the reader wants to choose a method or find a starting point, not to reproduce reported numbers.

Method

The survey has no method of its own. It covers the following, in order: the core GAN objective and its common variants §2; three generator layouts §3; text-to-image models §4; image-to-image translation models, with their losses summarised in tab. 1 §5; and evaluation metrics §6. Figures are described in words below; the paper's figures are not reproduced here.

Core GAN. A generator G maps noise z to a sample. A discriminator D scores real and generated samples. The original objective is a min-max game on the value function given below §2. The generator is trained to maximise log D(G(z)) rather than to minimise log(1 - D(G(z))), so its gradient does not saturate when D is confident §2. The first GAN uses fully connected layers; DCGAN moved to convolutional and transposed-convolutional layers §2. DCGAN's generator uses transposed convolution, batch normalisation and ReLU; its discriminator uses convolution, batch normalisation and LeakyReLU fig. 6 §3.1.

$\min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$ §2
Conditional GAN. A condition c (class, attributes or a text embedding) is concatenated with z and fed to G §2.1.
Auxiliary classifier. D gets an extra classifier head trained on a task label, which the survey says can also help against mode collapse §2.2 fig. 2.
Encoder. BiGAN and ALI add an encoder E, and D judges (x, E(x)) pairs against (G(z), z) pairs §2.3 fig. 3.
VAE-GAN. A VAE loss is added, with the reconstruction term measured in a discriminator layer rather than in pixels, under a Gaussian observation model with identity covariance §2.4 fig. 4.
Mode-collapse remedies. Minibatch features let D compare a sample with a batch §2.5. MRGAN adds an encoder whose reconstruction loss acts as a mode regulariser §2.5. WGAN replaces Jensen-Shannon divergence with the Wasserstein distance §2.5. WGAN-GP replaces weight clipping with a gradient penalty §2.5.
Three generator layouts. Direct: one G and one D §3.1. Hierarchical: two G and two D with different roles, such as structure and style, or foreground and background; SS-GAN generates a surface normal map first and then an image, and LR-GAN separates foreground and background generators §3.2 fig. 5. Iterative: several G at increasing resolution, each adding detail. LAPGAN predicts a residual per Laplacian-pyramid level, StackGAN has two levels, SGAN stacks feature-level generators, and GRAN shares weights across recurrent steps §3.3 fig. 5.
Other generation methods. PPGN uses activation maximisation with a denoising-autoencoder prior §3.4. ProgressiveGAN grows generator and discriminator from 4×4 up to 1024×1024 by adding layers §3.4.
Text-to-image. GAN-INT-CLS concatenates a sentence embedding to z. In D, the embedding of length K is spatially replicated to W×H×K and joined to the image features §4. Training uses three input types: matched image and text, matched image with wrong text, and generated image with matched text. Caption interpolation adds more text embeddings §4. TAC-GAN adds an auxiliary classifier to GAN-INT-CLS §4. GAWWN adds location constraints, either a bounding box through a spatial transformer or user keypoints as masks §4.1. StackGAN uses two generators, each with its own discriminator, and a conditioning augmentation that samples from a Gaussian predicted from the text embedding, with a KL term toward N(0,1) §4.2. StackGAN++ uses more stages and a colour-consistency term §4.2. AttnGAN adds word-level attention and a DAMSM matching loss §4.2. PPGN samples by iterative updates that combine a realism gradient, a caption-match gradient and noise §4.3. The survey notes that all these models do poorly on multi-object scenes such as MS-COCO §4.4.
Pseudocode
```
# Iterative sampling step, as summarised for PPGN  [§4.3]
# Input:  current image x (H, W, 3); target caption y
# Output: updated image x

def sample_step(x, y, eps1, eps2, eps3):
    # Step toward a more realistic image, ignoring the caption  [§4.3]
    x = x + eps1 * grad_log_prior(x)
    # Step toward an image that the captioner matches to y  [§4.3]
    x = x + eps2 * grad_log_caption(y, x)
    # Add small noise to explore the latent space  [§4.3]
    x = x + normal(0, eps3 ** 2)
    return x
```
Image-to-image translation. Pix2Pix combines a conditional GAN loss with an L1 loss, uses a U-Net generator with skip connections, and a PatchGAN discriminator that scores local patches §5.1. The noise z is replaced by dropout at training and test time, since z had little effect §5.1. Pairs of images are required. PLDT adds a pair discriminator that tells whether two images from different domains belong together §5.2. CycleGAN and DualGAN use two generators and two discriminators with a cycle-consistency loss; CycleGAN uses least-squares GAN loss and a history buffer of 50 generated images for the discriminators §5.3. DistanceGAN adds pair-wise distance and self-distance losses that keep distances between images similar before and after translation §5.4. DTN uses a feature-constancy loss, an identity mapping loss on the target domain, a ternary discriminator and a total-variation term §5.5. DAAC adds a classifier and a content-similarity loss that uses masks §5.6. UNIT assumes a shared latent space with weight sharing and a VAE-style cycle term §5.7. StarGAN uses one generator conditioned on the target domain label, with an auxiliary classifier and a cycle loss §5.8. tab. 1 lists which loss each model uses. Task-specific work: face editing with IcGAN, residual-image editing, IAN and TP-GAN §5.10.1; super-resolution with SRGAN §5.10.2; video prediction with VGAN and Adv-GDL §5.10.3.
Pseudocode
```
# Cycle-consistency term for two generators  [§5.3]
# Input:  batch a (B, H, W, 3) from domain A; batch b from domain B
# Output: scalar cycle loss

def cycle_loss(a, b, G_AB, G_BA):
    # Translate A to B and back to A  [§5.3]
    a_cycled = G_BA(G_AB(a))
    # Translate B to A and back to B  [§5.3]
    b_cycled = G_AB(G_BA(b))
    # L1 reconstruction error in both directions  [§5.3]
    return mean(abs(a - a_cycled)) + mean(abs(b - b_cycled))
```
$\mathcal{L}_{cyc} = \mathbb{E}_{a}[\|a - G_{BA}(G_{AB}(a))\|_1] + \mathbb{E}_{b}[\|b - G_{AB}(G_{BA}(b))\|_1]$ §5.3

The total CycleGAN objective adds the two adversarial losses and λ times the cycle loss; λ is not stated §5.3.
Metrics. Inception Score is the exponential of the expected KL between p(y|x) and p(y) under the classifier §6. FCN-score uses a segmentation network's accuracy on generated images §6. FID compares Gaussians fitted to Inception features of real and generated images §6. The survey also names the Parzen window, GAM and MODE score §6.
Discriminator as learned loss. The survey argues that the discriminator learns how to score samples during training, unlike a fixed perceptual loss §7.

Hyperparameters

The survey itself has no training runs. The values below are the ones it states for methods it reviews, each attributed to the cited work. Everything marked "not stated" is absent from this paper.

Name	Value	Where	Notes
Pix2Pix L1 weight λ	not stated	§5.1	only "balances the two terms"
Pix2Pix PatchGAN patch size	70×70 for 256×256 images	§5.1	best in the cited experiments
Pix2Pix noise	dropout at train and test time	§5.1	replaces z
CycleGAN cycle weight λ	not stated	§5.3
CycleGAN discriminator history	50 previously generated images	§5.3	taken from SimGAN
CycleGAN GAN loss	least squares	§5.3
DistanceGAN weights α1A, α1B, α2A, α2B, α3A, α3B, α4	not stated	§5.4	survey notes other combinations unexplored
DistanceGAN μ, σ	pre-computed pair-wise distance statistics per domain	§5.4
DTN weights α, β, γ	not stated	§5.5	MSE for both distance measures
DAAC weights α, β, γ	not stated	§5.6
UNIT weights λ3, λ4	not stated	§5.7
ProgressiveGAN resolution schedule	4×4 up to 1024×1024	§3.4	layers added incrementally
PPGN step sizes ε1, ε2, ε3	not stated	§4.3	ε3 scales the noise
StackGAN KL term weight	not stated	§4.2	pulls conditioning distribution toward N(0,1)
GAN-INT-CLS text embedding	length K, replicated to W×H	§4	concatenated at D's feature map

Results

Survey. No experiments of its own, so no setup, dataset or baseline is reported here. The rows below are what the paper says about cited work. Author-reported.

Setting	Metric	Baseline	Reported claim	Source
TAC-GAN vs GAN-INT-CLS and StackGAN, Oxford-102	Inception Score	GAN-INT-CLS, StackGAN	higher	§4
AttnGAN vs PPGN, MS-COCO	Inception Score	PPGN	higher, but the survey says the shown examples do not look visibly better	§4
Pix2Pix among the general translation methods discussed	visual sharpness	other §5 methods	sharpest	§5.9
Supervised vs unsupervised translation	visual quality	unsupervised	supervised Pix2Pix still outperforms CycleGAN by a noticeable margin	§5.3, §5.9
LAPGAN vs GAN	sharpness	GAN	sharper	§3.3
SRGAN vs baselines	several metrics	baselines	small margin, hard to see without zooming	§5.10.2

Paper vs code

No official code. Checked: the paper body and footnotes (the footnotes point to a tips repository and to the CycleGAN project page, neither by these authors); the arXiv metadata, which has no code link; and a GitHub repository search for "image synthesis GAN survey", which returned no results. A review has no method to implement, so no repository was cloned.

Pitfalls

Survey only. Nothing here is measured by the authors. Every number in Results is a claim the survey repeats from another paper; check the cited source before relying on it.
Version read. The source read is arXiv v2 (updated 2018-11-17). The published date on the card is the first version's date (2018-03-12).
Equation numbers are not cited. The source numbers equations in the order they occur, but this card cites equations by section, because the compiled numbering was not checked against the PDF.
Printed formulas with typos. The TV term in DTN §5.5 is printed with a misplaced square and unbalanced parentheses. Read it as the usual squared difference of neighbouring pixels in each direction, summed and square-rooted. The LSGAN term in CycleGAN §5.3 writes the fake-sample part as a squared D without a clear target; the standard reading is target 0 for fakes and 1 for real samples, as the real-sample term shows.
Cycle loss normalisation. The survey writes the cycle loss as expectations of L1 norms §5.3. Whether to average over elements or sum over pixels is not stated; the pseudocode averages, which is a choice.
DistanceGAN sign and goal. The survey points out that the paper's stated goal (maximise the sum of distance products) differs from the loss it uses (minimise absolute differences) §5.4. Implement the loss, not the rationale.
Iterative sampling needs many steps. PPGN's sampler §4.3 runs repeated gradient steps at test time; the survey says this is slower than feed-forward methods but gives higher resolution. The step count is not stated.
Metric limits. Inception Score is insensitive to the label prior and misses intra-class collapse; FID needs Inception features and the survey cites a claimed correlation with quality §6. Neither is a sufficient measure alone.
Multi-object scenes. The survey's own conclusion is that methods work on single-object datasets (CUB, Oxford-102) and fail on MS-COCO §4.4 §8.
Licensing. The source is CC BY 4.0. Figures are not reproduced; the taxonomy table tab. 1 is restated as a list of models per loss.

Lineage

Builds on: the original GAN of Goodfellow et al. (2014), described in §2; DCGAN, WGAN and CycleGAN as the main reference points in §2.5 and §5.3.
Compared against: the survey compares methods within its own taxonomy rather than against a fixed baseline §3, §5.9.
Companion reviews: the NIPS 2016 GAN tutorial and a 2017 GAN review, which this survey says are broader and not image-specific §1.

Source and license

Adapted from “An Introduction to Image Synthesis with Generative Adversarial Nets” by He Huang, Philip S. Yu, Changhu Wang (arXiv:1803.04469), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.