Generative Adversarial Transformers

In plain words

A GAN image generator usually starts from one random vector and turns it into a picture in stages. StyleGAN, the strong recent design, uses that vector to set the look of the whole picture at once, like choosing a single colour scheme and lighting for an entire room. That makes it good at global style, but it has little direct control over one spot in the image.

GANformer changes the generator so that the random input is split into several separate pieces, each acting like a small "helper" that is responsible for part of the scene. At each stage, the helpers and the growing image look at each other through attention, so each helper can pick out the region it should control and style that region on its own. Information flows from the helpers into the image and back, so the two sides refine each other. The attention is built so that its cost grows in proportion to the image size rather than with the square of it, which is what makes this affordable for larger pictures.

The paper reports that the duplex version gets a lower FID (a common image-quality score) than StyleGAN2 on each of the four datasets it tests, and that the helpers tend to line up with recognisable parts of a scene. The results come from the authors' own experiments and training budget.

TL;DR

• GANformer is a GAN whose generator uses a bipartite transformer: k latent variables and image features attend to each other, with multiplicative (scale and shift) control of the features §3.1 §3.2.
• Attention runs in one direction (latents to image, "simplex") or in both (adding a centroid step, "duplex") §3.1.1 §3.1.2.
• Cost is linear in the number of image positions for a fixed number of latents, versus quadratic for full self-attention §3.1.3.
• Author-reported FID at 256 px, equal training steps: CLEVR 9.17 (duplex) vs 16.05 (StyleGAN2); LSUN-Bedrooms 6.51 vs 11.53 tab. 1.
• Several design details in the paper differ from the released TensorFlow code's defaults (duplex centroids, integration form, latent counts, learning rate); see Paper vs code and Pitfalls.

Problem

Generative adversarial networks (GANs) are usually built from convolutions, which have a local receptive field. The paper argues that this makes long-range relations and global shape hard to model §1. Transformers model long-range relations, but full self-attention over all pixels costs O(n²) in the number of positions, which the authors say blocks high-resolution use §1 §3.1.3.

StyleGAN2 models global style well through a single latent vector that modulates each convolution layer, but it offers no direct means to control the style of a local region §3.2. The paper's aim is an architecture that keeps long-range interaction, supports region-level modulation, and stays linear in cost §1 §3.2.

Assumptions: experiments use 256×256 images with the StyleGAN2 training recipe and a fixed training length §4 app. A.

Method

The generator stays a StyleGAN-style design, with the convolutional blocks replaced in part by bipartite attention layers. The discriminator has no attention in the paper's final version §3.2 §3.2 footnote.

1. Latent split. The input vector z is cut into k components. Each component goes through a shared mapping network, giving k intermediate latent variables Y = [y_1, ..., y_k] §3.2. The paper uses k between 8 and 32 §4.
2. Generator stack. The stack starts from a 4×4 grid. It alternates bipartite attention, a convolution with kernel size 3 and LeakyReLU, and up-sampling, until the target resolution. Sinusoidal positional encodings are added to image positions; trained embeddings are used for latents §3.1.3.
3. Simplex update. Image features X attend to the latents Y. The attended summary sets a per-element scale and bias for the normalised features §3.1.1. Layer normalisation was reported to work better than instance normalisation §3.1.1 footnote.
4. Duplex update. Each latent first gets a centroid key K, which is an attention-weighted average over image positions. The image then attends to these centroids, with values from the latents, and the result sets scale and bias as in step 3 §3.1.2.
5. Bidirectional loop. Duplex attention alternates a latent update from the image (bottom-up) and an image update from the latents (top-down) §3.1.2.
6. Training. The paper adopts the StyleGAN2 recipe: style mixing, stochastic variation, weight moving average, non-saturating logistic loss and lazy R1 regularisation §3.2. Optimiser and batch details are in app. A.

Equations are unnumbered in the paper; they are cited by section.

Attention and modulation, as stated in the paper §3.1.1 §3.1.2:

a(X, Y) = \text{Attention}(q(X), k(Y), v(Y)) \quad [\S 3.1.1]

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^{T}}{\sqrt{d}}\right) V \quad [\S 3.1]

u^{s}(X, Y) = \gamma\big(a(X, Y)\big) \odot \omega(X) + \beta\big(a(X, Y)\big) \quad [\S 3.1.1]

\omega(X) = \frac{X - \mu(X)}{\sigma(X)} \quad [\S 3.1.1]

K = a(Y, X), \qquad u^{d}(X, Y) = \gamma\big(A(Q, K, V)\big) \odot \omega(X) + \beta\big(A(Q, K, V)\big), \quad Q = q(X) \quad [\S 3.1.2]

In these formulas, d is the feature dimension and γ, β are mappings to dimension d. The paper presents one head and says that multiple heads are used in practice §3.1.

# Bipartite update of a target group from a source group  [§3.1.1] [§3.1.2]
# Input:  target (B, n, d), source (B, m, d); n = H*W for images, m = k latents
# Output: updated target (B, n, d)

def simplex_update(target, source):
    # Queries come from the target, keys and values from the source  [§3.1]
    queries = to_queries(target)          # (B, n, d)
    keys = to_keys(source)                # (B, m, d)
    values = to_values(source)            # (B, m, d)
    # Attention over the source elements, scaled by sqrt(d)  [§3.1]
    attended = softmax(queries @ keys^T / sqrt(d)) @ values  # (B, n, d)
    # Normalise the target, then scale and shift it  [§3.1.1]
    normalized = layer_norm(target)       # (B, n, d)
    scale = gamma(attended)               # (B, n, d)
    shift = beta(attended)                # (B, n, d)
    return scale * normalized + shift


# Duplex update: image features read centroids of the latents  [§3.1.2]
def duplex_update(features, latents):
    # Centroid keys: each latent averages the image features it attends to,
    # K = a(Y, X)  [§3.1.2]
    centroid_keys = attend_over_positions(latents, features)  # (B, k, d)
    # Image queries attend to the centroids; values come from the latents
    # [§3.1.2]
    queries = to_queries(features)                            # (B, n, d)
    attended = softmax(queries @ centroid_keys^T / sqrt(d)) @ to_values(latents)
    normalized = layer_norm(features)                         # (B, n, d)
    return gamma(attended) * normalized + beta(attended)


# One duplex step in a generator layer  [§3.1.2]
def duplex_layer(features, latents):
    # Bottom-up: latents read the image, Y := u^a(Y, X)  [§3.1.2]
    latents = simplex_update(latents, features)
    # Top-down: image reads the latents, X := u^d(X, Y)  [§3.1.2]
    features = duplex_update(features, latents)
    return features, latents

Note on the pseudocode: attend_over_positions is the paper's a(Y, X). The paper describes K as weighted averages over the image positions, using the attention distribution from comparing them with the latents §3.1.2. This card reads that as a softmax over image positions for each latent; the paper's text does not name the axis explicitly.

Hyperparameters

Values per dataset are from the paper's hyperparameter table tab. 4. Training settings are from app. A. The code column gives the default in the released TensorFlow repository, where it differs or where the paper is silent.

Model size tab. 5: generator 34–38 M parameters across baselines and GANformer variants (36 M for both GANformer variants), discriminator 29 M, with 16 latent variables and latent dimension 512.

Results

Author-reported. Setup: 256×256 images, the same number of training steps for all models (about one week on 2 NVIDIA V100 per model) §4. Baselines are the paper's own GAN, k-GAN, SAGAN and StyleGAN2, all run inside the authors' TensorFlow codebase §4. FID, Inception Score, precision and recall are computed over 50k samples tab. 1. The paper also reports a 1k-sample study for CLEVR tab. 2 and disentanglement metrics over 1k CLEVR samples tab. 3.

Table 1, headline comparison tab. 1. ↓ lower is better, ↑ higher is better.

Other author-reported results:

• Training efficiency: on CLEVR the model reaches high-quality images in about a third of the steps of the second-best method, and it stays data-efficient down to as few as 1k training images §4.1.
• Attention maps: latent variables attend to coherent regions that match semantic classes such as windows, pillows, sidewalks and cars, measured with a pretrained segmenter over 1k samples §4.2 app. B.

Table 2, CLEVR chi-square statistics over 1k samples. The paper does not spell out the direction in words; the table bolds the largest value per row, and the text calls the GANformer's scores "better" tab. 2 §4.4:

Table 3, disentanglement metrics (DCI and modularity) over 1k CLEVR samples tab. 3:

Paper vs code

Code: official repository github.com/dorarad/gansformer (linked from the abstract; owner is the first author), commit 3a9efa45, MIT licence. The paper's stated framework is TensorFlow app. A; the TensorFlow tree (training/) was compared. The repository also has a PyTorch tree (pytorch_version/), which was not compared.

Confirmed in code:

• Duplex is switched on by --kmeans, which the GANformer preset enables run_network.py network.py.
• Attention scale 1/√d per head network.py.
• Per-dataset R1 weights match the paper run_network.py tab. 4.
• Mirror augmentation for FFHQ and Cityscapes only run_network.py.
• Adam β1 = 0.0 and β2 = 0.99 run_network.py.
• StyleGAN2-style losses: non-saturating logistic for G, logistic with R1 for D run_network.py loss.py.
• Eight mapping layers run_network.py.

Differences:

• Latent counts and dimensions. The paper's table gives FFHQ 8 variables of dimension 16 (overall 128) and CLEVR 16 of dimension 32 (overall 512) tab. 4. The preset gives 16 variables and overall 512 for FFHQ, Cityscapes and Bedroom, and 8 variables and overall 128 for CLEVR run_network.py. The preset therefore swaps the paper's FFHQ and CLEVR settings. The code's per-variable dimension is derived from these two values, so it differs as well.
• Integration form. The paper's update has a scale γ on the normalised features and a bias β §3.1.1. With the preset's integration = "mul" run_network.py, the code multiplies by (1 + γ) and does not add β network.py.
• Duplex centroids are learned by default. The paper defines K = a(Y, X), computed from the image features §3.1.2. In the code, iterative defaults to False network.py, so parametric is True network.py. Then the centroid tensor is a learned variable (toasgn_init) that overwrites the assignment-based centroids network.py. Assignments from the same layer enter only in a second k-means iteration, which needs kmeans_iters > 1 (default 1) network.py run_network.py.
• No bottom-up latent update in the default generator. The paper's duplex alternates a latent update from the image with an image update from the latents §3.1.2. The code's bottom-up step runs only with --g-img2ltnt, which the preset does not set network.py run_network.py. Without it, each layer takes its latent from the mapping output and the latents are not refined through the stack network.py.
• Heads. The paper says multi-head attention is used in practice §3.1. The preset leaves num_heads at 1 run_network.py.
• Learning rate. The paper states an equalised learning rate of 0.001 app. A. The code defaults to 0.002 for both networks run_network.py, and the preset does not change it.
• Duplex in the code. The only switch the code calls duplex is --kmeans. The repository does not hold the run commands for the simplex and duplex rows of Table 1, so which switches produced those rows is not verifiable from the repo.

Not compared: the discriminator code, the PyTorch tree, and any run scripts beyond the preset.

Pitfalls

• Latent settings. Take the per-dataset values from tab. 4, not from the code preset. The preset's FFHQ and CLEVR latent sizes differ from the paper (see Paper vs code). An agent copying the preset will not reproduce the paper's setting.
• Modulation form. Implement the paper's γ ⊙ ω(X) + β with layer normalisation §3.1.1. If you follow the code preset, the form is (1 + γ) ⊙ ω(X) with no β network.py.
• Duplex is not the paper's duplex by default. A duplex implementation that follows §3.1.2 needs the bottom-up latent update and same-layer centroids. The default code path has learned centroids and no bottom-up update (see Paper vs code).
• Learning rate conflict. The paper says 0.001 and the code says 0.002. The paper does not say which was used for its results. Treat it as unresolved and sweep if reproducing.
• Unstated values. Not stated in the paper: lazy-regularisation intervals, style-mixing probability, attention dropout, number of heads used in the experiments, and the exact training length in steps. The code defaults above are the only concrete values available.
• Results are only partly comparable. All models here were trained for the same number of steps, which is much shorter than the training behind the StyleGAN2 paper's own FID values. The paper's footnote attributes those lower published FFHQ and Bedroom FIDs to 5 to 7 times longer training §4 footnote. Do not compare these numbers to those published in the StyleGAN2 paper.
• Text vs table. The text says the model "matches or outperforms" prior work §4. On Cityscapes, the simplex variant has FID 14.23 against StyleGAN2's 8.35, and its recall is far lower (2.03 vs 27.82) tab. 1. The text does not discuss this.
• Table 2 wording. The text says the model is better on "almost all" properties §4.4. In tab. 2, GAN has the highest co-occurrence score (13.532 vs 13.020 for the duplex variant), and k-GAN is not in the table.
• Variance. No error bars or seed variance appear in the compiled text tab. 1. A commented-out sentence in the TeX source mentions averaging over repeated runs, but it is not in the published text. Assume single-run numbers.
• Memory. Attention maps are n × k per head per image. For a 256 × 256 final layer, n = 65,536 positions and k = 16 gives about 1.05 M entries per image per head. This is arithmetic from the stated sizes §3.1.3 app. A, not a figure the paper reports. The paper claims linear cost in n for fixed k §3.1.3.
• Compute. The paper gives a rough budget of one week on two V100 GPUs per model §4. The code's default training length is 25,000 kimg run_network.py. The comparison figures in the supplement state 5k to 15k kimg app. Supplementary Material. The run length behind the reported numbers is not stated as one value.
• Figures and table numbers. Table numbers were counted in TeX source order, which should match the compiled paper. Figures are cited by section because no compiled PDF was checked.
• Attention resolution range. The code's default range for generator attention is 0 to 8 in logarithmic resolution run_network.py. The paper says the stack starts at a 4×4 grid §3.1.3 but does not give the range of layers that carry attention. Check the code's range before reusing it.

Lineage

• Builds on: StyleGAN and StyleGAN2, for global style modulation and the training recipe §3.2 app. A; the Transformer, for attention §3.1; key-value memory networks, for the latent keys and values §3.1.2; k-means and slot attention, for the centroid view of duplex attention §3.1.2. The paper gives these as names only; no arXiv ids appear in the source.
• Compared against: a baseline GAN, k-GAN, SAGAN and StyleGAN2 §4. VQGAN appears in the supplementary sample images only app. Supplementary Material.
• Superseded by: not known from content in this wiki.