Adversarial Generation of Continuous Images

In plain words

Most image generators build a grid of pixels. This paper builds a small neural network instead: given any (x, y) position, it returns the colour at that point. Because the picture is a function of position, you can ask it for more or fewer pixels, or for points outside the original frame.

The generator is a second network that takes random noise and writes out the weights of that small picture network, one set of weights per image. The paper adds two tricks that make this practical: a cheap way to make the weights vary per image without storing a full weight matrix for each one, and a stack of blocks that work at increasing resolution so the picture network does not have to process every pixel at full size. The result is a GAN that generates images in this continuous form and comes close to pixel-based GANs in image quality, while the authors report several side benefits such as zooming out past the edge of the image and generating small images faster.

TL;DR

• A GAN whose generator is a hypernetwork that outputs the weights of a coordinate MLP (an INR) for each sample §3.1.
• Two architectural pieces: factorized multiplicative modulation (FMM) for the per-sample weights §3.2 and multi-scale INR blocks for high resolutions §3.3.
• Author-reported FID 5.09 (LSUN Churches 256²), 4.96 (LSUN Bedrooms 256²), 16.32 (FFHQ 1024²) §1 tab. 1 tab. 2. It is still behind StyleGAN2 on the same datasets tab. 1.
• Trained with the StyleGAN2 setup, with only the generator replaced §3.1 app. A. The released code departs from the paper in several training hyperparameters (see Paper vs code).

Problem

Pixel-based decoders evaluate a grid of fixed size, so a model trained at one resolution cannot produce another without retraining or resampling §1. An implicit neural representation (INR) decouples the output from the grid, but earlier INR-based image generators were limited to MNIST-like data §1 §2. The two obstacles named by the paper are:

• Hypernetworks that emit full weight matrices are unstable to train and need very many parameters §1.
• Evaluating a coordinate MLP at every pixel of a high-resolution grid is costly, which limits INR decoders to low resolutions §1 §3.3.

The paper assumes the usual GAN setting (a fixed discriminator and losses) and changes only the generator §3.1.

Method

Model overview §3.1. The generator G is a hypernetwork. It maps a latent z ~ N(0, I) to parameters θ of an INR F. To render an image, F is evaluated at every point of a fixed grid over [0, 1]², whose size equals the target resolution (for example 256² = 65,536 points for LSUN bedrooms) §3.1. The (x, y) coordinates are first embedded with Fourier-style features, u = sin(γ U p), where the embedding matrix U is produced by G from z rather than fixed §3.1. The embedding is described again in app. C with a cosine companion term eq. 1.

FMM layer §3.2. A naive hypernetwork writes out every weight matrix W^l of F. The output size of that layer grows with n_in × n_out §3.2. The paper instead:

1. keeps one shared full-rank learnable matrix W_s^l per layer;
2. has G output two thin matrices A^l and B^l whose product is a low-rank modulating matrix W_h^l = A^l B^l;
3. forms the weight as the shared matrix times a sigmoid of the modulation, W^l = W_s^l ⊙ σ(W_h^l) §3.2;
4. produces the bias vector directly, since it is small §3.2.

The paper states the rank is r = 10 for all layers, except the first and last, which are small enough to generate directly §3.2. §4.1 fixes the rank at 10 for all experiments. The authors note that a plain low-rank factorisation W = AB, without the shared full-rank part, performed much worse §3.2. They also state that the sigmoid is important for stability §3.2 §4.1.

Multi-scale INRs §3.3. The INR F is split into K blocks. Each block runs at its own resolution, and only the last block runs at the target resolution. The first block starts at 64² and the resolution doubles per block until the target is reached. Each block has 2 to 4 layers, and Fourier coordinate features are concatenated to the hidden features at the start of every block. Between blocks, the low-resolution features are nearest-neighbour upsampled to the next grid §3.3. app. A lists the block counts and widths used for each dataset.

Training §4.1 app. A. The paper keeps the StyleGAN2 discriminator (its "small" version, config-e), losses, optimisers and schedule, and changes only the generator app. A. The loss is the non-saturating logistic GAN loss with an R1 gradient penalty of weight 10, which the paper applies at every iteration rather than lazily app. A §4.1. Adam uses β1 = 0 and β2 = 0.98, ε = 1e-8 app. A. The paper does not use StyleGAN2 tricks such as progressive growing, equalised learning rate, path regularisation, noise injection, pixel normalisation or style mixing for the main results app. A.

Pseudocode: FMM weight for one layer

The paper's formulation §3.2. The released code uses a different default (see Paper vs code).

# Per-sample weight of one FMM layer  [§3.2]
# Input:  shared (c_out, c_in), factors (B, (c_out + c_in) * r)
# Output: weight (B, c_out, c_in)

def fmm_weight(shared, factors, rank):
    # Split the generator output into the two thin factors  [§3.2]
    left = factors[:, : c_out * rank].view(B, c_out, rank)
    right = factors[:, c_out * rank :].view(B, rank, c_in)
    # Low-rank modulating matrix W_h = A B  [§3.2]
    modulating = left @ right
    # Bound the modulation with a sigmoid  [§3.2]
    modulating = sigmoid(modulating)
    # Full-rank shared matrix scaled element-wise  [§3.2]
    weight = shared * modulating
    return weight                             # (B, c_out, c_in)

Pseudocode: multi-scale forward pass

Paraphrase of the block structure §3.3. The coordinate feature function and block depth are the paper's; the exact widths are in app. A.

# Multi-scale INR forward  [§3.3]
# Input:  latent (B, 512), target resolution R
# Output: RGB image (B, 3, R, R)

def multi_scale_inr(latent, target_res):
    # Start with 64x64 and double until the target is reached  [§3.3]
    resolution = 64
    hidden = None
    while resolution <= target_res:
        # Coordinate features at this grid, concatenated to the hidden state
        # at the start of every block  [§3.3]
        coords = fourier_features(grid(resolution))
        hidden = concat(upsample_nearest(hidden), coords)
        # Layers of the block, weights produced per sample  [§3.2] [§3.3]
        for layer in block(resolution):
            hidden = layer(hidden, latent)
        resolution = resolution * 2
    return to_rgb(hidden)

Pseudocode: Keypoints Prediction Loss (KPL)

The evaluation protocol of §4.2 and alg. 1 app. B. Written from the paper's algorithm, with the same symbols.

# Keypoints Prediction Loss  [alg. 1] [app. B]
# Input:  generator G, keypoint extractor K, embedding E, real test images
# Output: KPL score (non-negative)

def kpl(G, K, E, real_test, n_train=10_000):
    # Sample latent codes and decode them into synthetic images  [app. B]
    latents = [sample_latent() for _ in range(n_train)]
    fake_images = [G(w) for w in latents]
    # Keypoints of the synthetic and real images  [alg. 1] [app. B]
    fake_keypoints = [K(x) for x in fake_images]
    real_latents = [E(x) for x in real_test]  # projected, N_ts = 256
    real_keypoints = [K(x) for x in real_test]
    # Fit a linear map from latent to keypoints on synthetic data  [alg. 1]
    A, b = least_squares(latents, fake_keypoints)
    # Sum of squared errors on the real test set (not a mean)  [alg. 1]
    score = sum(
        squared_error(A @ w + b, k)
        for w, k in zip(real_latents, real_keypoints)
    )
    return score

Hyperparameters

Values in the "Paper" column are as stated in the paper. "Code" gives the value the released code uses by default, for comparison. Where the paper gives no value, the row says not stated.

Results

Author-reported. Setup: LSUN Bedrooms 128², LSUN Bedrooms 256², LSUN Churches 256², FFHQ 1024² §4.1. FID on 50k images app. A. GMACs and parameter counts are per generator tab. 1. "O/M" means out of memory at batch size 1 on a 32 GB V100 tab. 1.

Table 1 tab. 1: decoder comparison (FID lower is better)

The paper says INR-GAN is "about three times more efficient" than StyleGAN2 and has about three times as many parameters §4.1. The parameter count is dominated by G's output projection, which the paper puts at 90–99% of all parameters §4.1.

Table 2 tab. 2: additional ablations (FID; img/sec on 1 V100)

The paper reports that removing the sigmoid from FMM worsens FID tab. 2 §4.1. It also reports that for bilinear upsampling INR-GAN beats StyleGAN2 on Churches §4.1.

Table 3 tab. 3: Keypoints Prediction Loss on FFHQ 256² §4.2 (lower values mean that the latent space predicts keypoints better; the authors' statement is in §4.2 and app. B)

Table 4 tab. 4: multi-class datasets (FID, IS at 300k iterations) app. F

The headline FIDs in the introduction (5.09, 4.96, 16.32) are the Table 2 "+ StyleGAN2 architecture" row for the two 256² datasets and the FFHQ 1024² row of Table 1 §1 tab. 1 tab. 2. The headline row is not the same configuration as the main INR-GAN row in Table 2.

Properties (qualitative, no table unless stated) §4.2. The paper shows out-of-grid extrapolation (zoom-out) and image-space interpolation that it calls more meaningful than pixel-space interpolation §4.2. It also reports faster generation at lower resolutions, because the INR can be evaluated on a sparser grid, with MAC counts compared against StyleGAN2 in the FFHQ 1024² setting §4.2. Out-of-the-box super-resolution is scored by UpsampledFID, where the paper reports up to 50% better scores than nearest, bilinear or bicubic upsampling §4.2. The discussion section says the INR-based decoder uses about three times fewer MACs for inference, which it attributes to not aggregating local context §5 tab. 1. The limitations section says INR decoders lack spatial context and that high-frequency features can produce a texture artefact that grows with resolution §6.

Paper vs code

Official repository: universome/inr-gan at commit 5c498e6da5cdfdb13841951c5d42df3509b7781b. The owner handle matches the first author. The project page the paper links redirects to another site, which I did not check for a link to the repository. I read src/training/networks.py, src/training/layers.py, src/training/training_loop.py, src/training/loss.py, src/train.py and configs/inr-gan.yml. The repo has no LICENSE file, so the code is cited, not quoted.

• FMM activation. The paper uses W = W_s ⊙ σ(W_h) §3.2. The default in the released config is demod, which multiplies by (1 + W_h/√r) and then row-normalises the weight to unit L2 norm networks.py inr-gan.yml. The code also has a sigmoid option, but it is centred: σ(·) − 0.5 is added to 1 networks.py. Which variant produced the paper's Table 1 and Table 2 numbers is not stated.
• Rank scope. The paper gives rank 10 for all layers except the first and last §3.2. The code applies the FMM branch to every SynthesisLayer whenever fmm.enabled is set networks.py. I did not find an exception for the first or last layer.
• Fourier frequencies. The paper says the embedding matrix U is predicted by G from z §3.1. In the released config the frequencies are fixed: a log-spaced basis and a random basis are registered as buffers [code: src/training/layers.py#L127-L131], plus a learned constant embedding per resolution layers.py layers.py. The generator-predicted path (predictable_emb_size) is not set in the config, so it is inactive layers.py layers.py.
• Starting resolution. The paper starts the multi-scale blocks at 64² §3.3. The config sets the input resolution to 32 inr-gan.yml, and blocks with a target at or below that size are built with resolution 32 networks.py.
• Channel widths. The paper gives 512-dimensional layers in the 256² blocks app. A. With fmaps: 1, the code's channel count per block is min(32768 / resolution, 512), which gives 256 channels at 128² and 128 at 256² train.py networks.py.
• Mapping depth. The paper gives 4 mapping layers §4.1 or 3 non-linear layers with hidden size 1024 app. A. The code's auto spec uses 2 mapping layers train.py train.py. The paper's text is inconsistent with itself, and the code agrees with neither.
• Adam β2. The paper gives β2 = 0.98 app. A. The code uses 0.99 train.py.
• Learning rates. The paper gives 1e-5 (G), 5e-4 (INR shared parameters) and 3e-3 (D) app. A. The code has one learning rate for G and D: 0.0025 below 1024² and 0.002 at 1024² train.py train.py. No optimiser override is set in the INR-GAN config.
• R1 schedule. The paper applies R1 every iteration app. A. The code defaults to lazy regularisation every 16 iterations for D [code: src/training/training_loop.py#L107], and scales the learning rate and Adam betas by 16/17 for that phase training_loop.py. I found no override of D_reg_interval in src/train.py.
• Training length. The paper trains for 800k iterations §4.1. The auto spec sets 25,000 kimg train.py. The batch size the paper used is not stated, so the two are not directly comparable.
• Checkpoint FIDs. The README reports Churches 256² FID 4.45 and Bedrooms 256² FID 5.71 for INR-GAN README.md, "Performance" table. The paper's Table 2 gives 7.12 and 6.27 for its INR-GAN row. The README FFHQ FID is at 256², not 1024², so it is not in the paper's tables. The difference is not explained.
• Model fidelity. The code additionally has an exponential moving average of G's weights, set through the spec train.py. The paper does not mention EMA.

Pitfalls

• Ambiguous block spec. The FFHQ 1024² description in app. A reads "512 resolution had 2 layers of resolution 128", which is not a consistent block description. Take the block list from the code, or ask the authors, before implementing that setting.
• Activation choice. The paper says the sigmoid in FMM is crucial §3.2 §4.1. The released default is demodulation inr-gan.yml. If you implement the paper, the sigmoid is the stated form. The paper does not report results for the code's demodulation default, so the two are not interchangeable.
• Per-sample weights. FMM produces a weight tensor per image. The paper does not report the memory cost of the batched matrix products §3.2. The README gives 23.54 GB at batch 50 on the released 256² models, and the coordinate embeddings are cached at test time [code: README.md, "Performance" table].
• Output projection size. G's output layer is roughly 10³ × 10⁵ and holds most of the parameters §4.1. It is the main memory cost, not the INR.
• Nearest-neighbour upsampling kills spatial noise. The paper reports that with nearest-neighbour upsampling, the model learns to set the noise strength to zero, so noise injection is useless and was dropped app. A. The config keeps noise off inr-gan.yml.
• Truncation matters. The README says truncation 0.9 (Bedrooms) or 0.95 (for the bilinear variant) is crucial for those checkpoints README.md. The paper uses 0.9 only for the FFHQ sample grids app. B.
• Discriminator size. The paper says the discriminator is "small" (config-e) §4.1. The config uses fmaps: 1 inr-gan.yml. I did not verify that this matches config-e.
• KPL is a sum, not a mean. alg. 1 sums squared errors over the test set. Scores therefore depend on N_ts = 256 app. B. Compare only numbers computed the same way.
• KPL depends on the embedding. The real-image latents come from projecting each FFHQ image with the StyleGAN2 projector, with default hyperparameters, except that the injected noise is not optimised app. B. Projection quality is part of the metric.
• Outside-grid sampling. Zoom-out results use a larger evaluation grid, [−1.5, 1.5]², than the main figure's [−0.3, 1.3]² app. B. Both are evaluation choices, not training.
• Texture artefacts grow with resolution and are discussed as a limitation §6.
• Attribution to StyleGAN2. The paper builds on StyleGAN2 with only the generator replaced §3.1 app. A. Its loss, optimiser and discriminator code come from StyleGAN2-ADA, which the README says carries NVIDIA terms [code: README.md, "License"]. Do not reuse code from this repo without checking that licence.

Lineage

• Builds on: StyleGAN2 (generator swapped; discriminator, losses and training setup kept) §3.1 app. A. Hypernetwork background and low-rank modulation follow RHH and SENet, as the paper cites them §3.2 §2.
• Compared against: latent-code conditioned INR decoders §4.1; hypernetwork decoder of DeepMeta §4.1; StyleGAN2 generator tab. 1.
• Contemporary work with a related approach: CIPS (arXiv:2011.13775), an INR-based generator with learnable coordinate embeddings §1 §2.