wikillm

Paper · 2020-11-27 · cs.CV, cs.AI, cs.LG

Image Generators with Conditionally-Independent Pixel Synthesis

CIPS generates each pixel's RGB value from its (x, y) coordinate and one shared random latent, with no convolution, attention or upsampling between pixels. It adds StyleGANv2-style weight modulation, sine Fourier features and a learned embedding per pixel position. Author-reported FID at 256x256 is close to StyleGANv2 and lower on LSUN Churches and both satellite datasets.

Generative adversarial network (GAN)Implicit neural representation (INR)Positional encodingFourier featuresgenerative-modelsganpixel-synthesisimplicit-neural-representation

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Most image generators build a picture in stages, with filters that look at neighbouring pixels and mix their values. This one does not. For every pixel position it asks a small network for that single pixel's colour, given the position and one random code that the whole picture shares. No pixel ever looks at another, so each one can be computed on its own, in any order, in parallel, or only where you need it.

Picture a painter who fills each square of a grid by reading one recipe card for that square's coordinates, and never looks at the neighbouring squares. The recipe cards are the trained network. The hard part is making such a painter good enough to fool a judge: the authors train it inside a GAN, a setup where a second network tries to tell real images from fakes. At 256 by 256 pixels, the results on several photo datasets come close to StyleGANv2, a strong convolutional generator.

Two additions make the per-pixel approach work. One is a sine-based encoding of the position. The other is a learned vector for each pixel position. Without the learned vectors, the images are clearly worse. The per-pixel design also brings flexibility: the same network can draw a panorama on a different coordinate system, or a denser grid than it was trained on.

TL;DR

CIPS replaces the convolutional stack of a GAN generator with a per-pixel MLP. Each pixel's colour depends only on its coordinate and one latent code, so pixels never exchange information. Using StyleGANv2-style modulation, sine Fourier features and a learned embedding per position, the authors report FID close to StyleGANv2 at 256×256 (lower on three of five datasets) tab. 1. Use it when you need coordinate-driven sampling such as panoramas, foveated rendering or denser grids. Check the Paper vs code section first: in the released code, the learned position embedding is sampled by interpolation outside the training grid, which the paper does not describe.

Problem

Current unconditional image generators are built from spatial convolutions, and optionally self-attention, that synthesise images coarse-to-fine §1. The paper asks whether a generator with no spatial convolution, self-attention or upsampling can reach that quality on medium-resolution images (256×256) §1. Per-pixel coordinate networks had already shown that single images or scenes can be represented this way, but they had not been trained as image generators §1, §2.

Two further benefits are claimed for the per-pixel design. Non-grid topologies such as cylindrical panoramas need no special convolution §1. Pixels can be synthesised in memory-limited batches, or with more pixels in a region of interest (foveated synthesis) §1.

Assumptions: adversarial training with a discriminator §4.1; a fixed output resolution, 256×256 for almost all experiments §4.2. The paper states that convolutional upsampling can prevent a generator from learning the real spectral distribution, citing prior work, and that a pixel-wise design avoids that step §4.5.

Method

The generator G maps a shared latent z and a pixel position (x, y) to an RGB value. The full image is the set of G(x, y; z) over the whole grid, with z fixed for that image §3 eq. 1. A mapping network M, itself an MLP, converts z into a style vector w; the image's randomness enters only through w §3.

Each modulated fully connected (ModFC) layer computes ψ = B̂φ + b, where B̂ is a learnable weight matrix modulated per input by a scale vector s derived from w, then normalised per output row, followed by LeakyReLU §3 eq. 2. The modulation follows StyleGANv2 §3.

Positions are encoded in two parts §3.1 eq. 3 eq. 4. A sine layer maps the coordinates, rescaled to [-1, 1], to Fourier features. A separate vector is learned for every grid position (the coordinate embedding). The two are concatenated and fed to the network. Both parts are 512-dimensional §4.1.

The default configuration app. A fig. 2:

  1. Map z to the style w with M §3.
  2. Encode each position with Fourier features and its coordinate embedding, then concatenate §3.1 eq. 4.
  3. Project the 1024-dimensional concatenation to width 512 with one modulated layer app. A.
  4. Repeat seven times: two modulated layers, then an RGB projection; add the RGB outputs of all repeats §3 app. A.
  5. Output one RGB value per position §3.

The pseudocode below follows the paper's equations and the released code. Where they differ, the difference is noted in the Paper vs code section.

Pseudocode
# Generate one image from latent z  [§3] [eq. 1]
# Input:  z (latent), coords (H*W, 2), values in [-1, 1]  [§3.1]
# Output: rgb (H*W, 3), each row computed on its own

def generate(z, coords):
    # Mapping network M turns z into the style vector w  [§3]
    style = mapping_network(z)
    # Sine Fourier features of each position  [§3.1] [eq. 3]
    fourier = sin(fourier_layer(coords))
    # One learned embedding per grid position  [§3.1]
    embedding = coordinate_embedding(coords)
    # Concatenate both encodings  [§3.1] [eq. 4]
    features = concat(fourier, embedding)
    # First modulated layer projects 1024 -> 512  [app. A]
    features = modulated_layer(features, style)
    rgb = 0
    # Seven blocks, each with two modulated layers  [§3] [app. A]
    for block in range(7):
        features = modulated_layer(features, style)
        features = modulated_layer(features, style)
        # Project to RGB and add to earlier projections  [§3]
        rgb = rgb + to_rgb(features, style)
    return rgb
Pseudocode
# Modulated fully connected layer  [§3] [eq. 2]
# Input:  features (N, n); style w
# Output: features (N, m), after LeakyReLU

def modulated_layer(features, style):
    # Scale vector s, derived from the style by a small net A  [§3]
    scale = style_to_scale(style)                    # (n,)
    # Column j of the base weight is multiplied by s_j  [eq. 2]
    weight = scale * base_weight                     # (m, n)
    # Normalise each output row by its root sum of squares  [eq. 2]
    norm = sqrt(eps + sum(weight ** 2, axis=1))      # (m,)
    weight = weight / norm[:, None]                  # (m, n)
    out = features @ weight.T + bias                 # (N, m)
    # LeakyReLU, negative slope 0.2  [§4.1]
    return leaky_relu(out, slope=0.2)

Training, as stated in the paper §4.1: a non-saturating logistic GAN loss with an R1 penalty on the discriminator. The discriminator is a residual network from StyleGANv2, kept unchanged. The optimiser is Adam with learning rate 2e-3, β0 = 0, β1 = 0.99 and ε = 1e-8. The released code changes how these values are applied (see Hyperparameters and Paper vs code).

Pseudocode
# One adversarial step as implemented in the released code  [code: train.py#L118-L183]
# Discriminator loss: non-saturating logistic  [§4.1] [code: train.py#L53-L57]
# R1 penalty is applied lazily, not every step  [code: train.py#L155-L165]

def train_step(real, z, step):
    # Generator makes fake images from the latent  [code: train.py#L139]
    fake = generate_batch(z)
    # Discriminator scores real and fake images  [code: train.py#L141-L145]
    d_loss = softplus(-disc(real)).mean() + softplus(disc(fake)).mean()
    # Every 16 steps add R1 on real images, weighted  [code: train.py#L157-L163]
    if step % 16 == 0:
        d_loss = d_loss + r1_penalty(disc, real) * (10 / 2) * 16
    # Generator update with the same non-saturating loss  [code: train.py#L177]
    g_loss = softplus(-disc(generate_batch(z))).mean()
    return d_loss, g_loss

Hyperparameters

NameValueWhereNotes
Fourier feature dimension512§4.1sine layer, learnable 2×n weight eq. 3
Coordinate embedding dimension512§4.1one learned vector per grid position
Modulated layers14 in the main text; 15 counting the first projection§4.1 app. Athe appendix count includes the first projection
Layer width512§4.1the released default narrows the last two blocks, see Paper vs code
Residual variantresidual blocks scaled by 1/√2§4.3ablation only; default is cumulative RGB skips
Skip connectionsRGB projection every two layers, summed§3 app. Adefault configuration
LeakyReLU slope0.2§4.1the released code also applies a √2 gain, see Paper vs code
Sine activationfirst (Fourier) layer§3.1other layers use LeakyReLU
Mapping network depthnot stated§3released code: 8 equalised layers, lr multiplier 0.01 GeneratorsCIPS.py GeneratorsCIPS.py
Latent dimensionnot stated§3released code default: 512 train.py
Modulation epsilonnot statedeq. 2released code: 1e-8 blocks.py
Adam learning rate2e-3§4.1released code scales it per network, see Paper vs code
Adam β0 / β10 / 0.99§4.1released code scales β1 per network
Adam ε1e-8§4.1same as the PyTorch default used in the code
GAN lossnon-saturating logistic§4.1
R1 penalty weightnot stated§4.1released code: 10, applied every 16 steps with a matching factor train.py train.py
Style mixingoffapp. Dthe paper says no mixing regularisation was applied; code default 0 train.py
Batch size16 for ablations§4.3final models: not stated; released code default 4 per process, README uses 8 processes train.py
Training length300K iterations for ablations§4.3final models: not stated; released code default 1.2M train.py
Discriminator inputnot stated§4.1released code default: image plus coordinate channels; README uses RGB only train.py train.py
Discriminator width multipliernot stated§4.1released code: 2 train.py
Horizontal flipnot statedreleased code applies it to training images train.py
Generator EMAnot statedreleased code evaluates an averaged copy, decay about 0.9978 per step train.py
Coordinate range[-1, 1], linear grid§3.1matches code tensor_transforms.py
Image resolution256×256§4.2coordinate grids of 1024×1024 used at inference §4.7
DatasetsFFHQ 70,000; LSUN Churches 126,000; Landscapes 60,000; Satellite-Buildings 280,741 (300 px, cropped to 256); Satellite-Landscapes 2,608 (512 px, cropped to 256)§4.2
FID sample countnot statedreleased code default: 50,000 train.py

Results

Author-reported. Setup: 256×256, CIPS with skip connections, five datasets, FID with a Fréchet Inception setup §4.2 tab. 1. The StyleGANv2 value for LSUN Churches is taken from its original paper. The other StyleGANv2 models were trained by the authors without style mixing and path regularisation §4.2.

SettingMetricBaseline (StyleGANv2)This paper (CIPS)Source
FFHQFID, lower is better3.834.38tab. 1
LSUN ChurchesFID, lower is better3.862.92tab. 1
LandscapesFID, lower is better2.363.61tab. 1
Satellite-BuildingsFID, lower is better73.6769.67tab. 1
Satellite-LandscapesFID, lower is better51.5448.47tab. 1
FFHQ 256×256Precision0.6090.613tab. 2
FFHQ 256×256Recall0.5130.493tab. 2

The paper reports that the two models are not uniformly better. StyleGANv2 has the lower FID on FFHQ and Landscapes, and CIPS on LSUN Churches and both satellite sets §4.2. CIPS has higher precision but lower recall on FFHQ, which the authors read as more plausible samples with fewer covered modes tab. 2.

Ablations on FFHQ, 300K iterations, batch size 16 §4.3 tab. 3:

VariantFourier featuresCoordinate embeddingResidual blocksSkip to RGBSine activationFID
Baseyesyesnonono6.71
Without coordinate embeddingyesnononono12.71
Without Fourier featuresnoyesnonono10.18
Default (main)yesyesnoyesno6.31
Residual blocksyesyesyesnono6.52
Sine activationnoyesnonoyes10.0

Removing the coordinate embedding roughly doubles FID (6.31 to 12.71), and the authors call it the most damaging change §4.3. Using both encodings is needed for a good score tab. 3.

Positional-encoding analysis, FFHQ §4.4 fig. 4 fig. 5 fig. 6: Fourier features carry mostly low-frequency content, while the coordinate embeddings carry high-frequency detail. Zeroing the coordinate embeddings removes fine details such as facial hair and the forelock, which the authors take as evidence that they hold the high-frequency information §4.4 fig. 6.

Spectrum, FFHQ, 5000 samples per model §4.5 fig. 7: StyleGANv2 shows grid-like high-frequency artefacts that the CIPS variants do not. The variant without coordinate embeddings has a spectral tail closer to real images, but a worse FID §4.5.

Patch-based training, memory-constrained, with patches of 64 or 128 pixels and full images of 256 app. C:

DatasetPatch 256 (full image)Patch 128Patch 64
FFHQ FID4.389.0811.79
LSUN Churches FID2.927.0811.53

Parameter counts app. A: CIPS-base 43.8M, CIPS-NE 10.2M, CIPS default 45.9M, StyleGANv2 generator 30.0M.

Qualitative results, without numbers §4.6–§4.8:

  • Latent interpolation gives smooth morphs between sampled images §4.6 fig. 10.
  • Foveated synthesis samples 5%, 25%, 50% or 100% of pixels, weighted toward a central region, and fills the gaps by bicubic interpolation §4.7 fig. 8.
  • Sampling a 256-trained model on a 1024×1024 grid gives sharper eyelids and pupils than Lanczos upscaling of the 256 output, according to the authors §4.7 fig. 9.
  • Cylindrical panoramas are trained on Landscapes crops without any real panoramas and show blending of two panoramas in a style space §4.8 fig. 11 app. D.
  • Nearest-neighbour search with FaceNet finds no duplicates of training faces, so the authors argue against memorisation app. E.
  • Typical artefacts are wavy textures in hair and backgrounds, repeated line patterns in buildings, and glowing blobs. The authors attribute them to the sine layer and the LeakyReLU partitioning of the coordinate grid §4.9 fig. 12.

Paper vs code

Code: official repository advimman/CIPS at commit eadae6e45d8c1f3657faa88a065b59990747cd16. The README and LICENSE name the authors' lab (Samsung AI Center Moscow), the paper's affiliation. Searches also returned a third-party TensorFlow port (taki0112/CIPS-Tensorflow); it was not used.

  1. Output range. The paper says each RGB value lies in [0, 1] §3. The code trains against images normalised to [-1, 1] train.py, and the RGB output is a linear projection with no clamp or output activation GeneratorsCIPS.py blocks.py. Samples are saved with range (-1, 1) train.py.
  2. Coordinate embedding off the training grid. The paper defines one learned vector per position §3.1. The code uses that exact lookup only in training mode when the grid is 256×256 GeneratorsCIPS.py. In every other case, including evaluation mode at 256×256, it samples the learned 256×256 table with bilinear grid_sample and border padding GeneratorsCIPS.py. The call sets no align_corners argument, so PyTorch's default (False) applies. For linearly spaced coordinates on a 256 table, pixel i then reads table position i·256/255 − 0.5 rather than i. The sample images in train.py are drawn from the evaluation-mode EMA generator, so they use this path train.py train.py. The FID call receives the same EMA generator train.py. This is derived from reading the code; it was not run.
  3. Stochasticity. The paper says all randomness comes from the style vector §3. The code adds a per-pixel Gaussian noise term in every convolution, scaled by a learnable scalar that starts at zero blocks.py blocks.py. The generator forward pass does not pass a noise tensor, so fresh noise is drawn on every call GeneratorsCIPS.py. The paper does not mention this term.
  4. Width. The paper says the generator has width 512 §4.1. The default code configuration for 256 output uses 512 channels in the first five blocks and 256 and 128 in the last two GeneratorsCIPS.py GeneratorsCIPS.py.
  5. Depth. The main text says 14 modulated layers §4.1. The appendix counts 15, including the first projection app. A. The code has the first projection plus 14 modulated convolutions GeneratorsCIPS.py GeneratorsCIPS.py.
  6. Learning rate and betas. The paper gives lr 2e-3, β1 0.99 §4.1. The code scales them by the lazy-regularisation ratios: the generator gets lr 1.6e-3 (×4/5) and β1 ≈ 0.9920; the discriminator gets lr ≈ 1.88e-3 (×16/17) and β1 ≈ 0.9906 train.py.
  7. Path-length regularisation. The code has a g_reg_every option, but it only changes the learning-rate ratio. The path-length loss is fixed at zero train.py train.py. The paper does not describe path regularisation for CIPS; it says it was switched off for the StyleGANv2 baseline §4.2.
  8. Modulation. The paper describes the scale network A as a small net §3, fig. 2. The code uses a single equalised linear layer with bias initialised to 1 blocks.py blocks.py. It also multiplies the weights by 1/√(fan-in), which equation 2 does not show blocks.py.
  9. Activation gain. The code's LeakyReLU has a default gain of √2 fused_act.py. The paper gives only the slope §4.1.
  10. Fourier layer. The code's sine layer is a 1×1 convolution with a bias, initialised uniformly in ±√(9/2) for the first layer blocks.py blocks.py. The paper states neither the bias nor the initialisation §3.1.
  11. RGB projection. The code's RGB projection is a style-modulated 1×1 convolution without demodulation, plus a bias blocks.py. The paper calls it a projection to RGB app. A and does not give the detail.
  12. Patch training. The paper's patch sampling uses dilated sub-grids with spacing σ app. C. The code, when crop size is below image size, draws one random stride and offset per sample from the multi-scale sub-grids and picks one scale per iteration dataset.py train.py. With the default crop of 256, there is one scale only train.py.
  13. Augmentation. The code's training transform includes random horizontal flips train.py. The paper does not mention flips. It says the Satellite-Buildings images are randomly rotated §4.2; the repository's training transform has no rotation, and the data-preparation script only resizes prepare_data.py.
  14. Discriminator input. The paper does not state the input. The code defaults to five channels (RGB plus coordinates) and the README's training command uses RGB only train.py train.py train.py.
  15. Licensing of the code. The repository LICENSE is MIT LICENSE. The CUDA kernels op/fused_bias_act_kernel.cu and op/upfirdn2d_kernel.cu are NVIDIA Source Code License-NC (non-commercial) LICENSE-NVIDIA. The README states the same [README]. This card quotes no code from op/ and cites only the default value of one wrapper. Any implementation that uses those kernels is non-commercial.

Pitfalls

  • Off-grid coordinate embeddings. Inference at any size other than 256×256, and inference in evaluation mode, interpolate the learned table (Paper vs code, item 2). This applies to the 1024×1024 result §4.7 and to the panoramas and foveated samples §4.7–§4.8. An implementation that does an exact per-position lookup will not reproduce the released samples or FID.
  • Output range. Train on [-1, 1] and clip or map as the code does. The paper says [0, 1] §3.
  • Hidden noise. A learnable per-pixel noise term is present in the released model. Its scale starts at zero, but it is learned, so an implementation that drops it may not match the released checkpoints blocks.py.
  • Learning-rate adjustment. The code's learning rates and β1 are not the paper's nominal values (Paper vs code, item 6). Using 2e-3 and 0.99 directly is a different configuration.
  • Missing training details. Not stated: the R1 weight, the latent dimension, the mapping depth, batch size and iterations for the final models, EMA, and the discriminator input. The code's defaults are listed in Hyperparameters. They are the authors' code, not the paper's text.
  • Cost. Every pixel passes through every layer. The default network has 45.9M parameters against 30.0M for the StyleGANv2 generator app. A. Every pixel carries 512 channels in most blocks. The paper reports no runtime, FLOP or memory numbers, so these costs are not stated. The paper does say that pixels can be computed in parallel at inference §3.
  • Patch training costs quality. FID rises from 4.38 to 11.79 on FFHQ when patches shrink from 256 to 64 app. C. The authors call the samples meaningful but worse in quality and diversity app. C.
  • Table numbers are not comparable. The ablation FFHQ FID of 6.31 (Table 3) differs from the 4.38 in Table 1, because the main model was trained longer with a larger batch tab. 1 caption §4.3.
  • Author-reported baselines. The StyleGANv2 numbers come partly from a different training setup (no style mixing, no path regularisation) and partly from the original paper §4.2.
  • Spectrum versus FID. The variant without coordinate embeddings has the more realistic spectrum but the worse FID §4.5 tab. 3. Choose by which property you need.
  • Artefacts. Wavy textures in hair and backgrounds and repeated line patterns §4.9 are visible in the authors' failure cases fig. 12. The paper does not offer a fix.
  • License. Use the code from this repository only for non-commercial work if you need the CUDA kernels. The paper itself is CC BY 4.0.

Lineage

  • Builds on: StyleGANv2 for modulation, the mapping network, the residual discriminator, R1 and the training recipe §3 §4.1. StyleGAN for the mapping-network idea §3. SIREN and Fourier features for sine-based and Fourier coordinate encodings §3.1. Scene-representation and NeRF networks for coordinate-based MLPs §1 §2. CoordConv for feeding coordinates as inputs §2.
  • Compared against: StyleGANv2 on five datasets, with FID and precision/recall tab. 1 tab. 2. COCO-GAN and GRAF are discussed in related work without experimental comparison §2.
  • Superseded by: not determined from this wiki.

Source and license

Adapted from “Image Generators with Conditionally-Independent Pixel Synthesis” by Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, Denis Korzhenkov (arXiv:2011.13775), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/advimman/CIPS at commit eadae6e45d8c (MIT (repository LICENSE); op/ CUDA kernels under NVIDIA Source Code License-NC (non-commercial)). Any quoted code is shown under that license.