wikillm

Paper · 2017-02-22 · cs.CV, cs.AI

Unsupervised Diverse Colorization via Generative Adversarial Networks

Colorizes grayscale images with a fully convolutional conditional GAN. Stride-1 convolutions, the gray image concatenated at every layer and noise in early layers let one photo get many plausible colourings. On LSUN bedrooms, 80 people judged generated colourings real more than 62.6% of the time, against 70.0% for real images (author-reported).

Generative adversarial network (GAN)Image-to-image translationcolorizationimage-generation

arXivPDFCodeMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Colouring a black-and-white photo is a guessing game: many colours share the same grey, so a shirt could be red, blue or green. Many earlier systems learn one "safest" colour for each spot. Averaged over many possibilities, that tends to produce washed-out brown tones.

This paper asks a different question. Instead of matching the original colours, it trains a generator to produce colourings that look like real photos of bedrooms. A second network, the critic, tries to tell generated colourings from real ones, and the generator learns from its verdicts. Random noise is fed in at several depths, so the same grey photo can come out in different colours. The generator sees the grey image again at every layer so that it keeps the shapes in the right places.

Human judges were asked whether generated colourings looked real. They were about as convinced by them as by real photos, and the authors report no statistically significant difference.

TL;DR

The paper colorizes a grayscale image with a conditional GAN whose generator uses only stride-1 convolutions. The gray image is concatenated before every layer, and noise enters the first layers, so one input yields several colourings. Trained on 503,900 LSUN bedroom images at 64×64, the authors report that 80 human judges rated generated colourings as real at more than 62.6%, against 70.0% for real images. The released repository contains only the Wasserstein-GAN variant, which the paper says it did not use for its final model.

Problem

Colorization trained with a per-pixel distance tends to output the average of the plausible colours, which looks brownish when one grey level can mean many colours §1. A grey image alone cannot say what colour a wall or a girl's clothes are §1. The authors therefore want a model that outputs a plausible colour scheme rather than the original colours. They argue that distance measures such as RMSE or PSNR are unsuitable here, because many reasonable colourings differ from the original §5.2.

Method

The model is a conditional GAN §3.1. The generator G receives the grayscale image y and a noise vector z, and outputs colour eq. 2 fig. 2. The discriminator D sees real colour images and generated ones fig. 2 fig. 4. Training alternates D and G updates alg. 1.

Generator design §3.2 fig. 4:

  • Only convolutions, with stride 1 everywhere and no pooling or deconvolution. The authors argue this keeps spatial detail that encoder-decoder and U-Net designs would reduce §3.2.1 fig. 3.
  • Colour space is YUV. The Y channel is the grayscale input, so the generator predicts only two channels, U and V §3.2.2. Y is computed with the luminance weights of eq. 5.
  • Noise is projected to a spatial map and concatenated into the first three layers §3.2.3.
  • The grayscale image is concatenated before every layer, so conditioning persists through the network §3.2.4.
  • Each layer is convolution, batch normalisation and ReLU fig. 4.

Discriminator: fig. 4 shows its structure. The paper describes it only as the usual conditional-GAN discriminator §3.2; the layer details are not stated in the text.

RGB alternative: if RGB is predicted instead of YUV, the authors add an L1 term that keeps the generated gray level close to the input §3.2.2. The overall objective is eq. 6. The paper reports this variant gives structural misses §4.2.1 fig. 6.

Wasserstein GAN: the authors tried a WGAN version of the model but say its results were no better, so the final model does not use it §3.2.5 §4.2.4 fig. 9.

Key equations:

\mathcal{L}_{\text{cGAN}}(G,D) = \mathbb{E}_{x}[\log D(x)] + \mathbb{E}_{y,z}[\log(1 - D(G(y,z)))] eq. 2

G^* = \arg\min_G \max_D \mathcal{L}_{\text{cGAN}}(G,D) eq. 3

\mathcal{L}_{L1}(G) = \mathbb{E}_{y,z}[\| y - \mathrm{Gray}(G(y,z)) \|] eq. 4

\mathrm{Gray}(x) = 0.299\,r + 0.587\,g + 0.114\,b eq. 5

G^* = \arg\min_G \max_D \mathcal{L}_{\text{cGAN}}(G,D) + \lambda\,\mathcal{L}_{L1}(G) eq. 6

Training procedure with the paper's default parameters alg. 1:

Pseudocode
# One training iteration  [alg. 1]
# Defaults in the paper: k_D = 1, k_G = 1, m = 64, s_z = 100  [alg. 1]

def train_iteration(D, G):
    # Discriminator steps: real colour images against generated ones
    for step in range(k_D):
        noise = sample_noise(m, s_z)               # distribution not stated
        gray, color = sample_minibatch(m)          # color: (m, 3, 64, 64)
        fake = G(gray, noise)
        # Ascend log D(x) + log(1 - D(G(y, z)))  [eq. 2] [alg. 1]
        d_loss = -mean(log(D(color)) + log(1 - D(fake)))
        update(D, d_loss)
    # Generator steps
    for step in range(k_G):
        noise = sample_noise(m, s_z)
        gray, _ = sample_minibatch(m)
        # Mean of log(1 - D(G(y, z))), written as a gradient  [alg. 1]
        # Eq. 3 says G minimises, so this is a descent step  [eq. 3]
        g_loss = mean(log(1 - D(G(gray, noise))))
        update(G, g_loss)

Generator forward pass, as the paper describes it §3.2.1-§3.2.4:

Pseudocode
# Generator G  [§3.2] [fig. 4]
# Input:  gray (B, 1, 64, 64) in [-1, 1]; noise z (B, 100)
# Output: chroma (B, 2, 64, 64) in [-1, 1]  (U, V channels)

def generate(gray, z):
    # Turn z into a 64x64 noise map; the projection is not stated  [§3.2.3]
    noise = project_to_map(z)                  # (B, 1, 64, 64)
    h = noise                                  # layer 1 receives gray + noise
    for layer in range(1, num_layers):         # num_layers not stated
        # Gray image concatenated before every layer  [§3.2.4]
        inputs = concat(gray, h)
        # Noise also concatenated before layers 2 and 3  [§3.2.3]
        if 2 <= layer <= 3:
            inputs = concat(inputs, noise)
        # Stride-1 convolution, no pooling or deconvolution  [§3.2.1]
        h = relu(batch_norm(conv_stride1(inputs)))
    # Last layer outputs the two chroma channels  [§3.2.2]
    return tanh(conv_stride1(concat(gray, h)))

Testing with fresh noise alg. 2 §3.3:

Pseudocode
# Test-time sampling  [alg. 2]
# Input: gray images (m, 1, 64, 64); k_test rounds (value not stated)

def sample_colourings(gray_batch, k_test):
    # Batch norm makes outputs depend on the batch, so the same batch is
    # re-run with fresh noise every round  [§3.3]
    rounds = []
    for round in range(k_test):
        noise = sample_noise(m, s_z)
        rounds.append(G(gray_batch, noise))
    # Regroup so each image has its k_test colourings  [alg. 2]
    return regroup(rounds)

Hyperparameters

Values in the paper are cited to the paper. Rows marked "code" are read from the release, not stated in the paper, and are not the paper's values.

NameValueWhereNotes
DatasetLSUN bedroom, 503,900 random images§4.1Indoor scenes chosen for varied item colours §4.1 fig. 5
Preprocessingmaximum centre square, resized to 64×64§4.1Pixel scaling to [-1, 1] only in code utils_new.py
Batch size m64alg. 1 alg. 2Code default 64 main_wgan.py
Noise dimension s_z100alg. 1Noise distribution not stated; code uses uniform(-1, 1) model_wgan.py
Image size s64alg. 1
k_D, k_G1, 1alg. 1Code default k_D = 5 main_wgan.py
Noise layersfirst three layers§3.2.3Projection of z to a map: not stated
Condition concatenationevery generator layer§3.2.4
Generator stride1, no pooling§3.2.1
Generator layersnot stated in textfig. 4Code: 6 convolutions, kernels 7,5,5,5,5,5, filters 128,64,64,64,32,2 model_wgan.py
Discriminatornot stated in textfig. 4Code: 4 convolutions, 5×5, stride 2, 64/128/256/512, then linear 64 and 1 model_wgan.py
Gray weights0.299, 0.587, 0.114eq. 5Same weights in code utils_new.py
λ (L1 weight)not statedeq. 6Code flag l1_lambda default 1 is never used main_wgan.py
Epochs, GAN20§4.2.4Only the GAN run used in the WGAN comparison
Epochs, WGAN40§4.2.4Code default is 100, not overridden by its scripts main_wgan.py
Optimisernot stated§4.2.4Paper says WGAN should not use Adam. Code default RMSProp, decay 0.9 main_wgan.py
Learning rate, Dnot statedCode 2e-4 main_wgan.py
Learning rate, Gnot statedCode 1e-4 main_wgan.py
Weight clipping (WGAN)not statedCode 0.01 main_wgan.py
Adam β1 (code option)not statedCode 0.5 main_wgan.py
Gradient penalty λ (code option)not statedCode 10, off by default main_wgan.py
Test rounds k_testnot statedalg. 2Code runs 64 rounds with pickled z model_wgan.py

Results

Author-reported, from a questionnaire given to 80 participants. Each participant saw 20 questions, each with 5 images: one real and four generated, shuffled, and asked whether any was of poor reality §5.2. The paper gives no quantitative comparison with other colourisation methods; its method comparisons are visual §4.2 fig. 6 fig. 7 fig. 8 fig. 9.

SettingMetricBaselineThis paperSource
Human Turing-style survey, 80 participantsshare of images judged realground-truth images: 70.0%more than 62.6% of generated images§5.2
Significance test, ground truth vs generatedp-valuenone0.1359 (> 0.05, so no significant difference reported)§5.2
Credibility rank among 5 images per questionaverage rank (1 to 5)ground-truth images: 2.5generated ranks not given separately; the paper infers at least 37.5% of generated images rank above real ones§5.2
Qualitative RGB vs YUVstructural missesYUV: smoother transitionsRGB with L1: lacks item continuity§4.2.1 fig. 6
Qualitative single vs multi noisecolour diversitysingle-layer noisemulti-layer noise is more colourful§4.2.2 fig. 7
Qualitative single vs multi conditionstabilitysingle-layer conditionmulti-layer condition is more stable§4.2.3 fig. 8

Paper vs code

The repository is https://github.com/ccyyatnet/COLORGAN at commit 5547476836847a7a4caa501e2d81fdda6574fe6d. The paper links it §1. Its README says the code is for this paper README.md. It contains only wgan/ and a demo/ folder, with no LICENSE file.

  • The release implements the WGAN variant, not the final model. The paper's final model uses the conditional-GAN objective eq. 2 eq. 3 and says the WGAN version was not used §3.2.5. The code has a critic without a sigmoid, WGAN losses and weight clipping model_wgan.py model_wgan.py. Algorithm 1's generator loss is not the code's -mean(D(G)) alg. 1 model_wgan.py.
  • Noise is injected only at the input. The paper puts noise into the first three layers §3.2.3. The code concatenates the noise map once, at the first convolution model_wgan.py. multi_z is defined but never read main_wgan.py.
  • The conditional concatenation matches. The gray image is concatenated before every convolution model_wgan.py, as in §3.2.4. multi_condition is defined but never read main_wgan.py.
  • The L1 term is missing. l1_lambda is defined but unused, and the generator has no L1 term main_wgan.py model_wgan.py.
  • RGB mode cannot run. The RGB branch calls generator_colorization without its required image_Y argument model_wgan.py model_wgan.py. The paper's RGB comparison §4.2.1 cannot be reproduced from this code.
  • Step counts differ. Algorithm 1 sets k_D = 1 alg. 1. The code default is K_for_Dtrain = 5 main_wgan.py.
  • Epoch counts differ. The paper uses 20 epochs for GAN and 40 for WGAN §4.2.4. The code default is 100 main_wgan.py, and its scripts do not override it train_lsun_wgan.sh.
  • Discriminator layers are in the code, not the text. The release has 4 convolutions with stride 2, the first without batch norm model_wgan.py.
  • Test images are not held out. Test images are indexed from the same training list model_wgan.py model_wgan.py. Test rounds are fixed by the pickled z batches and run batch_size times model_wgan.py.
  • Batch norm is always in training mode. batch_norm defaults to train=True and is called without overriding it ops_new.py. This matches the batch-dependence the paper describes §3.3.
  • YUV conversion. The Y column matches eq. 5 and adds 127.5 offsets to U and V utils_new.py. The inverse clips to [0, 255] utils_new.py. The forward conversion does not clip utils_new.py.
  • Dataset path and data. The data list is read from ../lsun_64/, not from the README's data/lsun_64/ path model_wgan.py README.md. The list has 503,900 lines, matching §4.1 bedroom_train_valid.lst. Only one sample image is committed .
  • Scripts. train_lsun_wgan.sh calls mian_wgan.py, which does not exist train_lsun_wgan.sh. The other two scripts pass --data_set, not the defined --dataset flag train_lsun_improved_wgan.sh test_lsun_wgan.sh. The improved script uses Adam, which the paper says it did not use for WGAN §4.2.4.
  • Discriminator and generator details. The generator kernel and filter sizes are not in the paper's text. The code uses 7×7 then 5×5 kernels [code: wgan/model_wgan.py#L215-L235].

No discrepancies were checked for the human study, which has no code.

Pitfalls

  • Read the paper, not only the code. The shipped code is a WGAN that the paper says it did not use. An implementer following the code gets different losses, clipping and optimiser behaviour than the paper's final model.
  • Algorithm 1's generator step has no sign. It is written as a gradient of the mean of log(1 − D(G)). Eq. 3 says G minimises, so take a descent step on that term. The release instead uses a WGAN loss.
  • Unstated values that the code fills in. Noise distribution, learning rates, optimiser, λ, k_test, and the projection from z to a map are not in the paper. The code uses uniform(-1, 1) noise, learning rates 2e-4 and 1e-4, and RMSProp main_wgan.py. Treat these as code-only.
  • The disabled settings block. The TeX source has an \iffalse block listing generator kernels of 5 and discriminator stride 2 [paper source, colorGAN.tex lines 361-371]. It is not in the compiled paper, so do not treat it as the paper's settings.
  • Outputs depend on the batch. Batch norm runs in training mode at test time ops_new.py. Algorithm 2's re-running of the batch is required for this reason §3.3. The paper does not state k_test.
  • Y is not predicted. The output concatenates the input Y channel, so the network predicts only UV §3.2.2 model_wgan.py. The input gray is the Y channel of the same YUV conversion, not a separate grayscale image model_wgan.py.
  • Compute and memory. Every layer is stride 1 at full 64×64 resolution, so activation memory stays high in every layer §3.2.1. The paper reports no cost figures.
  • Epoch semantics. Each outer iteration consumes K_D + K_G = 6 minibatches of 64 model_wgan.py. One pass over 503,900 images is 7,873 minibatches model_wgan.py, so about 1,312 outer iterations.
  • The survey metric is underspecified. The paper does not define the 62.6% denominator precisely. It also states that the ground-truth image is added as a reference in case participants find no image real §5.2. The "at least 37.5%" figure is the authors' own inference from the average rank §5.2.
  • Environment. Python 2 print statements and cPickle model_wgan.py and TensorFlow 1.x README.md are required.

Lineage

  • Builds on: conditional GANs as used for image-to-image translation, which the paper cites as Isola et al. §2.2. The paper changes the generator to stride-1 convolutions, with noise and gray concatenation at several layers §3.2.
  • Compared against: Zhang et al. (ECCV 2016), colourisation as classification with class rebalancing §2.1. Deshpande et al., VAE-based colourisation §2.1. Cheng et al., colourisation with extra training per scheme §2.1 §1. The Wasserstein GAN of Arjovsky et al. was tried and not used §3.2.5.
  • Superseded by: not known from wiki content.

Source and license

Adapted from “Unsupervised Diverse Colorization via Generative Adversarial Networks” by Yun Cao, Zhiming Zhou, Weinan Zhang, Yong Yu (arXiv:1702.06674), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ccyyatnet/COLORGAN at commit 554747683684 (not stated (no LICENSE file in repository)). Any quoted code is shown under that license.