GLEAN: Generative Latent Bank for Large-Factor Image Super-Resolution

In plain words

Enlarging a tiny image by 16 times leaves almost no detail, so the upscaler has to invent plausible texture. One approach trains an upscaler to paint textures itself, and the results often look artificial. Another searches a picture-generating model for an image that matches the small input, which is slow for every image and can lose the true face.

GLEAN keeps a picture generator fixed and uses it as a bank of ready-made visual knowledge. A small encoder reads the low-resolution picture and produces a compact code plus several intermediate feature maps. These steer the generator, which makes its textures, and a decoder combines that output with the encoder's own features to give the final picture. One pass through the network produces the result, with no search for each image.

The bank can be swapped for another pretrained generator, so the same design covers different kinds of image, such as faces, cats, cars, bedrooms and towers.

TL;DR

GLEAN is an encoder, latent bank and decoder for super-resolution at large factors (8x to 64x) §1. It uses a pretrained, frozen StyleGAN as the bank, conditioned on latent vectors and multi-resolution encoder features, so no per-image optimisation is needed §3. The author-reported 16x face result has the highest ArcFace similarity of the compared methods tab. 1. The official code is in MMEditing (now MMagic, Apache-2.0), not in the authors' own repository, and it differs from the paper in several details listed under Paper vs code.

Problem

Large magnification factors discard most of the detail, so the output needs strong priors §1. Two existing uses of GANs fall short. Training the upscaler with an adversarial loss makes the generator carry both fidelity and natural texture, which yields artifacts and unnatural texture §1. GAN inversion optimises a latent code for each image at run time, which is slow, and a low-dimensional code does not keep spatial structure, so fidelity drops §1 §2. Reference-based methods need a similar reference image and costly global matching, which fails at low resolution §3.2 §5.

The paper trains and tests on FFHQ and CelebA-HQ for faces, and on LSUN-derived sets for cats, cars, bedrooms and towers app. A, tab. 3. Training uses the bicubic kernel §5.

Method

GLEAN has three parts fig. 2. An encoder turns the low-resolution image into latent vectors and a series of feature maps §3.1. These condition the latent bank, a pretrained StyleGAN whose weights are frozen, and the bank returns a set of feature maps §3.2. A decoder with skip connections fuses the bank features with the encoder features and produces the output §3.3.

Encoder §3.1. An RRDB trunk, as in ESRGAN, gives the first feature map f_0. Each later stage halves the resolution with a stride-2 convolution followed by a stride-1 convolution eq. 1. A final convolution and a fully connected layer map the coarsest map to the latent matrix C eq. 2.

f_i = E_i(f_{i-1}), \quad i \in \{1, \dots, N\} eq. 1

C = E_{N+1}(f_N) eq. 2

Generative latent bank §3.2. StyleGAN is changed in three ways. (1) Each block takes its own latent vector c_i, so C holds one vector per block; the paper reports fewer artifacts from this change §3.2. (2) Each style block has one extra convolution that fuses an encoder feature map §3.2. (3) The bank outputs its intermediate features g_i rather than an image, and passes them to the decoder §3.2.

g_i = \begin{cases} S_0(\mathbf{c}_0, f_N), & \text{if } i = 0, \\ S_i(\mathbf{c}_i, g_{i-1}, f_{N-i}), & \text{otherwise} \end{cases} eq. 3

Decoder §3.3. The decoder starts from f_0 and progressively fuses the bank features. Each stage is a 3x3 convolution. Every convolution except the final output layer is followed by a pixel-shuffle upsampling step §3.3.

d_i = \begin{cases} D_0(f_0) & \text{if } i = 0, \\ D_i(d_{i-1}, g_{N-2+i}) & \text{otherwise} \end{cases} eq. 4

Training §3.4 app. A. The bank weights stay fixed. The paper says fine-tuning the bank gave no clear gain and could bias the bank toward the training distribution §3.4. The generator loss is a weighted sum of a pixel loss, a VGG16 perceptual loss and an adversarial loss app. A, eq. 5-8. The discriminator is trained with the standard real/fake objective app. A, eq. 9. The paper states "Cosine Annealing" with Adam, 300K iterations and initial learning rate 1e-4 app. A.

\mathcal{L}_{mse} = \frac{1}{N} \lVert \hat{y} - y \rVert_2^2 app. A, eq. 5

\mathcal{L}_{percep} = \frac{1}{N} \lVert f(\hat{y}) - f(y) \rVert_2^2 app. A, eq. 6

\mathcal{L}_{gen} = \log\left(1 - D(\hat{y})\right) app. A, eq. 7

\mathcal{L}_g = \mathcal{L}_{mse} + \alpha_{percep} \cdot \mathcal{L}_{percep} + \alpha_{gen} \cdot \mathcal{L}_{gen} app. A, eq. 8

\mathcal{L}_d = \log\left(1 - D(\hat{y})\right) + \log D(y) app. A, eq. 9; the discriminator maximises this

Here f is the VGG16 feature embedding app. A, y is the ground truth, and ŷ is the output. The paper gives alpha_percep = alpha_gen = 1e-2 app. A.

Pseudocode for the forward pass, written from §3 and eq. 1-4. Indices follow the paper:

# GLEAN forward pass: one pass, no per-image optimisation  [§3]
# Input:  lr_image (B, 3, s, s), s = in_size
# Output: sr_image (B, 3, S, S), S = out_size

def glean(lr_image):
    # Encoder E_0: RRDB trunk gives f_0 at input resolution  [§3.1]
    feats = [rrdb_trunk(lr_image)]          # f_0
    # Each stage: stride-2 conv, then stride-1 conv  [§3.1] [eq. 1]
    for i in range(1, N + 1):
        feats.append(encoder_stage(i, feats[i - 1]))    # f_i
    # Final conv and fully connected layer give the latents  [eq. 2]
    latents = latent_head(feats[N])         # C: k vectors
    # Bank block 0 uses the coarsest encoder map  [§3.2] [eq. 3]
    bank = [bank_block(0, latents[0], None, feats[N])]  # g_0
    # Later blocks: own latent, previous bank output, matching map  [eq. 3]
    for i in range(1, k):
        prev = bank[i - 1]
        bank.append(bank_block(i, latents[i], prev, feats[N - i]))  # g_i
    # Decoder starts from f_0  [§3.3] [eq. 4]
    dec = decoder_stage(0, feats[0])        # d_0
    # Later stages fuse bank output g_{N-2+i}  [§3.3] [eq. 4]
    # M: number of decoder stages, not stated in the paper
    for i in range(1, M):
        dec = decoder_stage(i, dec, bank[N - 2 + i])    # d_i
    # Output layer: a convolution without pixel shuffle  [§3.3]
    return output_conv(dec)                 # (B, 3, S, S)

# Generator objective, weights from the appendix  [app. A]
# hr = ground truth, sr = glean(lr_image)

def generator_loss(sr, hr, vgg, disc):
    # Pixel fidelity: mean squared error  [app. A, eq. 5]
    loss_mse = mean((sr - hr) ** 2)
    # Perceptual: squared distance of VGG16 features  [app. A, eq. 6]
    loss_percep = mean((vgg(sr) - vgg(hr)) ** 2)
    # Adversarial term as written in the paper  [app. A, eq. 7]
    loss_gen = log(1 - disc(sr))
    # Weighted sum, both weights 1e-2  [app. A, eq. 8]
    return loss_mse + 1e-2 * loss_percep + 1e-2 * loss_gen

Hyperparameters

Values in the paper are cited to the paper. Values taken from the released code are labelled as such and cited to code. "not stated" means neither the paper nor its appendix gives a value.

Results

Author-reported. Setup: 16x super-resolution §4. Faces are CelebA-HQ, with 100 images for the ArcFace test §4. Each category in the PSNR/LPIPS test uses 100 images §4. Compared methods: PULSE, mGANprior, DGP, SinGAN, and ESRGAN+ (a larger ESRGAN with similar FLOPs to GLEAN) §4. Hardware for inference is not stated.

ArcFace cosine similarity on CelebA-HQ faces, 16x tab. 1:

PSNR / LPIPS on 16x super-resolution, by category tab. 2. The table bolds the best value in each row.

In the bedroom row, ESRGAN+ is best on both metrics and GLEAN is close on PSNR. The paper does not report variance, repeat runs or significance tests.

Qualitative results are shown in fig. 1 (16x examples of the input, ESRGAN, PULSE, GLEAN and ground truth), fig. 3 (16x faces), fig. 4 (other categories), fig. 5 (up to 64x), and fig. 6 (non-aligned and non-face inputs). The ablations in §5 are qualitative only, with no tables: fig. 7 shows the effect of encoder features, fig. 8 the latent bank features, fig. 9 the decoder, and fig. 10 the comparison with dictionary-based methods. The retouching application is also qualitative fig. 11 app. B.2 fig. 19.

Paper vs code

Code location. The authors' repository, ckkelvinchan/GLEAN, is linked from the project page for the CVPR 2021 paper. Its README says it contains only dataset metadata and that code and models are in MMEditing. The clone holds only the README and metadata files. The GLEAN implementation is in open-mmlab/mmediting (renamed MMagic), which is maintained by OpenMMLab, not by the authors. It was used as the code home because the authors' repository names it. The clone is at the commit listed in the frontmatter, and its license is Apache-2.0. The code is a later release than the 2021 paper, so defaults may have changed since publication.

Discrepancies. Each item below has both citations.

1. Generator adversarial loss. The paper writes the generator term as log(1 − D(ŷ)) app. A, eq. 7. The code computes the generator's GAN loss against the real label, with target_is_real=True srgan.py. The target label is set to the real value (1.0) gan_loss.py glean_x16_2xb8_ffhq.py. The exact vanilla formula sits in the loss module and was not checked line by line.
2. Discriminator initialisation. The paper says the discriminator "corresponds to the StyleGAN discriminator" app. A and does not say how it is initialised. The code loads pretrained StyleGAN2 discriminator weights and trains them with Adam glean_x16_2xb8_ffhq.py [code: configs/base/models/base_glean.py#L27-L29].
3. Latent bank version. The paper uses StyleGAN or StyleGAN2, depending on which pretrained model is available §4. The code uses only the StyleGAN2 generator class and its weights glean_styleganv2.py.
4. Generator weights. The code loads the EMA generator (generator_ema) glean_x16_2xb8_ffhq.py. The paper does not say which weights it uses.
5. Output stage. The paper says every convolution except the final output layer is followed by pixel shuffle §3.3. The code's output stage is a 3x3 convolution to 64 channels, a LeakyReLU, and a 3x3 convolution to 3 channels glean_styleganv2.py. Pixel shuffle is used in the earlier decoder stages glean_styleganv2.py.
6. Learning-rate schedule. The paper says cosine annealing over 300K iterations app. A. The code sets T_max to 600,000 and eta_min to 1e-7 [code: configs/base/models/base_glean.py#L33-L34], with 300,000 max iterations [code: configs/base/models/base_glean.py#L17]. By the standard cosine formula, the learning rate at 300K is about 5e-5, so training ends at half a cosine cycle, not at the minimum. This value is derived from the code, not stated in the paper.
7. Batch size. The paper gives 8 for faces and 16 for other categories app. A. The configs are named 2xb8 and set batch_size 8 per dataloader glean_x16_2xb8_ffhq.py glean_x16_2xb8_cat.py. By the naming convention this is 8 per GPU on 2 GPUs, so the total batch is 16. That total does not match the paper's 8 for faces. The GPU count is inferred from the name, not stated in the config.
8. Training degradation. The paper says GLEAN is trained on the bicubic kernel §5. The 16x and 8x configs read pre-made BIx16_down and BIx8_down folders and apply only a flip glean_x16_2xb8_ffhq.py glean_x8_2xb8_cat.py. The configs do not say how those folders were made. A separate 128→1024 face config applies random blur, resizing, Gaussian noise and JPEG compression glean_in128out1024_4xb2-300k_ffhq-celeba-hq.py. That config is not described in the paper.
9. Encoder count and latents. N and k are not stated in the paper. The code gives N = log2(s) − 2 glean_styleganv2.py and k = 2·log2(S) − 2 glean_styleganv2.py.
10. Encoder fusion range. The paper's equation fuses an encoder map in every bank block §3.2 eq. 3. The code fuses only at resolutions at or below the input size glean_styleganv2.py, and passes to the decoder only the bank maps above the input size glean_styleganv2.py.
11. RRDB depth. The paper does not state the RRDB count. The code default is 23 blocks glean_styleganv2.py.

No code was found for the pretrained StyleGAN weights themselves; they are downloaded from URLs listed in the code docstring glean_styleganv2.py.

Pitfalls

• Adversarial term. Implement the paper's formula or the code's real-label target, not both. The two differ (see Paper vs code, item 1). Check which one the reported numbers used; the paper does not say.
• Perceptual normalisation. The paper's 1/N factor for the VGG term is not defined for feature maps. The code uses criterion='mse' on VGG16 features at index 21 glean_x16_2xb8_ffhq.py. The averaging scheme is not stated.
• Batch size. Use 8 per GPU and check the GPU count before copying the paper's batch sizes.
• Learning rate at the end. A 300K run with T_max = 600K ends near 5e-5, not near zero. Copy the code's schedule if you want to match the released setup.
• Latent count. The number of latent vectors is set by output size, not by the paper. Check the config before changing the output resolution.
• Decoder indexing. Equation 4 uses g_{N-2+i}. The released code concatenates bank maps above the input size, so indexing must be re-derived from the code, not the equation.
• Pretrained weights and licence. The bank and discriminator weights are StyleGAN2 checkpoints hosted by OpenMMLab and originally released with StyleGAN2 §4 app. A. Their licence was not checked for this card. Confirm it before any commercial use.
• Cost. The paper says GLEAN is computationally efficient §3.2 but gives no runtime, memory or FLOP figures for it. The only FLOP statement is that ESRGAN+ has similar FLOPs, with no value §4.
• Evaluation. The quantitative tests use 100 images per category §4 tab. 1 tab. 2. The paper reports no variance across seeds or repeated runs.
• Ablations. The ablations are qualitative §5 fig. 7-10. Do not treat them as measured effects.
• Figure and equation numbers. No PDF was available. Figure numbers in this card follow source order in the LaTeX, and appendix equation numbers (5-9) assume the default LaTeX counter. Check against the compiled PDF before quoting numbers.
• Training kernel. The paper says bicubic §5, but the released 128→1024 face config trains with random degradations. Do not assume the bicubic setup when using that config.

Lineage

• Builds on: StyleGAN [karras2018style] and StyleGAN2 [karras2019analyzing] as the latent bank; the RRDB encoder from ESRGAN [wang2018esrgan]; the perceptual loss [johnson2016perceptual]; the training recipe of ESRGAN-style SR networks §3.4; Adam [kingma2014adam]; cosine annealing [loshchilov2016sgdr].
• Compared against: GAN-inversion methods PULSE [menon2020pulse], mGANprior [gu2020image] and DGP [pang2020exploiting]; SinGAN [shaham2019singan]; ESRGAN+ [wang2018esrgan]; reference-based DFDNet [li2020blind] and SRNTT [zhang2019image] in §5.
• The paper's own extension: the same design is described as applicable to other generators, such as BigGAN §3.2, and to denoising, inpainting and colorisation §7.