Blind Super-Resolution Kernel Estimation using an Internal-GAN

In plain words

A low-resolution photo is usually made by blurring a sharper picture and shrinking it, and the exact blur is rarely known. Super-resolution models, which try to enlarge such photos with sharp detail, mostly assume one standard blur. When the real blur is different, their results get worse.

KernelGAN estimates the blur for one specific image, without any training data. It trains a small GAN on that image alone. The generator learns to shrink the image so that small patches of the shrunken copy look like patches of the original to a discriminator. The generator is built only from plain linear layers, so after training the whole shrinking step collapses into one small filter, and that filter is the blur estimate. Give that filter to an existing super-resolution method and, according to the authors, the enlarged result improves.

TL;DR

• Estimates a per-image downscaling kernel with KernelGAN, a GAN trained on the test image alone at test time, with no training data §2 §5.2.
• The generator is a deep linear network: no activations, five hidden 64-channel layers, and the product of its filters is the kernel §4.1 §4.2.
• A non-blind SR method (ZSSR or SRMD) takes the estimated kernel as input §5.1 tab. 1.
• Author-reported: with ZSSR, 30.363 dB / 0.8669 at x2 and 26.810 dB / 0.7316 at x4 on DIV2KRK tab. 1.
• The official code is non-commercial and differs from the paper in several details. Read Paper vs code before implementing.

Problem

Super-resolution (SR) methods usually assume the low-resolution (LR) image came from a fixed, ideal downscaling kernel, such as bicubic. Real LR images rarely do, and SR results degrade when the assumed kernel is wrong §1. The authors cite earlier work finding that an incorrect kernel hurts SR more than the choice of image prior §1. Non-blind SR methods cannot use an image-specific kernel at test time, so on real images they produce poor results §1.

The previous image-specific estimator, by Michaeli and Irani, uses patch recurrence across scales. The authors say it tends to fail above x2 and has a very long runtime §1. They report 45 minutes per image on their DIV2KRK benchmark §5.3.

Assumptions: the degradation is a single linear model, the image is blurred by one kernel and then subsampled by the scale factor eq. 1 §1, and that kernel is the same across the whole image §5.3. The kernel is estimated for x2 and the x4 kernel is derived from it §4.2.

Method

Citations use the section order of the LaTeX source: §1 to §6, with the appendix unnumbered. Equation numbers follow the three numbered equation environments in order eq. 1 to eq. 3.

Overview §2. Two fully convolutional networks train on one LR image. A generator G downscales it. A patch discriminator D outputs a map with one score per local patch §2 §3 fig. 2. D is trained to score crops of the input as 1 and crops of G's output as 0. G is trained so that D's map on its output scores as 1 §2. Once training converges, G is the image's implicit downscaler §2. The paper's objective is eq. 2; its sign convention is discussed under Pitfalls.

G^*(I_{LR}) = \arg\min_G \max_D \left\{ \mathbb{E}_{x \sim \mathrm{patches}(I_{LR})} \big[ |D(x) - 1| + |D(G(x))| \big] + \mathcal{R} \right\} eq. 2

Training runs for 3,000 iterations. Each iteration is one G step and one D step, with ADAM (β1 = 0.5, β2 = 0.999). The learning rate is 2e-4 and is divided by ten every 750 iterations §2.

Discriminator §3 fig. 3. A 7×7 convolution followed by six 1×1 convolutions, with no pooling or stride, so each output location depends on a 7×7 input patch. Middle layers use batch normalisation and ReLU. The last layer uses a sigmoid, so the output map lies in [0, 1]. The figure also lists spectral normalisation; see Paper vs code. The loss is an absolute error between the output map and a map of ones (real) or zeros (fake) §2 §3.

Deep linear generator §4.1 fig. 4. G has no non-linearity, so it is a linear map. Five hidden layers have 64 channels, with kernel sizes 7, 5, 3, 1 and 1, and a final 1×1 output layer with stride 2 gives the downscaling §4.1 fig. 4. The first three filters give a 13×13 receptive field, which sets the size of the estimated kernel §4.1. The authors conjecture that a single strided layer cannot find the right kernel, because the loss surface is not convex in its weights, while a deep linear stack avoids this §4.1. The depth ablation reports a 3.8 dB (x2) and 1.6 dB (x4) gain over the single-layer version on DIV2KRK fig. 6. During the first iterations, G is pushed to be close to a bicubic downscaling of the input, and that constraint is then removed §4.1.

Kernel extraction and regularizer §4.2. The kernel is obtained from G's weights by convolving all filters in sequence with stride 1. This operation is differentiable, so the regularizer can be back-propagated §4.2. The regularizer has four terms eq. 3:

• sum-to-one, weight α = 0.5, keeps the kernel sum near 1;
• boundaries, weight β = 0.5, penalises mass far from the centre with an exponentially growing mask;
• sparsity, weight γ = 5, penalises small non-zero values to avoid over-smoothing;
• centre, weight δ = 1, keeps the centre of mass at the kernel centre §4.2.

\mathcal{R} = \alpha \mathcal{L}_{\mathrm{sum}} + \beta \mathcal{L}_{\mathrm{bnd}} + \gamma \mathcal{L}_{\mathrm{sparse}} + \delta \mathcal{L}_{\mathrm{center}} eq. 3

Other scales §4.2 app.. The x4 kernel follows from the x2 kernel as k4 = k2 convolved with a copy of k2 whose samples are spread apart by a factor of two, with zeros in between §4.2. The appendix derives this in one dimension app..

Training crops §5.3. Each iteration uses one crop for G and one for D. Crops are drawn with probability proportional to their gradient content. The paper gives a 64×64 crop for G and a 32×32 crop for D; see Paper vs code. The same kernel applies to the whole image, so a fixed number of crops is enough §5.3.

Pseudocode for one iteration of the training loop. The paper's text and the code agree on the structure; the details that differ are listed in Paper vs code.

# KernelGAN: estimate the x2 kernel of one image  [§2] [§4]
# [code: kernelGAN.py#L74-L120] [code: train.py#L14-L18]
# Input: LR image (H, W, 3); output: kernel (13, 13)

def estimate_kernel(image):
    # Build the linear generator G and the patch discriminator D  [§4.1] [§3]
    generator, discriminator = build_networks()
    # 3,000 alternating steps: one G update, then one D update  [§2]
    for step in range(3000):
        # Draw a G crop and a D crop, weighted by gradient content  [§5.3]
        g_crop, d_crop = sample_crops(image)
        # G step: the downscaled output should score as real (1)  [§2]
        # (1, 3, 26, 26) for a 64 px crop; see Paper vs code
        fake = generator(g_crop)
        generator_loss = l1(discriminator(fake), 1) + constraints(generator)
        update(generator, generator_loss)
        # D step: real crops score 1, fake crops score 0  [§2]
        real_score = discriminator(d_crop)
        fake_score = discriminator(fake.detach())
        discriminator_loss = (l1(real_score, 1) + l1(fake_score, 0)) / 2
        update(discriminator, discriminator_loss)
        # Learning-rate and constraint schedule  [§2] [§4.1]
        update_schedule(step)
    return extract_kernel(generator)

Pseudocode for reading the kernel out of G and for the regularizer, with the paper's values.

# Kernel from G's weights  [§4.2] [code: kernelGAN.py#L58-L63]

def extract_kernel(generator):
    # Start from a single unit impulse  [code: kernelGAN.py#L60]
    kernel = unit_impulse()
    # Convolve each layer's filter in turn, stride 1  [§4.2]
    # The kernel grows with every layer
    for layer in generator.layers:
        kernel = convolve(kernel, layer.weight)
    # Flip to the usual orientation  [code: kernelGAN.py#L63]
    return flip(kernel)                     # (13, 13) with the default setup

# Regularizer on the kernel, with the paper's values  [§4.2] [eq. 3]
# The code uses means, another exponent and a switch-on schedule
# See Paper vs code

def regularizer(kernel):
    # Sum-to-one: the kernel should sum to 1  [§4.2]
    sum_term = abs(1 - sum(kernel))
    # Boundaries: penalise mass far from the centre  [§4.2]
    # The mask grows exponentially with distance
    boundary_term = sum(abs(kernel * boundary_mask))
    # Sparsity: penalise small non-zero values, exponent 1/2  [§4.2]
    sparse_term = sum(abs(kernel) ** 0.5)
    # Centre of mass should sit at the kernel centre  [§4.2]
    centre_term = norm(centre_of_mass(kernel) - kernel_centre)
    # Weights alpha, beta, gamma, delta  [§4.2]
    return (0.5 * sum_term + 0.5 * boundary_term
            + 5 * sparse_term + 1 * centre_term)

Hyperparameters

Units: pixel values are in the range given by each row. "Where" names the paper section, figure or table, and the code path with line numbers where the code also gives a value.

Training

Generator

Discriminator

Regularizer and constraints

Evaluation

Results

Author-reported throughout. Setup: 100 images from DIV2KRK, a synthetic benchmark the authors built, with blur kernels that differ per image §5.2. Each row is an SR method run on the LR image with the kernel shown. "Class 3" is the blind pipeline: an estimator plus a non-blind SR method §5.1. Scores are PSNR (dB) and SSIM on the HR ground truth tab. 1.

Claims in the text, and what the table gives:

• ZSSR with the KernelGAN kernel beats the best non-blind and NTIRE rows. The text says "1 dB" in one place and "1.1 dB" in another at x2, and "0.47 dB" at x4 §5.3. The table gives 1.17 dB over RCANplus at x2 and 0.47 dB over PDN at x4 tab. 1.
• Against Michaeli & Irani's kernel, both with ZSSR, the gap is 1.0 dB (x2) and 0.73 dB (x4) tab. 1 §5.3.
• KernelGAN with SRMD trails PDN in PSNR at x4 but is higher in SSIM §5.3 tab. 1.
• Real photographs are compared visually only §5.3 fig. 1. The paper reports no numbers for them.
• Runtime: 61 s (V100) or 102 s (K80) per image §5.3.

Paper vs code

The code is at github.com/sefibk/KernelGAN, at commit 7218a37f0933519513aac97f40b8a72d3676eec9. Its README says "Official implementation" and cites this paper README.md. The owner name matches an author, Sefi Bell-Kligler, but the paper does not link the repository and I could not open the project page from here, so that link rests on the owner name and the README.

I read configs.py, networks.py, loss.py, kernelGAN.py, learner.py, data.py, util.py, train.py, the README, LICENSE.txt and KernelGAN.yml. I did not read ZSSRforKernelGAN/, did not run the code, and did not check the dataset generator, which lives in another repository README.md.

1. Sparsity exponent. Paper: |k|^(1/2) §4.2. Code: power 0.2 loss.py.
2. Centre term. Paper: the norm of the offset between the centre of mass and the kernel centre §4.2. Code: squared error against a fixed point of 6.5 on each axis for a 13-px kernel at x2 loss.py. The centre index of a 13-px kernel is 6 util.py.
3. Averages, not sums. The paper's R terms are written as sums eq. 3. The code uses means over kernel entries loss.py loss.py.
4. Constraint schedule. The paper gives fixed weights and says the bicubic constraint is dropped once met §4.1 §4.2. The code keeps a bicubic term, divides its weight by 100 every 200 iterations down to 5e-6, and switches on the sparsity and centre terms (5 and 1) only after that weight falls below 5e-3 learner.py kernelGAN.py.
5. Crop sizes. Paper: 64 px to G, 32 px to D, and a 32×32 D-map §5.3 fig. 3. Code: D's crop is the size of G's output. A 64 px G input gives a 26 px output (13-px kernel, stride 2), so D sees 26 px and outputs 20×20 networks.py networks.py data.py. This follows from the layer sizes; I did not run it.
6. Spectral normalisation. The figure lists it among the discriminator blocks fig. 3. The code applies it to every discriminator convolution, first and last included networks.py networks.py networks.py. Batch norm and ReLU are in the middle layers only networks.py.
7. Post-processing. The paper's active text has none; the paragraph that describes it is commented out in the LaTeX source (method.tex line 78) and uses a 75th-percentile rule. The code zeroes values below 0.75 times the 40th largest, renormalises, and shifts the kernel so its centre of mass sits at 6.5 for x2 configs.py util.py util.py.
8. The x4 kernel. Paper: k4 = k2 convolved with dilated k2, no crop §4.2 app.. The uncropped result is 37×37. Code: the same product is cropped by 6 pixels on each side and renormalised, giving 25×25 util.py. Table 1 does not say which form produced its x4 numbers tab. 1.
9. Adversarial loss. The overview text says "pixel-wise MSE" §2, while eq. 2 uses absolute values. Code uses the L1 loss loss.py.
10. Inputs. Images are mapped to [-1, 1] util.py. Discriminator inputs get noise of standard deviation 1/255 data.py kernelGAN.py. Unless --real is set, a 10 px border is cropped from every image data.py configs.py. The paper mentions none of these.
11. Colour channels. One generator is applied to each RGB channel as a batch of three single-channel images, so the kernel is shared across channels networks.py util.py. The paper does not say this.
12. Generator initialisation and bias. Xavier normal with gain 0.1 and no bias networks.py networks.py. The paper says neither.
13. Boundary mask scale. The paper gives the mask as exponentially growing, with no scale. The code uses a scale of 30 loss.py.

Pitfalls

• Equation 2 sign convention. As printed, eq. 2 puts the outer maximum over D on a sum that contains |D(x) − 1| and |D(G(x))|. Read literally, that makes D push real scores away from 1, which is the opposite of the discriminator loss the text describes. The code uses the text's convention: D minimises |D(x) − 1| + |D(G(x))|, and G minimises |D(G(x)) − 1| kernelGAN.py kernelGAN.py. Implement the code's convention.
• The code is non-commercial. LICENSE.txt grants "academic and noncommercial use only" LICENSE.txt. Do not copy the code into a commercial product. The pseudocode above is our own paraphrase, and no code is quoted.
• Random sampling, no seed. Crops and noise come from NumPy's global random generator data.py data.py. I found no seed in train.py, configs.py or data.py. Two runs can give different kernels, and the paper does not report run-to-run variance.
• GPU only as written. The generator, discriminator, losses and the Gaussian buffers call .cuda() kernelGAN.py loss.py. A CPU run needs edits.
• Old pins and a removed NumPy alias. The environment file pins Python 3.6, PyTorch 1.0 and NumPy 1.15 KernelGAN.yml. np.int in kernel_shift util.py was removed in NumPy 1.24, so the code fails on current NumPy.
• Each image is a full training run. Runtime is 61 s (V100) or 102 s (K80) per image, and there is no shared model to reuse §5.3.
• The ×2 centring shift matters for ZSSR. Table 1 combines the kernel with ZSSR tab. 1. The shift to 6.5 is in the code and not described in the paper util.py. Reproducing the table with a different centre would change the alignment ZSSR uses.
• The ablation's single-layer baseline is undescribed. The 3.8 dB and 1.6 dB gains fig. 6 do not say how the single-layer generator was trained.
• Evidence on real images is visual. The benchmark numbers use the authors' synthetic blur model §5.2. Real-photo quality is judged by eye fig. 1.
• Benchmark not in the repository. DIV2KRK is a download from the authors' site, and its generator is in another repository README.md. I did not check the downloads.

Lineage

• Builds on: Internal-GAN for learning one image's patch distribution §1 §2; ZSSR for training on the test image §1; Michaeli and Irani's kernel estimator, which uses patch recurrence across scales §1; the theory of deep linear networks §4.1; the LSGAN loss §2; the patch discriminator §3.
• Compared against: Michaeli and Irani with ZSSR and SRMD tab. 1; NTIRE 2018 winners PDN and WDSR tab. 1; EDSRplus and RCANplus on bicubic data tab. 1. The paper also mentions IKC as concurrent work §1.
• Superseded by: not stated in this card.