Exploiting Diffusion Prior for Real-World Image Super-Resolution

In plain words

Super-resolution means turning a small, blurry or noisy photo into a sharp large one. Image generators that were trained on huge collections of pictures (here, Stable Diffusion) already know what sharp fur, text and fabric look like. StableSR reuses that knowledge instead of training a super-resolution network from scratch. The big generator stays frozen. A small add-on network looks at the blurry input and nudges the generator at every step of its drawing process, so the result stays faithful to the input.

Two extra controls make the method practical. A single dial lets a user trade realism for faithfulness to the original. A second trick lets the generator handle images of any size by drawing overlapping tiles and blending them. The authors report that the results look more detailed than earlier methods on real photos, while scoring lower on classic pixel-accuracy measures.

TL;DR

• Adapts a frozen text-to-image diffusion model (Stable Diffusion 2.1-base) for blind real-world super-resolution §1 §4.1.
• Trains only a time-aware encoder and scale-and-shift layers; the encoder sees the diffusion step, so its guidance changes over the trajectory §3.1.
• An optional fidelity module (CFW, weight w, default 0.5) and overlapping-patch sampling for arbitrary sizes are added on top §3.2 §3.3.
• Author-reported: best FID and no-reference scores on DIV2K-Valid and the real benchmarks, but lower PSNR, SSIM and LPIPS than several baselines §4.3.
• Costly: 200 sampling steps take 15.16 s for a 512x512 output on one V100; the 4-step SD-Turbo variant takes 0.83 s tab. 5.

Problem

Real-world low-resolution photos have unknown and mixed degradations. Methods that train a super-resolution diffusion model from scratch need large compute and can lose the generative prior of a pre-trained model §1. Methods that steer a frozen diffusion model with hand-built constraints need the degradation to be known, so they generalise poorly §1. Diffusion sampling is also sensitive to image size: it was trained at 512x512, and patch-wise processing leaves seams §3.3.

Method

Stage 1: time-aware encoder with modulation (training). A lightweight encoder reads the low-resolution image as a latent. It is built like the contracting path of the Stable Diffusion U-Net but with fewer channels, and it has a time-embedding layer, so its features depend on the diffusion step §3.1 app. A. Its multi-scale features modulate the residual blocks of the frozen U-Net through an affine transform per spatial position eq. 1. The diffusion weights stay frozen; only the encoder and the modulation layers are trained §3.1.

The authors argue that the encoder should give strong guidance where the signal-to-noise ratio is near 5e-2, which is where image content forms §3.1 fig. 3. Their cosine-similarity measurements peak at the same place §3.1.

Colour correction (inference, optional). Diffusion output can drift in colour. The default corrects each channel's mean and standard deviation to those of the input §3.1 eq. 2. A wavelet variant keeps the high-frequency part of the output and takes the low-frequency part from the input §3.1 eq. 3-5.

Stage 2: controllable feature wrapping (CFW, optional). CFW adds encoder features of the autoencoder into the VAE decoder, scaled by a user weight w in [0, 1]. w near 0 favours realism and w near 1 favours fidelity §3.2 tab. 4. It is trained after stage 1: 100k synthetic 512x512 pairs are made with the stage-1 model, and the autoencoder is trained on them §4.1. Because it needs extra GPU memory, the authors make it optional §3.2.

Aggregation sampling (inference, arbitrary sizes). The latent is cut into overlapping 64x64 patches. At each diffusion step, each patch's noise prediction is blended with Gaussian weights, and overlaps are averaged; then one sampler step is taken on the full latent §3.3 eq. 7 alg. 1. At the right and bottom edges, the overlap is adjusted so the output has the target size app. B. Without this blending, the tiles leave visible seams §3.3 fig. 4.

Negative-prompt guidance (optional). The model was trained with a null prompt, but it reacts to prompts. Classifier-free guidance with a negative prompt and a scale s is described in §5.1 eq. 8. The default (s = 1 with the null prompt) is unchanged.

Fast sampling. DDIM with 20 steps §5.2, and SD-Turbo with 4 steps without further fine-tuning §5.2 fig. 17.

Pre-cleaning (severe degradation, optional). Run an existing restorer first (Real-ESRGAN+ for general images, CodeFormer for faces), then downsample by 2x with bicubic filtering before StableSR app. C.

Equations and algorithm, as in the paper.

\hat{F}^n_{\rm dif} = (1 + \alpha^n) \odot F^n_{\rm dif} + \beta^n; \quad \alpha^n, \beta^n = \mathcal{M}^n_\theta(F^n) eq. 1

y^c = \frac{\hat{y}^c - \mu^c_{\hat{y}}}{\sigma^c_{\hat{y}}} \cdot \sigma^c_x + \mu^c_x eq. 2

L^i = \mathcal{C}_i(L^{i-1}, k), \quad H^i = L^{i-1} - L^i eq. 3

k = \begin{bmatrix} 1/16 & 1/8 & 1/16 \\ 1/8 & 1/4 & 1/8 \\ 1/16 & 1/8 & 1/16 \end{bmatrix} eq. 4

y = H^l_y + L^l_x eq. 5

F_m = F_d + \mathcal{C}(F_e, F_d; \theta) \times w eq. 6

\epsilon_\theta(Z^{(t)}, F, t) = \sum_{n=1}^{M} \frac{w_{\Omega_n}}{\hat{w}} \odot f\left(\epsilon_\theta(Z^{(t)}_{\Omega_n}, F_{\Omega_n}, t)\right), \quad \hat{w} = \sum_n w_{\Omega_n} eq. 7

\tilde{\epsilon}_\theta = \epsilon_\theta(Z^{(t)}, F, c, t) + s\left(\epsilon_\theta(Z^{(t)}, F, [], t) - \epsilon_\theta(Z^{(t)}, F, c, t)\right) eq. 8

Symbols: F is the LR latent feature; F^n_dif the frozen U-Net features at scale n; c a negative prompt; [] the null prompt; f pads a patch to the full grid with zeros; w_{Omega_n} a Gaussian weight map that is zero outside its patch §3.1 §3.2 §3.3 §5.1.

Algorithm 1: progressive patch aggregation alg. 1

# Progressive patch aggregation sampler  [alg. 1] [§3.3]
# [code: ldm/models/diffusion/ddpm.py#L2567-L2685]
def aggregated_sample(lr_latent, steps, patch=64, overlap=32):
    # Start from Gaussian noise over the whole latent grid  [alg. 1]
    latent = sample_gaussian(lr_latent.shape)
    # Run the sampler from the noisiest step down to the last one  [alg. 1]
    for step in steps:
        # Cut overlapping patches; the last row and column are shifted
        # inward so the grid fits the image  [app. B]
        patches = cut_patches(latent, patch, overlap)
        # Predict noise per patch, conditioned on the same LR patch  [§3.3]
        noise_per_patch = []
        for each patch:
            noise_per_patch.append(predict_noise(patch, lr_patch, step))
        # Blend the patch predictions with Gaussian weights; overlaps
        # are divided by the summed weights  [eq. 7]
        noise = weighted_average(noise_per_patch, gaussian_weights)
        # One sampler step with the blended noise  [alg. 1]
        latent = sampler_step(latent, noise, step)
    return latent

Stage 1 training step §3.1 ddpm.py

# Stage 1 training step: time-aware encoder with modulation  [§3.1]
# [code: ldm/models/diffusion/ddpm.py#L2294-L2316]
def training_step(lr_latent, gt_latent, null_text):
    # Draw one diffusion step per sample, uniformly  [§3.1]
    step = sample_uniform_steps(batch_size, num_steps)
    # Add Gaussian noise to the clean latent at that step  [§3.1]
    noise = sample_gaussian(gt_latent)
    noisy_latent = add_noise(gt_latent, noise, step)
    # Encoder reads the LR latent and the step  [§3.1]
    struct_features = time_aware_encoder(lr_latent, step)
    # Frozen U-Net with trainable modulation layers predicts the noise  [§3.1]
    predicted = unet(noisy_latent, step, null_text, struct_features)
    # Mean squared error on the noise  [§4.3]
    return mean_squared_error(predicted, noise)

Scale-and-shift modulation eq. 1 openaimodel.py

# Modulate one frozen U-Net feature map with encoder features  [§3.1] [eq. 1]
def modulate(features, struct_features):
    # Small conv net maps the encoder feature to per-pixel scale
    # and shift  [eq. 1]
    scale, shift = small_conv_net(struct_features)
    # Affine modulation of the frozen feature  [eq. 1]
    return (1 + scale) * features + shift

CFW fusion in the decoder eq. 6 model.py

# Controllable feature wrapping inside the VAE decoder  [§3.2] [eq. 6]
def cfw_fuse(encoder_features, decoder_features, w):
    # Trainable residual from the paired encoder and decoder features  [eq. 6]
    residual = fusion_block(encoder_features, decoder_features)
    # Scale the residual by the user weight w in [0, 1]  [eq. 6]
    return decoder_features + w * residual

Colour correction, pixel domain (default) §3.1 eq. 2

# Colour correction in the pixel domain  [§3.1] [eq. 2]
# [code: scripts/sr_val_ddim_text_T_negativeprompt_canvas_tile.py#L44-L56]
def match_channel_stats(generated, lr_image):
    # Per channel: mean and standard deviation over height and width  [eq. 2]
    gen_mean, gen_std = channel_stats(generated)
    lr_mean, lr_std = channel_stats(lr_image)
    # Standardise the output, then give it the input's statistics  [eq. 2]
    normalised = (generated - gen_mean) / gen_std
    return normalised * lr_std + lr_mean

Colour correction, wavelet (optional) §3.1 eq. 3-5

# Wavelet colour correction  [§3.1] [eq. 3-5]
# [code: scripts/wavelet_color_fix.py#L73-L118]
def wavelet_color_fix(generated, lr_image, levels):
    # Low-pass the generated image: colour survives, fine detail goes  [eq. 3]
    low_generated = low_pass(generated, levels)
    # Low-pass the LR input the same way  [eq. 3]
    low_lr = low_pass(lr_image, levels)
    # Keep the generated high frequencies, take the LR low frequencies  [eq. 5]
    return (generated - low_generated) + low_lr

Negative-prompt guidance §5.1 eq. 8 ddpm.py

# Classifier-free guidance with a negative prompt  [§5.1] [eq. 8]
def guided_noise(latent, step, null_text, negative_text, scale):
    # Noise predicted with the null prompt, the default  [§5.1]
    noise_null = predict_noise(latent, step, null_text)
    # Noise predicted with the negative prompt  [§5.1]
    noise_negative = predict_noise(latent, step, negative_text)
    # Scale 1 gives the default; scale 0 gives the negative prompt
    # alone  [eq. 8]
    return noise_negative + scale * (noise_null - noise_negative)

Hyperparameters

Results

Author-reported. Setup: synthetic DIV2K-Valid (3k pairs, 128 to 512 px, Real-ESRGAN degradations), real benchmarks RealSR and DRealSR (centre crops, 128 to 512 px), DPED-iPhone (no reference, so only no-reference metrics) §4.2 §4.3. Baselines are used as released, except LDM, which the authors fine-tuned with the same settings §4.2. The paper's best/second-best colouring (red/blue) is not reproduced here; the text notes that StableSR is best on FID and on the no-reference scores §4.3.

Table 1: quantitative comparison tab. 1

Text claims §4.3: on DIV2K-Valid, StableSR's FID is 7.7% lower than LDM's and at least 32.9% lower than the GAN methods. A user study on 40 internet images with 35 subjects (1400 votes) gave StableSR over 80% of the votes against Real-ESRGAN+, SwinIR-GAN and LDM §4.3 fig. 7.

Table 2: comparison with follow-up ControlNet-style methods tab. 2

The caption attributes SUPIR's weaker numbers to a resolution mismatch: the test crops are 512 px and SUPIR's prior is 1024 px tab. 2. The text says SUPIR wins on outputs beyond 1024 px §4.3.

Table 3: ablation of time-aware guidance and colour correction tab. 3

Table 4: effect of the CFW weight w tab. 4

Table 5: complexity, 128 px input, 4x SR, one V100, batch 1, mean of ten runs tab. 5

Text §4.5 §5.2: StableSR's 200-step runtime is comparable to IF_III and slower than the GAN methods. The 4-step SD-Turbo variant is said to be 6.3 times faster than the 200-step LDM §5.2. DDIM with 20 steps keeps faithful detail, while DDIM on plain LDM loses quality badly §5.2 fig. 17.

Table 6: prompts and guidance scale tab. 6 (positive prompt (1) is "(masterpiece:2), (best quality:2), (realistic:2), (very clear:2)"; (2) is "Good photo."; negative (a) is "3d, cartoon, anime, sketches, (worst quality:2), (low quality:2)"; (b) is "Bad photo.")

Text §5.1: higher guidance raises CLIP-IQA and MUSIQ (sharper output), but s = 7.5 can over-sharpen fig. 16.

Paper vs code

Code: official repository IceClear/StableSR at commit 398ee9383777e255540ea027a704c8ce1f32145b, licensed under the S-Lab License 1.0, which permits non-commercial use only. Nothing from the code is quoted here; code is cited by path and line only.

• Optimiser. Paper: Adam §4.1. Code: AdamW ddpm.py.
• Parameterisation. Paper: loss is l2 on the noise §4.3. Code, 512 config: the v-parameterisation line is commented out, so noise prediction is used v2-finetune_text_T_512.yaml. The released 768 config sets parameterisation "v" v2-finetune_text_T_768v.yaml. The released 768 checkpoint therefore is not the paper's model.
• Ground-truth sharpening. Paper: not mentioned. Code, 512 config: unsharp-mask sharpening of ground truth is on v2-finetune_text_T_512.yaml ddpm.py; the 768 config turns it off v2-finetune_text_T_768v.yaml.
• FFHQ faces. Paper: 5000 images §4.2. Code: num_face 10000 v2-finetune_text_T_512.yaml, taking the first sorted files realesrgan_dataset.py.
• Modulation layer. Paper: scale-and-shift per position, (1 + alpha) * F + beta eq. 1. Code: a SPADE block, with normalisation followed by (1 + gamma) * x + beta, where gamma and beta come from a 3x3 conv with 128 hidden channels spade.py. It is applied after the block's output convolution and before the skip connection openaimodel.py. The code comment calls this a built-in SPADE component used instead of SFT, with no significant effect expected openaimodel.py.
• CFW fusion block. Paper: a convolution stack C(F_e, F_d) scaled by w eq. 6. Code: a residual block, two RRDB blocks and another residual block model.py. Fusion is applied only at the two middle decoder resolutions, not at the lowest or the highest one (with the 512 autoencoder's four levels) model.py.
• CFW adversarial weight. Paper: fixed 0.025, not self-adjusting §4.1. Code: the weight is the 0.025 multiplier times an adaptive ratio of gradient norms (norm of NLL gradient over norm of GAN gradient, clamped to 1e4) contperceptual.py contperceptual.py. If that ratio raises, the fallback is 1.0 times the 0.025 weight contperceptual.py.
• Wavelet colour correction. Paper: y = H^l_y + L^l_x, with a single high-frequency band H^l eq. 5. Code: the high frequencies are the sum over all bands, which telescopes to generated minus its low-pass wavelet_color_fix.py. Paper dilation for scale i is 2^i with i from 1 to l §3.1; code uses radius 2^i for i from 0 to 4, so its first level has dilation 1 wavelet_color_fix.py. Code's level count (5) is not in the paper.
• Default colour correction. Paper: pixel domain is the default §3.1. Script default: "nofix" (none) sr_val_ddim_text_T_negativeprompt_canvas_tile.py. The help text says AdaIN is the paper's method; the paper's formula is per-channel mean and std matching, which is the same operation sr_val_ddim_text_T_negativeprompt_canvas_tile.py. The README example uses wavelet.
• DDPM step count. Paper: 200 steps §4.1. Script default: 1000 sr_val_ddpm_text_T_vqganfin_oldcanvas.py. The code keeps the encoder's timestep as the original index, even with fewer steps ddpm.py.
• Gaussian weights. Paper: Gaussian filter, with no parameters given §3.3. Code: variance 0.01, with the exponent scaled by the squared tile width, so the effective spread is about 0.1 times the tile width (6.4 latent cells for 64). The vertical midpoint is H/2 rather than (H-1)/2 ddpm.py.
• Overlap at the edges. Paper: the last row and column are shifted to fit §3.3 app. B. Code: the same, with the offsets set to fit the image ddpm.py.
• Decoder-stage tiling. Not in the paper. For inputs above 1280 px, the scripts decode tiles in pixel space with stride 1000 and blend them with Gaussian weights sr_val_ddim_text_T_negativeprompt_canvas_tile.py util_image.py util_image.py.
• Input preparation. Paper: LR is upsampled to the output size; images under 512 px get a shorter side of 512 §4.1 §4.2. Code: bicubic upscaling by the larger of 512 divided by the short side and the upscale factor (default 4), padding to a multiple of 32, and a resize back at the end sr_val_ddim_text_T_negativeprompt_canvas_tile.py.
• Training length. Paper: 117 epochs §4.1. Code: max_steps 800000 with no epoch count v2-finetune_text_T_512.yaml.
• Negative-prompt guidance. Matches eq. 8: unconditional term from the negative prompt, conditional from the null prompt ddpm.py.
• Encoder and frozen weights. Matches §3.1: only encoder and modulation layers train ddpm.py.

Pitfalls

• Which modulation layer. The paper says SFT. The code uses SPADE with a convolutional gamma and beta, plus an extra normalisation step. An implementation built from the paper's equation alone will differ from the released model.
• CFW is under-specified in the paper. The paper's one-line formula does not give the fusion block's depth, its placement in the decoder, or its normalisation. Follow the code if the goal is to reproduce the release.
• Adversarial weight. The paper's 0.025 is fixed; the code's is adaptive (see above). Copying the 0.025 into a plain GAN loss will change training.
• Wavelet correction. The paper's equation keeps one band; the code keeps all bands, which is a different output. Also check the dilation offset and the number of levels (not given in the paper).
• Checkpoints. The paper's model is SD 2.1-base at 512 px with noise prediction. The released 768 model and its config use v-prediction and 768 px. Results will not transfer one-to-one.
• Step count. The paper's 200 DDPM steps are not the script default; the README's 768 example uses DDIM with 20 steps and passes dec_w 0.0, not the default 0.5.
• Fidelity dial. w = 0 gives the best perceptual scores; w = 0.75 gives the best PSNR and SSIM on several sets; w = 0.5 is the default because it balances the two §3.2 tab. 4. CFW needs extra GPU memory.
• Negative prompts. Off by default. Turning them on raises CLIP-IQA and MUSIQ but lowers PSNR and SSIM, and s = 7.5 over-sharpens tab. 6 fig. 16.
• Cost. Training from scratch needed 2.06 times the GPU memory of StableSR in the authors' test §4.4. The 200-step sampler is 15.16 s per 512 px output on one V100 tab. 5. Tiled sampling multiplies the number of network calls by the number of tiles at every step.
• Metrics. The paper argues that PSNR, SSIM and LPIPS favour methods trained with l2 or perceptual losses, and that diffusion models are not trained that way §4.3. This is the authors' conjecture, not a measured result. Compare on FID, CLIP-IQA and MUSIQ with care when reproducing.
• Failure cases. Small text, faces and fine patterns are weak because the diffusion prior has the same limits §6 fig. 18. A face-only variant was trained by fine-tuning on FFHQ for 50 epochs with CodeFormer's degradations app. C.
• Unstated values. Overlap size, Gaussian width, sampler schedule, weight decay and the exact number of CFW training steps are not in the paper.

Lineage

• Builds on: Stable Diffusion, a latent diffusion model (Rombach et al., CVPR 2022), which supplies the frozen prior and the autoencoder §1 §3.
• Builds on: CodeFormer (Zhou et al., NeurIPS 2022) for the fidelity-control idea of CFW §3.2.
• Builds on: the mixture-of-diffusers aggregation of Jiménez (2023) for overlapping-patch sampling §3.3.
• Builds on: Real-ESRGAN (Wang et al., 2021) for the training degradation pipeline §4.2.
• Compared against: BSRGAN, Real-ESRGAN+, RealSR, DASR, FeMaSR, SwinIR-GAN, LDM (fine-tuned by the authors), DeepFloyd IF stage III, ControlNet-tile (Zhang et al., ICCV 2023) §4.3, and DiffBIR and SUPIR as follow-up work tab. 2.
• Superseded by: not known from the content already in this wiki.