SeeSR: Towards Semantics-Aware Real-World Image Super-Resolution

In plain words

Upscaling a damaged photo with a diffusion model can make the model draw the wrong things. The damage makes the picture ambiguous, and the model fills the gap with confident but incorrect objects. SeeSR tells the model what is in the picture. A tagging network is first retrained so that its guesses on a damaged photo match its guesses on the clean original. Its object labels, written as text, and its internal features then steer a pretrained image generator through a side network that is trained with it. At inference the damaged photo also serves as the starting point for sampling, instead of pure random noise. That reduces invented texture in flat areas such as sky. The authors report better scores than earlier upscalers on several measures, though not on all of them.

TL;DR

SeeSR runs Stable Diffusion for x4 real-world super-resolution with two semantic signals taken from the low-resolution input: tag text, and embeddings from a fine-tuned RAM tagger called DAPE §3.2. A ControlNet copy and extra cross-attention layers take in these signals, and sampling starts from the low-resolution latent noised to step 999 §3.3, §3.4. Author-reported: best FID, CLIPIQA and MUSIQ on all four test sets, and below the best baseline on PSNR and SSIM for the three paired sets tab. 2. Expect a large model (2283.7M parameters) and about 7 s per 512x512 output on one V100 [suppl. complexity table].

Problem

Real-world super-resolution faces degradations that are complex and unknown, so models trained on simple synthetic degradations do poorly on real photos §1. GAN-based methods built on synthetic degradation pipelines produce artifacts from unstable adversarial training §1. Diffusion models trained on text and image pairs carry strong natural-image priors, but StableSR and DiffBIR use only the low-resolution (LR) image as control, so they ignore semantic text §1. PASD adds prompts from off-the-shelf captioning and tagging models, but it struggles with scenes that have many objects and with severe degradation §1.

The paper argues that a useful semantic prompt must cover many objects and must be robust to degradation. Classification-style prompts are robust but give one label for the whole image. Caption-style prompts add redundant words and make semantic errors on degraded input. Tag-style prompts list many objects but misfire on degraded input §3.1 tab. 1. Separately, most SD-based methods start sampling from random Gaussian noise, although the model was trained to denoise from noised data rather than from pure noise. This mismatch lets the model treat degradation as content to enhance, especially in smooth regions §3.4.

Method

Prose overview. The method has two training stages and one inference change.

1. Prompt-style analysis §3.1 tab. 1. The authors compare three prompt styles on degraded inputs and find that tag-style prompts give the most object coverage, but that tags become wrong under degradation. This motivates making the tagger degradation-aware.
2. Stage 1: DAPE §3.2 eq. 1. DAPE starts from the RAM tagger. The clean HR image passes through a frozen RAM copy to give anchor targets: a representation embedding and tag logits. The LR image passes through the trainable DAPE. Training minimises a mean-squared error on the representation embeddings plus a cross-entropy term on the logits, weighted by lambda eq. 1. Only LoRA parameters are updated §4.1.
3. Prompts from DAPE §3.2. Hard prompts are the predicted tags, passed as text to the frozen text encoder of the T2I model. Their number depends on a preset threshold, which the paper does not give. Soft prompts are the representation embeddings, which do not depend on a threshold.
4. Stage 2: SeeSR §3.3 eq. 2. The base model is Stable Diffusion 2-base §4.1. Its UNet, text encoder and VAE are frozen. The ControlNet is cloned from the UNet encoder. The soft prompts enter through representation cross-attention (RCA) modules, placed after the text cross-attention modules and following the PASD design. RCA modules are cloned with the encoder. The LR image enters a trainable image encoder that has the ControlNet structure and feeds the ControlNet. The objective is the standard noise-prediction mean squared error eq. 2. Trainable parts are the image encoder, ControlNet and RCA modules; the rest is frozen §3.3.
5. LR embedding (LRE) at inference §3.4 fig. 3. Instead of pure noise, the LR latent is noised according to the training noise schedule and used as the starting latent. The authors say LRE is used in all SeeSR experiments §3.4. They report that it removes false sky texture fig. 3.
6. Sampling §4.1. Spaced DDPM sampling with 50 steps. The classifier-free guidance scale is not stated in the paper.

Training stage 1 (DAPE):

# Stage 1: fine-tune DAPE on degraded inputs  [§3.2] [eq. 1]
# Input:  LR image y, HR image x (training pairs)
# Output: DAPE whose LR outputs match the frozen tagger's HR outputs

def dape_loss(lr_image, hr_image):
    # Frozen tagger on the clean image gives the anchor targets  [§3.2]
    rep_hr, logits_hr = frozen_tagger(hr_image)
    # Trainable DAPE on the degraded image  [§3.2]
    rep_lr, logits_lr = dape(lr_image)
    # Mean squared error on representation embeddings  [§3.2] [eq. 1]
    loss_rep = mse(rep_lr, rep_hr)
    # Per-tag binary cross-entropy with soft targets  [§3.2]
    # [code: basicsr/models/dape_model.py#L135-L137]
    loss_logits = binary_cross_entropy(sigmoid(logits_lr), sigmoid(logits_hr))
    # Balance weight lambda, set to 1  [§4.1]
    return loss_rep + 1.0 * loss_logits

Training stage 2 (SeeSR):

# Stage 2: train the controlled diffusion model  [§3.3] [eq. 2]
# Input:  HR image x, LR image y, tag text p_h, DAPE embeddings p_s
# Output: noise-prediction loss; only ControlNet and RCA parts are updated

def seesr_loss(hr_image, lr_image, tag_text):
    # Clean latent from the frozen VAE  [§3.3]
    z_0 = vae_encode(hr_image)
    # Random diffusion step and Gaussian noise  [§3.3]
    t = sample_step()
    eps = random_normal_like(z_0)
    z_t = add_noise(z_0, eps, t)
    # Hard prompt: tags as text through the frozen text encoder  [§3.2]
    text_states = text_encoder(tag_text)
    # Soft prompt: DAPE representation embeddings of the LR image  [§3.2]
    soft_states = dape_embeddings(lr_image)
    # LR image enters the ControlNet branch  [§3.3]
    eps_pred = controlled_unet(z_t, t, text_states, soft_states, lr_image)
    # Noise-prediction mean squared error  [§3.3] [eq. 2]
    return mse(eps_pred, eps)

Inference (LRE and sampling):

# Inference: LR-embedded start, 50 DDPM steps  [§3.4] [§4.1]
# Input:  LR image y, already upscaled x4 (see Paper vs code)
# Output: super-resolved image

def seesr_sample(lr_image):
    # Tags and soft embeddings from the LR image via trained DAPE  [§3.2]
    tag_text = dape_tags(lr_image)
    soft_states = dape_embeddings(lr_image)
    # LR latent noised to training step 999, used as the start  [§3.4]
    lr_latent = vae_encode(lr_image)
    latents = add_noise(lr_latent, random_normal_like(lr_latent), 999)
    # Spaced DDPM sampler, 50 steps  [§4.1]
    for t in spaced_ddpm_steps(50):
        # Guidance scale is not stated in the paper  [code: test_seesr.py#L251]
        noise_pred = controlled_unet_cfg(latents, t, tag_text, soft_states)
        latents = ddpm_step(noise_pred, t, latents)
    return vae_decode(latents)

Hyperparameters

Results

Author-reported. Setup §4.1: x4 super-resolution. Test sets are DIV2K-Val (3K random 512x512 crops, degraded with the training pipeline), RealSR and DRealSR (LR images centre-cropped to 128x128, following the StableSR protocol) and RealLR200 (about 200 real LR images collected from the literature, DiffBIR, VideoLQ and the internet). Baselines: BSRGAN, Real-ESRGAN, LDL, DASR, FeMaSR, LDM, StableSR, ResShift, PASD and DiffBIR §4.1. LDM is not run on RealLR200 because its code has no tiling tab. 2 caption.

Quantitative comparison tab. 2. "Best other" is the best value among the baselines. The paper's text says SeeSR is best on FID, CLIPIQA and MUSIQ on all four sets, and best on LPIPS and DISTS on DIV2K-Val §4.2. The table shows exactly that and no more:

Semantic preservation §4.3. The authors upscale degraded COCO-Val images (5K images resized to 512x512, with the original as ground truth). They run a COCO- trained OpenSeeD model for detection and segmentation. The table reports the ground-truth row, the zoomed LR and the best other method per task:

The paper says SeeSR is about three to four times better than the zoomed LR on these tasks §4.3.

User study §4.2. Twenty participants evaluated six methods. On synthetic data they judged which of two HR images matched the LR input, and the confusion rate is reported. On real data they picked the best output, and the best rate is reported. SeeSR: 38.6% confusion on synthetic data (next best DiffBIR 12.5%) and 57.1% best rate on real data (next best DiffBIR 15.2%). The ground truth beats every method on synthetic data §4.2.

Ablations and complexity (supplementary source, not in the main build; see Pitfalls).

• LRE, DIV2K-Val, without vs with LRE: PSNR 20.58 vs 21.04, LPIPS 0.3942 vs 0.3876, FID 32.53 vs 32.79, CLIPIQA 0.7314 vs 0.6834. On DrealSR: PSNR 26.55 vs 27.90, LPIPS 0.3952 vs 0.3299, FID 158.04 vs 151.88, CLIPIQA 0.7248 vs 0.6708 [suppl. LRE table]. The authors say LRE helps full-reference metrics and lowers CLIPIQA, which they attribute to less over-generation.
• Tagging on degraded COCO-Val, RAM vs DAPE: overall precision 0.7929 vs 0.8940, overall recall 0.3711 vs 0.3751, average precision 52.3 vs 63.0 [suppl. tagging table].
• Prompt ablation on DIV2K-Val, FID: no DAPE, no prompts (exp. 1) 37.35; RAM instead of DAPE 46.34; hard prompts only 38.92; soft prompts only 38.77; full SeeSR 32.79 [suppl. DAPE ablation table]. Removing the DAPE and RCA modules leaves a plain ControlNet, which the authors describe as producing blurry results.
• Complexity: SeeSR has 2283.7M parameters, 50 steps, 7.24 s per 512x512 output on one V100. DAPE adds about 300M parameters. StableSR is slower (18.70 s, 200 steps) [suppl. complexity table].

Paper vs code

Code: official repository https://github.com/cswry/SeeSR, linked from the abstract, at commit b13b8ac3bbcc6b9a6baa970ee5349628070877ee. Apache-2.0 LICENSE. The repository also contains a later SeeSR-Turbo variant README.md, which the paper does not describe. The DAPE checkpoint (preset/models/DAPE.pth) that the scripts load is not in the repository.

Differences that change what an implementer builds:

• Training tags come from HR images. The paper gives no source for the training tags. The code runs the tagger on ground-truth images to write the tag files make_tags.py make_tags.py. The dataset reads those files as text prompts paired_dataset.py, while the soft prompts come from the LR image paired_dataset.py train_seesr.py. At inference, both come from the LR image test_seesr.py. Training and inference therefore see differently derived tag text.
• Null tag text. The paper does not state a null-prompt rate. The code drops the tag text in half of the samples by default train_seesr.py paired_dataset.py.
• Batch size and length. The paper reports batch 192 for 150K iterations §4.1. The scripts give an effective batch of 32 train_seesr.sh train_seesr.sh. No step limit is set, so the default of 1000 epochs applies train_seesr.py.
• Optimiser. The paper says Adam §4.1. The code uses AdamW with weight decay 1e-2 train_seesr.py train_seesr.py.
• DAPE training. The paper says LoRA fine-tunes "the entire DAPE" for 20K iterations at batch 32 §4.1. The code freezes every parameter except LoRA weights dape_model.py. The shipped config uses 300K iterations, milestones at 20K and 30K, and batch 8 per GPU dape.yaml dape.yaml. LoRA rank 8 is applied to the Swin qkv projection and BERT query and key projections swin_transformer_lora.py bert_lora.py.
• Tag threshold. The paper says a preset threshold controls the tag count but gives no value §3.2. The code uses 0.68 by default, overridden per tag by a file ram_lora.py ram_lora.py.
• LR start. The paper says the LR image is embedded into the noise §3.4. The code first upscales the LR image x4 with PIL resize, then encodes that upscaled image with the VAE test_seesr.py pipeline_seesr.py. The start level 999 is set in code test_seesr.py pipeline_seesr.py.
• Inference extras not in the paper. The code appends "clean, high-resolution, 8k" to the prompt by default test_seesr.py test_seesr.py, uses a fixed negative prompt "dotted, noise, blur, lowres, smooth" test_seesr.py, and applies an AdaIN colour correction to the output by default test_seesr.py test_seesr.py wavelet_color_fix.py. The guidance scale 5.5 and conditioning scale 1.0 are also defaults in code test_seesr.py.
• Degradation pipeline. The paper says the Real-ESRGAN pipeline is used §4.1. The repository has a concrete version with its own ranges and probabilities make_paired_data.py. Training LR images are read from a precomputed sr_bicubic folder, not degraded on the fly paired_dataset.py.
• Tagging input size. RAM receives images resized to 384x384 without keeping the aspect ratio test_seesr.py paired_dataset.py. The paper does not mention this.
• Script bug. In scripts/train_seesr.sh, the single-GPU and multi-GPU commands end the --null_text_ratio line without a backslash, so the next flags run as separate commands train_seesr.sh train_seesr.sh. Fix this before reusing the scripts.

Other points checked and consistent with the paper: the noise-prediction loss matches eq. 2 train_seesr.py; the trainable UNet parts are the RCA modules named image_attentions train_seesr.py unet_2d_condition.py; the DAPE loss has an MSE term and a per-tag term with weight 1 dape_model.py; DAPE uses LR and HR pairs of 512 px resized to 384 for RAM dape_dataset.py.

Pitfalls

• Tags differ between training and inference. Training tags come from HR images, but inference tags come from the LR image make_tags.py test_seesr.py. The model never sees LR-derived tag text during training. If tags on degraded inputs look wrong, check this first.
• Defaults that the paper does not state. The appended prompt suffix, negative prompt, AdaIN colour correction, guidance scale and conditioning scale are code defaults test_seesr.py. Any reported number could depend on them, and the paper does not say which were used.
• Threshold. The tag threshold sets the trade-off between precision and recall, and the paper gives no value §3.2. The shipped per-tag threshold file belongs to RAM. Whether it suits the fine-tuned DAPE was not checked.
• Null-tag rate. The 0.5 rate in code is not in the paper train_seesr.py. The paper does not say how often the tag text was dropped during training, which decides how well the model handles the unconditional branch used by guidance.
• Compute. Batch 192 for 150K iterations §4.1 is not what the scripts run. With the shipped scripts the effective batch is 32, and no step limit is set train_seesr.sh train_seesr.py.
• Ablations are not in the main build. The main text cites ablation and complexity results as supplementary material. In the source, these are in sec/suppl.tex, which main.tex does not include. Check which version of the supplement the arXiv PDF carries before quoting those numbers.
• Semantic test uses COCO. The detector and segmenter are trained on COCO and the test images are COCO-Val §4.3. The gap is in-domain, so it does not test robustness to a different label set.
• RealLR200 is not in the repository. The 200-image set is built from several sources, and only one test image ships in the repository (preset/datasets/test_datasets/179.png). Reproducing the RealLR200 column needs the set assembled from the paper's description §4.1.
• Reference metrics favour GANs. On three paired sets SeeSR is below the best baseline on PSNR and SSIM tab. 2. The paper says this is expected for diffusion models §4.2. Do not select SeeSR for fidelity-only pipelines.
• Costs the abstract hides. Model size is 2283.7M parameters, and sampling takes about 7 s per 512x512 output with 50 steps on one V100 [suppl. complexity table]. This is far slower than the GAN baselines, which take about 0.1 s.
• Small text. The authors list small-scale scene text as a failure case §5.

Lineage

• Builds on: RAM tagger, arXiv:2306.03514, for DAPE initialisation §3.2. LoRA, arXiv:2106.09685, for DAPE fine-tuning §4.1. PASD cross-attention, arXiv:2308.14469, for the RCA design §3.3. ControlNet and the Stable Diffusion 2-base model are the other building blocks §3.3 §4.1.
• Compared against: Real-ESRGAN, BSRGAN, LDL, DASR, FeMaSR, LDM, StableSR (arXiv:2305.07015), ResShift, PASD and DiffBIR (arXiv:2308.15070) §4.1.
• Superseded by: not known from content already in this wiki.