RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors

In plain words

Dehazing removes the grey veil that fog or smog lays over a photo. Most dehazing networks are trained on hazy photos that a computer made from clean ones, and real hazy photos look different from those fakes. The fakes also tend to use one simple fog model with clean lighting, so the networks often fail on real scenes.

This paper makes two changes. The first is a richer recipe for the fake hazy photos: it adds dim light, noise, a colour cast in the haze and JPEG compression. The second is a picture dictionary that the network can lean on. The dictionary is learned by a model trained only to rebuild high-quality images, so its entries describe what clean texture looks like. The network looks up the entries that match the hazy input and rebuilds a clean image from them. A dial lets the user push that lookup towards stronger or softer dehazing.

The dictionary helps when the hazy input hides too much detail to guess the right answer. The cost is an extra pretraining stage and a more involved network.

TL;DR

• A VQGAN is pretrained on DIV2K and Flickr2K to learn a codebook of 1024 vectors of width 512, then the codebook and decoder are frozen and an encoder plus a new fusion decoder are trained §4 §5.2 app. A.1.
• Training pairs come from 500 clean images with RA-Depth maps, degraded by a phenomenological haze model rather than only the physical scattering model §3.
• At inference, Controllable HQPs Matching (CHM) multiplies each code's distance by exp(alpha · f̃), where f̃ is that code's activation-frequency gap between hazy and clean images. The paper's chosen value is alpha = 21.25 §4.3.
• Author-reported on RTTS, RIDCP has the best BRISQUE (18.782) and NIMA (4.4267), and 55.6% of user-study votes tab. 1 §5.3.
• RTTS has no ground truth, so every score is a no-reference metric §5.3.
• The official code is CC BY-NC 4.0. Cite it, never copy from it. Several code paths differ from the paper; see Paper vs code.

Problem

Paired real hazy and clean photos are scarce, and the physical scattering model eq. 1 does not describe real haze well §1 §2.1. Hand-made priors such as the dark channel prior fail in some regions, for example the sky §2.1. Real-image methods based on domain adaptation or prior losses (DAD, D4, PSD) still give limited results §1.

A codebook prior has a second problem. Codes are chosen by nearest-neighbour matching, and the gap between synthetic and real inputs biases that choice. The paper measures this: the code activation frequencies on real hazy inputs shift away from those on clean images §4.3 fig. 4.

Method

The method has three parts: a pretrained codebook prior (stage 1), the RIDCP network (stage 2), and a synthetic hazy data pipeline (data). Stage 2 also uses CHM at inference.

Stage 1: codebook prior from a VQGAN

A VQGAN has an encoder E_vq, a codebook Z with K entries of width n, and a decoder G_vq. Each latent vector is replaced by its nearest code, and the image is reconstructed from the codes eq. 3 eq. 4 §4.1. The VQGAN is pretrained only on high-quality images, so the codes hold clean-image structure §1 §4.1.

Observation 1 is that the pretrained VQGAN removes thin haze and restores colour on a hazy input without fine-tuning, but it leaves distorted textures and often picks the wrong code §4.1 fig. 3. The paper therefore keeps the codebook and decoder fixed and trains an encoder that predicts codes and a new decoder that also uses the decoder's features §4.1 §4.2.

Stage 2: RIDCP network

The encoder E follows SwinIR. A shallow head of residual layers downsamples the input 4×, then 4 residual Swin Transformer blocks run on the features §4.2 app. A.1.

Each encoder feature is matched to its nearest code eq. 3, and the matched code is fed to the frozen decoder. The trainable decoder fuses its own features F with the frozen decoder's warped features F_vq through the Normalized Feature Alignment (NFA) §4.2:

• warping: F_w = DCONV(F_vq, CONV(Concat(F_vq, F))) eq. 5;
• normalised addition: F = F + (ΣF / ΣF_w) · F_w eq. 6.

Eq. 6 forces the two terms to the same magnitude, because the ratio of their values was unstable §4.2. The DCONV is a deformable convolution whose offsets come from a CONV on the concatenated features §4.2.

The final result is the residual branch's output. The paper trains the encoder with a feature-matching loss, a style loss and an adversarial loss, and trains the decoder with an L1 loss and a perceptual loss. Gradients of the decoder loss do not reach the encoder app. A.2.

Data: phenomenological haze synthesis

The paper replaces the physical model with a pipeline that combines several degradations on top of a clean image and its depth map §3 eq. 2:

• dim light: a gamma factor of 1.5 to 3.0, applied to half the samples §3;
• Gaussian noise §3 eq. 2;
• haze from a transmission map built from a depth map, with haze density beta in [0.3, 1.5] §3;
• atmospheric light A in [0.25, 1.0], with a per-channel colour bias in [-0.025, 0.025] §3;
• JPEG compression on half the samples, which the paper says removes the JPEG artefacts that dehazing tends to amplify §3.

The hazy image is generated on the fly during training, from 500 clean images §3.

Controllable HQPs Matching (CHM) at inference

CHM changes nearest-neighbour matching by weighting each code's distance §4.3:

• matching rule: M'(ẑ) = argmin_k F(f̃_k, α) · ||ẑ − z_k|| eq. 7;
• weight: F(f̃, α) = exp(α · f̃) eq. 8;
• f̃_k = f_h^k − f_c^k, the gap in code activation frequency between 200 real hazy inputs and 200 clean images §4.3.

The paper picks alpha by minimising the forward KL divergence between the clean and hazy code distributions, with a binary search. The result is alpha = 21.25, and the paper says other values can be chosen for personal preference §4.3 eq. 9. Larger alpha gives stronger enhancement and negative alpha gives weaker enhancement §5.4 fig. 7.

CHM is an inference-time step. It is not used during training §4.3 [code: options/RIDCP.yml#L59].

# Synthetic hazy pair from a clean image and its depth  [§3] [eq. 2]
# Input:  clean (H, W, 3) in [0, 1]; depth (H, W) from RA-Depth  [§3]
# Output: hazy (H, W, 3) in [0, 1]

def synthesize_haze(clean, depth):
    # Min-max normalise depth to [0, 1]
    # [code: basicsr/data/haze_online_dataset.py#L113]
    depth = (depth - depth.min()) / (depth.max() - depth.min())
    # Dim light: raise to a gamma in [1.5, 3.0], half the time
    # [§3] [code: #L127-L128]
    if random() < 0.5:
        clean = clean ** uniform(1.5, 3.0)
    # Gaussian noise, half the time; the code also varies the noise model
    # [code: #L130-L131]
    if random() < 0.5:
        clean = clean + gaussian_noise(clean)
    # Haze density beta in [0.3, 1.5]  [§3]
    beta = uniform(0.3, 1.5)
    # Transmission. The paper writes exp(beta * d) [§3]; the code uses the
    # negative exponent below, with depth inverted and a factor 2
    # [code: #L115-L116]
    transmission = exp(-2.0 * beta * (1.0 - depth))[..., None]
    # Atmospheric light in [0.25, 1.0], plus per-channel bias in
    # [-0.025, 0.025]  [§3]
    light = uniform(0.25, 1.0) + uniform(-0.025, 0.025, size=3)
    # Physical haze composition  [eq. 1] [code: #L133-L134]
    hazy = clean * transmission + light * (1.0 - transmission)
    # JPEG compression, half the time; the code draws quality uniformly
    # from 30 to 95  [§3] [code: #L137-L138]
    if random() < 0.5:
        hazy = jpeg(hazy, quality=randint(30, 95))
    return hazy

# RIDCP inference with CHM  [§4.2] [§4.3] [eq. 3] [eq. 5] [eq. 6] [eq. 7]
# Input:  hazy (B, 3, H, W); codebook (K, n) with K = 1024, n = 512  [app. A.1]
# Output: dehazed (B, 3, H, W)

def dehaze(hazy, alpha, codebook, f_tilde):
    # Reflect-pad so that H and W are multiples of 32
    # [code: basicsr/archs/dehaze_vq_weight_arch.py#L567-L573]
    padded = pad_to_multiple(hazy, 32)
    # Encoder: 4x downsample, then 4 residual Swin blocks  [§4.2] [app. A.1]
    features = encoder(padded)                 # (B, 512, H/4, W/4) after 1x1
    # Nearest code under the CHM weight, using unsquared distances  [eq. 7]
    # Code's implementation uses squared distances  [code: #L63-L69, #L94-L98]
    weights = exp(alpha * f_tilde)                 # (K,)
    distances = distance(features, codebook)       # (B, H/4, W/4, K)
    index = argmin(weights * distances, axis=-1)   # (B, H/4, W/4)
    quantised = codebook[index]
    # NFA fusion at each decoder stage  [§4.2] [eq. 5] [eq. 6]
    warped = deform_conv(frozen_feats, offsets(concat(frozen_feats, feats)))
    feats = feats + (sum(feats) / sum(warped)) * warped
    # Residual branch gives the final output  [app. A.2] [code: #L474-L479]
    return residual_branch(features, feats)

# Choosing alpha by minimising KL(P_c || P_h)  [§4.3] [eq. 9]
# Input:  code counts on 200 clean images and on 200 real hazy images  [§4.3]
# Output: alpha_hat (paper reports 21.25)

def fit_alpha(clean_freq, hazy_freq_for, low, high):
    # Binary search for alpha; bounds and stopping rule are not stated
    # [§4.3]
    while not converged(low, high):
        mid = (low + high) / 2
        # Forward KL between clean and hazy code distributions  [eq. 9]
        divergence = kl(clean_freq, hazy_freq_for(mid))
        # Narrow [low, high] using divergence; the update rule is not
        # stated  [§4.3]
        low, high = update_bounds(low, high, mid, divergence)
    return mid

Hyperparameters

Results

Author-reported. Setup: RTTS, over 4,000 real hazy images, no ground truth. Metrics are no-reference: FADE and BRISQUE (lower is better), NIMA (higher is better), and the share of user-study votes §5.1 §5.3 tab. 1. Table 1 marks red for best and blue for second best in the paper. The user study used 100 images with 5 experts and 5 non-expert observers §5.3.

The paper says RIDCP improves BRISQUE by 25.57% and NIMA by 1.86% over the best baseline. On FADE, PSD ranks first; the paper argues that PSD's lead is misleading because PSD over-enhances §5.3 fig. 5. The other comparisons are visual only: RTTS and Fattal's images in figures 5 and 6, the NFA variants in figure 8, and the pipeline ablation on OTS and Haze4K in figure 9. The paper gives no numbers for them §5.4.

Paper vs code

Official code: https://github.com/RQ-Wu/RIDCP_dehazing, at code_commit. The paper links it. Licence: CC BY-NC 4.0 (non-commercial), so cite only and do not copy code LICENSE. Searches: not needed, since the paper links the repository.

• Transmission. The paper writes t(x) = exp(beta · d) §3. The code uses exp(−2 · beta · (1 − d̂)), with min-max normalised depth and a factor 2 haze_online_dataset.py. The paper's sign and depth direction are therefore not what the code does.
• Colour bias and probabilities. The paper gives no probability for the colour bias. The config applies it every time RIDCP.yml. The code adds Gaussian noise half the time, and the paper does not describe that step haze_online_dataset.py.
• CHM distance. The paper weights unsquared distances eq. 7. The code's distance is squared, because it expands ||x||² + ||y||² − 2xy [code: basicsr/archs/dehaze_vq_weight_arch.py#L63-L69], and the weight multiplies that . Multiplicative weights do not commute with the square root, so the chosen code can differ.
• CHM sign. The paper recommends alpha = +21.25 §4.3. The README inference command uses --alpha -21.25 README.md. The CLI default is 1.0 inference_ridcp.py. The weight tensor is loaded from pretrained_models/weight_for_matching_dehazing_Flickr.pth [code: basicsr/archs/dehaze_vq_weight_arch.py#L58-L59], which is not in the repository, so the sign convention of the stored tensor cannot be checked here.
• CHM fitting. The paper's binary search and frequency statistics are not in the repository. The README lists CHM weight acquisition as a to-do [code: README.md#L17]. basicsr/active_codebook.py only counts code indices over 100 RTTS images and plots them active_codebook.py. It does not compute alpha or the weight tensor.
• NFA. The paper writes the ratio as sums eq. 6. The code uses the ratio of means over the whole batch tensor dehaze_vq_weight_arch.py. These give the same ratio for tensors of equal size. The paper's offsets come from one convolution with no mask eq. 5. The code's DCNv2 uses a modulation mask (sigmoid) and a second offset convolution .
• Encoder shapes. Table 2 gives the input stem a 3 × 3 kernel app. A.1. The code uses a 4 × 4 kernel with padding 1 dehaze_vq_weight_arch.py. Table 2 gives the post-quantisation Conv3 kernel as 1. The code uses 3 × 3 ridcp_utils.py. The Swin depth per block is not in the paper. The code uses 6 dehaze_vq_weight_arch.py.
• Encoder loss. The paper's encoder loss has feature MSE with weight 1, plus style and adversarial terms app. A.2. The code's codebook term is beta · MSE with beta = 0.25, plus a Gram loss with weight 1.0 [code: basicsr/archs/dehaze_vq_weight_arch.py#L122-L128] [code: options/RIDCP.yml#L99]. So the effective weight on the MSE is 0.25.
• Adversarial loss. The code applies a hinge loss with weight 0.1 to the encoder features RIDCP.yml [code: basicsr/models/dehaze_vq_model.py#L188-L193]. The paper gives no weight.
• Perceptual loss. The paper uses squared L2 on VGG16 features app. A.2. The code uses LPIPS with a VGG backbone losses.py.
• Final output loss. The code's L1 loss is on the residual branch output dehaze_vq_model.py, which is also the output at test time dehaze_vq_weight_arch.py.
• Training length. The paper trains RIDCP for 10K iterations §5.2. The config sets 45,000 RIDCP.yml.
• Semantic term. The config sets a semantic weight of 0.1 [code: options/RIDCP.yml#L117], but use_semantic_loss defaults to off and the RIDCP config does not turn it on dehaze_vq_weight_arch.py. So the term is inactive during RIDCP training.
• Frozen modules. The config freezes the quantiser, decoder group, after-quant group and output conv RIDCP.yml [code: basicsr/models/dehaze_vq_model.py#L51-L58]. This matches the paper's fixed codebook and decoder §4.1.

Pitfalls

• Transmission sign. The paper's text writes a positive exponent. Follow the code, where transmission falls with depth, unless you have a reason to reverse it. Check the depth model's convention too. The code treats larger normalised values as nearer, which is what the formula implies.
• Alpha sign and scale. The paper recommends +21.25. The README runs with −21.25. Confirm the sign with the released checkpoint before use. Visual strength is roughly linear in alpha §5.4 fig. 7.
• Distance form. Compute CHM on unsquared distances to match eq. 7, or on squared distances to match the code. The two choose different codes in some cases. The paper does not say which it used for the reported result.
• Stage 1 is not reproducible from the repo. The VQGAN config file is empty and the pretraining code is a to-do. Only the released checkpoint is available, so stage 1 cannot be retrained from this release.
• Undocumented inference resizing. The CLI halves images above 1500² pixels before inference and upsamples the output afterwards [code: inference_ridcp.py#L24] inference_ridcp.py. The model reflect- pads to multiples of 32 dehaze_vq_weight_arch.py. Images above 8000² pixels are tiled dehaze_vq_model.py.
• Depth dependency. Training needs RA-Depth maps for every clean image §3. The repo's data table points to them, but they are not included.
• Non-reference evaluation. RTTS has no ground truth, and the paper says quantitative metrics do not reflect dehazing quality well §5.3. PSD scores best on FADE by over-enhancing §5.3.
• Known limits. The paper says RIDCP cannot handle non-homogeneous haze, that the enhancement-only design struggles with very dense haze, and that dehazing methods are hard to compare fairly §6.
• Gram loss scaling. The code's texture loss is the mean squared difference of Gram matrices, normalised by h·w dehaze_vq_weight_arch.py. The paper does not give its normalisation.
• Cost. The paper does not report inference time, memory or parameter count. The architecture uses Swin blocks and deformable convolutions §4.2 app. A.1.

Lineage

• Builds on: VQGAN (Esser et al., "Taming Transformers") for the codebook prior §4.1 §2.2.
• Related codebook restoration work: VQFR, CodeFormer and FeMaSR, cited for face restoration and super-resolution §2.2.
• Compared against: MSBDN, Dehamer, DAD, PSD and D4 tab. 1 §5.3.
• Superseded by: not stated in this card's sources.