wikillm

Paper · 2021-08-20 · cs.CV

Structure-Preserving Deraining with Residue Channel Prior Guidance

SPDNet removes rain streaks from one image while keeping object edges. A wavelet multi-level backbone is guided by the residue channel prior (max minus min over RGB), re-extracted from each of three stages' outputs. Author-reported PSNR on Rain200H: 31.30 versus 30.24 for RCDNet.

Wavelet transformChannel attentionMulti-scale feature fusionImage enhancementimage-deraininglow-level-visionwaveletresidual-network

arXivPDFCodeMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Rain streaks cover a photo and hide the edges of the things in it. Most learned deraining models try to find the streaks and subtract them. Where the streaks are dense or uneven, they remove too much or too little, and the shapes of objects come out blurred or broken.

SPDNet starts from a different angle. It builds a rough map of the scene's brightness structure directly from the rainy photo, using the gap between the brightest and darkest colour channel at each pixel. That map keeps the outlines of objects even though it is computed from the rainy image. The network uses it as a guide while it rebuilds the clean picture, so the outlines stay sharp.

The model does not run once. It produces a result, builds a cleaner version of the guide map from that result, and repeats this. Each pass gives a better guide for the next one. The work is in how the guide is fused with the image features, and in a backbone that splits the features into several resolutions so that details are not lost when the image is shrunk and expanded again.

TL;DR

  • Single-image deraining network that keeps the structure of objects, not only the streaks removed §1.
  • Guided by the residue channel prior (per-pixel max minus min over the colour channels), refined over three stages §3.2 eq. 15.
  • Author-reported: best PSNR on the four synthetic sets and on SPA-Data, with 3.04M parameters tab. 1.
  • Code is public but has no LICENSE file, is written for PyTorch 0.4.1 and hard-codes CUDA; the checks below list what differs from the paper.

Problem

Learned deraining methods mostly predict the rain-streak layer and subtract it. Because streak density varies, this removes too much or too little, and the structure of the underlying scene is lost §1. Hand-chosen image priors also fail: total variation smooths texture, sparse priors need extra domain knowledge, and edge detectors react to the streaks §1. The authors want a prior that keeps structure, can be computed from the rainy image alone, and needs no extra parameters §1 §3.2.1.

Method

The network has three stages. Each stage is a wavelet-based multi-level backbone (WMLM) that receives the rainy-image features and an RCP feature. The RCP for the first stage comes from the rainy image. For stages two and three it comes from the previous stage's output §3.2.4 eq. 15.

1. Residue channel prior (RCP). Under a simplified image-formation model the rain term is achromatic, so it cancels after colour normalisation. The RCP at each pixel is the largest minus the smallest value over the three colour channels §3.2.1 eq. 4 eq. 5 eq. 6. The paper says the RCP still works without colour normalisation, which is the form the code uses (see Paper vs code).

2. RCP extraction module. A convolution, then an SE-ResBlock stack (SRiR), turns the one-channel RCP into a feature map §3.2.2. The paper names one convolution; the code uses two (see Paper vs code).

3. Wavelet-based multi-level module (WMLM). Downsampling and deconvolution are replaced by a discrete wavelet transform (DWT) and its inverse (IWT) to keep frequency and location information §3.1. Each level applies a convolution after the DWT eq. 1. An SRiR block then processes each level eq. 2. Upsampling is an IWT of a convolved feature, added to the feature of the finer level eq. 3.

4. Interactive fusion module (IFM). The rainy features and the RCP features each pass through a 3×3 convolution eq. 7 eq. 8. Their element-wise product goes through a sigmoid, giving a gate in (0, 1) eq. 9. The gate multiplies each feature map, and the result is added back to that feature map eq. 10 eq. 11 eq. 12 eq. 13. The two outputs are concatenated eq. 14 §3.2.3 fig. 5.

5. Iterative guidance. Stage n produces a restored image B_n from its WMLM output. The next stage's prior is the RCP of B_n. The paper uses three WMLMs and three RCP extraction modules §3.2.4 eq. 15 §4.1. Each stage's output is supervised §3.3, and stage n's features are concatenated with earlier stages' features as an ensemble §3.2.4.

Pseudocode. The paper has no algorithm box. The two blocks below are our reconstruction from the equations and the code, not a copy of either.

Pseudocode
# SPDNet forward pass: three stages, each guided by the previous output
# Input:  rainy image (B, 3, H, W), values 0 to 255
# Output: three restored images; the last one is the result
# Reconstructed from [eq. 15] and [code: src/model/spdnet.py#L280-L295]

def forward(rainy):
    # Shallow features of the rainy image  [code: src/model/spdnet.py#L283]
    shallow = shallow_features(rainy)                 # (B, C, H, W), C = 32
    # Prior of stage 1: RCP of the rainy image  [§3.2.1] [eq. 6]
    prior = residue_channel_prior(rainy)              # (B, 1, H, W)
    rain_in = shallow
    stage_feats = []
    outputs = []
    for n in range(3):
        # Features of the current prior  [§3.2.2]
        prior_feats = rcp_extractor[n](prior)         # (B, C, H, W)
        # Wavelet backbone with interactive fusion  [§3.1] [§3.2.3]
        feats = wmlm[n](rain_in, prior_feats)         # (B, C, H, W)
        stage_feats.append(feats)
        # Restored image of this stage  [eq. 15]
        outputs.append(output_conv[n](feats))         # (B, 3, H, W)
        # Ensemble: concatenate with earlier features  [§3.2.4]
        rain_in = fuse_ensemble[n](concat(stage_feats + [shallow]))
        # Clearer RCP from this stage's output, for stages 2 and 3  [eq. 15]
        if n < 2:
            prior = residue_channel_prior(outputs[-1])
    return outputs
Pseudocode
# Interactive fusion of rainy and prior features  [§3.2.3]
# Input:  rain_feats, prior_feats (B, C, H, W)
# Output: concatenated features (B, 2C, H, W)

def interactive_fusion(rain_feats, prior_feats):
    # 3x3 convolutions map both inputs  [eq. 7] [eq. 8]
    rain_mapped = conv_rain(rain_feats)           # (B, C, H, W)
    prior_mapped = conv_prior(prior_feats)        # (B, C, H, W)
    # Similarity gate in (0, 1), element-wise product  [eq. 9]
    gate = sigmoid(rain_mapped * prior_mapped)    # (B, C, H, W)
    # Gate each input, then add it back to that input  [eq. 10-13]
    rain_out = gate * rain_feats + rain_feats     # (B, C, H, W)
    prior_out = gate * prior_feats + prior_feats  # (B, C, H, W)
    # Concatenate the two branches  [eq. 14]
    return concat(rain_out, prior_out)            # (B, 2C, H, W)
Pseudocode
# Residue channel prior of an image  [§3.2.1] [eq. 6]
# Input: image (B, 3, H, W); Output: prior (B, 1, H, W)

def residue_channel_prior(image):
    # Per-pixel maximum over R, G, B  [eq. 6]
    largest = max_over_channels(image)            # (B, 1, H, W)
    # Per-pixel minimum over R, G, B  [eq. 6]
    smallest = min_over_channels(image)           # (B, 1, H, W)
    return largest - smallest                     # (B, 1, H, W)

Key loss, summed over the three stage outputs eq. 16:

\mathcal{L} = \sum_{i=1}^{3} \left\| \mathcal{B}_i - \hat{\mathcal{B}} \right\|^2 eq. 16

Hyperparameters

Values marked "code" are not stated in the paper text and come from the repository.

NameValueWhereNotes
OptimizerAdam§4.1Betas, epsilon and weight decay not stated in the paper; code defaults are 0.9, 0.999, 1e-8 and 0 option.py
Learning rate5e-4§4.1 README.mdCode default is 1e-3 option.py; the README training command passes 5e-4
Learning-rate schedulenot stated in paperoption.py utility.pyCode: multi-step decay at epochs 100, 150, 200, 230, 260, 280, 300 with factor 0.5
Batch size16§4.1 README.md
Patch size128 × 128§4.1 README.mdRandom crop of the same region from input and target common.py
Epochs300 (Rain200L/H, Rain800, Rain1200)§4.1 README.md
Epochs, SPA-Data6§4.1Learning rate stated as 5e-4. The SPA-Data loader is not in the repository (see Paper vs code)
Number of WMLMs3§4.1 §4.3Table 4 shows 1 and 2 as well tab. 4
Number of RCP extraction modules3§4.1One per stage spdnet.py
Feature channels Cnot statedoption.py README.mdCode default and README value are 32
Residual blocks per SRiRnot statedoption.py README.mdCode default and README value are 3
Waveletnot stated§3.1 spdnet.pyCode uses Haar, one decomposition level
SE reduction rationot statedspdnet.pyCode uses reduction 1, so no bottleneck in the SE gate common.py
LossMSE per stage, summed over 3 stageseq. 16 trainer.pyCode uses mean-reduced MSE, see Paper vs code
Skip thresholdnot statedtrainer.py option.pyCode skips a batch whose loss exceeds 1e6 times the previous epoch's loss
Augmentationnot statedcommon.py srdata.pyHorizontal flip with probability 0.5 only
Input rangenot statedcommon.pyInputs stay in 0 to 255 with no mean subtraction
Weight initialisationnot statedspdnet.py spdnet.pyCustom initialiser is commented out; PyTorch default is used

Results

Author-reported. Setup: PSNR and SSIM in YCbCr space, following RCDNet, on four synthetic sets and SPA-Data §4.2.1 tab. 1. "Best other" is the strongest baseline in each row, as underlined in the paper's table.

DatasetBest other methodBest other PSNR / SSIMSPDNet PSNR / SSIMSource
Rain200LRCDNet39.87 / 0.987540.59 / 0.9880tab. 1
Rain200HRCDNet30.24 / 0.909831.30 / 0.9217tab. 1
Rain800RCDNet28.59 / 0.913730.21 / 0.9152tab. 1
Rain1200DCSFN34.31 / 0.954534.57 / 0.9561tab. 1
SPA-DataRCDNet41.47 / 0.983443.55 / 0.9875tab. 1

Cost on 128 × 128 inputs, averaged over 100 images §4.2.1 tab. 1: SPDNet 3.04M parameters and 0.055 s; RCDNet 3.17M and 0.068 s; DCSFN 6.45M and 0.253 s. The timing hardware is not stated; training used one NVIDIA Titan Xp §4.1.

Real-world images without references §4.2.2 tab. 2: NIQE (lower is better) 3.4603 for SPDNet against 3.5326 for DCSFN; PI (lower is better) 2.6254 against 2.6427 for DCSFN.

Detection on COCO350 with YOLOv3, mAP-50 §4.4 tab. 6: rainy input 0.551, DRDNet 0.652, DCSFN 0.665, RCDNet 0.658, SPDNet 0.692.

Ablations on Rain200H §4.3:

VariantPSNRSSIMSource
SPDNet31.300.9217tab. 3
without IFM (concatenation instead)31.050.9167tab. 3
without ensemble30.980.9142tab. 3
1 / 2 / 3 WMLMs29.85 / 30.90 / 31.300.9006 / 0.9166 / 0.9217tab. 4
RCP guidance rows 0, 1, 2 (update), 3 (update)30.56, 30.82, 31.04, 31.300.9144, 0.9161, 0.9197, 0.9217tab. 5
3 stages, RCP of rainy image kept (no update)31.120.9191tab. 5

Paper vs code

Code: official repository Joyies/SPDNet at commit c03b8d3b7d60705fdcefda73b3a39afc5198dcd4. The paper links it in its abstract. The repository owner is presumably the first author's account, judging by the shared handle; this is inferred, not stated. The repository has no LICENSE file, so no code is quoted here.

  • RCP normalisation. The paper defines the RCP on a colour-normalised image eq. 5 eq. 6. The code computes max minus min directly on the raw 0 to 255 input, with no normalisation spdnet.py. The paper says the RCP still works without normalisation §3.2.1, so this is consistent with the text but not with eq. 5.
  • IFM. The paper's IFM is gate × feature + feature, then concatenation eq. 10-14. The code adds learned 3×3 gating convolutions on the concatenated gated and raw features, and a 3×3 convolution after concatenation spdnet.py spdnet.py spdnet.py.
  • SRiR count. The text says three SRiR blocks are used, one per level §3.1 eq. 2. The code's subnet has three branches, each with three RIR units, so nine per WMLM spdnet.py.
  • Upsampling. The paper's upsampling is IWT of a convolved feature, added to the finer feature eq. 3. The code inserts a learned transposed-convolution block with a residual unit (the Fuse class) between the IWT and the addition spdnet.py spdnet.py.
  • RCP extraction. The paper names one convolution before SRiR §3.2.2. The code uses two 3×3 convolutions before a residual stack spdnet.py.
  • Loss. The paper writes a sum of squared L2 norms eq. 16. The code uses nn.MSELoss, which averages over elements, and sums the three stage losses trainer.py [code: src/loss/init.py#L24-L25]. This is a constant scale factor; the paper does not state a loss reduction.
  • Metric. The paper uses YCbCr PSNR and SSIM §4.2.1. The in-training test in the code computes RGB PSNR after shaving scale plus six border pixels utility.py. The README says the reported numbers come from a MATLAB Y-channel script README.md.
  • SPA-Data. The paper trains on SPA-Data for six epochs §4.1. The repository only has the RainHeavy loader, with placeholder paths rainheavy.py rainheavytest.py. No SPA-Data loader is in the repository.
  • Learning rate. The paper states 5e-4 §4.1; the code default is 1e-3, and the README command overrides it to 5e-4 option.py README.md.
  • Pre-trained weights. The README links pre-trained weights on Google Drive, not in the repository README.md.
  • Environment. README: PyTorch 0.4.1, torchvision 0.2.0, Python 3.6, pytorch_wavelets README.md. The DWT modules call .cuda() directly spdnet.py.

No discrepancies were found in the count of stages, the three-output training, the use of the last stage's output at test time, or the patch size, batch size and epoch count (all match the README and §4.1).

Pitfalls

  • Not runnable on CPU as written. The DWT modules call .cuda() spdnet.py. Dataset paths are placeholders (********) that must be edited before training rainheavy.py.
  • Old toolchain. The code targets PyTorch 0.4.1 and Python 3.6 README.md. We did not run it, so the parameter count (3.04M) and timings are the paper's figures, not verified.
  • Test uses the last stage only. The test code keeps only the third stage's output and quantises it to 8 bits before the metric trainer.py utility.py. The RCP of stages 2 and 3 is computed from the unquantised outputs of stages 1 and 2 spdnet.py.
  • Loss scale. Inputs and targets are kept at 0 to 255 with no mean subtraction common.py. The MSE values are therefore on that scale. This matters if you port the loss or change the learning rate.
  • Batch skipping. The trainer skips any batch whose loss is more than 1e6 times the previous epoch's loss, and the first epoch's reference value is 1e8 trainer.py trainer.py. This is not in the paper and can hide divergence.
  • Augmentation. Only horizontal flips are used; the vertical flip and 90-degree rotation are commented out common.py.
  • Initialisation. The custom initialiser is commented out, so PyTorch's default applies spdnet.py.
  • Ambiguous ablation rows. The caption of Table 5 says "Iteration" is the number of RCP guidance steps, but the rows labelled 0 and 1 show no RCP update, and the paper does not explain what they mean tab. 5. Read them as labelled, and check Figure 10 for the visual version fig. 10.
  • Timing hardware. The timing hardware is not stated, so the speeds in Table 1 cannot be compared across papers without a re-run tab. 1.
  • SE reduction. The SE gate uses reduction 1, not the usual 16, which changes the parameter count spdnet.py common.py.

Lineage

  • Builds on: RCDNet (Wang et al., 2020). The README says the code borrows from RCDNet README.md. RCDNet is the strongest baseline in four of the five rows of tab. 1.
  • Residue channel prior: from Li et al. (2018, 2019, 2020), cited in §3.2.1. Their work is not summarised in this wiki.
  • Compared against: DDN, RESCAN, PReNet, DRDNet, DCSFN, RCDNet tab. 1.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “Structure-Preserving Deraining with Residue Channel Prior Guidance” by Qiaosi Yi, Juncheng Li, Qinyan Dai, Faming Fang, Guixu Zhang, Tieyong Zeng (arXiv:2108.09079), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0. Code citations refer to https://github.com/Joyies/SPDNet at commit c03b8d3b7d60 (none (no LICENSE file in repository)). Any quoted code is shown under that license.