Real Image Denoising with Feature Attention

In plain words

Photos taken in the real world carry noise that is not a clean, even speckle: it depends on brightness, the camera and the image processing that follows. Many denoising networks are trained on artificial noise and do poorly on real photos. Others need a first stage that estimates the noise and a second stage that removes it, and often a person has to tune them.

This paper proposes one network that does the job in a single pass and needs no manual setting. It learns the noise itself and subtracts it from the photo, so the network predicts the noise rather than the clean image. Its main building block is a group of modules with several shortcut paths, which let the image's low-frequency content pass through unchanged while the network focuses on what to remove. Each module also has a small "feature attention" step that learns which of its internal channels matter for the current image and scales them up or down.

TL;DR

• Single-stage, blind denoiser for real photographs, plus synthetic noise, built from four enhancement attention modules §3.2 §3.3.
• Uses residual-on-residual skip connections and channel feature attention §3.2 §3.2.1.
• Trained with one L1 loss eq. 4. Author-reported: 39.23 dB PSNR and 0.9526 SSIM on DnD (blind), above CBDNet's 38.06 dB tab. 5.
• The official repository has the test model only; training code is not released README.md.

Problem

Earlier CNN denoisers were trained on synthetic, spatially invariant noise and lose accuracy on real photographs §1. Real-noise pipelines of the time used two stages (noise estimation, then denoising) or needed manual intervention to pick a noise level §1. The authors also note that real noise is spatially variant, non-Gaussian and signal dependent, so the assumption of spatially invariant noise used by many algorithms does not hold §4.3.4.

Method

Overview. The network has three parts: one convolution for feature extraction, a feature-learning body made of residual-on-residual modules, and one convolution for reconstruction. The output is the noisy input plus the predicted residual §3.1 fig. 2.

Steps.

1. Feature extraction: a single convolution on the noisy input gives initial features f_0 §3.1 eq. 1.
2. Feature learning: four enhancement attention modules (EAM) are cascaded. Each EAM output is added to its own input, and the sum of the body and the extracted features forms the group output §3.2 eq. 5 eq. 6.
3. Each EAM §3.2 fig. 2: a. a merge-and-run unit: two branches applied to the same input, each described as passing through dilated convolutions (the text does not say how many per branch; the code uses two per branch), then concatenated and passed through one more convolution §3.2; b. a residual block of two convolutions §3.2; c. an enhanced residual block (ERB) of three convolutions, the last one 1 x 1, which compresses the features §3.2; d. feature attention §3.2.1; e. the attention output is added to the EAM input §3.2.
4. Feature attention §3.2.1 fig. 3: global average pooling produces one value per channel eq. 7. A 1 x 1 convolution reduces the channels by a factor of 16, a soft-shrinkage function is applied, a second 1 x 1 convolution restores the channel count, and a sigmoid produces a gate per channel eq. 8. Each channel of the input is multiplied by its gate eq. 9.
5. Reconstruction: one convolution maps the group output back to the image channels, and the noisy input is added to it §3.2 eq. 3.
6. Training: the mean over the batch of the per-image L1 norm of the difference between the output and the ground truth eq. 4.

# RIDNet forward pass  [§3.1] [§3.2] [eq. 1] [eq. 3] [eq. 5] [eq. 6]
# Input:  noisy image noisy (B, C, H, W), C = 3 (color) or 1 (gray)
# Output: denoised image (B, C, H, W)

def ridnet(noisy):
    # Feature extraction: one 3 x 3 convolution to 64 channels  [§3.1] [eq. 1]
    first = conv3x3(noisy)  # (B, 64, H, W)
    # Four cascaded EAMs, each fed the output of the previous one  [eq. 5]
    features = first
    for module in eam_modules:
        # Each EAM output is also added to its input  [§3.2]
        features = eam(module, features) + features
    # Group skip: add the extracted features back  [§3.2] [eq. 6]
    group_out = first + features
    # Reconstruction: one convolution to C channels, then the input skip  [eq. 3]
    return conv3x3(group_out) + noisy

# Enhancement attention module (EAM)  [§3.2] [fig. 2]
# Input:  features (B, 64, H, W)
# Output: features (B, 64, H, W)

def eam(features):
    # Merge-and-run unit: two branches on the same input  [§3.2]
    # Dilation rates are not stated in the text; see Paper vs code.
    branch_a = dilated_pair(features, dilation_rates=(d1, d2))
    branch_b = dilated_pair(features, dilation_rates=(d3, d4))
    merged = conv3x3(concat(branch_a, branch_b))  # (B, 64, H, W)
    # Residual block of two 3 x 3 convolutions  [§3.2]
    learned = residual_block(merged)
    # Enhanced residual block: three convolutions, the last one 1 x 1  [§3.2]
    compressed = enhanced_residual_block(learned)
    # Channel reweighting by feature attention  [§3.2.1]
    attended = feature_attention(compressed)
    # The attention output is added to the EAM input  [§3.2]
    return attended + features

# Feature attention  [§3.2.1] [fig. 3] [eq. 7] [eq. 8] [eq. 9]
# Input:  features (B, C, H, W), C = 64
# Output: rescaled features (B, C, H, W)

def feature_attention(features):
    # Global average pooling over all spatial positions  [eq. 7]
    descriptor = mean(features, over=(H, W))  # (B, C, 1, 1)
    # Channel reduction by 16 with a 1 x 1 convolution  [§3.3] [eq. 8]
    reduced = conv1x1(descriptor, out_channels=C // 16)  # (B, 4, 1, 1)
    # Soft-shrinkage; the threshold is not stated  [eq. 8]
    reduced = soft_shrink(reduced)
    # Channel upsampling back to C, then sigmoid  [eq. 8]
    gate = sigmoid(conv1x1(reduced, out_channels=C))  # (B, C, 1, 1)
    # Rescale each channel by its gate  [eq. 9]
    return gate * features

# Training loss  [eq. 4]
# Input:  noisy_batch and clean_batch, each (N, C, H, W)
# Output: scalar loss

def training_loss(noisy_batch, clean_batch):
    # Predicted denoised batch  [§3.2]
    predictions = ridnet(noisy_batch)  # (N, C, H, W)
    # Per-image L1 norm of the error, averaged over the N images  [eq. 4]
    return mean(sum(abs(predictions - clean_batch), over=(C, H, W)))

Key equations (as in the paper):

\hat{y} = M_r(f_g) + x §3.2

L(\mathcal{W}) = \frac{1}{N} \sum_{i=1}^N \lVert \mathrm{RIDNet}(x_i) - y_i \rVert_1 eq. 4

f_m = EAM_m(f_{m-1}) eq. 5

f_g = f_0 + M_{fl}(\mathcal{W}_{w,b}) eq. 6

r_c = \alpha(H_U(\delta(H_D(g_p)))) eq. 8

where \delta is soft-shrinkage, \alpha is the sigmoid, and H_D and H_U are the channel reduction and upsampling operators eq. 8.

Hyperparameters

Results

Author-reported. Set-up: PSNR is the metric; baselines are the methods in each table. The DnD scores come from the public online benchmark §4.3.1, so the ground truth is not visible to the authors.

DnD sRGB, blind and non-blind, PSNR (dB) and SSIM tab. 5

BSD68 grayscale, PSNR (dB) tab. 2

12 classical images, grayscale, PSNR (dB) tab. 3

BSD68 color, PSNR (dB) tab. 4

Real noisy datasets, PSNR (dB) tab. 6

SSID numbers use the validation split, because the test images were not released §4.3.1.

Ablation, BSD68 PSNR (dB), 2e5 iterations tab. 1. Columns follow the table's check marks (transcribed from the source table):

The highest value uses all connections §4.2.1 tab. 1.

Paper vs code

Code: official repository saeed-anwar/RIDNet at commit d19ba1bb168f0e701588141c8db4a6f6ae973b3e. The README links the first author's page and the paper README.md. The model code is in TestCode/code/model/. The repository has no LICENSE file, and the GitHub API reports no license. Snippets are not reproduced here.

Architecture settings agree with the paper: four modules, 64 features, reduction 16, 1 x 1 kernels in the ERB last conv and attention units [code: TestCode/code/model/ridnet.py#L61-L66] ops.py option.py option.py.

Differences:

• Feature attention activation: the paper uses soft-shrinkage §3.2.1 eq. 8; the code uses a ReLU after the channel-reduction convolution ridnet.py ops.py.
• EAM skip: the paper adds the attention output to the EAM input §3.2. The code's block returns the attention output with no addition ridnet.py.
• Group skip: the paper's f_g = f_0 + M_fl eq. 6 adds the extracted features to the body output. The code's forward pass does not add the head output anywhere; only the global input skip is present ridnet.py.
• Per-EAM sum f_m = f_m + f_{m-1} §3.2 is not in the code, because the block loop passes each output directly to the next block ridnet.py.
• Input addition in merge-and-run: the code adds the input to the unit's output ops.py. The paper does not state this.
• Dilation rates: the paper gives no numbers. The code uses rates (1, 2) and (3, 4), one pair per branch ops.py.
• Feature extraction: eq. 1 is a plain convolution, but the code's head adds a ReLU ridnet.py ops.py.
• ReLU after residual addition: the residual and enhanced residual blocks apply ReLU after the addition ops.py. The paper does not state this.
• Input mean subtraction: the code subtracts per-channel RGB means at input and adds them back at output ridnet.py common.py. The paper does not mention it. The frozen flag is set on the module rather than on its parameters common.py.
• Training: the README says "Will be added later" README.md. The optimizer, learning-rate schedule, batch, patch size and loss in the paper therefore cannot be checked against code. Only the test model is released.
• Internal paper inconsistency: §4.3.4 gives 3.82 dB and 4.14 dB over NC and NI, but tab. 5 gives 3.80 dB and 4.12 dB (39.23 minus 35.43 and 35.11).

No discrepancies were found in the reported ablation or result tables; the code's test model was not re-run here.

Pitfalls

• The training data is ambiguous. §4.1 lists both synthetic and real sources, while §4.3.4 says the method uses only real noisy images with the L1 loss. Which data trained the reported real-noise model is not stated.
• Noise levels used during training are not stated. The paper says only that test noise uses sigma 15, 25 and 50 §4.3.1.
• The SSID comparison uses the 1280-pair validation split, not an official test split §4.3.1. Results are not directly comparable to a test-set leaderboard.
• Nam: 110 random 512 x 512 crops were chosen from the cropped images, with no seed stated §4.3.1.
• Equation 7 writes both sums with index i, so the inner sum is ambiguous; the intended index is j eq. 7.
• The soft-shrinkage threshold and the sigmoid details are not stated eq. 8. The code uses ReLU instead (see Paper vs code).
• The paper gives no memory footprint. Four modules of 64-channel convolutions run at full resolution, so memory use is high for large inputs. The timing §3.3 does not state its hardware.
• Figure labels: the paper reuses fig:Nam for two figures in the source, so "Figure 8" is the Nam figure fig. 8.
• Weight initialisation is not stated. The code's init helper does nothing ops.py.

Lineage

• Builds on: residual-on-residual from RCAN §3.2, long skip connections from EDSR §2, squeeze-style channel gating from Hu et al. §3.2.1.
• Compared against: CBM3D, WNNM, EPLL, CSF, MLP, TNRD, DnCNN, IrCNN, CNLNet, FFDNet, CBDNet, NC, NI, TWSC, MCWNNM, CIMM and others §4.3 tab. 5.
• Superseded by: not stated in this wiki.