Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalization

In plain words

Photos taken with a real camera contain noise that does not look like the neat random grain that most denoisers are trained on. A denoiser trained on artificial noise often does well on artificial noise and poorly on real photos. Collecting real noisy photos paired with clean versions is slow and costly, so there are few of them.

This paper proposes a denoiser that first estimates how strong the noise is at each pixel, then uses that estimate to adjust its internal features. The adjustment happens inside every block, so the network can respond to changing noise across the picture. The authors train it on artificial noise and then adapt it to real photos by changing only a small part of the network. The adaptation uses a small number of real pairs.

The authors report that this approach works well on two public benchmarks of real photos, and that the adaptation needs only a few examples to help.

TL;DR

• AINDNet = a noise-level estimator plus a U-Net whose residual blocks use adaptive instance normalisation (AIN) driven by that estimate §3.1 fig. 2.
• Trained with an L1 loss and a noise-map loss on synthetic noise §3.2 eq. 6. Real-noise adaptation updates only the AIN modules, the estimator and the last convolution, with an L1 loss §3.2 eq. 7 fig. 4.
• Author-reported: 39.53 dB PSNR on DND with synthetic training only, the best DND value in the authors' table among rows without self-ensemble tab. 3.
• Author-reported: 39.37 dB on DND and 38.95 dB on SIDD validation with transfer learning tab. 3 tab. 4. With 32 real pairs, the transfer model reaches 38.75 dB on SIDD validation, above RIDNet trained on all 320 pairs (38.71 dB) tab. 5 §4.3.
• Official code exists but contains only test scripts and the network definition. Training settings cannot be checked against code model.py.

Problem

Real-world noise is not Gaussian. Its strength varies across the image and it depends on the camera's processing pipeline §1 §2. Denoisers trained on synthetic noise tend to overfit the noise model they saw, so they generalise poorly to real photos §1. Real noisy-clean pairs are hard to collect, and training a deep network on them alone overfits to one camera §1.

The method assumes that a noise model for synthetic training data is available (here the one from CBDNet) and that a small number of real pairs can be gathered for the adaptation step §3 §3.2.

Method

Overview. The network has two parts: a noise-level estimator that predicts a noise map from the noisy image, and a reconstruction network that takes the image and the map and outputs the denoised image §3.1 eq. 1 fig. 2.

Noise-level estimator §3.1 eq. 1 fig. 2. The estimator has a coarse branch and a fine branch. The coarse branch works at a reduced resolution, which gives it a larger receptive field. The two outputs are blended with weights 0.8 (coarse, upsampled) and 0.2 (fine). The paper describes the coarse map at one-quarter resolution; the code's coarse map is at one-eighth resolution (see Paper vs code).

AIN module §3.1 eq. 2 eq. 3 eq. 4. Each feature map is normalised per channel over its spatial positions. Then a scale and shift are computed for every pixel from the noise map, and they replace the normalised values' affine parameters. Thus the normalisation is channel-wise, and the affine transform is pixel-wise and depends on the noise level §3.1.

AIN-ResBlock §3.1 fig. 3. Two blocks, each consisting of an AIN module, a Leaky ReLU and a 3x3 convolution. A residual connection adds the block's input to its output model.py.

Reconstruction network §3.1 fig. 2. A U-Net with three scales (channels 64, 128, 256; scale factors 1, 2 and 4). Downsampling uses average pooling. Upsampling uses a transposed convolution and adds the matching encoder feature. Every convolution is 3x3 except the last one, according to the figure caption fig. 2. The code's last convolution is 1x1 (see Paper vs code).

Transfer learning §3.2 fig. 4. Stage 1 trains the synthetic-noise (SN) denoiser on synthetic pairs. Stage 2 copies its weights to a real-noise (RN) denoiser, freezes everything except the AIN modules, the noise-level estimator and the last convolution, and trains those on real pairs with an L1 loss only §3.2 eq. 7. The paper argues that the noise-level estimator needs no target of its own in this stage; it is trained through the reconstruction loss §3.2.

Forward pass (own rendering of §3.1, fig. 2 and the code; the code adds the input back to the network output, see Paper vs code):

# AINDNet inference  [§3.1] [fig. 2] [code: code/model.py#L132-L139]
# Input:  noisy image (B, H, W, 3), values in [0, 1]
# Output: denoised image (B, H, W, 3)

def aindnet(noisy):
    # Estimate a per-pixel noise map  [§3.1] [eq. 1]
    noise_map = noise_level_estimator(noisy)
    # Reconstruct the residual from the image and the map  [§3.1]
    residual = reconstruction_network(noisy, noise_map)
    # Code adds the noisy input back; the paper's equation does not
    # [code: code/model.py#L137]
    return residual + noisy

AIN module. The paper's equation writes the pixel-wise scale without the "1 +" that the code uses; the code's form is shown here.

# AIN module: normalise, then a pixel-wise affine map  [§3.1] [eq. 2]
# Input:  features (B, H', W', C), noise_map (B, H, W, 3)
# Output: features (B, H', W', C)

def ain(features, noise_map):
    # Resize the noise map to this feature resolution  [code: code/model.py#L39]
    noise_here = resize_bilinear(noise_map, size=(H', W'))
    # Per image and channel, normalise over space  [§3.1] [eq. 3] [eq. 4]
    mean = mean_over_space(features)
    std = sqrt(variance_over_space(features) + 1e-5)
    normalised = (features - mean) / std
    # Shared hidden layer on the noise map  [code: code/model.py#L42-L45]
    hidden = relu(conv5x5(noise_here, channels=64))
    # Pixel-wise scale and shift; code writes the scale as 1 + gamma
    # [code: code/model.py#L46-L54]
    gamma = conv3x3(hidden, channels=C)
    beta = conv3x3(hidden, channels=C)
    # Affine transform  [eq. 2]
    return normalised * (1 + gamma) + beta

AIN-ResBlock.

# AIN-ResBlock  [§3.1] [fig. 3] [code: code/model.py#L57-L67]
# Input:  features (B, H', W', C), noise_map (B, H, W, 3)
# Output: features (B, H', W', C)

def ain_resblock(features, noise_map):
    # First AIN module, then Leaky ReLU, then a 3x3 convolution  [fig. 3]
    x = ain(features, noise_map)
    # Leaky ReLU; slope not stated in paper  [code: code/model.py#L60]
    x = leaky_relu(x, slope=0.02)
    x = conv3x3(x, channels=C)
    # Second AIN module, Leaky ReLU and 3x3 convolution  [fig. 3]
    x = ain(x, noise_map)
    x = leaky_relu(x, slope=0.02)
    x = conv3x3(x, channels=C)
    # Residual connection  [code: code/model.py#L67]
    return x + features

Noise-level estimator. The paper does not state which image the fine branch takes; the code computes it from the upsampled coarse map.

# Noise-level estimator  [§3.1] [eq. 1] [fig. 2] [code: code/model.py#L69-L82]
# Input:  noisy image (B, H, W, 3)
# Output: noise map (B, H, W, 3), non-negative

def noise_level_estimator(noisy):
    # Two 3x3 convolutions with 32 channels and ReLU  [fig. 2]
    x = relu(conv3x3(noisy, channels=32))
    x = relu(conv3x3(x, channels=32))
    # 4x4 average pooling; the paper says the result is at H/4  [§3.1]
    x = avg_pool(x, size=4)
    x = relu(conv3x3(x, channels=32))
    x = relu(conv3x3(x, channels=32))
    # 2x2 average pooling; code's total scale is H/8  [code: code/model.py#L76]
    x = avg_pool(x, size=2)
    # Coarse map, 3 channels, ReLU  [code: code/model.py#L77]
    coarse = relu(conv3x3(x, channels=3))
    # Fine branch: upsample the coarse map, then two 3x3 convolutions
    # [code: code/model.py#L79-L81]
    up = resize_bilinear(coarse, size=(H, W))
    fine = relu(conv3x3(relu(conv3x3(up, channels=3)), channels=3))
    # Blend coarse and fine with weights 0.8 and 0.2  [§3.1] [eq. 1]
    # [code: code/model.py#L136]
    return 0.8 * up + 0.2 * fine

Multi-scale asymmetric loss (synthetic stage only) §3.2 eq. 5. It penalises the noise-map error at two scales and treats under- and over-estimates differently.

# Multi-scale asymmetric loss for the noise map  [§3.2] [eq. 5]
# Input:  predicted and ground-truth maps at scales 1 and 4 (dicts)
# Constants: alpha = 0.25; weights w_1 = 0.2, w_4 = 0.8  [§3.2]

def asymmetric_noise_loss(pred, true):
    total = 0
    # Scale 1 with weight 0.2, scale 4 with weight 0.8  [§3.2]
    for scale, weight in [(1, 0.2), (4, 0.8)]:
        diff = pred[scale] - true[scale]
        # Indicator: 1 where the estimate is below the truth  [eq. 5]
        under = (diff < 0).float()
        # Asymmetric weight |alpha - indicator|  [eq. 5]
        asym = abs(0.25 - under)
        # Reduction over pixels is not stated; a sum is shown  [eq. 5]
        total = total + weight * sum(asym * diff ** 2)
    return total

Total losses. Stage 1 (synthetic): L1 reconstruction loss plus 0.05 times the asymmetric noise loss §3.2 eq. 6. Stage 2 (real): L1 reconstruction loss only §3.2 eq. 7. Both use Adam §3.2.

# Two training losses  [§3.2] [eq. 6] [eq. 7]
# Stage 1: synthetic pairs (y_s, x_s), with noise-level targets
loss_sn = l1(model(y_s) - x_s) + 0.05 * asymmetric_noise_loss(pred, true)
# Stage 2: real pairs (y_r, x_r); only AIN, estimator and last conv train
loss_rn = l1(model(y_r) - x_r)

Noise is synthesised with the CBDNet noise model §3. The paper does not restate the parameters of that model in this card's source.

Hyperparameters

Results

Author-reported. The paper's numbers come from its own training and from benchmark servers (DND and SIDD scores were obtained by submitting denoised images to the benchmark sites) §4.1.

Estimator accuracy on Kodak24 (MAE and STD of the noise-level error, L1 training) tab. 1:

Estimator accuracy, full grid (16 noise settings, Kodak24, FCN from [guo2019toward]) app. "More Noise Level Estimation Results" tab. 8:

AWGN denoising, PSNR (dB) tab. 2. Set12 and BSD68, sigma = 15, 25, 50. The source's bold or colour marking of best and second-best values is not reproduced here.

DND benchmark, PSNR and SSIM tab. 3. Training data column as the paper states it (Synthetic, Real, All, or none for classical methods). The asterisk marks geometric self-ensemble results.

The source writes one row as "AIDNet + RT" (here AINDNet+RT); the SIDD table does the same.

SIDD benchmark, PSNR and SSIM tab. 4:

The SIDD validation set and the test server numbers are not separated in the table; the paper states the SIDD benchmark numbers §4.1.

Real-noise adaptation versus number of real pairs, SIDD validation PSNR (dB) tab. 5. The column headings are the number of real training images:

Reconstruction block comparison, SN training tab. 6:

Transfer from AWGN, SIDD validation PSNR (dB) tab. 7. The source defines AINDNet(AWGN) as a denoiser trained on Gaussian noise, and "TF_1" as adaptation with a single real image:

Noise-map loss ablation, SN training tab. 9:

Which parameters to update in transfer, SIDD validation PSNR (dB) tab. 10:

Qualitative comparisons on SIDD and RNI15, and a transfer example, are shown in fig. 5, fig. 6 and fig. 7; they are described in words only here.

Paper vs code

The repository is terryoo/AINDNet at code_commit. The paper says the code "will be" released at that address §5 and the README links the same address README.md. The paper itself names this address, and the owner (terryoo) matches the author-chosen name in the source's e-mail addresses, so the repository is taken to belong to the authors.

Searched: the paper's own link, which opened the repository directly. No other search was needed.

What the repository contains: test scripts (code/test_images.py, code/test_dnd.py), the network definition (code/model.py), a README, a test set (RNI15), and a checkpoint index file. The trained weights are on a Google Drive link in the README and were not downloaded checkpoint. Training code is absent: there are no loss, optimiser, learning-rate, batch or epoch settings in the repository.

The repository has no LICENSE file, so its license is not stated. The card does not quote code. Reuse of the code in a commercial product needs the authors' permission.

• Coarse map scale. The paper puts the coarse estimate at H/4 §3.1. The code pools by 4 and then by 2, so the coarse map is H/8 model.py.
• Blend weights. Paper: 0.8 on the coarse branch and 0.2 on the fine branch §3.1 eq. 1. Code: the same weights model.py. No discrepancy.
• Fine branch input. The paper does not say what the fine branch reads. The code computes it from the upsampled coarse map, not from the image model.py.
• Scale of AIN. Paper: the pixel-wise scale multiplies the normalised features directly eq. 2. Code: the scale is (1 + gamma), so a zero prediction gives identity model.py.
• Last convolution. Paper figure caption: all 3x3 except the last fig. 2. Code: the last convolution is 1x1 model.py.
• Output. Paper: the denoised image is the reconstruction network's output §3.1. Code: the network output is added to the noisy input model.py. The residual form is an implementation choice that the paper does not state.
• Channels and block counts. Channels 64, 128 and 256 per scale match the figure caption fig. 2. The number of AIN-ResBlocks per scale (2, 2, 5 on the way down; 3 and 2 on the way up) is only in the code model.py.
• Skip path. The encoder skip is passed through an extra plain residual block (no AIN) before the transposed-convolution sum model.py. The paper does not describe it.
• Downsampling blocks. After each pooling, the code applies a plain 3x3 convolution with ReLU and no AIN model.py model.py.
• Leaky ReLU slope. Code 0.02 model.py. Not stated in the paper.
• Self-ensemble. Starred results use geometric self-ensemble §4.2. Neither test script flips or rotates inputs; a search for flip and rotate calls found none test_images.py test_dnd.py.
• Test-time DND crop. The code crops an 80-pixel margin around each of the 20 boxes per image, runs the network on the crop and trims the margin test_dnd.py test_dnd.py. The paper does not describe this step.
• Ground-truth sigma unused. The DND noise parameters are loaded test_dnd.py but not passed to the network, which estimates noise by itself.
• Reported numbers. The README gives its own numbers for an uploaded checkpoint: DND 39.46 dB (paper 39.37), SIDD validation 38.96 dB (38.90), SIDD test 39.01 dB (38.95) readme.md. The README says the checkpoint is higher than the paper "with fixing minor bugs". The bugs are not described.
• Initialisation. Variance-scaling factor 0.02 on the AIN and residual convolutions model.py; deconvolution filters truncated normal with stddev 0.02 model.py. Not stated in the paper.

Pitfalls

• Training is not fully specified. The paper gives Adam, a 256 x 256 patch size and the loss terms, but no learning rates, batch sizes, epoch or iteration counts, or schedule for either stage §3.2 §4.1. The repository has no training code, so these values cannot be recovered from code.
• Noise model for synthetic data. The paper says it uses the CBDNet noise model with heteroscedastic Gaussian noise and in-camera simulation, applied to Waterloo images §3 §4.1. The noise parameters and camera response functions are not restated in this paper.
• Asymmetry direction. With alpha = 0.25, under-estimates get weight 0.75 and over-estimates 0.25 under eq. 5 as written (derived, not stated). The paper's wording is "to prevent under estimation" §3.2. The direction follows the equation.
• Loss reduction. Whether the asymmetric and L1 terms are summed or averaged over pixels and batch is not stated. This changes the effective weight of the 0.05 term.
• Coarse-map size. The paper's ground-truth 1/4-scale noise map is a 4x4 average pool of the full-resolution map §3.2. The code's coarse prediction is 1/8 of the input, so a model that follows the code cannot be supervised with the paper's 1/4 target without changing the pooling.
• Transfer-stage schedule. The paper says the transfer stage updates the AIN modules, estimator and last convolution §3.2. It gives no epochs or learning rate, and the estimator has no direct loss in this stage.
• Freezing AIN is harmful. Freezing the AIN modules in the transfer stage gives 34.60 dB, below the synthetic-only model's 35.21 dB tab. 10. Freezing the estimator or last convolution costs 0.1 to 0.2 dB [app. "Ablation Study"].
• Self-ensemble adds about 0.1 to 0.25 dB on DND. AINDNet(S) rises from 39.53 to 39.77 dB tab. 3. Implement it only if the comparison requires it.
• "Best on DND" depends on the rows. Table 3's colour coding marks the self-ensemble row as best. Among rows without self-ensemble, AINDNet(S) at 39.53 dB is the highest value (derived from the table).
• Synthetic-only results on SIDD are much lower. AINDNet(S) reaches 35.66 dB on SIDD tab. 4, against 38.73 dB for the real-trained model. The authors attribute this to SIDD's similar scenes and objects §4.2.
• Benchmark numbers need a server. DND and SIDD scores were obtained from the benchmark sites §4.1. A local reimplementation can only compare on the RNI15 images, which have no ground truth §4.1.
• Input size. The paper does not state how test images are sized. The network pools twice. The code uses "same" padding and resizes the estimator output to the input size, so odd sizes are not excluded by the code. Not tested (inferred from code).
• Estimator comparison set-up. Table 1 and the full grid train the FCN and the estimator with L1 regression on noise-level targets §4.2. The FCN baseline is the one the authors cite as widely used in CBDNet and Unprocessing §4.2.
• Efficiency claim. The paper says the network has about 13.7 M parameters against about 160 M and 190 M for two challenge methods §4.2 fig. 2. The count is not checked against code.
• Naming slips. The source writes "AIDNet + RT" in its DND and SIDD tables, where it means AINDNet+RT. The tables above keep the paper's values; labels are corrected only where noted.

Lineage

• Builds on: instance normalisation and adaptive instance normalisation, cited in §3.1 and §3.2 by name; the convolutional noise-model and asymmetric loss of CBDNet §2 §3.2; the code's README names SPADE as the base of its implementation README.md.
• Compared against: BM3D, TNRD, DnCNN, UNLNet, FFDNet and RIDNet on AWGN tab. 2; CDnCNN-B, MLP, TNRD, FFDNet, BM3D, WNNM, GCBD, KSVD, TWSC, CBDNet and RIDNet on DND tab. 3; the same and more on SIDD tab. 4. Ablations use IN + Concat and a Gated-ResBlock tab. 6.
• Superseded by: not stated in this card.