Zero-Shot Noise2Noise: Efficient Image Denoising without any Data

In plain words

Denoising means cleaning grain or speckle out of a picture. The usual way is to train a large neural network on many clean and noisy image pairs, which needs a big dataset, and the result can fall apart on photos that look different from the training set. This paper asks what you can do with only the one noisy picture you have.

The trick is that neighbouring pixels in a natural photo are alike, while the noise on each pixel is random. The method makes two smaller copies of the picture by averaging pixels in a fixed pattern. Each copy has the same scene but different random noise, so one copy can act as a noisy "target" for the other. A very small network is trained on this pair, using only the one image. It is also pushed to keep its output consistent when the picture is shrunk first and then cleaned, and when it is cleaned first and then shrunk. Once the network is fitted, it is run on the original picture to give the cleaned result.

The appeal is that no dataset and no noise model are needed, and the network is small enough to run on a CPU. The paper reports that at high noise levels it trails the competing S2S method, which is much slower.

TL;DR

ZS-N2N fits a 2-layer CNN to one noisy image. Two fixed 2x2 averaging filters produce a pair of half-size images with similar signal and independent noise; the network learns to predict one from the other, with residual, symmetric and consistency terms, and the denoised output is the input minus the network's noise estimate. It needs only pixel-wise independent, zero-mean noise. Author-reported: it is the only dataset-free method that stays competitive across the tested Gaussian, Poisson, camera and microscope settings, and is far faster than S2S. Code is not published as a repository; only a Colab notebook is linked.

Problem

Supervised and self-supervised denoisers need large image collections §1 §2. Dataset-free methods exist, but they each have a cost §1 §2:

• BM3D needs the noise level and is tuned for Gaussian noise §2 §4.2.
• Blind-spot networks (Noise2Void, Self2Self) need long optimisation or ensembling and perform poorly at low noise §1 §4.2.
• Deep Image Prior needs an early-stopping point that is hard to choose in advance §2 §4.1.
• Noise2Fast is close in spirit but uses a larger network and needs early stopping §2.

The paper assumes the noise is zero-mean and pixel-wise independent, with no known noise distribution or level §1 §3.2.

Method

Background. Noise2Noise trains a network on two noisy observations of the same scene. With zero-mean, independent noise, mapping one noisy observation to the other has the same minimiser as mapping to the clean image §3.1 eq. 1. Neighbour2Neighbour creates such pairs from one noisy image by sub-sampling §3.1.

ZS-N2N §3.2:

1. Split the noisy image y into two half-resolution images with fixed stride-2 convolutions, channel-wise, so any channel count works §3.2. D1 uses kernel [[0, 0.5], [0.5, 0]] and D2 uses [[0.5, 0], [0, 0.5]]. Both are averages of two pixels from each 2x2 block §3.2 fig. 2.
2. Fit a small image-to-image network f by minimising a loss built from D1(y) and D2(y) and from the full-resolution y. The network is never trained on other images §3.2.
3. Apply the fitted network to the full noisy image and output x_hat = y - f(y) §3.2.

The loss is the sum of a residual term and a consistency term §3.2:

• Residual term. The network predicts the noise in D1(y), so the difference to D2(y) is fitted. A plain squared loss on f(D1(y)) against D2(y) is the starting point eq. 2. The residual form replaces it with || D1(y) - f(D1(y)) - D2(y) ||^2 eq. 3.
• Symmetry. The residual loss is averaged over the two directions, swapping D1 and D2 eq. 4.
• Consistency term. Denoising the full image and then downsampling should agree with downsampling first and then denoising. This term is averaged over both directions and is the only term where the network sees the full-resolution image §3.2 eq. 5 eq. 6.

The paper states that residual learning, the symmetric loss and the consistency term were each needed for good results §3.2 app. A, tab. 5.

# ZS-N2N: fit a small net on one noisy image, then denoise it  [§3.2]
# Input:  noisy image y, (H, W, C)
# Output: denoised image x_hat, (H, W, C)

def zs_n2n(y):
    # Fixed downsamplers, not learned  [§3.2]
    down_1 = fixed_stride2_filter([[0, 0.5], [0.5, 0]])
    down_2 = fixed_stride2_filter([[0.5, 0], [0, 0.5]])
    f = small_net_init()                         # 2 conv 3x3 + 1 conv 1x1 [§3.2]
    # Optimise only f; the iteration count is stated only as 1k to 2k [§3.2]
    for step in range(num_iterations):
        loss = residual_loss(f, y) + consistency_loss(f, y)
        f = gradient_step(f, loss)               # optimiser not stated
    # Final estimate uses the full-resolution input  [§3.2]
    return y - f(y)


def residual_loss(f, y):
    # Symmetric residual term, half-resolution inputs  [eq. 3] [eq. 4]
    d1 = down_1(y)                               # (H/2, W/2, C)
    d2 = down_2(y)                               # (H/2, W/2, C)
    term_a = squared(d1 - f(d1) - d2)
    term_b = squared(d2 - f(d2) - d1)
    return (term_a + term_b) / 2


def consistency_loss(f, y):
    # Symmetric consistency term, full-resolution f(y)  [eq. 5] [eq. 6]
    d1 = down_1(y)
    d2 = down_2(y)
    residual = y - f(y)                          # (H, W, C)
    term_a = squared(d1 - f(d1) - down_1(residual))
    term_b = squared(d2 - f(d2) - down_2(residual))
    return (term_a + term_b) / 2

Network §3.2. Two 3x3 convolutions followed by one 1x1 convolution. No normalisation or pooling layers. The paper gives about 20k parameters §3.2; the compute table lists 22k tab. 4; the ablation's default is also 22k tab. 6. Hidden width, activation, padding, bias and output channels are not stated.

Schedule. Convergence "typically" takes 1k to 2k gradient iterations, under half a minute on a GPU and about a minute on a CPU §3.2. The optimiser, learning rate, initialisation and any stopping rule are not stated. The paper says the output keeps improving with more iterations, shown for one Kodak24 image app. A fig. 5.

Equation numbers: eq. 1 is the Noise2Noise equivalence (appendix proof app. B); eq. 2 is the plain pair loss; eq. 3 residual; eq. 4 symmetric residual; eq. 5 consistency with a generic D; eq. 6 symmetric consistency with D1, D2.

Hyperparameters

Results

Author-reported. Setup: Kodak24 and McMaster18 synthetic noise §4.2; PolyU and SIDD real camera noise, 20 random images per dataset §4.3; Fluorescence Microscopy dataset, three listed images §4.3; a 256x256 colour image for timing §4.4. Baselines: N2C (UNet, supervised), NB2NB and N2V (self-supervised, trained on 500 ImageNet images), DIP and S2S (zero-shot), BM3D (classical), and N2F for microscopy. DIP is early-stopped with ground truth, so its numbers are optimistic §4.1. S2S is the original 50-network ensemble; S2S* is the ensemble-free version §4.1.

Synthetic noise, PSNR in dB tab. 1. Columns: Kodak24 at the three levels, then McMaster18 at the same three levels. "Known" means the dataset-based network was trained at that exact level; "no" means the blind-range network §4.2.

In the Poisson rows the three levels are lambda = 50, 25, 10 in that order tab. 1.

Real camera noise, PSNR in dB tab. 2, 20 random images per dataset §4.3:

Microscope noise, PSNR in dB tab. 3, three listed rows:

Compute tab. 4. Time to denoise one 256x256 colour image; parameter counts. The S2S GPU time is half of the value the S2S authors report, following the paper's argument that only half the iterations are needed §4.4.

Ablations (Kodak24, Gaussian sigma = 25 unless stated) app. A:

• Loss variants tab. 5: default 29.07; without symmetric loss 28.65; without consistency 28.01; without residual 17.93 (the network then learns the identity).
• Network size tab. 6: UNet 3.3M gives 21.01; default 22k gives 29.07; 4k: 28.66; 2k: 28.28; 1k: 28.07; 500: 27.71. The paper attributes the UNet result to overfitting on the single image.
• Training-set size for N2C fig. 4: 500 images 28.27; 4000 images 30.67; 10000 images 30.98. Dataset-free methods are constant in this plot.

Headline summary (text): ZS-N2N is on par with or better than the other dataset-free baselines on Gaussian, Poisson, camera and microscope noise, except at high noise levels where S2S is better §4.2 app. B. The paper states it needs about 1/200 of S2S's denoising time and 2% of its memory §4.2.

Paper vs code

No official code. The abstract and the paper link only a Google Colab notebook (colab.research.google.com/drive/1i82nyizTdszyHkaHBuKPbWnTzao8HF9b). The notebook is not a repository and its page could not be read (only the Colab sign-in shell returned). A GitHub repository search for "zero-shot noise2noise" and "ZS-N2N" returned only repositories owned by people who are not the paper's authors (for example Li-Tong-621/ZS-N2N, herryi/ZS-N2N, Marvel-Gu/Zero_Shot_Noise2Noise). These are third-party reimplementations and were not used. No code, code_commit or code_license is recorded. The Colab notebook is the authors' own demo, and the abstract says it contains the code and hyperparameters. An implementer should check it directly before relying on the values the paper leaves unstated.

Pitfalls

• "Diagonal" is ambiguous. The text says D1 averages the diagonal pixels of each 2x2 block and D2 the anti-diagonal ones §3.2. The kernel printed for D1 has its non-zero entries at the top-right and bottom-left positions. Implement the kernels exactly as written, and check the output of the Figure 2 example fig. 2.
• Eq. 5 is generic. It writes D without saying which downsampler. Eq. 6 makes clear that each of D1 and D2 is used in turn. Using one downsampler in eq. 5 gives a different loss.
• Loss weights. The paper sums residual and consistency terms with weight 1 §3.2. No other weighting is stated, and the ablation does not test one.
• Network details missing. Width, activation, padding, bias and output channels are not stated. The parameter count is 20k in the text and 22k in the tables §3.2 tab. 4 tab. 6. With no activation, the three linear convolutions would be a linear map; the paper does not say whether one is used.
• Optimiser and stopping rule missing. No optimiser, learning rate, initialisation or iteration count is given beyond "typically 1k to 2k" §3.2. The reported numbers depend on the count used. The paper says quality still improves with more iterations for one image app. A, fig. 5, so a fixed count is a choice, not a convergence test.
• Residual output. The final image is y - f(y), computed at full resolution §3.2. The downsampled images only enter the loss; the full-resolution pass happens in the consistency term, so each iteration costs one full-resolution forward pass.
• Poisson scaling. Lambda values are said to lie in [0, 1] §4.2, but the mapping from the image to the Poisson counts is not stated. Reimplementers must choose it.
• Baseline fairness. DIP's early stopping uses ground truth, so the DIP figures are an upper bound §4.1. S2S's timing is half the authors' reported figure §4.4. The ensemble S2S (50 networks) is not directly comparable with the single-network methods; S2S* is the like-for-like row §4.1.
• Small real-data samples. Real camera results use 20 random images per dataset §4.3; the microscope table has three image rows tab. 3. The differences between methods on these samples are small (fractions of a dB in several rows).
• Timing arithmetic. The text says ZS-N2N is 35 times faster than DIP on CPU §4.4. Table 4 gives 45 min against 80 s, which is about 34 times. The 200 times figure for S2S (4.5 h against 80 s) does check out.
• Known weaknesses. At high noise (Gaussian sigma = 50, Poisson lambda = 10) ZS-N2N trails S2S app. B. The method is motivated for pixel-wise independent, zero-mean noise [abstract] §1; the paper does not test correlated noise.
• Noise2Fast comparison. The Noise2Fast baseline is run on grayscale only, the only form for which code was available §4.3. The paper reports sharper edges than Noise2Fast on a knee MRI example fig. 3; this is a visual judgement, not a metric.

Lineage

• Builds on: Noise2Noise, which trains on pairs of noisy observations §3.1 eq. 1; Neighbour2Neighbour, which creates such pairs from one image by sub-sampling §3.1. Neither is linked to a card here.
• Compared against: Noise2Clean with a UNet, NB2NB, Noise2Void, DIP, Self2Self, BM3D and Noise2Fast §4.1 §4.3. Noise2Fast is the closest prior method §2.
• Related: Noisier2Noise and Anscombe-based approaches, cited as explicit noise-model methods that the paper excludes §4.1. Self-supervised pretraining with a symmetric loss follows SimSiam §3.2.
• Superseded by: not known from content in this wiki.