Image deblurring

Overview

A blurry photo is a sharp image smeared by camera or subject motion. Single-image deblurring predicts the sharp image from the blurry one alone. The strongest earlier methods stack sub-networks that each work at a different image scale, from coarse to fine. Each stage refines the previous estimate. This is accurate but costs time and memory, because every stage adds compute 2108.05054 §1.

The variants share one interface: a blurry image goes in and a sharp image comes out, trained against paired sharp images such as the GoPro set [arxiv:2108.05054 §4.1] 2303.09472 §5.4. They differ in where the compute goes. MIMO-UNet keeps the coarse-to-fine supervision but runs it inside one network, with one decoder output per scale 2108.05054 §3. DiffIR runs a diffusion model, but on a compact learned prior rather than on pixels 2303.09472 §4. Two papers are covered so far.

Variants

Single U-Net with multi-scale inputs and outputs (MIMO-UNet)

A U-Net with three encoder blocks and three decoder blocks 2108.05054 §3. Three changes distinguish it from a plain U-Net 2108.05054 §1, §3:

• Multi-input encoder. The blurry image is also fed at half and quarter size. Each smaller input gets a shallow feature module, and a fusion module merges those features into the encoder at its level 2108.05054 §3.1.
• Multi-output decoder. Each decoder block produces a sharp estimate at its own scale. The blurry input at that scale is added to the estimate, so the network predicts a residual 2108.05054 §3.2, eq. 5. The three estimates act as intermediate supervision, which stacked sub-networks would otherwise provide 2108.05054 §3.2.
• Asymmetric feature fusion. Each decoder level receives the encoder features of all three levels, resampled to its size 2108.05054 §3.3, eq. 6. The paper gives this module only as a figure 2108.05054 fig. 4(c).

Training minimises an L1 loss on each estimate, summed over the three scales, plus a frequency-domain L1 loss weighted by 0.1 2108.05054 §3.4, eqs. 7-9.

The paper evaluates three models of this design 2108.05054 §4.2:

• MIMO-UNet: 8 residual blocks per encoder and decoder block.
• MIMO-UNet+: 20 residual blocks per block.
• MIMO-UNet++: MIMO-UNet+ with a geometric self-ensemble at inference. The transforms the ensemble uses are not stated 2108.05054 §4.2.

Author-reported on GoPro, the paper's main benchmark 2108.05054 §4.1: MIMO-UNet reaches 31.73 dB PSNR at 0.008 s and 6.8M parameters; MIMO-UNet++ reaches 32.68 dB at 0.040 s 2108.05054 tab. 1. Card: Rethinking Coarse-to-Fine Approach in Single Image Deblurring.

Ablation on GoPro, starting from a baseline without any of the four components: the full model adds 0.57 dB, of which the frequency loss contributes 0.27 dB 2108.05054 §4.3, tab. 4.

Diffusion over a compact restoration prior (DiffIR)

DiffIR is a general image-restoration method. Its deblurring result is one of several tasks in the paper. It does not diffuse pixels. A stage-1 encoder reads the ground-truth and the blurry image and produces an image-restoration prior (IPR) of 4C' = 256 values, with C' = 64 2303.09472 §4.1, §5.1. A transformer restorer (DIRformer) uses that prior to modulate its feed-forward and attention blocks, and is trained with an L1 reconstruction loss 2303.09472 §4.1, eqs. 6-9.

In stage 2 a diffusion model learns to predict the IPR from the blurry image alone, with T = 4 diffusion steps 2303.09472 §4.2, §5.1. The stage-2 objective trains the condition encoder, the denoiser and the restorer jointly. It adds an L1 term that matches the predicted IPR to the stage-1 IPR 2303.09472 §4.2, eq. 13. At inference, the model draws the IPR from noise, runs the reverse steps, and passes the result to the restorer. No sharp image is needed 2303.09472 §4.2, alg. 2.

On GoPro, trained on GoPro only, the paper reports 33.20 dB PSNR and 0.963 SSIM. On HIDE it reports 31.55 dB and 0.947 2303.09472 tab. 3. The next entries in that table are Restormer at 32.92 dB and MPRNet at 32.66 dB on GoPro 2303.09472 tab. 3. The paper says its deblurring Mult-Adds are 78% of Restormer's 2303.09472 §5.4. The paper gives no deblurring runtime.

For deblurring, training uses L1 only, Adam with β = (0.9, 0.99), and a learning rate of 2e-4 with cosine decay to 1e-6 over 300K iterations. Patch size grows in stages 2303.09472 app. F. The released deblurring code adds an encoder-only warm-up that the paper does not mention; see the paper card before reproducing the schedule. Card: DiffIR: Efficient Diffusion Model for Image Restoration.

When to use which

The paper's timings were measured on an Intel i5-8400 CPU with an NVIDIA Titan XP GPU 2108.05054 §4.1. Other hardware is not covered by the paper. DiffIR reports no deblurring latency, so the two papers cannot be compared on speed. Only the MIMO-UNet timings are available.

Implementation sketch

Language-agnostic outline of the MIMO-UNet forward pass. Module names are ours. This sketch covers the MIMO-UNet variant only. DiffIR's two-stage procedure is on its paper card. The paper's equations omit some decoder layers that the released code contains; see the MIMO-UNet card before implementing.

# Input pyramid: blurry image at full, half and quarter size [arxiv:2108.05054 §3.1]
def mimo_unet(blurry):
    half = resize(blurry, 0.5)
    quarter = resize(half, 0.5)

    # Encoder: fuse each smaller input into its level [arxiv:2108.05054 §3.1]
    enc1 = encoder_block_1(stem(blurry))
    enc2 = encoder_block_2(fuse(strided_conv(enc1), shallow(half)))
    enc3 = encoder_block_3(fuse(strided_conv(enc2), shallow(quarter)))

    # Asymmetric fusion of all encoder levels for each decoder level [§3.3, eq. 6]
    skip1 = aff_1(enc1, up(enc2, 2), up(enc3, 4))
    skip2 = aff_2(down(enc1, 2), enc2, up(enc3, 2))

    # One estimate per scale, residual on the blurry input [§3.2, eq. 5]
    dec3 = decoder_block_3(enc3)
    out_quarter = head_3(dec3) + quarter
    dec2 = decoder_block_2(merge(up(dec3, 2), skip2))
    out_half = head_2(dec2) + half
    dec1 = decoder_block_1(merge(up(dec2, 2), skip1))
    out_full = head_1(dec1) + blurry
    return [out_quarter, out_half, out_full]  # coarsest first

# Training loss: L1 per scale plus 0.1 times an L1 on Fourier coefficients
# [arxiv:2108.05054 §3.4, eqs. 7-9]
def loss(estimates, sharp):
    content = sum(l1(e, t) for e, t in zip(estimates, sharp))
    freq = sum(l1(fft2(e), fft2(t)) for e, t in zip(estimates, sharp))
    return content + 0.1 * freq

Training settings from the paper: initial learning rate 1e-4, halved on a fixed epoch schedule, batches of four images, 256 × 256 random crops, and horizontal flips 2108.05054 §4.1. The paper does not state the optimizer or weight decay.

Contradictions

• MIMO-UNet+ runtime. The text gives 0.014 s 2108.05054 §4.2. The table gives 0.017 s 2108.05054 tab. 1.
• Reported timing against later re-measurement. The paper's timings are 0.008 s for MIMO-UNet and 0.040 s for MIMO-UNet++ 2108.05054 tab. 1. The official repository's README reports re-measured GPU times on an RTX 3090 with synchronisation, which are much larger, for example 0.130 s for MIMO-UNet. Details are on the paper card. Compare timings only from the same hardware and method.
• DiffIR's margin over MIMO-UNet+. The text says DiffIR beats MIMO-UNet+ on GoPro by 0.54 dB 2303.09472 §5.4. The same paper's table gives 33.20 dB against 32.45 dB, a margin of 0.75 dB 2303.09472 tab. 3. Use the table values. The DiffIR card lists this and other text-table mismatches.

Papers

• Rethinking Coarse-to-Fine Approach in Single Image Deblurring: single U-Net with multi-scale inputs and outputs, asymmetric feature fusion and a frequency loss.
• DiffIR: Efficient Diffusion Model for Image Restoration: diffusion over a 256-value restoration prior in 4 steps, with two-stage training; also covers inpainting and super-resolution.