---
kind: concept
slug: image-deblurring
title: Image deblurring
summary: >-
  Recover a sharp image from one blurry photo. Coarse-to-fine stacks are accurate but costly. MIMO-UNet puts the multi-scale idea in one U-Net, with a small gap to top GoPro results. DiffIR diffuses a 256-value prior in 4 steps and tops its GoPro table, after two-stage training.
tags: [computer-vision, image-restoration, deblurring, multi-scale, u-net, diffusion]
papers: ["2108.05054", "2303.09472"]
related: [image-denoising, image-enhancement, image-super-resolution, multi-scale-feature-fusion, diffusion-models]
status: draft
updated_at: 2026-10-24
---

## Overview

A blurry photo is a sharp image smeared by camera or subject motion. Single-image
deblurring predicts the sharp image from the blurry one alone. The strongest
earlier methods stack sub-networks that each work at a different image scale,
from coarse to fine. Each stage refines the previous estimate. This is accurate
but costs time and memory, because every stage adds compute [arxiv:2108.05054 §1].

The variants share one interface: a blurry image goes in and a sharp image comes
out, trained against paired sharp images such as the GoPro set [arxiv:2108.05054
§4.1] [arxiv:2303.09472 §5.4]. They differ in where the compute goes. MIMO-UNet
keeps the coarse-to-fine supervision but runs it inside one network, with one
decoder output per scale [arxiv:2108.05054 §3]. DiffIR runs a diffusion model,
but on a compact learned prior rather than on pixels [arxiv:2303.09472 §4]. Two
papers are covered so far.

## Variants

### Single U-Net with multi-scale inputs and outputs (MIMO-UNet)

A U-Net with three encoder blocks and three decoder blocks [arxiv:2108.05054 §3].
Three changes distinguish it from a plain U-Net [arxiv:2108.05054 §1, §3]:

- **Multi-input encoder.** The blurry image is also fed at half and quarter
  size. Each smaller input gets a shallow feature module, and a fusion module
  merges those features into the encoder at its level [arxiv:2108.05054 §3.1].
- **Multi-output decoder.** Each decoder block produces a sharp estimate at its
  own scale. The blurry input at that scale is added to the estimate, so the
  network predicts a residual [arxiv:2108.05054 §3.2, eq. 5]. The three estimates
  act as intermediate supervision, which stacked sub-networks would otherwise
  provide [arxiv:2108.05054 §3.2].
- **Asymmetric feature fusion.** Each decoder level receives the encoder features
  of all three levels, resampled to its size [arxiv:2108.05054 §3.3, eq. 6]. The
  paper gives this module only as a figure [arxiv:2108.05054 fig. 4(c)].

Training minimises an L1 loss on each estimate, summed over the three scales,
plus a frequency-domain L1 loss weighted by 0.1 [arxiv:2108.05054 §3.4, eqs. 7-9].

The paper evaluates three models of this design [arxiv:2108.05054 §4.2]:

- **MIMO-UNet:** 8 residual blocks per encoder and decoder block.
- **MIMO-UNet+:** 20 residual blocks per block.
- **MIMO-UNet++:** MIMO-UNet+ with a geometric self-ensemble at inference. The
  transforms the ensemble uses are not stated [arxiv:2108.05054 §4.2].

Author-reported on GoPro, the paper's main benchmark [arxiv:2108.05054 §4.1]:
MIMO-UNet reaches 31.73 dB PSNR at 0.008 s and 6.8M parameters; MIMO-UNet++
reaches 32.68 dB at 0.040 s [arxiv:2108.05054 tab. 1]. Card:
[Rethinking Coarse-to-Fine Approach in Single Image Deblurring](/papers/2108.05054).

Ablation on GoPro, starting from a baseline without any of the four components:
the full model adds 0.57 dB, of which the frequency loss contributes 0.27 dB
[arxiv:2108.05054 §4.3, tab. 4].

### Diffusion over a compact restoration prior (DiffIR)

DiffIR is a general image-restoration method. Its deblurring result is one of
several tasks in the paper. It does not diffuse pixels. A stage-1 encoder reads
the ground-truth and the blurry image and produces an image-restoration prior
(IPR) of 4C' = 256 values, with C' = 64 [arxiv:2303.09472 §4.1, §5.1]. A
transformer restorer (DIRformer) uses that prior to modulate its feed-forward and
attention blocks, and is trained with an L1 reconstruction loss
[arxiv:2303.09472 §4.1, eqs. 6-9].

In stage 2 a diffusion model learns to predict the IPR from the blurry image
alone, with T = 4 diffusion steps [arxiv:2303.09472 §4.2, §5.1]. The stage-2
objective trains the condition encoder, the denoiser and the restorer jointly.
It adds an L1 term that matches the predicted IPR to the stage-1 IPR
[arxiv:2303.09472 §4.2, eq. 13]. At inference, the model draws the IPR from
noise, runs the reverse steps, and passes the result to the restorer. No sharp
image is needed [arxiv:2303.09472 §4.2, alg. 2].

On GoPro, trained on GoPro only, the paper reports 33.20 dB PSNR and 0.963 SSIM.
On HIDE it reports 31.55 dB and 0.947 [arxiv:2303.09472 tab. 3]. The next
entries in that table are Restormer at 32.92 dB and MPRNet at 32.66 dB on GoPro
[arxiv:2303.09472 tab. 3]. The paper says its deblurring Mult-Adds are 78% of
Restormer's [arxiv:2303.09472 §5.4]. The paper gives no deblurring runtime.

For deblurring, training uses L1 only, Adam with β = (0.9, 0.99), and a learning
rate of 2e-4 with cosine decay to 1e-6 over 300K iterations. Patch size grows in
stages [arxiv:2303.09472 app. F]. The released deblurring code adds an
encoder-only warm-up that the paper does not mention; see the
[paper card](/papers/2303.09472) before reproducing the schedule. Card:
[DiffIR: Efficient Diffusion Model for Image Restoration](/papers/2303.09472).

## When to use which

| Situation | Use | Why |
|---|---|---|
| One network, tight latency or memory budget | MIMO-UNet | 6.8M parameters and 0.008 s in the authors' timing, 31.73 dB on GoPro [arxiv:2108.05054 tab. 1] |
| Highest GoPro PSNR, extra inference time is acceptable | MIMO-UNet++ | 32.68 dB against 32.66 dB for MPRNet, the table's best other entry, at 0.040 s against 0.162 s [arxiv:2108.05054 tab. 1]. The margin is 0.02 dB. The ensemble transforms are not stated |
| Best GoPro and HIDE PSNR in DiffIR's table, diffusion sampling is acceptable | DiffIR | 33.20 dB on GoPro and 31.55 dB on HIDE, against 32.92 dB and 31.22 dB for Restormer [arxiv:2303.09472 tab. 3]. The author-reported compute is 78% of Restormer's [arxiv:2303.09472 §5.4] |
| Single network trained on one objective, no pretrained prior network | MIMO-UNet | One combined loss over three scales and the frequency term [arxiv:2108.05054 §3.4]. DiffIR needs a stage-1 pretraining and then a joint stage-2 run [arxiv:2303.09472 §4.1, §4.2] |
| Real blurry photos (RealBlur) | MIMO-UNet+ or MIMO-UNet++ | 31.92 dB and 32.05 dB on RealBlur, against 31.76 dB for MPRNet [arxiv:2108.05054 tab. 2]. These are author-reported; see Contradictions. DiffIR reports no RealBlur result |
| Training compute is limited | Plan for long schedules | MIMO-UNet: 3000 epochs on GoPro [arxiv:2108.05054 §4.1]. DiffIR deblurring: 300K iterations at batch 64 [arxiv:2303.09472 app. F] |

The paper's timings were measured on an Intel i5-8400 CPU with an NVIDIA Titan
XP GPU [arxiv:2108.05054 §4.1]. Other hardware is not covered by the paper.
DiffIR reports no deblurring latency, so the two papers cannot be compared on
speed. Only the MIMO-UNet timings are available.

## Implementation sketch

Language-agnostic outline of the MIMO-UNet forward pass. Module names are ours.
This sketch covers the MIMO-UNet variant only. DiffIR's two-stage procedure is
on its [paper card](/papers/2303.09472). The paper's equations omit some decoder
layers that the released code contains; see the
[MIMO-UNet card](/papers/2108.05054) before implementing.

```pseudocode
# Input pyramid: blurry image at full, half and quarter size [arxiv:2108.05054 §3.1]
def mimo_unet(blurry):
    half = resize(blurry, 0.5)
    quarter = resize(half, 0.5)

    # Encoder: fuse each smaller input into its level [arxiv:2108.05054 §3.1]
    enc1 = encoder_block_1(stem(blurry))
    enc2 = encoder_block_2(fuse(strided_conv(enc1), shallow(half)))
    enc3 = encoder_block_3(fuse(strided_conv(enc2), shallow(quarter)))

    # Asymmetric fusion of all encoder levels for each decoder level [§3.3, eq. 6]
    skip1 = aff_1(enc1, up(enc2, 2), up(enc3, 4))
    skip2 = aff_2(down(enc1, 2), enc2, up(enc3, 2))

    # One estimate per scale, residual on the blurry input [§3.2, eq. 5]
    dec3 = decoder_block_3(enc3)
    out_quarter = head_3(dec3) + quarter
    dec2 = decoder_block_2(merge(up(dec3, 2), skip2))
    out_half = head_2(dec2) + half
    dec1 = decoder_block_1(merge(up(dec2, 2), skip1))
    out_full = head_1(dec1) + blurry
    return [out_quarter, out_half, out_full]  # coarsest first
```

```pseudocode
# Training loss: L1 per scale plus 0.1 times an L1 on Fourier coefficients
# [arxiv:2108.05054 §3.4, eqs. 7-9]
def loss(estimates, sharp):
    content = sum(l1(e, t) for e, t in zip(estimates, sharp))
    freq = sum(l1(fft2(e), fft2(t)) for e, t in zip(estimates, sharp))
    return content + 0.1 * freq
```

Training settings from the paper: initial learning rate 1e-4, halved on a fixed
epoch schedule, batches of four images, 256 × 256 random crops, and horizontal
flips [arxiv:2108.05054 §4.1]. The paper does not state the optimizer or weight
decay.

## Contradictions

- **MIMO-UNet+ runtime.** The text gives 0.014 s [arxiv:2108.05054 §4.2]. The
  table gives 0.017 s [arxiv:2108.05054 tab. 1].
- **Reported timing against later re-measurement.** The paper's timings are
  0.008 s for MIMO-UNet and 0.040 s for MIMO-UNet++ [arxiv:2108.05054 tab. 1].
  The official repository's README reports re-measured GPU times on an RTX 3090
  with synchronisation, which are much larger, for example 0.130 s for
  MIMO-UNet. Details are on the [paper card](/papers/2108.05054). Compare
  timings only from the same hardware and method.
- **DiffIR's margin over MIMO-UNet+.** The text says DiffIR beats MIMO-UNet+ on
  GoPro by 0.54 dB [arxiv:2303.09472 §5.4]. The same paper's table gives 33.20 dB
  against 32.45 dB, a margin of 0.75 dB [arxiv:2303.09472 tab. 3]. Use the table
  values. The DiffIR card lists this and other text-table mismatches.

## Papers

- [Rethinking Coarse-to-Fine Approach in Single Image Deblurring](/papers/2108.05054): single U-Net with multi-scale inputs and outputs, asymmetric feature fusion and a frequency loss.
- [DiffIR: Efficient Diffusion Model for Image Restoration](/papers/2303.09472): diffusion over a 256-value restoration prior in 4 steps, with two-stage training; also covers inpainting and super-resolution.