Rethinking Coarse-to-Fine Approach in Single Image Deblurring

In plain words

A blurry photo is the sharp picture smeared by camera or subject motion. Deblurring tries to recover the sharp picture. The usual approach stacks several networks, one after another: the first works on a tiny copy of the image and produces a rough result, and each later network refines that result at a larger size. It works, but every stacked network costs time and memory, which makes it hard to run on phones, cars or robots.

This paper keeps the coarse-to-fine idea but puts it inside one network. A single U-shaped network reads the blurry image at three sizes and, on the way out, produces a sharp estimate at each of those sizes. The estimates at small sizes are cheap guides for the large one. Extra links move information between the sizes, so the detail found at one scale can help at the others.

The result is a deblurring network that gets close to the best slower methods on the standard GoPro test set while running much faster, by the authors' own timing.

TL;DR

MIMO-UNet is a single encoder-decoder network for single-image deblurring §3. It reads the blurry image at full, half and quarter size, its decoder emits three sharp estimates, and asymmetric feature fusion (AFF) connects all encoder levels to all decoder levels §3.3. It is trained on a multi-scale L1 loss plus a frequency-domain L1 loss with weight 0.1 §3.4. Author-reported: 31.73 dB PSNR on GoPro at 0.008 s for the smaller model, and 32.68 dB for the larger ensemble variant tab. 1.

Problem

Coarse-to-fine deblurring networks stack several sub-networks, each working on a different image scale. These stacks reach high accuracy but have high computational cost and memory use, which makes them hard to use where time and compute are scarce §1. A lighter single-network alternative was reported to lose too much accuracy §1. The paper asks whether one encoder-decoder can deliver the coarse-to-fine benefit without the stack §1.

Method

The model is a U-Net with three encoder blocks (EB) and three decoder blocks (DB), built on the standard U-Net design §3. Three changes distinguish it from a plain U-Net: multi-input single encoder (MISE), multi-output single decoder (MOSD) and asymmetric feature fusion (AFF) §1, §3.

Multi-input single encoder (MISE) §3.1. The blurry image is also fed at half and quarter size. A shallow convolutional module (SCM) turns each smaller input into features fig. 4(a). Each SCM applies two stacks of 3×3 and 1×1 convolutions, joins the result with its own input, and refines the join with a 1×1 convolution §3.1. SCMs are used for the second and third levels §3.1. At each level, a stride-2 convolution downsamples the previous encoder output, which is then fused with the SCM output by a feature attention module (FAM) §3.1. FAM multiplies the two features element-wise, passes the product through a 3×3 convolution, and adds the result back to the previous-level feature §3.1, fig. 4(b). Eight residual blocks follow §3.1.

Multi-output single decoder (MOSD) §3.2. Each decoder block works at a different scale. Each produces a feature map, which a single convolution layer maps to an image. The blurry image at the same scale is added to that image eq. 5. The decoder thus produces three estimates, one per scale, and the paper treats these as the intermediate supervision that stacked sub-networks would otherwise provide §3.2.

Asymmetric feature fusion (AFF) §3.3, eq. 6, fig. 4(c). Each decoder level receives a fusion of the encoder outputs of all three levels. Features are resampled to the target size first eq. 6. The fused result goes to its decoder block §3.3. The paper describes the AFF structure only in the figure fig. 4(c).

Loss §3.4, eqs. 7-9. The content loss is the L1 distance between each estimate and its target at that scale, each averaged over its element count t_k and summed over the K = 3 levels eq. 7. The paper reports that L1 beat MSE here §3.4. The multi-scale frequency reconstruction (MSFR) loss applies the same L1 distance to the Fourier transforms of each estimate and target eq. 8. The total loss is the content loss plus λ times MSFR, with λ = 0.1 eq. 9.

Pseudocode (own wording). Pieces the paper does not name are marked in comments.

# MIMO-UNet forward pass  [§3, fig. 3]
# Input:  blurry image, shape (B, 3, H, W)
# Output: estimates at quarter, half and full scale

def mimo_unet(blurry):
    # Input pyramid at 1/2 and 1/4 scale  [§3.1]
    blurry_half = resize(blurry, 0.5)
    blurry_quarter = resize(blurry_half, 0.5)

    # Shallow features from the two smaller inputs  [§3.1, fig. 4(a)]
    half_feats = scm_2(blurry_half)
    quarter_feats = scm_3(blurry_quarter)

    # Level 1: stem, then encoder block  [§3]
    enc_1 = eb_1(stem(blurry))

    # Level 2: strided conv, fuse with the half-scale SCM  [§3.1]
    enc_in_2 = fam_2(strided_conv(enc_1), half_feats)
    enc_2 = eb_2(enc_in_2)

    # Level 3: same pattern at quarter scale  [§3.1]
    enc_in_3 = fam_3(strided_conv(enc_2), quarter_feats)
    enc_3 = eb_3(enc_in_3)

    # Asymmetric feature fusion, one output per decoder level  [§3.3, eq. 6]
    skip_1 = aff_1(enc_1, resize(enc_2, 2), resize(enc_3, 4))
    skip_2 = aff_2(resize(enc_1, 0.5), enc_2, resize(enc_3, 2))

    # Coarsest decoder block and its quarter-scale estimate  [§3.2, eq. 5]
    dec_3 = db_3(enc_3)
    out_quarter = head_3(dec_3) + blurry_quarter

    # Upsample and merge with the level-2 fusion output  [§3.2]
    merged_2 = merge_2(up_3(dec_3), skip_2)
    dec_2 = db_2(merged_2)
    out_half = head_2(dec_2) + blurry_half

    # Finest decoder block and its full-scale estimate  [§3.2, eq. 5]
    merged_1 = merge_1(up_2(dec_2), skip_1)
    dec_1 = db_1(merged_1)
    out_full = head_1(dec_1) + blurry
    return [out_quarter, out_half, out_full]

# Training loss  [§3.4, eqs. 7-9]
# Inputs: estimates (3 scales) and sharp targets at the same scales

def total_loss(estimates, sharp):
    loss_content = 0.0
    loss_freq = 0.0
    for estimate, target in zip(estimates, sharp):
        # L1 averaged over all elements  [eq. 7]
        loss_content += mean(abs(estimate - target))
        # L1 on real and imaginary parts of the FFT difference  [eq. 8]
        diff = fft2(estimate) - fft2(target)
        loss_freq += mean(stack([real(diff), imag(diff)]).abs())
    # Weighted sum with lambda = 0.1  [eq. 9]
    return loss_content + 0.1 * loss_freq

# Training loop  [§4.1]
# Optimizer and weight decay are not stated in the paper

def train(model, pairs, num_epochs):
    # Initial learning rate 1e-4  [§4.1]
    optimizer = adam(model.parameters(), lr=1e-4)
    # Learning rate halves every 500 epochs (GoPro) or 200 (RealBlur)  [§4.1]
    scheduler = step_decay(optimizer, factor=0.5)
    for epoch in range(num_epochs):
        for blurry, sharp in batches(pairs, batch_size=4, shuffle=True):
            # Random 256x256 crop, same window for both images  [§4.1]
            blurry, sharp = random_crop(blurry, sharp, size=256)
            # Horizontal flip with probability 0.5, applied to both  [§4.1]
            blurry, sharp = random_flip(blurry, sharp, p=0.5)
            optimizer.zero_grad()
            estimates = model(blurry)
            loss = total_loss(estimates, sharp)
            loss.backward()
            optimizer.step()
        scheduler.step()

Hyperparameters

Values stated in the paper are cited to the paper. Code-only values are marked. Repository paths are relative to the repository root at code_commit.

Results

Author-reported. The paper evaluates on the GoPro test set (1,111 pairs), training only on GoPro train §4.1. Table 1 is copied in full, with the runtime columns as the paper gives them: one value measured on the authors' PC and one reported in each original paper, in seconds tab. 1. MIMO-UNet's single value spans both columns.

Source: tab. 1. The paper says MIMO-UNet+ runs at 0.014 s in its text §4.2, which disagrees with the 0.017 s in the table (see Pitfalls).

Setup of the variants §4.2. MIMO-UNet uses 8 residual blocks per EB and DB; MIMO-UNet+ uses 20; MIMO-UNet++ is MIMO-UNet+ with a geometric self-ensemble §4.2. The paper cites Lim et al. for the ensemble but does not state which transforms it uses §4.2.

RealBlur tab. 2. Author-reported PSNR and SSIM on RealBlur §4.2:

Feature fusion in MISE tab. 3. Fusion method on GoPro PSNR: concatenation 31.66 dB, element-wise sum 31.60 dB, FAM 31.73 dB §3.1, tab. 3.

Ablation of components tab. 4, §4.3. GoPro PSNR and parameters (M), starting from a baseline with none of the four components:

Source: tab. 4. The text gives the gains: MOSD +0.17 dB, AFF +0.17 dB, MISE + AFF +0.23 dB, all three +0.30 dB, and MSFR a further +0.27 dB, for +0.57 dB in total §4.3.

Object detection §4.4, fig. 7, fig. 8. The paper runs a pretrained detector (PFPNet, trained on PASCAL VOC) on blurry and deblurred GoPro images. Figure 7 shows a qualitative example fig. 7. Average precision per class is shown only in a figure, so the numbers are not stated in the text fig. 8.

Paper vs code

The official repository is chosj95/MIMO-UNet. The paper's abstract links it [abstract], and its README lists the first author's GitHub profile [README.md]. It was cloned at code_commit for this card. The repository has no LICENSE file, so code_license is recorded as "none". No code is quoted verbatim; every point below is our own description with citations.

• Optimizer. Paper: not stated. Code uses Adam, learning rate 1e-4, weight decay 0 train.py.
• GoPro schedule. Matches the paper: halve every 500 epochs, 3000 epochs main.py.
• RealBlur schedule. The paper gives 1000 epochs with halving every 200 §4.1. The repository has no RealBlur training configuration, and its defaults are the GoPro settings main.py. The README says the RealBlur test code "will be released soon" [README.md, Model section]. The RealBlur numbers cannot be reproduced from the code alone.
• Ground-truth downsampling. Paper: not stated. Code uses bilinear resizing for the training targets train.py.
• Input and feature resampling. Paper: not stated. Code uses F.interpolate with its default mode, which is nearest-neighbour, for the input pyramid, the AFF inputs and upsampling MIMOUNet.py. This differs from the bilinear targets.
• Decoder structure. Equation 5 shows a decoder block fed by an AFF output and the previous decoder output. The code also applies a 4×4 transposed convolution with ReLU for upsampling, concatenates it with the AFF output, and reduces it with a 1×1 convolution plus ReLU [code: models/MIMOUNet.py#L88-L89, #L150, #L153-L154, #L160-L161]. None of this appears in the paper's equations.
• Output head. Paper: "a single convolutional layer" §3.2. Code uses 3×3 convolutions MIMOUNet.py.
• Encoder downsampling. Paper: a stride-2 convolution §3.1. Code applies ReLU after the stride-2 convolution MIMOUNet.py.
• AFF. Paper: structure given only in fig. 4(c). Code: a 1×1 convolution with ReLU, then a 3×3 convolution with no activation MIMOUNet.py.
• Dropout. Not in the paper. MIMO-UNet+ in the code applies Dropout2d(0.1) to both AFF outputs. This is active only in training mode, since validation calls eval() MIMOUNet.py.
• Frequency loss. Paper: L1 on the Fourier transform, normalised by t_k eq. 8. Code uses torch.rfft with onesided=False, which gives a trailing dimension of 2 for real and imaginary parts, and averages L1 over both parts [code: train.py#L59-L69, #L71]. The effective normalisation therefore differs from the element count t_k by the factor of 2 for the real and imaginary parts. The README pins PyTorch 1.4, and torch.rfft was later removed from PyTorch.
• Content loss. Matches: L1 averaged over elements, summed over three levels train.py.
• Geometric self-ensemble (MIMO-UNet++). Not in the code. The README says the ensemble ran as two inference passes at batch size 2 [README.md, Performance section]. The transforms used are not stated.
• Evaluation. Test and validation use the full-scale output and clamp it to [0, 1] before computing PSNR eval.py. PSNR is computed per image and averaged eval.py. The code computes no SSIM.
• Timing. The timing code calls time.time() without CUDA synchronisation eval.py. The README reports re-measured synchronised times on an RTX 3090: MIMO-UNet 0.012 s asynchronous, 0.130 s synchronised; MIMO-UNet+ 0.025 s and 0.282 s; MIMO-UNet++ 0.049 s and 1.115 s [README.md, GPU synchronisation section]. These differ from the paper's table.
• Learning-rate milestones. The code defines milestones 500 to 3000 [code: main.py#L53]. The 3000 milestone is at the last epoch and has no effect.

Pitfalls

• Missing decoder detail. Equation 5 does not show the upsampling layers, the concatenation with the AFF output, or the 1×1 merge convolutions. An implementation built from the paper's equations alone would lack them. Use the code structure MIMOUNet.py.
• Output index order. The estimates are returned coarsest first: index 0 is the quarter scale, index 1 the half scale and index 2 the full scale [code: models/MIMOUNet.py#L151, #L158, #L164]. The training loss and the evaluation both depend on this order train.py.
• Resampling method. The paper does not say how it downsamples the input or the targets. The code uses bilinear for targets and nearest-neighbour for the input pyramid and fusion features (see Paper vs code). Changing either alters the result.
• Frequency-loss normalisation. The FFT term in the code averages over real and imaginary parts, so its magnitude differs from the paper's t_k normalisation by that factor. The paper sets λ = 0.1 "experimentally" §3.4 and does not say which normalisation that value was chosen with, so the effective weight is uncertain.
• Outdated framework calls. The training loss uses torch.rfft, which the README's pinned PyTorch 1.4 provides. Porting to a current PyTorch needs the torch.fft API.
• RealBlur not reproducible from the code. The paper's RealBlur schedule and any RealBlur test script are missing from the repository (see Paper vs code).
• Ensemble not specified. MIMO-UNet++'s self-ensemble is cited to another paper, and the transforms are not given. The code does not include it.
• Runtime figures disagree. The text gives MIMO-UNet+ at 0.014 s §4.2; the table gives 0.017 s tab. 1. The 4.05× speed-up over MPRNet uses the measured MPRNet runtime of 0.162 s in the table divided by MIMO-UNet++'s 0.040 s §4.2, tab. 1. The timing method is not synchronised (see Paper vs code).
• Text typo. The paper writes "SVDM" where Table 1 uses SVDN §4.2, tab. 1.
• Dropout only in MIMO-UNet+. The dropout exists only in the MIMO-UNet+ class in the code MIMOUNet.py. The paper does not mention dropout, so whether the 8-block model should have it is not stated.
• Epoch versus iteration. The paper describes sampling four random images per iteration §4.1. The code shuffles the full training set into batches of 4 each epoch data_load.py. For 2,103 GoPro pairs that is 526 iterations per epoch, the last with 3 images.
• Augmentation is limited. Only random crop and horizontal flip are used, so vertical flips and rotations are not in the training setup data_load.py.
• Parameter counts not checked. The paper reports 6.8M parameters for MIMO-UNet and 16.1M for MIMO-UNet+ tab. 1. The card does not recount them; treat them as reported values. The default code configuration has 8 residual blocks per encoder and decoder block MIMOUNet.py.

Lineage

• Builds on: DeepDeblur (Nah et al., multi-scale stacked sub-networks and the GoPro dataset) §2.1; PSS-NSC (shared-parameter U-Net sub-networks) §2.2; MT-RNN (iterated single U-shaped network) §2.3; the U-Net §3; eight modified residual blocks following SRN §3.1; dense intra-scale connections from Kim et al. §3.3. None of these has a card in this wiki, so no link is given.
• Compared against: DeepDeblur, SRN, PSS-NSC, DMPHN, SAPHN, MT-RNN, RADN, SVDN, MPRNet tab. 1; DeblurGAN-v2 and MPRNet on RealBlur tab. 2.
• Superseded by: not stated in this wiki.