Event-Based Fusion for Motion Deblurring with Cross-modal Attention

In plain words

A photo taken while the camera or the subject moves comes out smeared. A normal camera records one frame at a time, so the smear is baked in. An event camera works differently: each pixel reports only when its brightness changes, and it does so with very fine timing. Those reports carry the motion that happened during the photo.

This paper builds a neural network that takes both inputs, the smeared frame and the stream of changes, and returns a sharp frame. The network first turns the events into a compact grid that keeps their timing. It then mixes the frame's features with the event features at several scales, letting the frame look at the events that matter for the blur and ignore noise. A second refinement pass cleans up the result, and a gate passes information from the first pass to the second only where it helps.

The authors also recorded a new dataset with a real event camera and real blur, because most earlier work trained on simulated events. The network beats the earlier image-only and event-based methods the authors could compare against, on both the simulated and the real data.

TL;DR

• Single-frame motion deblurring that uses events from an event camera as a second input §1.
• Core ideas: a symmetric cumulative event voxel grid (SCER) §3.3, cross-modal channel attention between image and event features (EICA) §3.4, and an event-mask gated connection between two stages (EMGC) §3.5.
• Author-reported: 35.46 dB PSNR and 0.972 SSIM on GoPro with simulated events tab. 1; 38.12 dB and 0.975 on REBlur with real events tab. 2.
• Code is public (Apache-2.0) at a first-author repository. The released configs differ from the paper in optimizer, schedule and gradient clipping (see Paper vs code).

Problem

Image-only deblurring methods fail on extreme or real blur, because a single blurry frame does not record the motion that caused it §1, §2. Earlier event-based methods added events, but simple concatenation or weak fusion gave only small gains over image-only networks §1, §2. The authors also note that simulated events for training differ from real events, and that no large real-world event-blur dataset with ground truth existed §4.

Method

Event model. An event is a tuple of pixel position, time and polarity. The paper uses the exposure-time model in which the blurry frame is the average of the latent sharp frames over exposure time, and the sharp frames are linked to events through the contrast threshold §3.1. Real events are noisy and the threshold varies across the sensor, so the paper does not invert this model directly. It learns the mapping instead, with a two-stage design: an initial estimate from the blurry image and the events, then a denoising mapping on that estimate §3.2.

Overall architecture. EFNet has two U-Net-like stages, each with two down-sampling and two up-sampling layers and a residual convolution block at each level §3.2 fig. 1. An event branch encodes the event voxel grid in parallel with the image branch. Features from the two branches are fused at several encoder levels by EICA. A supervised attention module (SAM, from MPRNet) passes the first-stage features and an image estimate to the second stage §3.2 fig. 1. Between the two stages, EMGC forwards encoder and decoder features from the first stage, gated by an event mask §3.5.

Step 1: event representation (SCER) §3.3 fig. 2. The exposure time is split into 2N equal intervals, with N = 3 in the paper's implementation. For each interval edge, the signed polarity sum is accumulated between the edge and the centre of the exposure. Edges before the centre give negative channels and edges after it give positive ones. The centre slice, which is all zeros, is dropped, leaving 2N channels. Channels near the outer edges describe the blurred area; channels near the centre describe sharp edges §3.3. The paper also reports that this representation beats a fixed-interval voxel grid (SBT) and plain stacking of events in its ablation tab. 4.

# Symmetric cumulative event representation  [§3.3]
# [code: scripts/data_preparation/base_dataset.py#L675-L714]
# Input: events in one exposure window (x, y, t, p), num_bins = 2N = 6
# Output: voxel grid (2N, H, W)

def scer(events, num_bins=6):
    # Centre of the exposure window  [§3.3]
    middle = (window_start + window_end) / 2
    left = []
    # Edges before the centre: accumulate from the edge to the centre,
    # then negate  [§3.3]
    for i in range(num_bins // 2):
        edge = window_start + (window_end - window_start) / num_bins * i
        left.append(-accumulate(events, from_time=edge, to_time=middle))
    # Edges after the centre: accumulate from the centre to the edge  [§3.3]
    right = []
    for i in range(num_bins // 2, num_bins):
        edge = window_start + (window_end - window_start) / num_bins * (i + 1)
        right.append(accumulate(events, from_time=middle, to_time=edge))
    # The centre channel is all zero and is dropped (keep_middle = False)
    return stack(left + right)  # (2N, H, W)

The SCER equation the pseudocode implements, in the paper's notation:

\mathbf{SCER}_i = \mathrm{sgn}(i-N) \sum_{j:\, m \le t_j \le M} p_j\,\delta_{x_j y_j} §3.3

Here m and M are the interval edges around the centre f of the exposure, and \delta is the Kronecker delta over pixel positions §3.3.

Step 2: event-image cross-modal attention (EICA) §3.4 fig. 3. At each fused level, the image features supply the queries and the event features supply the keys and values. Both branches first pass through normalisation and 1×1 convolutions. Attention is computed over channels rather than pixels, so the attention map is c × c and not hw × hw. That keeps the cost linear in the number of pixels §3.4. The attention output is added to the image features, and a two-layer GELU MLP follows §3.4.

\mathrm{Attention}(Q_{\text{image}}, K_{\text{event}}, V_{\text{event}}) = V_{\text{event}}\,\mathrm{softmax}\!\left(\frac{Q_{\text{image}}^{T} K_{\text{event}}}{\sqrt{d_k}}\right) §3.4

# Event-image cross-modal attention  [§3.4]
# [code: basicsr/models/archs/arch_util.py#L282-L349]
# Inputs: image_feats (B, C, H, W), event_feats (B, C, H, W)
# Output: fused (B, C, H, W)

def eica(image_feats, event_feats, num_heads):
    # Normalise each branch, then project to queries, keys and values  [§3.4]
    queries = conv_q(norm_image(image_feats))
    keys = conv_k(norm_event(event_feats))
    values = conv_v(norm_event(event_feats))
    # Split channels into heads; each head works on C / heads channels
    # [code: arch_util.py#L305-L307]
    queries = split_heads(queries, num_heads)  # (B, heads, c, H*W)
    keys = split_heads(keys, num_heads)
    values = split_heads(values, num_heads)
    # Unit-normalise queries and keys over pixels, then scale by a learned
    # per-head temperature  [code: arch_util.py#L309-L312]
    queries = l2_normalize(queries, dim=-1)
    keys = l2_normalize(keys, dim=-1)
    scores = (queries @ transpose(keys)) * temperature
    # Channel-by-channel attention map, c x c per head  [§3.4]
    attention = softmax(scores, dim=-1)  # (B, heads, c, c)
    out = project_out(merge_heads(attention @ values))
    # Add to the image features, then apply a GELU MLP with its own residual
    # [§3.4] [code: arch_util.py#L334-L349]
    fused = image_feats + out
    fused = fused + mlp(norm_mlp(fused))
    return fused

Step 3: event mask gated connection (EMGC) §3.5 fig. 1(b). The paper's mask marks pixels with no events as 1 and pixels with events as 0. Encoder features are gated so that regions with events are masked out, and decoder features are gated so that regions without events are masked out. A direct skip path is added beside the gate, and the result goes into the second stage §3.5. The gate is applied with a 3×3 convolution on each branch.

# Event mask gated connection  [§3.5] [fig. 1(b)]
# [code: basicsr/models/archs/EFNet_arch.py#L175-L201]
# Note: the code stores mask = 1 at event pixels (see Paper vs code)

def emgc(block_out, encoder_feat, decoder_feat, mask):
    # Encoder: keep features where there are no events  [§3.5]
    enc_gated = (1 - mask) * encoder_feat
    # Decoder: keep features where there are events  [§3.5]
    dec_gated = mask * decoder_feat
    # Gated paths are added to the stage-two features, beside the plain
    # skip paths  [§3.5] [code: EFNet_arch.py#L196-L200]
    out = block_out + conv_enc(encoder_feat) + conv_enc_mask(enc_gated)
    out = out + conv_dec(decoder_feat) + conv_dec_mask(dec_gated)
    return out

Training objective. The paper's body does not state the loss. The released code applies a PSNR-type loss to the output of each stage and sums them (see Paper vs code).

Data and training setup §5.1. Training crops are 256×256 from full GoPro frames. The paper lists random flips, random noise and hot pixels in the event voxels as augmentation, and trains from scratch. Fine-tuning on REBlur uses 600 iterations with a smaller learning rate. Full network widths are said to be in a supplement, which is not in the arXiv source.

Hyperparameters

Results

Author-reported. GoPro setup: 2,103 training and 1,111 test pairs, with events simulated by ESIM §5.1 tab. 1. REBlur setup: 486 training and 983 test pairs with real events, plus 4 extra sequences without ground truth §4. Baselines are image-only networks and event-based networks with public implementations, plus event-enhanced versions of SRN and HINet that receive the SCER voxel grid as extra input tab. 1 tab. 2.

GoPro, simulated events tab. 1

The paper reports a 2.75 dB PSNR gain over HINet and 2.47 dB over the best event-based method in the table, with a model of 8.47M parameters §5.2.

REBlur, real events tab. 2

Ablation on GoPro tab. 3 tab. 4. Relative to the image-only one-stage baseline (29.06 dB, 0.936), the full model is reported as 6.4 dB PSNR and 3.6% SSIM higher §5.3.

Event representation on GoPro tab. 4: no events 32.15 / 0.954; plain stacking 31.90 / 0.950; SBT with 6 bins 35.12 / 0.970; SCER 35.46 / 0.972. The paper says plain stacking is worse than no events §5.3.

Paper vs code

Repository: AHupuJR/EFNet, first author's repository, commit ebd8cb90498e3490bd1fc8fedc924ce16a84901a, Apache-2.0 LICENSE. The README links the paper and the pretrained weights README.md. The repository is public; the training configs point to cluster paths.

• Optimizer. Paper: Adam §5.1. Code: AdamW with weight decay 1e-4 and betas (0.9, 0.99) EFNet.yml.
• Training length and batch. Paper: 300k iterations, batch 8 §5.1. Code: total_iter 200000 EFNet.yml, batch 4 per GPU EFNet.yml. The documented launch uses four GPUs README.md, so the effective batch is 16 unless changed. Dataset enlargement of 4 applies on top EFNet.yml.
• Gradient clipping. Not in the paper. Code clips the global gradient norm to 0.01 by default image_event_restoration_model.py.
• Loss. Not in the paper body. Code: PSNR-type loss on each stage's output, weight 0.5, summed over stages image_event_restoration_model.py losses.py. A commented-out appendix in the LaTeX source states the same objective, but the appendix is not in the compiled paper and is not used as paper content here.
• Event mask. Paper: mask is 0 where an event occurred, from binarising the sum of the first and last SCER channels §3.5. Code: mask is 1 at event pixels, computed as any non-zero bin across all channels, then a 3×3 morphological close make_voxels_esim.py make_voxels_esim.py. The gate formulas in code, (1 − mask) on the encoder and mask on the decoder, give the same behaviour as the paper's stated convention. The mask definition differs.
• SCER channel count. Paper: 2N = 6 channels with the centre dropped §3.3. Code: the GoPro voxel script does not pass keep_middle, and the default is True base_dataset.py, which would add a zero centre channel. The model expects 6 input channels EFNet_arch.py. The released GoPro data is named "bin6", so the shipped files are presumably 6-channel; check the shape before training.
• EICA scaling. Paper: softmax over Q^T K divided by sqrt(d_k) §3.4. Code: L2-normalised queries and keys with a learned per-head temperature arch_util.py. The code also applies a 1×1 output projection arch_util.py that the paper formula does not show, and a residual GELU MLP with a 4× hidden width arch_util.py EFNet_arch.py.
• Attention size. Paper: attention map is c × c with c set by the 1×1 convolution §3.4. Code: c is the channel count per head, C / heads, with heads [1, 2, 4] EFNet_arch.py.
• Augmentation. Paper: flips, random noise and hot pixels applied as augmentation §5.1. Code: flips only in the loader h5_image_dataset.py. Noise (std 1.0, 5 % of entries) and hot pixels (range 20, fraction 2e-5) are added once when the GoPro voxels are built make_voxels_esim.py.
• Residual outputs. The stage-one estimate is formed by adding the input image inside the SAM module EFNet_arch.py, and the final output adds the input image to the stage-two result EFNet_arch.py. The paper body does not describe either residual.
• Event exposure. The GoPro script assumes a 1/240 s exposure make_voxels_esim.py. The paper does not state this value.
• Test time. GoPro test config has no flips, rotations or grids EFNet.yml. The paper does not mention ensembling, so no discrepancy, but the reported numbers are from the single-pass setting.
• Equation and figure numbers. Not checked against a compiled PDF; the card cites sections where numbering was uncertain.

Pitfalls

• Learning-rate and step settings are split. The paper and the released configs use different optimizers, step counts and batch sizes. Reproduce the paper's setup from the paper, or the configs, and say which.
• Gradient clipping at 0.01 is on by default in code and invisible in the paper. It changes training dynamics noticeably and should be a named option.
• Mask polarity. The paper's text puts 0 at event pixels; the code puts 1 there. Implementing the paper's binarisation literally, from the first and last SCER channels, gives a different mask from the code's union of all channels with a morphological close.
• SCER channel count and the centre channel. A default flag in the GoPro voxel script can produce 7 channels, while the model takes 6. Check the tensor shape before training.
• Simulated versus real events. The GoPro results use events from ESIM with a per-pixel threshold, and the simulator is not in this repository. Real-event results are on REBlur only. The paper reports the gap is real; treat simulated-event numbers as an upper bound for real cameras.
• Parameter count. The paper reports 8.47M for EFNet §5.2 and gives no channel widths in the body. The code default (width 64) was not used to recount parameters here.
• Costs not reported. The model has two stages, attention at several levels and a separate event encoder. The paper reports parameter count only, with no runtime or memory figures.
• Dataset licences. The repository README points to external dataset downloads. Their licences were not checked, and this card uses no dataset content.

Lineage

• Builds on: the event-based double-integral (EDI) blur model of Pan et al. §1, §3.1 (no card in this wiki yet); MPRNet's supervised attention module §3.2; the U-Net encoder–decoder §3.2; SBT as a baseline representation §3.3, tab. 4.
• Compared against: image-only SRN, HINet, MPRNet, DeblurGAN, DeblurGAN-v2, DMPHN, SPAIR, Suin et al. tab. 1; event-based BHA, D²Nets, LEMD, ERDNet and event-enhanced SRN+ and HINet+ tab. 1 tab. 2.
• Superseded by: not stated.