Event-camera video reconstruction

Overview

An event camera reports, per pixel, when its brightness changes, as a stream of timestamped events 2312.00113 §1. The task here is to recover colour frames inside a time window using only one colour frame from before that window and the events that follow it 2312.00113 §1, §4.2. The output can be requested at any time inside the window, not only at fixed frame times 2312.00113 §3.

The problem differs from frame interpolation. Interpolation needs a sharp frame on both sides of the target time. If the later frame is blurred or badly exposed, the result is blurred too, and the method must wait for that later frame, which adds latency 2312.00113 §1.

The paper's answer splits the work into a motion model, a colour model and a fusion step. The motion model is a set of learned time basis functions per pixel 2312.00113 §3.1. The colour model is a tri-plane representation built from the events and the initial frame 2312.00113 §3.2. Flow between the synthesised frame and the initial frame refines the result 2312.00113 §3.3, and a fusion network merges the warped pyramids into the output 2312.00113 §3.4.

Variants

The cards on this site contain one method for this task, so this section has one entry. Other methods are not covered until their papers have cards.

Single frame plus events, decoded at any time

The model runs once per window. It builds two event volumes, one for motion and one for synthesis 2312.00113 suppl. Input Representation. Each pixel's trajectory is a weighted sum of K = 5 learned basis functions of time 2312.00113 §3.1, eq. 2. The colour model is a tri-plane representation on three planes (x–y, x–t, y–t) at four scales. Samples from the three planes are multiplied elementwise and decoded to RGB 2312.00113 §3.2, eq. 3. The model runs once per window and then decodes any number of times 2312.00113 §3, fig. 3. Card: Event-based Continuous Color Video Decompression from Single Frames.

The paper's ablation shows that each branch alone trails the full model on BS-ERGB. Synthesis-only and motion-only reach 22.90 and 23.37 dB PSNR at gap 1, against 25.40 dB for the full model. At gap 3 the figures are 22.36 and 22.22 dB against 24.80 dB 2312.00113 tab. 4.

When to use which

The paper is the only source here, so the table below is limited to the constraints it tests.

Implementation sketch

Pseudocode of the inference path, in the paper's order. Training losses and schedules are on the card, not here.

# Decode one frame at time t from one event window  [arxiv:2312.00113 §3]
# Inputs: initial frame (B, 3, H, W); t in [0, 1] (window fraction)
#         motion volume and synthesis volume (B, C, H, W)
def decode_frame(initial, motion_vol, synth_vol, t):
    # Per-pixel coefficients for K = 5 basis functions  [§3.1]
    coeffs = motion_net(motion_vol)
    # Time basis evaluated at t  [§3.1] [eq. 2]
    basis = basis_mlp(t)
    # Per-pixel trajectory at time t  [eq. 2]
    flow_t = weighted_sum(coeffs, basis)
    # Tri-plane features from the events and the initial frame  [§3.2]
    planes = synthesis_net(synth_vol, initial)
    # Sample the three planes at (x, y, t), multiply, decode  [eq. 3]
    latent = decode_rgb(sample_planes(planes, t))
    # Flow between the initial frame and the synthesised frame  [§3.3]
    latent_flow = flow_net(initial, latent)
    # Splat pyramids along both flows, then fuse  [§3.4] [eq. 5]
    return fusion_net(splat(initial, flow_t, latent_flow), latent)

The event-reading networks run once per window, and the steps inside the loop over times are the only per-time work 2312.00113 §3, fig. 3.

Contradictions

The paper's own tables give different PSNR margins depending on the baseline. On BS-ERGB gap 3, the full model's 24.80 dB is 3.69 dB above DMVFN's 21.11 dB and 3.92 dB above RAFT (inpainted) at 20.88 dB 2312.00113 tab. 2. On E2D2 the margin over DMVFN is 2.75 dB, from 28.68 against 25.93 2312.00113 tab. 1. A reader quoting a single margin should name the baseline and the table.

The LPIPS figures are also reported against RAFT (inpainted): 0.095 against 0.142 on BS-ERGB gap 3 is about a third lower 2312.00113 tab. 2.

The authors' released code reports different numbers from the paper's tables. The card lists these differences in detail. This page does not resolve them; see the card.

Papers

• Event-based Continuous Color Video Decompression from Single Frames: reconstructs colour video at any time inside an event window from one colour frame, using a learned motion basis, tri-plane event features and flow-guided fusion.