Event-based Continuous Color Video Decompression from Single Frames

In plain words

A normal camera takes a fixed number of pictures per second, so fast motion between two pictures is lost, and a blurry picture ruins whatever is built from it. An event camera works differently: each pixel reports when its brightness changes, as a light stream of timestamped events.

This paper asks a simple question. If you have one good color photo and the events that happened after it, can you rebuild what the scene looked like at any moment in that window? The method learns how each pixel moves over time, separately learns what the colors should look like, and merges the two into one image for the moment you ask for. It never needs a second photo, so a blurry or badly exposed later frame cannot spoil the result.

TL;DR

• Input: one color frame plus the aligned event stream after it. Output: color frames at any requested time inside the events §1.
• Core parts: a continuous motion model (learned basis functions), a tri-plane synthesis of the colors from events, RAFT-based flow between the synthesized frame and the initial frame, and a multi-scale fusion network that uses softmax splatting §3.
• Author-reported gains over frame-interpolation, video-prediction and flow-unrolling baselines on BS-ERGB and on the authors' E2D2 dataset §4.2, tabs. 1–2.
• The official code does not reproduce the paper's table numbers, and it ships no training code. Read ## Paper vs code before reusing anything.

Problem

Frame interpolation needs two sharp frames around the target time. If the later frame is corrupted by motion or lighting, the interpolated frame is blurry too, and the method must wait for that later frame, which adds latency §1. Event cameras send only the changes in log intensity, so their bandwidth is much lower than that of a frame camera at the same rate §1.

The task here is harder than interpolation: the only frame available is the one before the window, and the output must be recovered from events alone plus that frame §1, §4.2. The paper assumes an event stream aligned with the initial color frame. It gets this alignment from a single-lens beam splitter rig, which it describes in [suppl. Dataset Design]. The BS-ERGB sequences use a static camera §4.1.

Method

The model runs once per window and then decodes any number of times §3, fig. 3. Four parts feed one fusion network: the continuous motion field (§3.1), the tri-plane synthesis (§3.2), the latent-frame flow (§3.3), and the fusion (§3.4). §3.5 gives the losses.

Step 1: event volumes. Events in a window are binned into two volumes of 60 channels in total, split over 6 intervals [suppl. Input Representation] voxel.py.

• Synthesis volume: positive and negative events in separate bins, 5 each per interval, interpolated in time and not normalized voxel.py.
• Motion volume: unsigned counts in 10 bins per interval, then normalized per interval by the 98th percentile of the non-zero values and clipped voxel.py.

Step 2: continuous trajectory field (§3.1). Each pixel's trajectory is a weighted sum of K learned basis functions of time, with K = 5 §3.1. The basis is a small MLP shared by all inputs, so it can be queried at any time [suppl. Model] motion.py. A U-Net predicts the per-pixel coefficients for x and y and a splatting weight, at several scales, with the coefficients refined coarse to fine [suppl. Model] motion.py. Equations 1 and 2 are the discrete and continuous forms of the trajectory:

\mathbf{x}_i(t_j) = \sum_{k=1}^{N_b} \alpha_k(\mathbf{u}_i)\, \theta_{k,j} eq. 1

\mathbf{x}_i(t) = \sum_{k=1}^{N_b} \alpha_k(\mathbf{u}_i)\, g^{\theta}_k(t) eq. 2

Step 3: tri-plane synthesis (§3.2). A U-Net turns the synthesis volume and the initial frame into features at 4 scales. Each scale carries three planes: x–y, x–t and y–t [suppl. Model] synthesis.py. To decode a pixel at (x, y, t), the model samples the three planes bilinearly, multiplies the three samples elementwise, and concatenates the products over scales §3.2 synthesis.py. A three-layer convolutional decoder then maps these features to RGB [suppl. Model] synthesis.py.

\hat{I}_\tau(x,y) = \phi\bigl(\mathbf{g}_{xy}(\pi_{xy}(\mathbf{q})),\, \mathbf{g}_{xt}(\pi_{xt}(\mathbf{q})),\, \mathbf{g}_{yt}(\pi_{yt}(\mathbf{q}))\bigr) eq. 3

Here q = (x, y, τ) and π maps q onto each plane §3.2.

Step 4: latent-frame flow (§3.3). The synthesized frame is not accurate in color, but it has enough texture for matching. RAFT builds a correlation volume between the features of the synthesized frame and of the initial frame, then refines the flow over 12 steps §3.3. The correlation is:

C\bigl(\mathbf{f}_t(i,j), \mathbf{f}_0(k,l)\bigr) = \mathbf{f}_t(i,j)^{\top}\,\mathbf{f}_0(k,l) eq. 4

Step 5: fusion (§3.4). Feature pyramids and image pyramids of the initial frame are splatted along the motion flow and along the latent-frame flow with softmax splatting. The splatted pyramids are concatenated with a pyramid of the synthesized frame §3.4:

\mathcal{G} = \mathcal{C}\bigl(\mathcal{T}_{M_t}(\Psi_0), \mathcal{T}_{M_t}(\mathcal{I}_0), \mathcal{T}_{\tilde{M}_t}(\mathcal{I}_0), \mathcal{T}_{\tilde{M}_t}(\Psi_0), \hat{\mathcal{I}}_t\bigr) eq. 5

A multi-level convolutional merging network turns the result into the final frame §3.4 continuitycam.py.

Step 6: training losses (§3.5). Three terms supervise the flow and the images.

• Self-supervised warping loss: the frame at time t is warped back to t = 0 along the backward flow, with a Charbonnier photometric penalty and a second-order smoothness penalty §3.5.

L_{\text{photo}}(I_0, \hat{I}_{\text{warp}}) = \rho(I_0 - \hat{I}_{\text{warp}};\, \beta) eq. 6

• Supervised flow loss against RAFT flow, computed from the original frames §3.5.

L_{\text{flow}}(M_t, W_t) = \lVert M_t - W_t \rVert_1 eq. 7

• Image loss: L1 plus VGG-19 perceptual loss, weighted equally, applied to the synthesized frame and to the final output §3.5.

\mathcal{L} = \lambda_{\text{photo}} L_{\text{photo}} + \lambda_{\text{flow}} L_{\text{flow}} + \lambda_{\text{reconstr}}(L_1 + L_p) eq. 8

Training is sequential: the motion branch and the synthesis branch are trained on their own first, then the whole system is trained jointly §3.5. The schedule is in the Hyperparameters table.

Inference pseudocode

# Decode one frame at time t from one window  [§3] [fig. 3]
# Inputs: initial (B, 3, H, W) in [-1, 1]; t (B,) in [0, 1] (window fraction)
#         motion_volumes, synthesis_volumes (B, 60, H, W)
def decode_frame(initial, motion_volumes, synthesis_volumes, t):
    # Per-pixel coefficients, one set per scale  [§3.1] [code: motion.py#L33]
    coefficients = motion_unet(motion_volumes)
    # Time basis at t, K = 5  [suppl. Model] [code: motion.py#L38-L40]
    basis = basis_mlp(t)                              # (B, 1, K)
    # Flow at t: coefficients weighted by the basis  [eq. 2]
    flows = weighted_basis_sum(coefficients, basis)
    # Zero the flow where the window has no events  [code: motion.py#L56-L57]
    flows = mask_by_events(flows, motion_volumes)
    # Tri-plane features  [§3.2] [code: synthesis.py#L77]
    planes = synthesis_encoder(synthesis_volumes, initial)
    # Sample three planes at (x, y, t), multiply, decode RGB  [eq. 3]
    latent = color_decoder(sample_triplanes(planes, t))  # (B, 3, H, W)
    # Flow from the initial frame to the synthesized frame  [§3.3] [eq. 4]
    latent_flow = raft(initial, latent)
    # Splat pyramids along both flows, then fuse  [§3.4] [eq. 5]
    splatted = splat_pyramids(initial, flows, latent_flow)
    return fusion_net(splatted, latent)

Hyperparameters

Values marked "code" appear only in the released code, not in the paper.

Results

Author-reported, with the authors' own baselines and metrics. PSNR and SSIM are higher-better; LPIPS is lower-better. Each table lists the paper's rows in full.

Table 1: E2D2 (authors' dataset), all frames averaged tab. 1.

Table 2: BS-ERGB, TimeLens protocol tab. 2. Gap 1 and gap 3 are the numbers of frames between key frames. Italic LPIPS values in the paper were copied from the TimeLens paper, and the metric implementation may differ tab. 2.

Table 3: fiducial tag detections on the downstream sequences tab. 3 (§4.3).

Table 4: ablation on BS-ERGB tab. 4 (§5). Motion-only uses just the motion branch and synthesis-only just the synthesis branch.

Downstream use (§4.3): 3D Gaussian Splatting built from the decompressed frames is sharper than one built from blurred inputs, using COLMAP poses §4.3. Qualitative comparisons with the baselines are in §4.2.

Paper vs code

Official repository found via the project page, which the abstract links [abstract]: github.com/ZiyunClaudeWang/event-continuitycam. The owner is the first author, and the MIT license is in LICENSE LICENSE. Commit 0c8ceaf095d14af6b5233ef90d10a494e9be1607, cloned on 2026-10-19.

1. Reported numbers differ. The release README gives BS-ERGB skip 1 as 26.30 PSNR / 0.083 LPIPS and skip 3 as 25.48 / 0.095 README.md. E2D2 is 29.26 / 0.066 / 0.811 README.md. The paper gives 25.40 / 0.088 and 24.80 / 0.095 tab. 2, and 28.68 / 0.077 / 0.802 tab. 1. The README attributes the gap to a U-Net fix and extra temporal augmentation (event windows of varying length) README.md.
2. Splatting weight. The supplement says a tanh activation is applied to the splatting weight [suppl. Model]. The code applies a sigmoid and multiplies by a learned scalar α motion.py.
3. RAFT weights. The paper uses RAFT weights pre-trained on FlyingChairs and FlyingThings3D §3.3. The code uses torchvision's default RAFT-large weights continuitycam.py. These are not the same checkpoint.
4. Flow masking. The code zeroes the motion flow where a window has no events continuitycam.py motion.py. The paper does not mention this.
5. Coarse-to-fine coefficients. The code adds the upsampled coarser-scale coefficients to each finer scale motion.py. The paper only describes per-pixel prediction.
6. Latent flow uses motion splatting weights. The code splats the latent flow with the same splatting metrics as the motion flow continuitycam.py. The paper does not say whether the latent flow has its own weights.
7. Event normalization. The supplement says the motion volume is normalized to a maximum of 1 and a minimum of 1 [suppl. Input Representation]. The code clips at the 98th percentile of the non-zero values voxel.py.
8. Feature channels. The supplement says the feature encoder outputs 24 channels, 8 per plane [suppl. Model]. The code uses 8 channels for x–y and produces x–t and y–t planes from 32 projection channels, resampled to 24 time samples synthesis.py.
9. No training code. The release has inference, data and evaluation code only. A search for loss, optimizer and Adam found nothing in the Python files evaluate.py. So the loss weights, the stop-gradient on the splatting input §3.4, and the training schedule cannot be checked against code.
10. Extra metric. The evaluation script adds a PSNR over moving pixels only, which the paper does not report evaluate.py.

Pitfalls

• The abstract's 3.61 dB does not match the tables. On BS-ERGB gap 3, the paper's own rows give 24.80 − 21.11 = 3.69 dB over DMVFN and 24.80 − 20.88 = 3.92 dB over RAFT (inpainted) tab. 2. On E2D2, 28.68 − 25.93 = 2.75 dB over DMVFN tab. 1. The LPIPS claim checks out: 0.095 vs 0.142 is a 33% decrease, and 0.077 vs 0.111 is about 31%. Treat the PSNR margin as unresolved.
• The "+20% tags" claim depends on the baseline. Against TimeLens (8,628), the detections are 10,342 / 8,628 ≈ +19.9%. Against FILM (8,853) it is about +16.8%, and the same holds against the original (8,854) tab. 3.
• Baseline fairness. The RAFT flow-unrolling baselines use the original ground-truth frames to compute flow [suppl. Baselines]. That is more information than the method under test uses, so these baselines are not a like-for-like test.
• Time normalization epsilon. The motion volume divides by a range plus 1e-6, and the file docstring says the time unit is 1e9 µs, which it describes as a 1 ms epsilon voxel.py. The arithmetic in the code divides timestamps by 1e9 directly. Check the units on your own data before reusing this code.
• Bins follow each window's own events. The synthesis and motion time bins run from each interval's first event to its last event, per polarity for the synthesis volume voxel.py. A sparse window shifts the bins. The paper does not say this.
• Any-time query is only partly validated. The paper queries frames at the protocol times §4.1. The README reports that frames very close to the key frame are less accurate than the rest, with the cut-off at t = 1/8 README.md, and gives per-window-length numbers for queries at t = k/r README.md. Treat queries at other times as unverified in the paper.
• Cost. The synthesis branch runs once per window; decoding each extra time costs one RAFT pass, one tri-plane sample and one fusion pass continuitycam.py. The README gives about 10 GB GPU memory at 970 × 625 README.md.
• Dataset access. E2D2 is distributed as a 41 GB OneDrive folder README.md. The paper and the README state no licence for the dataset, so do not redistribute it.

Lineage

• Builds on: K-Planes-style factorised planes, with a Hadamard product across planes §3.2; RAFT for correlation-based matching §3.3; softmax splatting for warping §3.4; the EV-FlowNet photometric loss §3.5; FILM's frozen feature extractor [suppl. Training].
• Compared against: TimeLens/TLXNet+ and FILM for frame interpolation, DMVFN for video prediction, RAFT and E-RAFT for flow unrolling §4.1.
• Superseded by: not known from this card's sources.