E2GS: Event Enhanced Gaussian Splatting

In plain words

A normal camera takes a photo over a short exposure. If the camera or the scene moves, the photo smears. An event camera works differently: each pixel reports when its brightness changes, so it keeps timing detail that a smeared photo loses. This paper uses both kinds of data from the same moment.

The method builds a 3D scene out of many small soft blobs, which is the Gaussian Splatting approach. It trains those blobs so that their renderings, averaged over a few sub-frame viewpoints, match each blurry photo. It also checks that the brightness changes the renders imply agree with what the event camera recorded. The result is a sharper 3D scene that can render new viewpoints, and it trains and renders much faster than the NeRF-based method it is compared with.

The approach works on still scenes only; the authors name moving subjects as future work.

TL;DR

E2GS adds event-camera data to 3D Gaussian Splatting, so the scene can be reconstructed from blurry multi-view RGB frames. Its loss has two terms: an image term that matches the average of sub-frame renders to the blurry photo, and an event term that matches per-pixel event counts. The authors report better synthetic deblurring than E2NeRF, mixed novel-view results, and much faster training and rendering. Scenes must be static. The official code is non-commercial.

Problem

Blurred frames make 3D reconstruction hard, and NeRF-based methods that use events (E2NeRF) need long training and slow rendering because of ray sampling §1 §2. Plain Gaussian Splatting renders of blurry inputs are themselves blurred fig. 1. The method assumes a static scene, a blurry image set and an event stream covering the exposure of each image §3.

Method

Overview

The input is a set of blurry images and the event stream of a static scene. Preprocessing turns each blurry image into N sharp frames at N timestamps and N camera poses, using the events §3 §3.2 fig. 2. Training then optimises Gaussians under two losses: an image rendering loss on the averaged render, and an event rendering loss on pairs of renders §3.3 eq. 12.

3.1 Preliminaries

Gaussian representation. Each Gaussian has a full 3D covariance Σ, which is built from a rotation R and a scale S as Σ = R S Sᵀ Rᵀ eq. 3. Its unnormalised value is G(x) = exp(−½ xᵀ Σ⁻¹ x) eq. 1. For rendering, the covariance in camera coordinates is Σ′ = J W Σ Wᵀ Jᵀ, where J is the Jacobian of the affine approximation of the projection and W the view transform eq. 2 §3.1.

Event model. A pixel (x, y) emits an event at time τ when its log intensity changes by more than the threshold C since its last event. The polarity p is +1 or −1 accordingly, with l = log L and τ′ the time of the previous event at that pixel eq. 4 §3.1.

3.2 Preprocessing

Preprocessing uses the events to make the sharp frames and camera poses that the 3D model needs §3.2.

1. Event bins. N timestamps split the event stream of an exposure into N−1 bins B_i, each holding the events with t_i < τ ≤ t_{i+1} eq. 5 §3.2. The text says the split is equal ("divides ... equally"); whether equal in time or in event count is not stated §3.2.
2. Sharp frames. Each sharp frame follows from the previous one by adding the events of the bin eq. 6 §3.2. The paper cites the event-based double integral (EDI) model for this step §3.2.
3. Blur consistency. The blurry image is the mean of the N sharp frames eq. 7 §3.2.
4. Poses and initial points. EDI gives N camera poses per view §3.2. COLMAP structure-from-motion runs on all N deblurred images per view to give the initial point cloud and poses §3.2. The paper says COLMAP fails on the blurry images §3.2 §4.1.

3.3 Loss function

Image rendering loss. N poses per view give N renders Î_i. Their mean is the blur estimate Î_blur eq. 8 §3.3. The loss is an L1 term plus a D-SSIM term, weighted by w_D-SSIM eq. 9 §3.3.

Event rendering loss. Two renders are picked at random, frames n < m §3.3. Their grayscale intensities are converted to the log domain. The difference between the two log values, divided by the threshold C, gives the per-pixel event count. The paper rounds down for a rise and up for a fall, which truncates toward zero eq. 10 §3.3. The count of ground-truth events between the same two frames is B′_nm. Positive and negative events cancel in it, because the model assumes monotonic intensity change between the two timestamps §3.3. The loss is the squared L2 norm of the difference between the estimated and ground-truth counts eq. 11 §3.3.

Total loss. L = L_blur + w_event · L_event eq. 12 §3.3.

# One E2GS training step  [§3.2] [§3.3]
# The paper has no algorithm box; this follows eq. 7-12.
# Input:  blurry (H, W, 3); event bins of the view; N = 5 poses
# Output: scalar loss

def training_step(blurry, event_bins, poses, scene):
    # Render one sharp frame per exposure pose  [§3.2] [eq. 7]
    frames = [render(scene, pose) for pose in poses]   # N x (H, W, 3)
    # Average the N frames to match the blurry photo  [eq. 8]
    blur_estimate = mean(frames)                       # (H, W, 3)
    # Blur loss: L1 mixed with D-SSIM  [eq. 9]
    blur_loss = (
        (1 - w_dssim) * l1(blur_estimate, blurry)
        + w_dssim * dssim(blur_estimate, blurry)
    )
    # Pick two frames n < m at random  [§3.3]
    n, m = random_pair(len(frames))
    # Per-pixel event count implied by the log-intensity change  [eq. 10]
    # Paper: floor for a rise, ceil for a fall (see Pitfalls)
    change = (log(gray(frames[m])) - log(gray(frames[n]))) / C   # (H, W)
    implied = where(
        gray(frames[n]) < gray(frames[m]),
        floor(change),
        ceil(change),
    )
    # Ground-truth signed event count between the same frames  [§3.3]
    # How bin indices map to frames n and m is not stated.
    counted = count_events(event_bins, n, m)           # (H, W)
    # Squared L2 difference of the two counts  [eq. 11]
    event_loss = sum((implied - counted) ** 2)
    # Combine the two terms  [eq. 12]
    return blur_loss + w_event * event_loss

Equations

All equations are numbered as they appear in the compiled paper, in source order. Equations 1 to 4 are in §3.1, 5 to 7 in §3.2, and 8 to 12 in §3.3.

G(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x}^{T}\mathbf{\Sigma}^{-1}\mathbf{x}} eq. 1

\mathbf{\Sigma}' = \mathbf{J}\mathbf{W}\mathbf{\Sigma}\mathbf{W}^T\mathbf{J}^T eq. 2

\mathbf{\Sigma} = \mathbf{R}\mathbf{S}\mathbf{S}^T\mathbf{R}^T eq. 3

p(x,y,\tau) = \begin{cases} +1 & \text{if } l(x,y,\tau) - l(x,y,\tau') > C \\ -1 & \text{if } l(x,y,\tau) - l(x,y,\tau') < -C \end{cases} eq. 4

B_i = \{ e_j(x_j,y_j,\tau_j,p_j) \}_{j=1}^{N_e^i} \quad (t_i < \tau_j \le t_{i+1}) eq. 5

I_{i+1} = I_i \sum_{j=1}^{N_e^i} \exp(C p_j) eq. 6 (as typed; see Pitfalls)

I_{\text{blur}} = \frac{1}{N} \sum_{i=1}^{N} I_i eq. 7

\hat{I}_{\text{blur}} = \frac{1}{N} \sum_{i=1}^{N} \hat{I}_i eq. 8

\mathcal{L}_{\text{blur}} = (1 - w_{\text{D-SSIM}}) \mathcal{L}_1 + w_{\text{D-SSIM}} \mathcal{L}_{\text{D-SSIM}} eq. 9

\hat{B}'_{nm} = \begin{cases} \lfloor (\log L_m - \log L_n) / C \rfloor & \text{if } L_n(x,y) < L_m(x,y) \\ \lceil (\log L_m - \log L_n) / C \rceil & \text{if } L_n(x,y) \ge L_m(x,y) \end{cases} eq. 10

\mathcal{L}_{\text{event}} = \lVert \hat{B}'_{nm} - B'_{nm} \rVert_2^2 eq. 11

\mathcal{L} = \mathcal{L}_{\text{blur}} + w_{\text{event}} \mathcal{L}_{\text{event}} eq. 12

Hyperparameters

Values in the first block come from the paper. The rest is taken from the official code, which says the other parameters follow 3D Gaussian Splatting defaults §4.1. Code values are reported as the code gives them.

Dataset settings §4.2:

Results

Author-reported. Setup: E²NeRF dataset; 6 synthetic scenes averaged for tables 1 and 2, 5 real scenes averaged for tables 3 and 4 §4.2. Baselines: 3D Gaussian Splatting (GS), E2NeRF, and GS trained with only the blur loss (GS w/ L_blur) §4.1. Metrics: PSNR, SSIM and LPIPS for synthetic data; BRISQUE for real data, which has no sharp reference §4.1.

Image deblurring, synthetic data tab. 1

Novel view synthesis, synthetic data tab. 2

Image deblurring, real-world data, BRISQUE ↓ tab. 3

Novel view synthesis, real-world data, BRISQUE ↓ tab. 4

Training time and rendering speed, synthetic data at 800 × 800 tab. 5 §4.5

The paper states training time falls to 1/60 and rendering time to 1/3500 of E2NeRF's §4.5. Hardware for the E2NeRF timing is not stated.

Qualitative results are in fig. 3 (real-world deblurring), fig. 4 (real-world novel view) and fig. 5 (synthetic). Figure numbers follow source order.

Paper vs code

Official repository: github.com/deguchihiroyuki/E2GS, first author's account, linked from the abstract §1. Cloned at commit 68e5614ee31432038d362d33fc35d6420c36250a. The repository is under the E2GS License, which allows non-commercial research and evaluation use only LICENSE.md. This card cites it by path and line and quotes nothing from it. Do not reuse the code in commercial work.

Only the training loss and its data handling were compared. The repository contains no EDI or COLMAP preprocessing, and no code produces events.pt README.md.

• Event-pair selection. Paper: two random frames per step §3.3. Code: all 10 pairs of 5 renders, drawn with random.sample(combination, 10) on every call, so each iteration uses all of them train_BlurAndEvent.py event_loss.py.
• Threshold values. Paper: C_pos = 0.2 for positive events and C_neg = 0.3 for negative events §4.1. Code divides the positive-side difference by 0.3 and the negative-side difference by 0.2 event_loss.py. The pairing is inverted relative to the paper's names. Which is intended is not settled by the paper or the code.
• Rounding. Paper: floor for a rise, ceil for a fall eq. 10. Code: no rounding; the quotient is used directly event_loss.py.
• Log transform. Paper: log of intensity §3.3. Code: lin_log, linear up to a threshold of 20 on a 0–255 scale and logarithmic above it event_loss.py event_loss.py. The linear region is not in the paper.
• Grayscale. Paper: "grayscale" with no weights §3.3. Code: weights 0.299, 0.587 and 0.114 train_BlurAndEvent.py.
• Event loss form. Paper: squared L2 norm eq. 11. Code: ±0.5 offsets added to the event counts, and the squared errors averaged over pixels and then over pairs event_loss.py event_loss.py. Neither the offsets nor the averaging is in the paper.
• Event weight. Paper: w_event = 0.005 §4.1. Code: event loss multiplied by 0.02 train_BlurAndEvent.py. A commented-out line would switch to 0.01 after iteration 7,000; it is not active train_BlurAndEvent.py.
• Image loss. Matches the paper: mean of the 5 renders, then L1 plus 0.2 × (1 − SSIM) train_BlurAndEvent.py [code: arguments/init.py#L83]. No discrepancy found.
• Initial point cloud. Paper: COLMAP on EDI-deblurred frames §3.2 §4.1. The repository README says that for synthetic data COLMAP can take the blurry images directly README.md.
• Event tensor. Paper: no file format. Code loads events.pt per view and indexes it with bin numbers 0 to 4 train_BlurAndEvent.py train_BlurAndEvent.py train_BlurAndEvent.py. Eq. 5 gives N−1 = 4 bins for 5 timestamps. How the file is built is not in the repository.
• Optimiser. Paper: not stated. Code: Adam with eps 1e-15 gaussian_model.py.
• Entry point. The README says python train.py README.md, but the repository has train_BlurAndEvent.py.

Pitfalls

• Eq. 10 is not differentiable as written. Floor and ceil have zero gradient almost everywhere. A literal implementation of the event loss would send no gradient to the Gaussians through that term. The released code avoids this by not rounding event_loss.py. Any rounding you add changes what the loss trains.
• Eq. 6 is ambiguous. As typed it multiplies I_i by a sum of exponentials of the event polarities. The text says events are "added" to the image §3.2. The paper does not restate EDI's exact form, and the code does not contain EDI. Implement the sharp-frame recovery from the EDI model's own definition, not from eq. 6 alone.
• Two thresholds, one symbol. Eq. 4 and eq. 10 use a single C. Section 4.1 gives C_pos and C_neg only. The paper does not say which value C should take in the event loss. The code uses 0.3 for the positive side and 0.2 for the negative side, the reverse of the paper's naming event_loss.py.
• Index ranges disagree. Eq. 7 indexes frames from i = 1 to N. §3.3 picks frames from {I_i} with i from 0 §3.3. Choose one base and apply it to the bins too. The bin-to-frame mapping for the event count is not stated.
• Equal split is undefined. "Divides the event stream equally" §3.2 could mean equal time or equal event count. The two give different bins when the event rate changes during an exposure.
• Loss normalisation. Eq. 11 is a squared L2 norm, which sums over pixels. The code takes means event_loss.py. Combined with the different event weight (0.005 vs 0.02), the event term's effective strength differs from the paper by a large factor. Pick one normalisation and document it.
• Cost per iteration. The code renders 5 poses per iteration [code: gaussian_renderer/init.py#L157]. It also copies the whole event tensor to the GPU once train_BlurAndEvent.py. The event term computes 10 pairwise terms per step, each with its own event sum event_loss.py. The paper does not report these costs.
• Timing scope. Training time in tab. 5 is 50 min for 30,000 iterations on an A5000 §4.1 §4.5. Whether EDI and COLMAP preprocessing time is included is not stated. Neither is the hardware for the 140 FPS rendering.
• Novel-view results are mixed. E2NeRF has higher PSNR and SSIM than E2GS on synthetic novel views tab. 2. E2GS is better on LPIPS. The text says E2GS is "better or comparable" §4.3.
• Real-world per-scene results are mixed. On BRISQUE, E2NeRF is lower (better) on lego for deblurring tab. 3 and on lego and plant for novel views tab. 4. BRISQUE measures naturalness, not fidelity, and no sharp reference exists for these scenes §4.1.
• Event data is simulated for synthetic scenes. The synthetic events come from V2E §4.2. Only the real-world set uses a sensor, the DAVIS346 colour event camera §4.2.
• Static scenes only. The method assumes a static scene §3. Dynamic scenes are future work §5.
• No error bars. The tables report single numbers. No seeds, variances or repeated runs are stated.
• Ratio claim. "60 times faster training" and "3500 times faster rendering" §1 follow from tab. 5 as the paper reports them. The stated training time of 2 days is the paper's figure; the E2NeRF setup behind it is not described.

Lineage

• Builds on: 3D Gaussian Splatting (cited as 3Dgaussians in the source bibliography, with no arXiv id there and no card in this wiki). The EDI event-based double integral model is used for deblurring §3.2. COLMAP provides structure-from-motion §3.2.
• Compared against: E2NeRF (Qi et al., ICCV 2023), the NeRF-based method that uses blurry images and events §2 §4.1. The ablation GS w/ L_blur tests the event loss §4.1.
• Superseded by: not known from content in this wiki.