Hybrid Gaussian Splatting for Novel Urban View Synthesis

In plain words

The task is to show a street scene from a camera position the car never drove through. You get video from one drive down the street and must render what a second drive along a neighbouring lane, or in the opposite direction, would have seen. This matters for testing self-driving software in simulation.

The authors use two steps. First they build a 3D model of the street from the training frames, using many points that come from a standard reconstruction tool, and render the new viewpoint from that model. The render has smudges and holes, because the model was never shown that angle. Second, a fast image model, trained to clean up exactly these smudges, turns the render into a believable frame. A single pass of that image model is enough.

TL;DR

A challenge entry (RealADSim-NVS, ICCV 2025 workshop) for novel street-view synthesis. It fits 3D Gaussian Splatting to the scene, using a dense point cloud from COLMAP, then cleans the renders with a one-step diffusion enhancer that was fine-tuned on about 36k pairs of artifact and clean frames. Author-reported score 0.432 on the test leaderboard, second of 28 submissions §1. Use it as a recipe for a 3DGS-plus-diffusion-refinement pipeline and for the data-curation and initialization lessons; the paper is short and leaves many training settings unstated.

Problem

Novel view synthesis must produce frames from camera poses held out from the training traversals of one scene §2. The challenge defines three levels: views shifted by one lane (Level 1), views shifted by two lanes (Level 2), and views from the opposite direction (Level 3) §2. Each team gets 12 sequences with training frames, camera poses and a point cloud §1, §2. Outputs are scored against held-out ground truth with PSNR, SSIM and LPIPS §2.

Pure 3DGS renders degrade as the target view moves away from the training views §3.2. The authors say the 3DGS render alone shows significant artifacts that grow with that distance §3.2. Extrapolating to a different trajectory, such as a perpendicular one at an intersection, remains the main failure mode §4.

Method

The pipeline has two stages §3 fig. 1. First, a 3DGS model of the scene is fitted from the source frames and their cameras, and rendered at each target pose. Second, a single-step diffusion enhancer maps each render, together with a clean reference frame from the source set, to the final frame §3.2.

Stage 1: 3DGS backend §3.1. The authors state that the initial point cloud matters most. The challenge point clouds are very sparse, on the order of 1e3 points for Paralane sequences §3.1. The authors re-estimate them from the images and known cameras with COLMAP: a sparse reconstruction (feature extraction, matching, triangulation) and then the dense pipeline (undistortion, depth and normal maps, fusion) §3.1 fig. 3. Their experiments with other primitives, densification or priors did not beat standard 3DGS once it was initialized carefully §3.1.

Stage 2: diffusion enhancement §3.2. The enhancer is derived from SD-Turbo and the public Difix checkpoint, then fine-tuned to remove 3DGS rendering artifacts §3.2. The reference frame is the source frame whose camera pose is closest to the target pose, measured as a weighted sum of translation distance and quaternion rotation distance §3.2 (footnote 1 gives the weights). The enhancer's inputs and output are:

S = 0.4 \times \frac{PSNR}{100} + 0.3 \times SSIM + 0.3 \times (1 - LPIPS) eq. 1

\mathcal{G} \leftarrow Fit(\{\xi, \pi, c\}, P_c) eq. 2

y^{gs}_j = Render(\mathcal{G}, p_j, c_j) eq. 3

\hat{y}_j = Difix(y^{gs}_j, x_{ref}) eq. 4

Here eq. 1 is the challenge score §2; eq. 2 fits the 3DGS model to frames, poses, intrinsics and the point cloud §3.1; eq. 3 renders a target view §3.1; eq. 4 enhances it using the reference frame §3.2. The reference is x_ref, chosen as the source frame whose pose is nearest the target §3.2.

Training data §3.2. Corrupted and clean frame pairs come from Paralane and EUVS. Paralane has 25 sequences with three clips each; each clip is fitted with 3DGS and used to render the other clips, giving 150 corrupted clips §3.2. EUVS contributes its 79 public scenes, rendered with its predefined train/test split §3.2. The mixed set has 36,231 training pairs §3.2. Source and target frames come from different traversals, so they contain different dynamic objects; these are masked with Grounded SAM 2 both when fitting 3DGS and in the enhancer's loss §3.2.

Training loss §3.2. The enhancer minimises a weighted sum of MSE, SSIM and LPIPS between its output and the clean frame, after masking dynamic objects in both. The paper does not give the weights.

Optional iterative refinement §3.2 fig. 4. Enhanced renders at interpolated poses can be added as pseudo source views and the 3DGS model refined, in K steps. The paper shows K=3 improves visuals but says the improvement is not reflected in the metrics, so it was left out of the final submission §4.

# Hybrid NVS pipeline for one scene  [§3] [fig. 1]
# Input:  source frames, poses, intrinsics; target poses
# Output: one enhanced frame per target pose

def render_scene(frames, poses, intrinsics, target_poses):
    # 1. Re-estimate the point cloud from the images  [§3.1]
    # Sparse reconstruction first, then dense fusion on the same cameras.
    points = sparse_reconstruction(frames, poses, intrinsics)
    points = dense_fusion(frames, poses, intrinsics, points)

    # 2. Fit 3DGS, leaving dynamic objects out of the loss  [§3.1] [§3.2]
    gaussians = fit_3dgs(frames, poses, intrinsics, points, dynamic_masks)

    enhanced = []
    for target_pose in target_poses:
        # 3. Render the coarse view from the fitted scene  [§3.1] [eq. 3]
        coarse = render_3dgs(gaussians, target_pose)

        # 4. Pick the source frame closest in pose as reference  [§3.2]
        reference = closest_frame(frames, poses, target_pose)

        # 5. Clean the render in one enhancer pass  [§3.2] [eq. 4]
        enhanced.append(enhancer(coarse, reference))
    return enhanced

# Enhancer training loss  [§3.2, "Training loss"]
# Input:  corrupted render, clean frame, reference frame, dynamic mask
# Output: scalar loss; term weights not stated

def enhancer_loss(corrupted, clean, reference, dynamic_mask):
    # 1. Enhance the corrupted render with its reference  [§3.2] [eq. 4]
    prediction = enhancer(corrupted, reference)

    # 2. Keep dynamic objects out of prediction and target  [§3.2]
    # How masked pixels enter the loss is not stated.
    masked_prediction = apply_mask(prediction, dynamic_mask)
    masked_clean = apply_mask(clean, dynamic_mask)

    # 3. Weighted sum of MSE, SSIM and LPIPS terms  [§3.2]
    # Weights and the SSIM term's sign convention are not stated.
    return (w_mse * mse(masked_prediction, masked_clean)
            + w_ssim * ssim_term(masked_prediction, masked_clean)
            + w_lpips * lpips(masked_prediction, masked_clean))

Hyperparameters

Results

Author-reported. Setup: the RealADSim-NVS challenge evaluation server §1 (test set) and the authors' devset §4. Baseline: the challenge's own point clouds fig. 3 and the pretrained enhancer tab. 1.

Leaderboard. Aggregated score 0.432, second place of 28 submissions §1. The paper does not break this down by level.

3DGS only, challenge sequences fig. 3. Each cell is source views / target views. The target-view score is the one used in the prose §4.

Enhancer on the devset tab. 1. Single-step enhancement, no iterative refinement §4.

Qualitative examples: source view, target view, 3DGS render and enhanced frame are shown side by side fig. 2. Failure cases are described in §4 fig. 2.

Paper vs code

No official code. The paper, its abstract and its TeX source give no repository or project page. The arXiv metadata in raw/arxiv/2510.12308.json contains no comments field. Searches run: GitHub repository search for "hybrid gaussian splatting urban" (no results); "realadsim" (three hits, none from the authors: wm-research/ViSE, UQHTy/RealADSim, hyzhou404/HUGSIM_Local_Server); and the Qualcomm-AI-research organisation's repository listing, read in part (the response was truncated) with no gaussian-splatting or NVS repository among the entries that were read.

Pitfalls

• Point-count figures disagree. §3.1 says challenge sequences have about 1e3 points for Paralane and that COLMAP sparse gives about 1e5 per sequence. fig. 3 lists about 30K for the challenge clouds and about 180K for COLMAP sparse. The table does not say whether counts are per-sequence means.
• Score direction in tables. Source-view and target-view numbers are shown side by side in fig. 3. The prose quotes target-view scores only (0.319 to 0.343) §4. The score formula eq. 1 reproduces the table's scores when applied by hand, so use it to check your own runs.
• Figure label. The initialization table is a figure environment labelled tab:initialization in the source fig. 3. Cite it as fig. 3.
• Figure and table numbers here follow source order and were not checked against the compiled PDF. If in doubt, cite the section.
• Iterative refinement helps visuals but not the metric, and was excluded from the submission §4. Do not assume K=3 is part of the final method.
• Masked loss is not a clear win. On the devset it leaves the score at 0.462 and gains only PSNR; the authors note that it benefits PSNR and SSIM but not the overall score §4 tab. 1.
• Devset is not the leaderboard. The devset (6 Paralane, 3 EUVS sequences) scores 0.462 tab. 1, while the leaderboard is 0.432 §1. Do not compare them directly.
• Clean-target construction is implied, not stated. §3.2 describes pairs of corrupted and clean frames. The text implies the clean frame is the real camera frame of the held-out traversal, but does not say so outright.
• Enhancer training inputs are not fully specified. The paper does not say which reference frame the enhancer sees during training, or how it is sampled.
• Weights and checkpoint licensing. The enhancer is fine-tuned from the public Difix checkpoint §3.2 §4. The paper does not state that checkpoint's license or the license of the SD-Turbo weights it derives from §3.2. Check both before reuse.
• Variants tried but not reported. The authors tested triangle primitives, MCMC densification, depth and normal priors and separate sky and ground handling, and say none beat standard 3DGS once initialized carefully §3.1. They give no numbers for these variants.
• Hidden costs. Dense COLMAP reconstruction produces several million points per sequence §3.1. The paper gives no runtime, memory or hardware figures.
• Generalization. Extrapolation to a different trajectory, such as a perpendicular path at an intersection, remains a failure mode §4.

Lineage

• Builds on: 3D Gaussian Splatting (Kerbl et al., 2023) for the scene model §3.1; SD-Turbo (Sauer et al., 2024) and Difix / Difix3D+ for the enhancer §3.2; COLMAP for point clouds §3.1; Grounded SAM 2 for masks §3.2.
• Compared against: the pretrained Difix enhancer and the challenge's own point clouds tab. 1 fig. 3. The paper also cites other hybrid diffusion-plus-3D methods for urban scenes §1 but does not benchmark them.
• Superseded by: not known from the content of this wiki.