DriveSplat: Unified Neural Gaussian Reconstruction for Dynamic Driving Scenes

In plain words

DriveSplat rebuilds a recorded driving scene so it can be viewed from places the car never drove. It does this with a large cloud of small soft 3D blobs (Gaussians), fitted to the recorded camera images. The method treats the road, buildings and other fixed things as one background. Blobs far from the camera are kept coarser and blobs near the camera are kept finer, so nearby detail stays sharp without wasting effort on distant areas.

Moving actors are handled separately. Each one is tracked as an object with a rigid motion, and extra learned deformation is added for non-rigid actors such as pedestrians. Pretrained models supply dense depth and surface-direction maps of each frame, and the fit is nudged to agree with them. The result is judged on the Waymo and KITTI driving datasets, where it renders new viewpoints better than the methods it is compared with.

TL;DR

• DriveSplat reconstructs large dynamic driving scenes with neural Gaussians, one unified representation for background and actors §1.
• Background uses a hashed multi-scale table with a near, mid and far split and view-dependent level selection §3.2.
• Actors use a tracked rigid pose plus an anchor-level MLP deformation, which is cheaper than per-Gaussian 4D deformation §3.3.
• Depth and normal priors from pretrained models supervise training §3.4.
• Author-reported best novel-view synthesis on Waymo and KITTI. The paper's own text overstates two reconstruction claims (see Pitfalls).

Problem

Large driving scenes mix very near and very distant structure, and dynamic actors move non-rigidly. Single-scale Gaussian densification adds redundant primitives in distant regions and hurts novel-view quality §1. LiDAR depth is sparse and uneven, so it constrains distant facades weakly §1. Per-object or object-centric methods do not cover the scale and geometry of driving scenes §1§2. The paper's aim is one representation that handles both static background and dynamic actors §1§3.

Method

Inputs. RGB images, an initial 3D point cloud and bounding boxes of dynamic actors, all from the dataset §3. Depth and normal priors come from pretrained models and are used only in the supervised stage §3.

Step 1. Point-cloud initialization §3.1

1. Support SfM, LiDAR and dense DUSt3R points §3.1.
2. For each tracked actor, estimate pose from its box, build a transform matrix, move each frame's points into actor-local coordinates and keep the points inside an axis-aligned box §3.1eq. 1.
3. The remaining points are static background §3.1eq. 2.
4. Without LiDAR, actor points are sampled at random inside their boxes; the background uses COLMAP points §3.1.

Step 2. Background: multi-scale neural Gaussians §3.2

1. Each anchor carries one resolution level l, chosen by the LOD rule of step 4 §3.2.
2. The level's grid resolution grows geometrically from a base value to a finest value §3.2eq. 3.
3. Position is quantized into integer cells at that resolution §3.2eq. 4.
4. Cells map to a feature index with a spatial hash (XOR of cell times primes, modulo table size) §3.2eq. 5.
5. The anchor reads one table entry at its own level. Features are not summed across levels §3.2eq. 6.

Step 3. Background: geometry-guided partition §3.2

1. Estimate the dominant direction of background points by PCA §3.2eq. 7.
2. Project each point onto that direction to get depth z §3.2eq. 8.
3. Cut depth at its 33rd and 67th percentiles into near, mid and far §3.2eq. 8.
4. Give each region a voxel size s_base / alpha_r, with alpha_near > alpha_mid > alpha_far §3.2eq. 9. The values of alpha_r are not stated.

Step 4. Background: learnable LOD allocation §3.2

1. Target level from camera distance d §3.2eq. 10: floor(log_s(d_max / d)), where s is a LOD scale and d_max a scene-dependent constant (neither value is stated).
2. An anchor is rendered only if its assigned level is at most its target §3.2.
3. The two cut depths and the soft-region sharpness are treated as learnable and are optimized with the rest of the model §3.2eq. 11.
4. Soft membership of an anchor in a region uses sigmoids of depth minus each cut §3.2eq. 11.

Step 5. Dynamic actors §3.3

1. Global motion: a tracked rotation and translation per frame move each actor from its canonical frame to the world §3.3eq. 12. This applies to rigid and non-rigid actors alike §3.3.
2. Non-rigid actors add AnchorFlow, an N-layer MLP (N not stated). It takes the anchor position, anchor feature and encoded time and predicts updated anchor positions P_a(t) §3.3.
3. A geometry update head moves each neural Gaussian with its anchor §3.3.
4. Appearance heads then update opacity and colour per Gaussian at time t §3.3.
5. Compose all actors with the background and splat them to RGB, depth and normal §3.3eq. 13eq. 14.

Step 6. Geometry priors and loss §3.4

1. Priors: DepthAnything-V2 for relative depth, ZoeDepth for absolute depth, Omnidata for normals §3.4§4.2.
2. Colour loss: L1 plus a weighted SSIM term, weight 0.2 §3.4eq. 16§4.2.
3. Depth loss: the paper uses an absolute value of a correlation between the rendered depth and the relative prior §3.4eq. 17. See Pitfalls.
4. Normal loss: an L1 term plus a cosine-similarity term against the normal prior §3.4eq. 18.
5. Mask loss: a cross-entropy term on dynamic-actor masks, weight 0.01 in the paper §3.4§4.2eq. 15. See Paper vs code.
6. Total loss is the sum with weights lambda_d, lambda_n, lambda_m §3.4eq. 15.

Pseudocode for the partition and LOD rules (steps 3 and 4):

# Partition background points by depth along the main axis  [§3.2] [eq. 7-8]
def partition_by_depth(points):
    # Main direction: top principal component of the centred points  [eq. 7]
    direction = principal_axis(points - mean(points))
    # Scalar depth of each point along that direction  [eq. 8]
    depth = (points - mean(points)) @ direction
    # Cuts at the 33rd and 67th percentiles  [§3.2]
    tau_1 = percentile(depth, 33)
    tau_2 = percentile(depth, 67)
    # Near, mid and far masks from the two cuts  [eq. 8]
    near = depth <= tau_1
    mid = (depth > tau_1) & (depth <= tau_2)
    far = depth > tau_2
    return near, mid, far

# Soft membership of an anchor in one region  [§3.2] [eq. 11]
def soft_region_weight(z, tau_low, tau_high, ell):
    # Sigmoid step up at tau_low, step down at tau_high  [eq. 11]
    # tau_low is minus infinity for near and tau_high is plus infinity for far
    up = sigmoid((z - tau_low) / ell)
    down = sigmoid((z - tau_high) / ell)
    return up - down

# Visibility of an anchor from its LOD and camera distance  [§3.2] [eq. 10]
def anchor_visible(level, distance, d_max, s):
    # Target level from log base s of the distance ratio, rounded down  [eq. 10]
    target = floor(log(d_max / distance) / log(s))
    # Render only if the anchor's level does not exceed the target  [§3.2]
    return level <= target

Pseudocode for the dynamic actor and loss (steps 5 and 6):

# Place an actor's canonical Gaussians into the world  [§3.3] [eq. 12]
def to_world(mu_obj, rot_obj, rot_track, trans_track):
    # Position: rotate the canonical centre, then translate  [eq. 12]
    mu = rot_track @ mu_obj + trans_track
    # Rotation: canonical rotation times the transposed track rotation  [eq. 12]
    rot = rot_obj @ transpose(rot_track)
    return mu, rot

# Anchor flow at time t  [§3.3]
def anchor_flow(anchor_pos, anchor_feat, t):
    # Encode position and time, then concatenate with the anchor feature  [§3.3]
    encoded = concat(encode(anchor_pos), anchor_feat, encode(t))
    # The N-layer MLP outputs the updated anchor positions P_a(t)  [§3.3]
    return mlp(encoded)

# Total training loss  [§3.4] [eq. 15]
def total_loss(render, prior):
    # Colour term: L1 plus SSIM term  [eq. 16]
    loss = l1(render.rgb, gt.rgb) + 0.2 * ssim_loss(render.rgb, gt.rgb)
    # Depth prior term, weighted by lambda_d = 0.01  [eq. 17]
    loss = loss + 0.01 * depth_term(render.depth, prior.depth)
    # Normal prior term, weighted by lambda_n = 0.01  [eq. 18]
    loss = loss + 0.01 * normal_term(render.normal, prior.normal)
    return loss

Hyperparameters

Values marked "paper" come from the card's source paper; "code" values come from raw/code/2508.15376 at the commit in the frontmatter. Where the paper gives no value, the table says so.

Results

Author-reported. Setup: Waymo at 1066 x 1600 and KITTI at 370 x 1226, with novel views every fourth frame held out §4.1§4.2. Baselines are the methods listed in each table. Bold and underline are as printed in the paper.

Waymo, reconstruction and novel-view synthesis tab. 1

S = static-only method, D = dynamic. D_opt and N_opt mark depth and normal supervision.

KITTI tab. 2

Scenes with mostly non-rigid actors (Waymo subset, PSNR = dynamic regions)* tab. 3

Challenging novel views, FID and FVD (lower is better), shifts of 1, 2 and 5 m tab. 4

Training and rendering cost tab. 5

The paper does not state the GPU used for the FPS measurements.

Ablation: point-cloud initialization tab. 6

Ablation: background modules, added cumulatively tab. 7

SMG = scene-aware multi-scale Gaussian representation, GSP = geometry-guided partitioning, LLOD = learnable LOD allocation tab. 7.

Ablation: depth and normal priors tab. 8

Ablation: dynamic-actor deformation tab. 9

Ablation: loss terms tab. 10

The paper's text names mask loss the largest single contributor in this ablation §4.6.

Paper vs code

The official code is at https://github.com/Michael-Evans-Savitar/DriveSplat, commit 0db6a70fa0a7d5f8863b9068e1f37bb7e81db05a. Its README calls it "the official code release" for this paper README.md. The repository owner is not one of the paper's authors. We use it because the paper's project page (physwm.github.io/drivesplat) links it under "Code". The repository has no LICENSE file at its root, so its license is unstated. Do not copy code from it. Use the paths below for reference only.

Discrepancies, each with both citations:

1. Partition method. Paper: cuts at the 33rd and 67th depth percentiles §3.2eq. 8. Code: a 3-component Gaussian mixture, with a 10th/90th percentile fallback hash_lod_partitioner.py.
2. Learnable cuts. Paper: the two cuts and sharpness are learnable and enter through sigmoids §3.2eq. 11. Code: the cuts are stored as parameters with requires_grad=False, and region labels are hard hash_lod_partitioner.py gaussian_model_bkgd.py. No sigmoid soft assignment was found in the code.
3. LOD rule. Paper: target level floor(log_s(d_max/d)) and render if level is at most the target §3.2eq. 10. Code: level is log(standard_dist / dist) / log(lod_base) plus an extra level term, minus distance-based bias, clamped and then compared with the anchor's level gaussian_model_bkgd.py. lod_base is the hash per-level scale when hashing is on, otherwise the fork value gaussian_model_bkgd.py. The per-anchor LOD bias defined in the partitioner (paper-style Eq. 12 in its comments) is never called in the background path hash_lod_partitioner.py.
4. Hash lookup. Paper: one table entry per anchor at its own level, read at a grid index §3.2eq. 6. Code: trilinear interpolation over the eight corners of each level, enabled by default hash_encoding.py [code: arguments/init.py#L185]. The tiny-cuda-nn path always interpolates hash_encoding.py.
5. Single-level rule for the background. Paper: single-level reads §3.2. Code: the shared default is single-level [code: arguments/init.py#L166], but the background override is off [code: arguments/init.py#L190], with a comment that the background uses its standard neural Gaussian features [code: arguments/init.py#L187-L189]. Which path runs in the released configuration was not resolved.
6. Two scale sets. Paper: one ratio alpha per region §3.2eq. 9. Code: voxel/10, /5, /1 for near, mid and far gaussian_model_bkgd.py, and separately 0.8, 1.0, 1.5 in the partitioner hash_lod_partitioner.py. Which set drives the anchor level was not fully traced.
7. Actor motion. Paper: global rigid transform from tracked poses §3.3eq. 12. Code: the actor pose handling was not traced in this pass.
8. Anchor deformation. Paper: an N-layer MLP predicts updated anchor positions §3.3. Code: an 8 by 256 MLP predicts a 3D anchor displacement and a 6D rotation, and the offset is rotated by the anchor's rotation anchor_deformation.py anchor_deformation.py anchor_deformation.py.
9. Deformation output. The deformation step returns zero rotation and scale deltas for each Gaussian deform_model.py. The paper's appearance heads for opacity and colour at time t §3.3 were not located in this pass.
10. Non-rigid ramp. Paper: not stated. Code: the residual is multiplied by zero before the start, by the square root of linear progress during warm-up, and by one after generate_neural_gaussian.py. The opacity threshold ramps from 0 to 0.001 over the same schedule generate_neural_gaussian.py.
11. Photometric loss form. Paper: L1 plus lambda times an SSIM loss §3.4eq. 16. Code: (1 - 0.2) L1 + 0.2 (1 - SSIM) train.py train.py. The code uses a convex combination with an explicit 1 minus SSIM.
12. Depth loss form. Paper: the L1 norm of a normalized covariance between rendered depth and the relative prior §3.4eq. 17. Code: 1 minus the Pearson correlation between the rendered depth and 1/(prior + 200) train.py train.py train.py train.py. The offset of 200 is not in the paper.
13. Absolute depth. Paper: supports an absolute prior with an L1 term §3.4. Code: the L1 branch exists, but depth_mode is set to "relative" in both places, so that branch is never taken train.py train.py.
14. Normal loss. Paper: an L1 term plus a cosine-similarity term §3.4eq. 18. Code: L1 plus (1 minus the dot product), with the prior rescaled from [0, 1] to [-1, 1] and rotated by the camera's rotation train.py train.py.
15. Mask loss. Paper: cross-entropy on actor masks with weight 0.01 §3.4§4.2. Code: the default weight is 0.000 waymo_default.py, and the only use of the semantic output in training is an activation check train.py. No cross-entropy term was found in the training loss. The renderer applies a log to the semantic output and comments that this is for cross-entropy drivesplat_render.py.
16. Phi constants. Paper: phi_1 = 10 and phi_2 = 4 §4.2. Code: no matching constants were found, so their meaning is not resolved.
17. Background hash table. Paper: no table size, feature dimension or primes given §3.2. Code: table 2^21, feature dimension 32, primes 73856093, 19349663 and 83492791 [code: arguments/init.py#L161] [code: arguments/init.py#L173] hash_encoding.py.

Pitfalls

• Text overstates Table 1 and 2. The text says DriveSplat has the best SSIM and LPIPS for reconstruction on Waymo §4.3. Table 1 has OmniRe at SSIM 0.939 against DriveSplat's 0.937 tab. 1. On KITTI the text says DriveSplat beats all baselines in reconstruction §4.3, but OmniRe's reconstruction PSNR is 28.68 against 28.59 tab. 2. Use the tables, not the prose.
• Sequence count differs. The text in §4.3 says "two select sequences" for the non-rigid comparison §4.3. The ablation in §4.6 says four sequences §4.6. The subset behind tab. 3 is unclear.
• Same name, different numbers. DriveSplat's PSNR is 33.83 in several tables and 30.96 in others, because the subsets differ. Compare like with like.
• Depth-loss form. Minimizing the paper's |correlation| would push the correlation toward zero, which does not match a prior that should agree with the rendered depth. The code minimizes 1 minus the correlation, which is smallest at full agreement. Implement the code's form and check the sign.
• Normal cosine term. The paper says "cosine similarity loss" §3.4 and gives no separate formula eq. 18. The code uses 1 minus the dot product. Use that.
• Background decisions are not fully specified. The paper leaves the scale ratio, the LOD scale, the table size, the level count and the feature dimension unstated §3.2. The code's values are in the Hyperparameters table, but they are code defaults, not paper claims.
• Learnable cuts are not in the code path we traced. Build the partition as hard labels from the chosen percentile or mixture method, and do not implement learnable cuts unless you verify them elsewhere.
• Depth and normal losses are gated. They run only after iteration 100 in the code, and the paper does not say so train.py train.py.
• Non-rigid schedule. Paper gives no schedule. Code uses a square-root ramp and a default of zero starting iteration in the arguments, but the renderer falls back to 3000 when the attribute is missing [code: arguments/init.py#L138-L139] generate_neural_gaussian.py.
• Reference environment is narrow. The README asks for Python 3.7, PyTorch 1.13, CUDA 11.7, a modified rasterizer, and a COLMAP binary on PATH; it warns against replacing the rasterizer with the upstream package README.md README.md.
• Data prep is a large step. Waymo needs gsutil, TFRecord conversion, LiDAR depth, sky masks via GroundingDINO and SAM, and monocular priors. Some priors are optional and controlled by config README.md.
• Paper states cost, not the GPU for FPS. 68 minutes is on one L20 for 30K iterations §4.2. The FPS figure has no stated hardware tab. 5.
• Editing is limited. Vehicles are only partly reconstructed from one viewpoint, so edits can expose holes §5.2. The paper says future work will add image-to-3D completion §5.2.
• No license for the code. Use it only for reading and for citations. Build from the paper and this card, not by copying repository files.

Lineage

• Builds on: neural Gaussians with anchors (cited as lu2024scaffold and ren2024octree) §2; deformation fields for dynamic scenes, following Deform-GS and 4D-GS §2; depth and normal priors for Gaussians §2.
• Compared against: 3D-GS, GaussianPro, Octree-GS, 4D-GS, Deform-GS, S3Gaussian, PVG, StreetGS, OmniRe, Desire-GS, AD-GS tab. 1 tab. 2.
• Superseded by: not known from the card's content.