Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting

In plain words

Driving simulators rebuild real street scenes from camera footage, so that a self-driving system can be tested in them. Most rebuilds are trained on daytime images. At night, headlights, street lamps and reflective signs make the light very uneven, and the rebuilds lose detail or get smeared.

The authors add a model of light to 3D Gaussian Splatting, a fast method that describes a scene as many small Gaussian blobs. Each blob gets a colour, a roughness, a metalness and a facing direction, the kind of properties a game engine uses to shade a surface. The light is split into two parts. Soft light that falls across the whole scene is predicted by a network from the camera position and the time. Sharp glints from lamps and shiny surfaces are modelled per blob with a few directional lobes. The two parts are added, compressed into the displayable brightness range, and then drawn as usual.

The method is tested on 11 night scenes from each of two driving datasets. The gains are the authors' own measurements. In their ablations, removing either light part makes the results worse.

TL;DR

• Adds physically based shading to composite 3D Gaussian Splatting scenes, so that nighttime driving footage can be reconstructed §3.2.
• Diffuse light comes from a global second-degree spherical-harmonics module. Specular light comes from four anisotropic spherical Gaussian lobes per Gaussian, constrained by a simplified Disney BRDF §3.4 §3.5.
• Author-reported: best reconstruction PSNR, SSIM and LPIPS against 3DGS, StreetGaussians and OmniRe on 11 Waymo and 11 nuScenes night scenes tab. 1 tab. 2.
• Novel-view gains are smaller. On Waymo, novel-view SSIM is second to StreetGaussians tab. 1. No official code was found, and the paper reports no frame rate or training cost.

Problem

Driving-scene reconstruction methods are mostly tested under normal light, and they degrade at night, where ambient light is low and sharp sources such as headlights and street lamps dominate §1 §2.1. Earlier relighting methods for 3DGS target static scenes. They sample environment maps for each ray and use heavier shading pipelines than vanilla 3DGS §2.2. The authors aim to improve nighttime reconstruction while keeping real-time rendering §1.

The method optimises each driving sequence separately (a per-sequence paradigm). The authors name this as a scalability limit for large digital-twin simulation §5.

Method

Overview §3.2 fig. 1. For a given camera and timestep, each Gaussian's appearance is split into a diffuse part and a specular part. A global lighting module predicts spherical-harmonics (SH) coefficients for the diffuse light. Per-Gaussian anisotropic spherical Gaussians (ASGs) model the incident specular light. Both parts are combined with per-Gaussian material attributes into HDR radiance, which is tone-mapped to LDR with Reinhard and then rasterised with the standard 3DGS rasteriser §3.2 §3.6.

Scene graph §3.1.2. The scene has three parts: a static background, dynamic actors and a sky map. Background Gaussians start from LiDAR points. Rigid vehicles are Gaussian nodes moved by their trajectory T_v in SE(3). Pedestrians are SMPL Gaussians, and cyclists and distant pedestrians are deformable Gaussians. The sky is an explicit cubemap sampled along the viewing direction. The graph follows earlier composite-graph work, and the paper does not describe its training in detail §3.1.2.

Gaussian base §3.1.1. Each Gaussian has an opacity, a mean, a covariance built from a rotation quaternion and a scale, and a colour. Pixel colour is alpha-blended over depth-ordered Gaussians eq. 1 eq. 2.

Material model §3.3. Each Gaussian carries an albedo b in [0, 1]^3, a roughness r in [0, 1], a metalness m in [0, 1] and a surface normal n in R^3. The paper follows the parameterisation of RelightableGS §3.3. The rendering equation is the reference model eq. 3.

Diffuse light: global lighting module §3.4. An MLP takes a normalised timestep embedding and a camera embedding and produces a latent code. One linear head per SH level (l = 0, 1, 2) turns it into coefficients §3.4 §4.2. The Lambertian diffuse term is approximated as a weighted sum over the SH coefficients eq. 4:

L_d = \frac{b}{\pi}\sum_{l=0}^{2}A_{l}\sum_{m=-l}^{l}c^{l}_{m}Y^{l}_{m}(n) eq. 4

Here A_l are cosine-kernel convolution factors, Y^l_m is the real SH basis evaluated at the Gaussian normal, and c^l_m are the coefficients from the global module §3.4.

Specular light: ASG lobes §3.5. Each Gaussian has four ASG lobes. An ASG is defined by a lobe frame [x, y, z], sharpness values lambda and mu, and an amplitude c eq. 5:

G(v;[x,y,z],[\lambda,\mu],c) = c\,S(v;z)\,e^{-\lambda(v\cdot x)^2-\mu(v\cdot y)^2} eq. 5

The specular term uses a simplified Disney BRDF eq. 6:

f_s(\omega_o,\omega_i) = \frac{D(h;r)\,F(\omega_o,h;b,m)\,G(\omega_i,\omega_o,h;r)}{(n\cdot\omega_i)(n\cdot\omega_o)} eq. 6

Because the normal distribution is modelled as a spherical Gaussian, the product of the distribution with the ASG lighting can be written in closed form. The specular radiance is then eq. 7:

L_s = G(\omega_i,\omega_o,h;r)\,F(\omega_o,h;b,m)\sum_i A_i\,s_i\,\mathrm{ASG}_i\!\left(\omega_r,[x_i,y_i,z_i],\left[\tfrac{\nu\lambda_i}{\nu+\lambda_i},\tfrac{\nu\mu_i}{\nu+\mu_i}\right],\,a_{ndf}\tfrac{\pi}{\sqrt{(\nu+\lambda_i)(\nu+\mu_i)}}\right) eq. 7

Here nu and a_ndf are the concentration and amplitude of the normal distribution, and omega_r is the reflection direction. No per-ray sampling is needed §3.5.

Sum and tone mapping §3.6. Diffuse and specular radiance are added in HDR eq. 8. The result is compressed with Reinhard tone mapping eq. 9. The LDR value is the per-Gaussian colour c_i used for rasterisation §3.6.

L_{HDR} = L_d + L_s eq. 8

L_{LDR} = \frac{L_{HDR}}{1 + L_{HDR}} eq. 9

Normals and regularisers §3.7. Accurate normals matter for lighting. The paper takes a normal map prior N from an off-the-shelf monocular model §3.7. During rasterisation, per-Gaussian normals are alpha-blended into a rendered normal map eq. 10:

\hat{N} = \frac{\sum_{i\in M} n_i\,\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j)}{\sum_{i\in M}\alpha_i\prod_{j=1}^{i-1}(1-\alpha_j)} eq. 10

A normal loss against the prior eq. 11, a per-view confidence weight from the agreement between depth-derived normals and the prior eq. 12, and a confidence-weighted depth-normal loss eq. 13 follow the depth-normal regulariser of VCR-Gaus §3.7:

L_n = \lVert\hat{N}-N\rVert_1 + (1-\hat{N}\cdot N) eq. 11

w = \exp\!\left(\frac{\hat{N}_d\cdot N-1}{\gamma}\right) eq. 12

L_{dn} = w\,\big(\lVert\hat{N}_d-N\rVert_1 + (1-\hat{N}_d\cdot N)\big) eq. 13

The total objective combines RGB, D-SSIM and depth-normal terms eq. 14:

L_{total} = w_{rgb}L_{rgb} + w_{\text{D-SSIM}}L_{\text{D-SSIM}} + w_{dn}(L_d + L_n) eq. 14

The loop below is our rendering of the per-frame shading in §3.2 to §3.6. Helper names are ours. The paper defines the terms they stand for only in the equations cited next to them.

# Shade one frame: per-Gaussian light, then rasterise  [§3.2]
# Inputs: gaussians (N Gaussians), camera, timestep
# Output: LDR image

def render_frame(gaussians, camera, timestep):
    # Global diffuse light: SH coefficients from camera and time  [§3.4]
    sh_coeffs = global_sh_module(camera, timestep)
    for g in gaussians:
        # Diffuse: Lambertian term from SH coefficients  [§3.4] [eq. 4]
        diffuse = diffuse_term(g.albedo, g.normal, sh_coeffs)
        # Specular: geometry and Fresnel terms times the four ASG lobes
        # [§3.5] [eq. 7]
        specular = geometry_fresnel(g) * asg_sum(g, camera)
        # Add in HDR, then Reinhard to LDR, per Gaussian  [§3.6] [eq. 8, 9]
        hdr = diffuse + specular
        g.color = hdr / (1 + hdr)
    # Alpha-blend LDR colours with the standard rasteriser  [§3.1.1] [eq. 2]
    return rasterize(gaussians, camera)

Hyperparameters

Values are from the paper unless noted. "Not stated" means the paper does not give a value, and no official code was found to check it (see Paper vs code).

Results

Author-reported. Setup: 11 low-light scenes per dataset, baselines 3DGS, StreetGaussians and OmniRe, PSNR, SSIM and LPIPS §4.1 §4.3. The paper does not define the "Reconstruction" split, and it does not say how many runs were made or report variance.

Waymo §4.4 tab. 1:

nuScenes §4.5 tab. 2:

The paper reports about 1.0 dB PSNR over OmniRe on nuScenes reconstruction §4.5. On Waymo novel view, the paper calls the PSNR and SSIM gains moderate and points to the LPIPS gain instead §4.4.

Ablation: lighting decomposition §4.6.1 tab. 3:

Ablation: SH vs ASG for specular light §4.6.2 tab. 4:

Ablation: BRDF constraint §4.6.3 tab. 5:

Without the BRDF constraint the paper reports a 1.0 dB PSNR drop with much higher LPIPS §4.6.3. The full-model row is identical in tab. 3, tab. 4 and tab. 5. The paper does not say whether these come from one run.

Qualitative figures (described, not reproduced). fig. 2 and fig. 3 show Waymo and nuScenes crops in the order ground truth, this paper, OmniRe, StreetGaussians, 3DGS. The paper reports clearer vehicles, headlight effects on the road and building windows §4.4 §4.5. fig. 4 shows that without ASGs, vehicle detail is lost, and without global SH, headlight-to-ground interaction and cast shadows are wrong §4.6.1. fig. 5 shows that SH specular maps give exaggerated, overly diffuse highlights §4.6.2. fig. 6 compares BRDF-constrained and unconstrained ASG evaluation §4.6.3. fig. 7 shows the albedo, diffuse, specular and final-render decomposition §4.6.4.

Paper vs code

No official code found. The search covered three sources:

1. Paper links. The TeX source has no code or project URL. The arXiv abstract page lists the comment "ICRA 2026" and no code or project link (checked 2026-10-24).
2. Project page. None linked.
3. GitHub repository search (WebFetch, search API). Queries "nighttime gaussian splatting driving", "physically based gaussian driving night", "nighttime 3dgs" and "nighttime+gaussian org:huawei-noah" returned zero results. Queries "low-light gaussian splatting autonomous", "physically based gaussian splatting", "relightable driving gaussian" and "huawei-noah nighttime" returned HTTP 403, so this step is incomplete.

Because no code is recorded, there are no code citations. Every point below is checked against the paper alone.

Pitfalls

• LPIPS weight with no LPIPS term. §4.2 sets omega_LPIPS = 0.025 §4.2, but the total loss has no LPIPS term eq. 14. Whether LPIPS is used in training is not stated.
• Which normal and depth terms are in the total loss. §3.7 defines the view-consistent L_dn eq. 13, but the total loss uses L_d + L_n eq. 14. L_d is never defined. The weight w_dn and the sharpness gamma are not stated.
• Tone mapping is per Gaussian, before blending. Reinhard is applied to each Gaussian's HDR colour §3.6, and the result is alpha-blended. The final pixel is therefore not the Reinhard value of the blended HDR radiance. The paper does not discuss this choice.
• Symbols in the specular term are undefined. A_i, s_i, nu, a_ndf and omega_r in eq. 7 have no stated values or formulas. The saturation S(v; z) in eq. 5 has no closed form. The D, F and G terms in eq. 6 have no explicit formulas. The mapping from roughness to the distribution width, and the mixing of albedo and metalness into F0, are not stated.
• Cosine-kernel factors. A_l in eq. 4 are described but not given.
• Coefficient layout. §3.4 says the module produces "three sets" of coefficients, one per SH level §3.4. eq. 4 sums over nine basis functions (l = 0 to 2, m = -l to l). Whether c^l_m is a scalar or per colour channel is not stated.
• Global module under-specified. Embedding sizes, the hidden width of the eight layers, the activation, the timestep normalisation and the camera encoding are not stated §3.4 §4.2.
• Normal prior. [wang2025moge] is cited for the normal prior, but the bib entry is MoGe-2 (arXiv:2507.02546). The checkpoint and output type are not stated. The method for deriving N_d from the rendered depth is not stated.
• Division by summed alpha. eq. 10 divides by the summed alphas, which is zero where no Gaussian covers a pixel. The paper gives no guard.
• Learning rates are partial. Only normals, roughness, metalness, lobe axes, the stretch parameters and amplitude have rates §4.2. The rate for positions, rotations, scales, albedo and the global MLP is not stated. The paper does not say whether the lobe sharpness pair is the "stretch" pair.
• Training scope. Batch size, densification, pruning, pose refinement, Gaussian count and image resolution are not stated. The composite scene graph is inherited from earlier work §3.1.2.
• Sky lighting. The sky is a cubemap §3.1.2. The paper does not say whether the sky receives the lighting model or what cubemap resolution it uses.
• No explicit light sources. Headlights and street lamps are not modelled as emitters. Their effects are carried by the SH and ASG terms, and the paper describes the specular term as capturing self-illuminating objects §4.6.4.
• Metric citations look swapped. §4.3 cites zhang2018unreasonable for SSIM and wang2004image for LPIPS. The bib entries are the reverse. Check before citing either metric.
• "Reconstruction" is undefined. The paper does not say whether the reconstruction columns are scored on training frames. Only the novel-view split is defined: every eighth frame §4.1.
• Abstract overstates the Waymo result. The abstract claims the method outperforms the state of the art. On Waymo novel-view SSIM it is second: 0.720 against StreetGaussians' 0.722 tab. 1.
• Real-time is claimed, not measured. §1 says real-time rendering is maintained, but the paper reports no frame rate, render time, memory or training time.
• No variance. The paper reports single values, with no seeds, runs or error bars.
• Scene selection is qualitative. The 11 low-light scenes per dataset are chosen by the criteria in §4.1, and no quantitative threshold is given.
• Same numbers across tables. The full model's numbers repeat in three ablation tables tab. 3 tab. 4 tab. 5. The paper does not say whether this is one run reused.

Lineage

• Builds on: 3D Gaussian Splatting [kerbl20233d] §3.1.1; the composite Gaussian scene graph of OmniRe [chen2025omnire], StreetGaussians [yan2024street], ArmGS [wu2025armgs], DrivingGaussian [zhou2024drivinggaussian] and others §3.1.2; the material model of RelightableGS [gao2024relightable] §3.3; the simplified Disney BRDF of NeILF [yao2022neilf] and RelightableGS §3.5; anisotropic spherical Gaussians [xu2013anisotropic] §3.5; the depth-normal regulariser of VCR-Gaus [chen2024vcr] §3.7.
• Differs from relighting work for static scenes: RelightableGS and NeILF sample environment maps per ray §2.2 §3.4. DarkGS and LL-Gaussian target static low-light scenes §2.2. Spec-Gaussian feeds ASGs to an MLP for RGB, whereas this paper evaluates ASGs as incident light §3.5.
• Compared against: 3DGS [kerbl20233d], StreetGaussians [yan2024street] and OmniRe [chen2025omnire] §4.3 tab. 1 tab. 2.