Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing

In plain words

A normal 3D Gaussian scene shows the lighting it was photographed under, baked into its colours. If you move the lights, nothing changes. This paper asks what a scene of small 3D blobs looks like if each blob also records what its surface is made of and how it reflects light.

Each blob gets a surface direction, a base colour (how it reflects light in every direction), a roughness (how blurred its reflections are), and a small amount of light that reaches it from other blobs. The scene is fitted to a set of ordinary photos, so the blobs settle on values that reproduce those photos under the original light.

Once the material and light are separated, the scene can be lit differently. The paper also adds a way to tell which blobs block the light from each direction, which gives shadows. Because the scene is still a set of points, it can be moved, combined with other objects, and relit without rebuilding a mesh.

TL;DR

• Adds a surface normal, an albedo and a roughness to each 3D Gaussian, plus incident light split into one shared environment map and a per-Gaussian spherical-harmonic term §3.3.
• Two training stages: first the geometry and normals with depth and normal regularisers, then the materials and light with the geometry fixed §5.1 §4.2.
• A bounding-volume-hierarchy ray tracer over Gaussians gives per-Gaussian visibility for shadows; visibility is precomputed once per stage 1 result §4.1 §4.2.
• Author-reported: NeRF-synthetic PSNR 31.22 (3DGS 33.88, which is not relightable); Synthetic4Relight relighting PSNR 31.00 and novel-view PSNR 36.80 tab. 1 tab. 2.
• The official code is non-commercial research only. Implement from the paper, not from the code, for any commercial use. See "Paper vs code" and "Pitfalls".

Problem

3D Gaussian Splatting renders novel views well, but its colours are fixed for the lighting seen in training, so it cannot be relit §1. Shadows and reflections need ray tracing, which is hard with points, because a point has no clear surface to hit §1. Earlier inverse-rendering methods for neural fields or meshes handle relighting but are hard to edit, and point-based pipelines did not combine relighting with editing §4.3. The paper targets static scenes; large scenes make the per-point cost grow quickly §6.

Method

Preliminaries. A scene is a set of Gaussians with mean, covariance (stored as rotation and scale), opacity and spherical-harmonic colour. Colour is alpha-blended front to back from projected 2D Gaussians §3.1 eq. 1 eq. 2.

Stage 1: geometry.

1. Give each Gaussian a normal vector and learn it by back-propagation. The paper starts from a random vector §3.2; the code starts from the point cloud's normals, and substitutes random vectors when those are all zero gaussian_model.py dataset_readers.py.
2. Render a depth and a normal per pixel as weighted averages over the Gaussians along the ray, with weights T_i α_i normalised over the ray §3.2 eq. 3.
3. Penalise the difference between the rendered normal and a pseudo-normal computed from the rendered depth §3.2 eq. 4.
4. Add an extra densification rule: split or clone a Gaussian when the gradient of its normal exceeds a threshold, in addition to the standard 3DGS rule §3.2.
5. Penalise the depth variance, which is the rendered squared depth minus the squared rendered depth §3.2 eq. 5.
6. When an object mask is available, apply a binary cross-entropy between the mask and the accumulated opacity §3.2 eq. 6.
7. Run the standard 3DGS photometric loss together with these terms §5.1 eq. 16.

Stage 2: materials and light, geometry fixed.

1. Compute the rendering equation per Gaussian, not per pixel §3.3 eq. 7. The result is a colour c'(ω_o) for each Gaussian, which is then alpha-blended like the original colour §3.3.
2. Use a simplified Disney-style BRDF: a Lambert diffuse term with albedo b / π, and a specular term with a normal distribution, a Fresnel term and a geometry term that depend on roughness r §3.3 eq. 8. The paper does not give the forms of D, F and G.
3. Represent incident light at a Gaussian as visibility times a shared direct term, plus a per-Gaussian indirect term §3.3 eq. 9. The direct term is one global 16×32 environment map. The indirect term is a third-order spherical-harmonic expansion §3.3.
4. Estimate the integral with N_s = 64 directions per Gaussian, placed by a Fibonacci spiral over the hemisphere around the normal §3.3 §5.1. Each sample is weighted by its share of the hemisphere §3.3 eq. 10.
5. Add regularisers: a near-white light prior on the incident light §3.4 eq. 11, and smoothness of roughness, normals and albedo that is weaker across strong image edges §3.4 eq. 12.
6. Use a total loss of photometric terms plus the regularisers §5.1 eq. 17.

Visibility by point-based ray tracing. The Gaussians are placed in a binary radix tree (BVH), with each leaf bounding a Gaussian and each inner node bounding its children §4.1. A ray is approximated as hitting each Gaussian at the point where that Gaussian's contribution peaks along the ray §4.1 eq. 13 eq. 14. The transmittance is multiplied along the ray, and the walk stops early once the transmittance falls below a threshold T_min, which the paper does not give §4.1 eq. 15.

Precomputation and pipeline. After stage 1, the visibility of every Gaussian is computed for its N_s hemisphere directions with the ray tracer, and the geometry is then frozen. Stage 2 uses these precomputed visibilities, so the ray tracer is not called during training §4.2. For relighting, the scene is composed with other objects, the visibilities are recomputed for the new layout, and the original per-object indirect light is discarded §4.3.

# Shade one Gaussian by Monte Carlo over its N_s directions  [§3.3] [eq. 10]
def gaussian_pbr_color(normal, albedo, roughness, view_dir, dirs, vis):
    # Direct light scaled by visibility, plus indirect SH  [eq. 9]
    light = vis * env_map(dirs) + indirect_sh(dirs)
    # Clamp the cosine to the upper hemisphere  [eq. 10]
    cos_term = clamp(dot(normal, dirs), min=0)
    # Lambert diffuse plus specular BRDF  [eq. 8]
    brdf = albedo / pi + specular(normal, view_dir, dirs, roughness)
    # Each sample covers 2*pi of solid angle  [code: utils/graphics_utils.py#L33]
    # Monte Carlo estimate over the hemisphere  [eq. 10]
    return sum(brdf * light * cos_term) * (2 * pi / len(dirs))

# Visibility of one direction from one Gaussian, by BVH walk  [§4.1] [eq. 14]
def visibility(origin, direction, gaussians):
    # Start with full transmittance  [§4.1] [eq. 15]
    transmittance = 1.0
    for g in bvh_leaves_hit(origin, direction):
        # Skip faint Gaussians and those facing away from the ray
        # [code: bvh/src/trace.cu#L233-L235]
        if g.opacity < 1.0 / 255.0 or faces_away(g, direction):
            continue
        # Place the hit where the contribution peaks  [eq. 14]
        t = peak_parameter(g, origin, direction)
        # Skip hits that are too close to the origin  [code: bvh/src/trace.cu#L238]
        if t < 0.01:
            continue
        alpha = g.opacity * exp(gaussian_power(g, origin + t * direction))
        transmittance *= 1.0 - alpha
        # Zero once transmittance falls below 0.9  [code: bvh/src/trace.cu#L251-L254]
        if transmittance < 0.9:
            return 0.0
    return transmittance

Key equations (reproduced from the paper; numbering as in the compiled paper):

\mathbf{c}'(\boldsymbol{\omega}_o) = \sum_{i=0}^{N_s} \big(f_d + f_s(\boldsymbol{\omega}_o, \boldsymbol{\omega}_i)\big)\, L_i(\boldsymbol{\omega}_i)\, (\boldsymbol{\omega}_i \cdot \mathbf{n})\, \Delta\boldsymbol{\omega}_i eq. 10

L_i(\boldsymbol{\omega}_i) = V(\boldsymbol{\omega}_i)\, L_{direct}(\boldsymbol{\omega}_i) + L_{indirect}(\boldsymbol{\omega}_i) eq. 9

t_j = \frac{(\boldsymbol{\mu} - \boldsymbol{r}_o)^\top \boldsymbol{\Sigma}\, \boldsymbol{r}_d}{\boldsymbol{r}_d^\top \boldsymbol{\Sigma}\, \boldsymbol{r}_d} eq. 14

\ell_{stage1} = \lambda_1 \ell_1 + \lambda_{ssim} \ell_{ssim} + \lambda_n \ell_n + \lambda_{s,n} \ell_{s,n} + \lambda_O \ell_O + \lambda_u \ell_u eq. 16

\ell_{stage2} = \lambda_1 \ell_1 + \lambda_{ssim} \ell_{ssim} + \lambda_l \ell_l + \lambda_{s,b} \ell_{s,b} + \lambda_{s,r} \ell_{s,r} eq. 17

Hyperparameters

Values stated in the paper are cited to the paper. Values taken from the released code are marked "code" and cited to the file. Where the two differ, see "Paper vs code".

Results

Author-reported. Setups: NeRF-synthetic (novel-view synthesis, scene averages) §5.2 tab. 1; Synthetic4Relight (novel-view synthesis, relighting, albedo, roughness) §5.3 tab. 2; qualitative relighting on Mip-NeRF 360 scenes §5.3 fig. 6. Higher PSNR and SSIM are better; lower LPIPS and MSE are better.

NeRF-synthetic, novel-view synthesis tab. 1

Synthetic4Relight tab. 2

Reading the table: the paper's albedo PSNR is below TensorIR and Nvdiffrec-MC, and its roughness error is above InvRender's. Its albedo SSIM and LPIPS are the best in the table, which is the claim made in the text §5.3. The albedo scale is fixed against ground truth before comparing (see "Pitfalls") §5.3.

Qualitative results (no numbers): NeRF-synthetic renders, normals and ambient-occlusion maps fig. 4; Synthetic4Relight decomposition and relighting fig. 5; Mip-NeRF 360 garden and kitchen relit under two extra maps fig. 6; ablations of normal densification, full versus simplified light, and the depth-variance loss fig. 7.

Not in this card: the supplementary DTU novel-view comparison §5.4 (supplement) is referenced in the source but its table file is absent from the archive, so no DTU number is recorded here.

Paper vs code

Searched: the arXiv abstract and the LaTeX source contain no code link. A GitHub repository search for the method name returned NJU-3DV/Relightable3DGaussian. The owner is the NJU-3DV organisation. Its README lists the paper's authors with Nanjing University affiliations (Jian Gao, Youtian Lin, Hao Zhu, Xun Cao, Yao Yao) and links the project page at nju-3dv.github.io. It is therefore accepted as the official repository. Cloned at a102f14c7e34b7bd5e421990e6f1593d39ebab83.

Licence: non-commercial. The root LICENSE.md is the Gaussian-Splatting License (Inria and MPII). It allows research use only and forbids commercial use LICENSE.md LICENSE.md. This card cites the code for comparison only. No code is quoted, and the code must not be used in any commercial product. Reimplementing the method from the paper is a separate matter; the paper is CC BY 4.0.

Discrepancies, each with both citations:

• Mask term weight. Paper: λ_O = 0.01 eq. 16 §5.1. Scripts pass 0.1 run_nerf.sh run_syn4.sh. Code default is 0 [code: arguments/init.py#L110].
• Normal smoothness weight in Synthetic4Relight. Paper: 0.01 eq. 16. Script: 0.02 run_syn4.sh. Code default is 0 [code: arguments/init.py#L118].
• Normal consistency. Paper: the 2-norm of the difference eq. 4. Code: mean squared error, averaged over elements, with the pseudo-normal detached render.py.
• Depth-variance loss. Paper: the variance itself, weight 0.01 eq. 5 §5.1. Code: the mean of the square root of the variance (its standard deviation), with a weight that grows as 0.01 × min(10^(iteration/5000), 100). That is 0.01 at the start, 0.1 at 5,000 steps, 1.0 at 10,000 steps, then held at 1.0 render.py.
• Stage 2 light regulariser. Paper: weight 0.0001 eq. 17 §5.1. Both scripts pass 0.01 run_nerf.sh run_syn4.sh. Code default is 0 [code: arguments/init.py#L127].
• Stage 2 smoothness weights. Paper: λ_s,b = λ_s,r = 0.01 eq. 17. NeRF script: both 0 run_nerf.sh. Synthetic4Relight: base colour 1, roughness 0.5 run_syn4.sh.
• Stage 2 geometry lock. Paper: geometry is locked §4.2. Synthetic4Relight locks it run_syn4.sh. NeRF script does not: position, normal, SH, opacity, scale and rotation keep non-zero rates run_nerf.sh.
• Stage 2 length on Synthetic4Relight. Paper: 10,000 steps §5.1. Script: 50,000 total from the 30,000 checkpoint, i.e. 20,000 new steps run_syn4.sh train.py.
• Stage 2 extra terms. The paper's Eq. 17 has one L1 and one SSIM pair. The code adds the photometric loss on the blended rendering and a second L1 and SSIM on the PBR image, with weight λ_pbr = 1 neilf.py [code: arguments/init.py#L126]. The code also has a light smoothness term and an environment-map total-variation term, which the paper does not have neilf.py.
• Light regulariser form. Paper: a sum over channels of each value minus the channel mean eq. 11. Code: an L1 deviation of the diffuse light from its channel mean, applied to the sample-averaged transport term neilf.py. The printed sum is identically zero, so the code's form is the working one.
• Smoothness form. Paper: the magnitude of the roughness gradient times exp of minus the image-gradient norm eq. 12. Code: per-channel absolute gradients weighted by exp of minus the absolute image gradient, summed over the two spatial directions loss_utils.py neilf.py. A norm-based variant is defined in the same file and imported, but no call uses it neilf.py.
• Stage 1 normal smoothness. The paper does not define L_{s,n}. Code: edge-aware smoothness of the rendered normal, guided by the ground-truth image render.py.
• Normal gradient in the BRDF. The code passes the normal detached into the rendering equation during training, so the PBR shading term does not send gradients to the normal neilf.py. The paper does not say this.
• Roughness parameterisation. Paper: roughness r in [0, 1] §3.3. Code: output in [0.09, 0.99] gaussian_model.py; GGX uses α = r² neilf.py. The geometry term uses k = (α + 2r + 1)/8 neilf.py.
• Albedo parameterisation. Paper: albedo in [0, 1]³ §3.3. Code: [0.03, 0.80] per channel gaussian_model.py.
• Fresnel. Paper: a term F with no form given §3.3. Code: fixed F0 = 0.04 with a 2^(polynomial in V·H) approximation neilf.py.
• Fibonacci sampling. Paper: Fibonacci sampling over the hemisphere §3.3. Code: the elevation is clamped to at least 10 degrees, so about 17 % of samples land on one ring at that elevation graphics_utils.py. Directions are computed once with random rotation off and reused each step gaussian_model.py.
• Albedo scale default. The code keeps a per-channel base-colour scale at 1 and overrides it only in the Synthetic4Relight evaluation gaussian_model.py eval_relighting_syn4.py.
• Point-peak intersection. Paper: Eq. 14 uses Σ §4.1. Code: the peak is computed with the inverse covariance, which is the correct form for the Gaussian's peak along a ray utility.cuh gaussian_model.py.
• Visibility output. Paper: visibility is a transmittance along the ray §4.1. Code: the ray returns zero as soon as transmittance drops below 0.9, and otherwise returns the leftover transmittance, so the output is close to binary trace.cu trace.cu. Back-facing Gaussians are skipped trace.cu.
• Visibility SH. The code also stores visibility as a 16-coefficient spherical-harmonic field with its own learning rate. The paper does not mention it. The default training path reads only the precomputed ray-traced values; the field is used by a fine-tuning routine that is off by default and by a separate composite renderer gaussian_model.py gaussian_model.py [code: arguments/init.py#L76] neilf_composite.py.
• Indirect light clamp. Local spherical-harmonic light is clamped to non-negative values neilf.py. The paper does not say so.
• Direct light. Paper: a shared 16 × 32 map §3.3. Code: the map is initialised randomly and reparameterised with softplus direct_light_map.py direct_light_map.py.
• Albedo scale in relighting evaluation. Paper: the albedo scale is aligned to ground truth §5.3. Code: a fixed per-object, per-channel scale is hard-coded for each Synthetic4Relight object eval_relighting_syn4.py. The alignment code prints a median ratio eval_relighting_syn4.py.

Pitfalls

• The official code is non-commercial. Implement from the paper for commercial work, and do not copy from the code. The paper itself is CC BY 4.0.
• Stage 2 lock is not uniform. Taking the NeRF script as the template leaves geometry trainable in stage 2. Taking the Synthetic4Relight script leaves it frozen. The paper says frozen §4.2.
• Visibility is computed once. Visibility, directions and areas are precomputed from the stage 1 normals and not refreshed in stage 2 train.py gaussian_model.py. Memory grows with N × 64 directions, each with a visibility value, a 3D direction and an area. The paper does not state this cost.
• T_min is not given. The paper leaves the transmittance threshold open §4.1. The code's 0.9 threshold is a hard cut-off, and the output is near-binary trace.cu. Treat it as a design choice, not a reproduction of the paper.
• Eq. 14 notation. The formula uses Σ, but the code computes the peak with Σ⁻¹ utility.cuh. Use the inverse, which gives the true peak of the Gaussian.
• Sampling is not uniform in the code. Samples below 10 degrees are clamped, so about 17 % of the directions sit on one ring rather than spread over the hemisphere graphics_utils.py. The estimator weights every sample by the same 2π, which assumes uniform sampling. Expect a different estimate near grazing angles if you follow the paper's sampling literally.
• Train length flag. The --iterations flag in the code is the absolute end step, not the number of stage 2 steps. Stage 2 starts from iteration 30,000 train.py train.py.
• Light and albedo scale ambiguity. Albedo and light scale can trade against each other. The Synthetic4Relight albedo and relighting numbers depend on the albedo scale, and the per-object constants in the code are not described in the paper eval_relighting_syn4.py §5.3.
• Depth-variance ramp. The effective depth-variance weight is 100 times larger after 10,000 steps than at the start render.py. Keep the ramp if you match the code, or treat the paper's fixed weight as the target.
• Ablation numbers are not given. The ablations (normal densification, full versus simplified light, depth uncertainty) are shown qualitatively only §5.4 fig. 7.
• Relighting in composed scenes. Composition needs visibility recomputed for the new layout. The original indirect light of each object is discarded §4.3, so inter-object light bounce is not modelled beyond the visibility term.
• Time and hardware. The timing in Table 2 has no stated GPU; the RTX 3090 line is commented out in the source §5.1 tab. 2.
• Scope. The method targets static objects. Large scenes slow optimisation because every point is shaded at every step, and the paper suggests deferred rendering as a remedy §6.
• Text versus table. In Table 1 the R3DG PSNR (31.22) is 0.02 dB below FreqPCR (31.24), and FreqPCR carries the second-best marking. The text says the method is superior to most point-based baselines tab. 1 §5.2.

Lineage

• Builds on: 3D Gaussian Splatting for the representation and projection §3.1; the Disney-style BRDF §3.3; Fibonacci sampling from NeILF §3.3; the BVH construction from Karras §4.1; the point-peak intersection approximation from Keselman et al. §4.1; the light-colour prior from NeRO §3.4; the smoothness priors from NeILF §3.4.
• Compared against: NPBG, NPBG++, FreqPCR, 3DGS, PhySG, NeILF++, Nvdiffrec, NeRFactor, Nvdiffrec-MC, InvRender and TensorIR tab. 1 tab. 2.
• Superseded by: not known from content already in this wiki.