---
kind: concept
slug: physically-based-rendering
title: Physically based rendering
summary: >-
  Physically based rendering computes colour from a material model and light, or traces rays through a scene. Four cards cover night Gaussian-splat shading, SDF inverse rendering for relighting, raindrop ray tracing for paired data and Kubric's path-traced scene generator. Code is released for PhySG and Kubric only.
tags: [rendering, relighting, material-model, ray-tracing, path-tracing, inverse-rendering, gaussian-splatting, autonomous-driving, low-light, synthetic-data, raindrop-removal]
papers: ["2104.00674", "2203.03570", "2602.13549", "2609.37870"]
related: [gaussian-splatting, differentiable-rendering, neural-implicit-surface, novel-view-synthesis, domain-randomization, data-augmentation]
status: draft
updated_at: 2026-10-24
---

## Overview

A physically based renderer computes the colour of a surface from a material
model and the light that reaches it. The usual split is a diffuse part, which
scatters light evenly, and a specular part, which reflects it in a preferred
direction. The result is then tone-mapped into the displayable range
[arxiv:2602.13549 §3.2].

The four cards use the term for three purposes. Two shade a scene
representation: one adds shading to Gaussian splats, the other fits a shape
and material to photos so that the object can be relit. Two use ray tracing
to make training data, one for raindrop pairs and one for general synthetic
video. In each case the physics is a model, not a fixed colour. The shading
and data uses share a name, not an algorithm. Check which one a paper means
before you reuse its code or numbers.

## Variants

### Physically based shading for composite Gaussian scenes at night

The variant adds a diffuse and a specular light model to composite 3D Gaussian
Splatting scenes of driving footage. The diffuse light is one global module.
It predicts second-degree spherical-harmonics coefficients from the timestep
and the camera [arxiv:2602.13549 §3.4]. The specular light is four
anisotropic spherical-Gaussian lobes per Gaussian, combined with a simplified
Disney BRDF so that no per-ray sampling is needed [arxiv:2602.13549 §3.5].
The sum is compressed with Reinhard tone mapping before rasterisation
[arxiv:2602.13549 §3.6]. Card: [Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting](/papers/2602.13549).

The authors report that this setup targets low light, where headlights and
street lamps dominate [arxiv:2602.13549 §1] [arxiv:2602.13549 §2.1].

### Inverse rendering with spherical Gaussians

This variant recovers the parts of a physical model from photos. The input is
posed RGB images of one glossy object under static, distant light. The output
is an SDF geometry, a spatially varying diffuse albedo, one monochrome
specular spherical-Gaussian lobe, and an environment map of 128 spherical
Gaussians [arxiv:2104.00674 §3]. Shading is a closed-form integral over
products of spherical Gaussians, so the light integral needs no Monte-Carlo
sampling [arxiv:2104.00674 §3]. Because material and light are separate
outputs, the object can be relit and its material edited afterwards. The
neural view-synthesis baselines in the paper fold both into one network and
cannot do this [arxiv:2104.00674 §1] [arxiv:2104.00674 §2]. Card: [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](/papers/2104.00674).

The model assumes distant light, one material per object, and no
self-occlusion or indirect light [arxiv:2104.00674 §3]. The authors report
novel-view PSNR 35.93 dB on their synthetic set, and 33.25 dB for relighting
[arxiv:2104.00674 tab. 1]. On the real scenes, IDR is slightly better on the
image metrics, which the authors attribute to bias in the physics-based
appearance model [arxiv:2104.00674 tab. 3].

### Ray-traced synthetic raindrops for paired training data

This variant uses ray tracing to make training data, not to shade a scene. A
camera looks through a tilted glass plane with spherical-cap drops on it, at
a background plane behind. Each camera pixel is traced. Rays that miss every
drop keep the background value. Rays that pass through a drop take the
background value where they refracted to. If total internal reflection
happens on leaving a drop, the pixel is black [arxiv:2609.37870 §3]. Glass
refraction is ignored [arxiv:2609.37870 §3]. The drop regions are then
blurred, and the drop positions give the mask for free [arxiv:2609.37870 §3].
The authors use this to build about 30,000 training images from the Cityscapes
training split, with 1,525 test pairs from its test split
[arxiv:2609.37870 §3] [arxiv:2609.37870 §5.1]. Card: [Learning from synthetic photorealistic raindrop for single image raindrop removal](/papers/2609.37870).

The sampling distributions for the drop parameters are not stated, and the
paper releases no renderer or dataset [arxiv:2609.37870 §3]. An implementer
has to rebuild the ray tracer from the prose.

### Path-traced synthetic scenes with physics

Kubric is a general framework for generating labelled scenes. PyBullet
simulates rigid-body motion and collisions, and Blender's Cycles path tracer
renders the frames [arxiv:2203.03570 §3]. A worker script builds one random
scene, and many workers run in parallel on one machine or on a cluster
[arxiv:2203.03570 §3.2]. Each frame can be exported with depth, instance
segmentation, optical flow, surface normals and object coordinates, together
with object trajectories and collision events [arxiv:2203.03570 §3].
The authors state that path tracing can be several orders of magnitude slower
than rasterisation [arxiv:2203.03570 §3.2]. Card: [Kubric: A scalable dataset generator](/papers/2203.03570).

Its optical-flow use case is author-run. RAFT pretrained on Kubric reaches
an end-point error of 4.82 on KITTI-15, against 7.63 for FlyingChairs and 4.66
for AutoFlow [arxiv:2203.03570 §4.2]. The paper does not state the training
schedule for that comparison [arxiv:2203.03570 §4.2]. Asset licences decide
what can be redistributed. The authors say that non-CC0 asset licences
blocked publishing their pose-estimation data [arxiv:2203.03570 §4.4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Night or low-light driving footage, with per-sequence training affordable | Shading variant | Reported gains over 3DGS, StreetGaussians and OmniRe on 11 scenes per dataset [arxiv:2602.13549 tab. 1] [arxiv:2602.13549 tab. 2] |
| Many sequences, or a single model across them | Not covered | The authors name per-sequence optimisation as a scalability limit [arxiv:2602.13549 §5] |
| No monocular normal estimates available | Not covered | The normal prior and depth-normal terms depend on them [arxiv:2602.13549 §3.7] |
| Daytime scenes, or static scenes with no need for relighting | Not covered | The paper evaluates low-light scenes only [arxiv:2602.13549 §4.1] |
| Strict real-time budget | Check first | The paper says real-time rendering is kept, but reports no frame rate or memory [arxiv:2602.13549 §1] |
| Relight or edit the material of one glossy object from photos under one static light | Inverse rendering | It is the only card that outputs shape, material and light as separate parts [arxiv:2104.00674 §3] |
| Matte (Lambertian) object, or lighting that is not distant | Not covered | Light and albedo are ambiguous without specular highlights [arxiv:2104.00674 §4] |
| Many objects, or a tight training budget | Check first | The paper trains 250k iterations on one 12 GB GPU per scene [arxiv:2104.00674 §3] |
| Paired clean and rainy images, with exact drop masks, for raindrop removal | Ray-traced data variant | Synthetic pairs with masks; the detection network reaches average precision 0.9973 on the synthetic set [arxiv:2609.37870 §5.1] |
| Real rainy images only, no synthetic data | Not covered | The paper's real-image test set is the dataset of Qian et al., where the authors' model is roughly on par with Qian et al. and not ahead of it [arxiv:2609.37870 §5.2] |
| Synthetic video with physical motion and dense ground truth for pretraining | Path-traced framework | It exports flow, depth, segmentation and object trajectories [arxiv:2203.03570 §3] |
| Limited compute for rendering | Check first | The authors say Kubric currently needs substantial compute [arxiv:2203.03570 §5] |

Novel-view gains are smaller than reconstruction gains. On Waymo novel view,
SSIM is second to StreetGaussians [arxiv:2602.13549 tab. 1].

On the synthetic raindrop set, the full model beats Qian et al. in PSNR,
41.29 dB against 37.79 dB. On real images the rough-mask model is 30.17 dB
against Qian et al.'s 30.82 dB [arxiv:2609.37870 tab. 1] [arxiv:2609.37870 tab. 2].

## Implementation sketch

The loop below is our own rendering of the per-frame shading in
[arxiv:2602.13549 §3.2] to [arxiv:2602.13549 §3.6]. It covers the shading
variant of the Gaussian card only. The other three variants have a different
core: the inverse-rendering card sphere-traces an SDF per ray and then shades
the surface point in closed form [arxiv:2104.00674 §3], the raindrop card has
no shading step [arxiv:2609.37870 §3], and the Kubric card leaves shading to
Blender's path tracer [arxiv:2203.03570 §3]. Helper names are ours. The paper
defines the terms they stand for only in the equations cited next to them.
Two parts are not stated in the paper and are left as helpers: the
roughness-to-width mapping and the Fresnel input.

```pseudocode
# Shade each Gaussian, then blend as in plain 3DGS  [§3.2]
# Inputs: gaussians (N Gaussians), camera, timestep
# Output: LDR image

def shade_frame(gaussians, camera, timestep):
    # Global diffuse light as degree-2 SH coefficients  [§3.4]
    sh = global_light_module(timestep, camera)
    for g in gaussians:
        # Lambertian diffuse term from albedo, normal and SH  [§3.4] [eq. 4]
        diffuse = diffuse_from_sh(g.albedo, g.normal, sh)
        # Specular: BRDF factors times four ASG lobes  [§3.5] [eq. 7]
        specular = brdf_factors(g, camera) * asg_lobes(g, camera)
        # Sum in HDR, then compress per Gaussian  [§3.6] [eq. 8] [eq. 9]
        hdr = diffuse + specular
        g.color = hdr / (1 + hdr)
    # Alpha-blend the LDR colours with the standard rasteriser  [§3.1.1]
    return rasterize(gaussians, camera)
```

## Contradictions

None known between the papers, since they report on different tasks and do
not compete on the same quantity. Two within-paper points are worth reading
before you cite a number. In the shading paper, the full model's numbers
repeat across three ablation tables, and the paper does not say whether they
come from one run [arxiv:2602.13549 §4.6.1] [arxiv:2602.13549 §4.6.3]. In the
inverse-rendering paper, IDR has the lower surface-normal error on the
synthetic set, 2.207 degrees against 2.528, and the authors describe their
geometry as close to IDR's [arxiv:2104.00674 tab. 2].

## Papers

- [Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting](/papers/2602.13549): physically based diffuse and specular shading added to composite Gaussian splatting for night driving scenes.
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](/papers/2104.00674): recovers SDF geometry, albedo and a spherical-Gaussian environment map from posed photos, with closed-form shading for relighting.
- [Learning from synthetic photorealistic raindrop for single image raindrop removal](/papers/2609.37870): ray-traces adherent raindrops onto Cityscapes images to build paired training data for a three-stage removal network.
- [Kubric: A scalable dataset generator](/papers/2203.03570): open-source framework that pairs PyBullet physics with Blender path tracing to generate labelled synthetic scenes at scale.