Physically based rendering

Overview

A physically based renderer computes the colour of a surface from a material model and the light that reaches it. The usual split is a diffuse part, which scatters light evenly, and a specular part, which reflects it in a preferred direction. The result is then tone-mapped into the displayable range 2602.13549 §3.2.

The four cards use the term for three purposes. Two shade a scene representation: one adds shading to Gaussian splats, the other fits a shape and material to photos so that the object can be relit. Two use ray tracing to make training data, one for raindrop pairs and one for general synthetic video. In each case the physics is a model, not a fixed colour. The shading and data uses share a name, not an algorithm. Check which one a paper means before you reuse its code or numbers.

Variants

Physically based shading for composite Gaussian scenes at night

The variant adds a diffuse and a specular light model to composite 3D Gaussian Splatting scenes of driving footage. The diffuse light is one global module. It predicts second-degree spherical-harmonics coefficients from the timestep and the camera 2602.13549 §3.4. The specular light is four anisotropic spherical-Gaussian lobes per Gaussian, combined with a simplified Disney BRDF so that no per-ray sampling is needed 2602.13549 §3.5. The sum is compressed with Reinhard tone mapping before rasterisation 2602.13549 §3.6. Card: Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting.

The authors report that this setup targets low light, where headlights and street lamps dominate 2602.13549 §1 2602.13549 §2.1.

Inverse rendering with spherical Gaussians

This variant recovers the parts of a physical model from photos. The input is posed RGB images of one glossy object under static, distant light. The output is an SDF geometry, a spatially varying diffuse albedo, one monochrome specular spherical-Gaussian lobe, and an environment map of 128 spherical Gaussians 2104.00674 §3. Shading is a closed-form integral over products of spherical Gaussians, so the light integral needs no Monte-Carlo sampling 2104.00674 §3. Because material and light are separate outputs, the object can be relit and its material edited afterwards. The neural view-synthesis baselines in the paper fold both into one network and cannot do this 2104.00674 §1 2104.00674 §2. Card: PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting.

The model assumes distant light, one material per object, and no self-occlusion or indirect light 2104.00674 §3. The authors report novel-view PSNR 35.93 dB on their synthetic set, and 33.25 dB for relighting 2104.00674 tab. 1. On the real scenes, IDR is slightly better on the image metrics, which the authors attribute to bias in the physics-based appearance model 2104.00674 tab. 3.

Ray-traced synthetic raindrops for paired training data

This variant uses ray tracing to make training data, not to shade a scene. A camera looks through a tilted glass plane with spherical-cap drops on it, at a background plane behind. Each camera pixel is traced. Rays that miss every drop keep the background value. Rays that pass through a drop take the background value where they refracted to. If total internal reflection happens on leaving a drop, the pixel is black 2609.37870 §3. Glass refraction is ignored 2609.37870 §3. The drop regions are then blurred, and the drop positions give the mask for free 2609.37870 §3. The authors use this to build about 30,000 training images from the Cityscapes training split, with 1,525 test pairs from its test split 2609.37870 §3 2609.37870 §5.1. Card: Learning from synthetic photorealistic raindrop for single image raindrop removal.

The sampling distributions for the drop parameters are not stated, and the paper releases no renderer or dataset 2609.37870 §3. An implementer has to rebuild the ray tracer from the prose.

Path-traced synthetic scenes with physics

Kubric is a general framework for generating labelled scenes. PyBullet simulates rigid-body motion and collisions, and Blender's Cycles path tracer renders the frames 2203.03570 §3. A worker script builds one random scene, and many workers run in parallel on one machine or on a cluster 2203.03570 §3.2. Each frame can be exported with depth, instance segmentation, optical flow, surface normals and object coordinates, together with object trajectories and collision events 2203.03570 §3. The authors state that path tracing can be several orders of magnitude slower than rasterisation 2203.03570 §3.2. Card: Kubric: A scalable dataset generator.

Its optical-flow use case is author-run. RAFT pretrained on Kubric reaches an end-point error of 4.82 on KITTI-15, against 7.63 for FlyingChairs and 4.66 for AutoFlow 2203.03570 §4.2. The paper does not state the training schedule for that comparison 2203.03570 §4.2. Asset licences decide what can be redistributed. The authors say that non-CC0 asset licences blocked publishing their pose-estimation data 2203.03570 §4.4.

When to use which

Novel-view gains are smaller than reconstruction gains. On Waymo novel view, SSIM is second to StreetGaussians 2602.13549 tab. 1.

On the synthetic raindrop set, the full model beats Qian et al. in PSNR, 41.29 dB against 37.79 dB. On real images the rough-mask model is 30.17 dB against Qian et al.'s 30.82 dB 2609.37870 tab. 1 2609.37870 tab. 2.

Implementation sketch

The loop below is our own rendering of the per-frame shading in 2602.13549 §3.2 to 2602.13549 §3.6. It covers the shading variant of the Gaussian card only. The other three variants have a different core: the inverse-rendering card sphere-traces an SDF per ray and then shades the surface point in closed form 2104.00674 §3, the raindrop card has no shading step 2609.37870 §3, and the Kubric card leaves shading to Blender's path tracer 2203.03570 §3. Helper names are ours. The paper defines the terms they stand for only in the equations cited next to them. Two parts are not stated in the paper and are left as helpers: the roughness-to-width mapping and the Fresnel input.

# Shade each Gaussian, then blend as in plain 3DGS  [§3.2]
# Inputs: gaussians (N Gaussians), camera, timestep
# Output: LDR image

def shade_frame(gaussians, camera, timestep):
    # Global diffuse light as degree-2 SH coefficients  [§3.4]
    sh = global_light_module(timestep, camera)
    for g in gaussians:
        # Lambertian diffuse term from albedo, normal and SH  [§3.4] [eq. 4]
        diffuse = diffuse_from_sh(g.albedo, g.normal, sh)
        # Specular: BRDF factors times four ASG lobes  [§3.5] [eq. 7]
        specular = brdf_factors(g, camera) * asg_lobes(g, camera)
        # Sum in HDR, then compress per Gaussian  [§3.6] [eq. 8] [eq. 9]
        hdr = diffuse + specular
        g.color = hdr / (1 + hdr)
    # Alpha-blend the LDR colours with the standard rasteriser  [§3.1.1]
    return rasterize(gaussians, camera)

Contradictions

None known between the papers, since they report on different tasks and do not compete on the same quantity. Two within-paper points are worth reading before you cite a number. In the shading paper, the full model's numbers repeat across three ablation tables, and the paper does not say whether they come from one run 2602.13549 §4.6.1 2602.13549 §4.6.3. In the inverse-rendering paper, IDR has the lower surface-normal error on the synthetic set, 2.207 degrees against 2.528, and the authors describe their geometry as close to IDR's 2104.00674 tab. 2.

Papers

• Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting: physically based diffuse and specular shading added to composite Gaussian splatting for night driving scenes.
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting: recovers SDF geometry, albedo and a spherical-Gaussian environment map from posed photos, with closed-form shading for relighting.
• Learning from synthetic photorealistic raindrop for single image raindrop removal: ray-traces adherent raindrops onto Cityscapes images to build paired training data for a three-stage removal network.
• Kubric: A scalable dataset generator: open-source framework that pairs PyBullet physics with Blender path tracing to generate labelled synthetic scenes at scale.