PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

In plain words

Take a set of photos of a shiny object under one fixed room light, and PhySG tries to work out three things at once: the object's shape, the paint-like colour and gloss of its surface, and the light that was in the room. Once those are known, you can render the object from new angles, change its material, or put it under different lights. Earlier neural methods could make new views but baked the light into the result, so you could not change it afterwards.

The shape is stored as a small neural network that answers "how far is this point from the surface?". The light is stored as a set of soft blobs on the sphere of directions, and the lighting maths is done approximately in closed form instead of by random sampling. The whole pipeline is trained from the photos alone, with no 3D scan and no RGB-D sensor. It assumes that the light comes from far away and that each object has one glossy material.

TL;DR

• Input: posed RGB photos of one glossy object under static, distant lighting, plus object masks §3.
• Output: an SDF geometry, a spatially varying diffuse albedo, one monochrome specular lobe, and an environment map of 128 spherical Gaussians §3.
• Shading is evaluated in closed form with SG products, so no Monte-Carlo sampling is needed for the light integral §3.
• It gives relighting and material editing that neural view-synthesis baselines cannot do. Its novel-view quality is close to IDR on real data, slightly worse tab. 3 §4.
• Author-reported results only. No indirect light and no self-occlusion are modelled §3.

Problem

Joint recovery of geometry, material and illumination from multi-view images normally needs varying light, such as a flashlight moved with the camera or a turntable, or RGB-D input or scanned geometry §1. Neural methods that need only RGB and static light (NeRF, IDR, DVR) render good novel views, but they fold material and light into one network and so cannot edit them §1 §2. The paper targets glossy objects under static, natural illumination with no scanned geometry and no known environment map §1.

The assumptions are: the light is distant, so the environment map does not depend on position §3; each object has one material with a shared monochrome, isotropic specular term §3; and the rendering omits self-occlusion and indirect illumination §3. Under Lambertian (purely matte) materials the light and the albedo are ambiguous, and the paper says light cannot be recovered without extra priors §4 "Robustness to material roughness".

Method

The pipeline has three parts that are all differentiable §3:

1. Geometry. The shape is a signed distance function S(x; Θ), with S > 0 outside the object §3 footnote 1. It is an MLP with positional encoding §3 "Geometry modeling". A ray starts at its intersection with a bounding volume and advances by the SDF value at each step (sphere tracing). The surface normal is the SDF gradient §3. Gradients reach the surface point through implicit differentiation, while the marching itself is not differentiated §3. The sample step is written in sample_network.py.
2. Appearance. The light is a mixture of M = 128 spherical Gaussians §3 "Appearance modeling". The diffuse albedo comes from an MLP of the surface point §3 "Appearance modeling". The specular term is one SG used as the normal distribution D, with an isotropic BRDF so that its axis equals the normal, and a monochrome amplitude §3 "Appearance modeling". The Fresnel and shadowing term M is in the supplement and not in the main text §3.
3. Forward rendering. For each pixel: sphere-trace the surface point x; take the normal n as the SDF gradient; evaluate the albedo a(x); and evaluate the rendering equation in closed form with the SG approximations §3 "Forward rendering". The rendering equation is eq. 1.

The rendering equation, with the paper's notation (ω_o outgoing, ω_i incoming) eq. 1:

L_o(\omega_o; x) = \int_{\Omega} L_i(\omega_i)\, f_r(\omega_o, \omega_i; x)\,(\omega_i \cdot n)\, d\omega_i eq. 1

An n-dimensional SG has the form eq. 2:

G(\nu; \xi, \lambda, \mu) = \mu\, e^{\lambda(\nu \cdot \xi - 1)} eq. 2

Here ξ is the lobe axis, λ the sharpness and μ the amplitude §3 "Appearance modeling". The clamped cosine ω_i·n is approximated by one SG with constants 0.0315, 32.7080 and 31.7003 §3 "Appearance modeling". The hemispherical integral of a product of SGs has a closed form §3. The three numerical constants are used as written in the code sg_render.py.

Pseudocode for one pixel, following the forward pass §3 "Forward rendering":

# Forward rendering of one camera ray  [§3 "Forward rendering"]
# Input:  ray origin (3), ray direction (3)
# Output: RGB colour (3)

def render_ray(origin, direction):
    # 1. Sphere-trace the SDF to the surface point  [§3 "Geometry modeling"]
    point = sphere_trace(sdf, origin, direction)     # about 10 MLP calls
    # 2. Surface normal is the gradient of the SDF  [§3 "Geometry modeling"]
    normal = sdf_gradient(point)                     # (3,)
    # 3. Spatially varying diffuse albedo  [§3 "Appearance modeling"]
    albedo = albedo_mlp(point)                       # (3,), values in [0, 1]
    # 4. Closed-form SG shading with the rendering equation  [§3] [eq. 1]
    colour = sg_shade(normal, direction, albedo)
    return colour

Shading uses the SG products described in §3 "Appearance modeling". Its numerical implementation is in sg_render.py. The specular lobe is warped about the reflection direction, and the product with each light lobe is taken with the closed-form "lambda trick" sg_render.py.

Training loss, per iteration, on a batch of sampled pixels §3 "Loss functions":

# Training loss for one iteration  [§3 "Loss functions"]
# Input:  object pixels with ground-truth colour, non-object pixels, SDF
# Output: scalar loss

def training_loss(object_pixels, nonobject_pixels, points):
    # 1. L1 colour error on the rendered object pixels  [§3 "Loss functions"]
    colour_term = l1(render(object_pixels), gt_colour(object_pixels))
    # 2. Keep the SDF non-negative on rays that miss the object  [§3]
    #    smooth form of max(-S, 0), with sharpness alpha
    miss_term = softplus(-alpha * min_sdf(nonobject_pixels)) / alpha
    # 3. Unit-norm SDF gradient on sampled points  [§3]
    eikonal_term = mean((norm(sdf_gradient(points)) - 1) ** 2)
    # 4. Penalise normal variance inside 2x2 patches of object pixels  [§3]
    smooth_term = variance_in_patches(normals)
    return colour_term + 100 * miss_term + 0.1 * eikonal_term + 10 * smooth_term

The weights 100, 0.1 and 10 are the paper's values §3 "Loss functions". The code uses a different smoothness weight, see Paper vs code.

Hyperparameters

Values come from the paper unless the "Where" column names the code. The defaults used in the official code are in default.conf.

Results

Author-reported. The synthetic set has test views outside the training cone, so the metrics measure view extrapolation §4 "Synthetic data". For the synthetic inverse-rendering table, the paper aligns each prediction to ground truth with per-channel median scale factors before computing metrics §4 "Synthetic data". The alignment is not in the repository (see Paper vs code).

Table 1, synthetic inverse rendering and relighting tab. 1

Table 2, geometry on synthetic data tab. 2 (normal error on test views; L1 Chamfer distance, unit-box normalised)

Table 3, novel-view quality against baselines tab. 3: rows are metric and method, columns are scenes. For synthetic data, HDR images are tone-mapped with exponent 1/2.2 before metrics tab. 3 caption.

Captions state that on the real data IDR is slightly better than PhySG on these metrics, which the authors attribute to bias in the physics-based appearance model tab. 3 caption.

Qualitative results are in fig. 4 (synthetic, compared with NeRF, IDR and DVR), fig. 5 (real-world relighting and editing), fig. 6 (environment maps under varying Ward roughness), and fig. 7 (synthetic test view, albedo, editing and relighting). Figure numbers follow source order in the LaTeX.

Geometry tab. 2: the authors report surface quality close to IDR's and well ahead of NeRF and DVR, and note that their method also supports relighting and material editing §4 "Comparison with baselines".

Paper vs code

The code is the repository Kai-46/PhySG at commit 030d590, which the paper's project page links. The repository is MIT-licensed; its LICENSE copyright line names Lior Yariv (2020), and the README says the codebase borrows heavily from the IDR code LICENSE README.md. No parts of the repository are quoted here.

Agreements, checked against the code: SDF MLP width and depth, skip index and 6 frequencies default.conf; albedo MLP 4×512 with 10 frequencies default.conf; 128 light SGs default.conf; 2048 rays per batch default.conf; 2×2 patches idr_train.py; α from 50 doubling to 1600 default.conf; β1 = 100 and β2 = 0.1 default.conf; 100 points per missed ray ray_tracing.py; the hemisphere-cosine constants 0.0315, 32.7080 and 31.7003 sg_render.py; the Fibonacci lattice for lobe directions sg_envmap_material.py.

Discrepancies, each with both citations:

1. Ray start volume. The paper starts sphere tracing at the bounding box §3 "Geometry modeling". The code intersects rays with a bounding sphere of radius 1 ray_tracing.py default.conf.
2. Eikonal samples. The paper uses N_x = 1024 random points in the box §3 "Loss functions". The code samples 1024 points uniformly in the cube [-1, 1]³ and also adds all 2048 ray-hit points, so the eikonal term runs on 3072 points implicit_differentiable_renderer.py.
3. Miss-term normalisation. The paper divides by N_nobj §3 "Loss functions". The code sums over the non-object samples and divides by the batch's total sample count (2048 for one image) loss.py. The code also applies the term to object pixels that the network misses, as target 1, which the paper's equation does not describe loss.py.
4. RGB loss scale. The paper takes the L1 norm over channels per pixel, then averages over N_obj §3 "Loss functions". The code uses PyTorch's mean-reduced L1, which averages over channels too, so the colour term is one third of the paper's scale for the same images loss.py loss.py.
5. Patch smoothness weight. The paper uses 10 §3 "Loss functions". The default config sets normalsmooth_weight = 1.0 default.conf. The code takes the variance per channel and averages over channels loss.py. The paper does not state the reduction.
6. Iterations. The paper says 250k iterations §3 "Loss functions". The README command uses the default max_niter of 200001 exp_runner.py README.md.
7. BRDF sharpness initial value. The paper draws the sharpness from [95, 125] §3 "Initialization". The code draws a roughness parameter from [1.5, 2.0] before a sigmoid, and sets the lobe sharpness to 2/r⁴ sg_envmap_material.py sg_render.py. By my calculation from those lines, the initial sharpness is about 3.3 to 4.5, far from [95, 125].
8. BRDF amplitude and specular albedo. The paper optimises the amplitude μ of the specular SG, initialised in [0.18, 0.26] §3 "Initialization". The code fixes the SG amplitude to 1/(π r⁴) from the roughness and learns a Schlick Fresnel reflectance F0 instead, initialised as the absolute value of a standard normal passed through a sigmoid sg_render.py sg_envmap_material.py sg_envmap_material.py. Values near 0.18 to 0.26 are not what this code produces.
9. Light initial intensity. The paper scales the lobe amplitudes so that the initial rendered intensity is about 0.5 §3 "Initialization". The code rescales the amplitudes so the total lobe energy is 2π sg_envmap_material.py (the comment in the code says "around 1"). No code path targets a pixel intensity of 0.5 in this file.
10. Image scaling. The paper scales all images of an object by one constant so that the median intensity is 0.5 §3 "Initialization". The dataset loader applies only a power (inverse gamma, default 1.0) scene_dataset.py. Scaling is not in the repository, so the data preparation must be reproduced from the paper.
11. Metric alignment. The paper aligns predictions to ground truth with per-channel median scales before computing metrics §4 "Synthetic data". The evaluation computes a masked PSNR on clipped images and has no median alignment eval.py eval.py. LPIPS and SSIM do not appear in the repository. The paper's Table 1 numbers therefore cannot be reproduced from this code alone.
12. Tone mapping. The paper tone-maps the synthetic data with exponent 1/2.2 tab. 3 caption. Evaluation applies the exponent 1/gamma, which is 1 in the README command eval.py README.md.
13. Specular Fresnel and shadowing. The paper defers the Fresnel and shadowing terms to the supplement §3 "Appearance modeling". The code uses a Schlick-type Fresnel with constants 5.55473 and 6.8316, and a shadowing term with k = (r+1)²/8 sg_render.py sg_render.py.
14. Unused IDR renderer. The code also has an MLP renderer from IDR. Its loss weight is 0 in the default config, so it is not trained by the PhySG loss default.conf implicit_differentiable_renderer.py.
15. Light RGB. The paper's SG amplitudes are RGB §3 "Appearance modeling". The default config sets white_light = False, so light amplitudes are RGB, and the code starts them as monochrome sg_envmap_material.py.

Pitfalls

• Ambiguous "bounding box". The paper leaves open whether the box is a cube or a sphere. The code uses a sphere for ray start and a cube for eikonal samples. Pick one, and keep the object inside the unit sphere, as the README requires README.md.
• Scale ambiguity. Light and reflectance are ambiguous up to a scale. Absolute radiance is not identifiable, so metrics depend on the alignment step, which the paper does not fully specify in code.
• Lambertian input. Without specular highlights, lighting and texture cannot be separated §4 "Robustness to material roughness".
• Monochrome versus RGB specular. The paper assumes a monochrome, isotropic specular term §3. The code's default uses white_specular = True default.conf, which matches. The light is RGB default.conf.
• Non-differentiable sphere tracing. Sphere tracing runs without gradients. Gradients reach the surface point through the implicit function §3 "Geometry modeling" sample_network.py. Re-deriving this step is easy to get wrong.
• Unstated activations and normalisation. The paper does not give the SDF MLP activation, weight normalisation or the albedo activation. The code uses Softplus with β = 100 for the SDF implicit_differentiable_renderer.py and weight normalisation implicit_differentiable_renderer.py, ELU for the albedo MLP sg_envmap_material.py and sigmoid outputs for albedo and reflectance sg_envmap_material.py.
• Numerically sensitive steps. The BRDF sharpness is divided by roughness to the fourth power sg_render.py, so small roughness gives huge lobes. The code clamps with TINY_NUMBER = 1e-6 sg_render.py and uses a numerically stable form of the hemisphere integral sg_render.py. Reproduce both, or the training diverges.
• Non-convergent rays. Rays that sphere tracing does not converge on are re-sampled with 100 points and then refined with up to 32 root-finding steps ray_tracing.py default.conf. The paper mentions neither count.
• Compute. The paper trains on a single 12 GB GPU for 250k iterations §3 "Loss functions". Each training iteration evaluates the SDF MLP on the surface, on 3072 eikonal points, and on the miss rays, with 128 SGs per pixel in shading implicit_differentiable_renderer.py.
• Initialisation sensitivity. An initial light that is too bright or too dark can leave the albedo MLP stuck at all-zero or all-one outputs. The paper's initialisation is meant to avoid this §3 "Initialization", but the code does not implement the intensity target (see Paper vs code, item 9).
• Figure and table numbering. Figure and table numbers in this card follow the LaTeX source order. Check them against the rendered PDF before using them in a citation.

Lineage

• Builds on: SDF surface representations, sphere tracing and implicit differentiation as in IDR and DVR §3 "Geometry modeling"; positional encoding as in NeRF §3; spherical Gaussians for approximate light transport and their closed-form products §3 "Appearance modeling". The source package has no bibliography file, so no arXiv ids are given for these works.
• Compared against: NeRF, IDR and DVR for novel-view extrapolation and geometry §4 "Comparison with baselines" tab. 2 tab. 3. A Monte-Carlo differentiable renderer (redner) was tried and failed to converge from a sphere initial mesh §4 "Comparison with baselines".
• Superseded by: not stated in the paper, and no later card in this wiki supersedes it.