---
kind: concept
slug: differentiable-rendering
title: Differentiable rendering
summary: >-
  Differentiable rendering makes an image-forming model differentiable, so a loss on the image can update pose, scene, material, face or mesh parameters by gradient descent. Variants fit faces, silhouettes or photos, score renders with a frozen CLIP or diffusion model, grow point clouds, refine SLAM maps, shade night driving scenes, or separate light from material.
tags: [3d-reconstruction, pose-estimation, volume-rendering, materials, optimization, text-to-3d, slam, voxels, relighting, autonomous-driving, point-cloud, mesh, view-synthesis, inverse-rendering, 3d-face-reconstruction]
papers: ["1612.00814", "1902.05978", "2012.09856", "2112.05131", "2201.08845", "2404.17569", "2112.01455", "2112.03221", "2203.13333", "2212.00774", "2211.10440", "2312.02126", "2409.10141", "2602.13549", "1912.08804", "2104.00674"]
related: [gaussian-splatting, neural-radiance-field, novel-view-synthesis, monocular-depth-estimation, instance-segmentation, diffusion-models, generative-prior-inverse-problems, simultaneous-localization-and-mapping, vision-language-pretraining, explicit-feature-grid, classifier-free-guidance, image-to-3d-generation, latent-diffusion, multi-view-stereo, physically-based-rendering, neural-implicit-surface, generative-adversarial-network, neural-style-transfer, fourier-features, gan-inversion, face-recognition, score-distillation-sampling]
status: draft
updated_at: 2026-11-01
---

## Overview

The problem is recovering scene parameters (a pose, a voxel grid, a material,
a Gaussian map, a mesh) from images, or generating them from a prompt, when a
rendering model that maps those parameters to pixels is available. The core
idea is to make the renderer differentiable with respect to its parameters, so
that a loss on the rendered image gives gradients for the parameters
[arxiv:2112.05131 §3.1] [arxiv:2012.09856 §3.2]. The shared invariant is that
the render is a function of the parameters, and a scalar loss is minimised by
a gradient method [arxiv:2012.09856 eq. 5] [arxiv:2112.05131 eq. 2]
[arxiv:2404.17569 §3.3] [arxiv:2312.02126 §3].

The loss does not always compare the render with photos. Dream Fields scores
the render with a frozen CLIP model [arxiv:2112.01455 §4.2], and SJC passes a
frozen diffusion model's score back through the renderer [arxiv:2212.00774 §4].
Magic3D uses the same kind of diffusion score in both of its stages, and
passes it through a volume renderer and then a mesh rasteriser [arxiv:2211.10440 §3]
[arxiv:2211.10440 §4.3].

Text2Mesh also scores renders with CLIP, but it changes only each vertex's
colour and normal offset, so the mesh keeps its shape [arxiv:2112.03221 §3.1]
[arxiv:2112.03221 §3.2].

The sixteen cards differ in what is optimised, what the renderer is, and what
judges the render:

- A GAN texture prior on a 3D morphable face model, rendered and compared with
  a single photo through deep face-recognition features [arxiv:1902.05978 §3.1]
  [arxiv:1902.05978 §3.3].
- A neural mesh renderer, compared against an instance mask and a monocular
  depth map, for a known object's pose [arxiv:2012.09856 §3.2].
- Volume rendering of a sparse voxel grid, compared against training photos,
  for a radiance field [arxiv:2112.05131 §3.1].
- A procedural material graph turned into texture maps and rendered with a
  microfacet BRDF, compared against a 2D painted image, for surface materials
  [arxiv:2404.17569 §3.3].
- A NeRF scored by a frozen CLIP model over random camera poses, with no
  photos [arxiv:2112.01455 §4.2].
- A voxel radiance field driven by the score of a pretrained 2D diffusion
  model [arxiv:2212.00774 §4].
- A Gaussian map, where each frame fits the camera pose and the map is then
  refined on keyframes, for dense RGB-D SLAM [arxiv:2312.02126 §3].
- A remeshed SMPL-X body, carved to generated normal and silhouette views,
  for a single photo of a person [arxiv:2409.10141 §3.2].
- A voxel grid predicted from one image, projected by a fixed perspective
  layer and compared with silhouettes from several views, for single-view
  reconstruction [arxiv:1612.00814 §3.2].
- A Gaussian scene whose per-Gaussian colour is shaded by a global light model
  and four specular lobes, compared with night driving footage, for
  nighttime reconstruction [arxiv:2602.13549 §3.2].
- A cloud of neural points with features and confidences, rendered by ray
  marching, and pruned and grown per scene, for novel views of posed photos
  [arxiv:2201.08845 §3] [arxiv:2201.08845 §4.2].
- A two-stage text-to-3D pipeline: a hash-grid field scored by a 64x64 prior,
  then a mesh scored by a 512x512 latent prior through a differentiable
  rasteriser [arxiv:2211.10440 §4.1] [arxiv:2211.10440 §4.3].
- A feature point cloud predicted from one image, splatted into a new view by
  a soft point renderer and refined by a GAN, with no depth labels
  [arxiv:1912.08804 §3.1] [arxiv:1912.08804 §3.2].
- A signed distance surface and a spherical-Gaussian light fitted to posed
  photos of one glossy object under fixed light, so that light and material
  can be changed afterwards [arxiv:2104.00674 §3].
- A fixed input mesh whose vertex colours and bounded normal offsets come from
  a neural field, scored by CLIP on augmented multi-view renders, for
  text-driven stylisation [arxiv:2112.03221 §3.1] [arxiv:2112.03221 §3.2].
- A Loop-subdivision surface started from a sphere, with a texture and a normal
  map, scored by CLIP and a diffusion prior through a differentiable renderer,
  with a Laplacian smoothness term, for a text-to-mesh output
  [arxiv:2203.13333 §3] [arxiv:2203.13333 eq. 4].

## Variants

### Mask and depth fit of a known object's pose

Scale, rotation and translation are optimised so the rendered silhouette
matches an instance mask, and the rendered depth matches a monocular depth
estimate [arxiv:2012.09856 §3.2] [arxiv:2012.09856 eq. 3] [arxiv:2012.09856 eq. 4].
The mask comes from a class-agnostic instance model, keeping the instance that
overlaps the detected hand box most [arxiv:2012.09856 §3.2]. The fit starts
from 1000 rotations, uniform in [−π, π], with translation at zero, and the
lowest-loss result is kept [arxiv:2012.09856 app. Implementation Details]. The
paper reports about 18 s for this step per image on a GTX 2080 Ti
[arxiv:2012.09856 app. Implementation Details]. Card: [Reconstructing
Hand-Object Interactions in the Wild](/papers/2012.09856).

### Joint objective with the rendered terms kept

The rendered mask and depth losses stay in one joint objective with a
symmetric chamfer term between hand and object vertices and a collision term
read from a voxel signed-distance grid [arxiv:2012.09856 §3.3] [arxiv:2012.09856 eq. 6]
[arxiv:2012.09856 eq. 8] [arxiv:2012.09856 eq. 9]. The paper also refines the hand afterwards
with a small learned network [arxiv:2012.09856 §3.4]. Card: [Reconstructing
Hand-Object Interactions in the Wild](/papers/2012.09856).

### Volume rendering fit of a voxel grid

Each ray colour is the NeRF volume rendering sum over samples along the ray
[arxiv:2112.05131 §3.1] [arxiv:2112.05131 eq. 1]. The grid stores opacity and
degree-2 spherical harmonic colour at each occupied corner, interpolated
trilinearly [arxiv:2112.05131 §3.2] [arxiv:2112.05131 §3.3]. The loss is a
mean squared error on ray colours plus a total-variation term, with RMSProp as
the optimiser [arxiv:2112.05131 §3.5] [arxiv:2112.05131 eq. 2]. Training starts
on a coarse grid, prunes empty cells, and splits the rest in half
[arxiv:2112.05131 §3.4]. No neural network is used. The authors report 31.71
PSNR on the 8 NeRF synthetic scenes, against 31.85 for JAXNeRF, and about 11
minutes per scene on one Titan RTX [arxiv:2112.05131 tab. 2]. The paper also
describes a slower JAX implementation [arxiv:2112.05131 §3.8]. Card: [Plenoxels:
Radiance Fields without Neural Networks](/papers/2112.05131).

### Material graph fit to a painted view

A procedural material graph is turned by DiffMat v2 into albedo, normal and
roughness maps. These are sampled with UV maps and rendered with a microfacet
BRDF, under a light field initialised by InvRenderNet and a residual light
that starts at zero [arxiv:2404.17569 §3.3] [arxiv:2404.17569 eq. 2]. Before
optimising, a search over UV scales and orientations picks the lowest Gram
loss [arxiv:2404.17569 supp.: material graph optimization]. The loss sums an
albedo term and a render term (masked L1 plus a VGG Gram style term), each with
weight 1 [arxiv:2404.17569 supp.: material graph optimization]. The graph is
first chosen by retrieval from a library, and the fit starts from that graph
[arxiv:2404.17569 §3.3]. The optimiser is Adam, with 300 iterations in
total [arxiv:2404.17569 §4.1]. Card: [MaPa: Text-driven Photorealistic Material
Painting for 3D Shapes](/papers/2404.17569).

### Silhouette projection of a predicted voxel grid (PTN)

A network predicts a 32³ occupancy grid from one 64×64 image [arxiv:1612.00814
§3.1] [arxiv:1612.00814 §4]. A fixed perspective layer samples the grid onto
each training camera and takes the maximum along depth, which the paper says
approximates ray tracing [arxiv:1612.00814 §3.2] [arxiv:1612.00814 eq. 4]. The
loss is the squared error between predicted and observed silhouettes, averaged
over views [arxiv:1612.00814 eq. 1]. A volume loss can be added with a weight,
but the paper's aim is training when ground-truth volumes are not available
[arxiv:1612.00814 §3.1] [arxiv:1612.00814 eq. 2]. The encoder is pre-trained
with a rotation curriculum so that its code does not depend on the viewpoint
[arxiv:1612.00814 §3.3] [arxiv:1612.00814 §4]. On chairs, the projection-only
model reports a test IoU of 0.5027, against 0.5067 with volume supervision and
0.4983 for a volume-only CNN [arxiv:1612.00814 tab. 1]. The released Torch code
takes its silhouette targets from projected ground-truth volumes, not from
image masks, so the no-volume model as released uses 3D data (see the card's
Paper vs code section). Card: [Perspective Transformer Nets: Learning
Single-View 3D Object Reconstruction without 3D Supervision](/papers/1612.00814).

### Text-scored NeRF with no photos (Dream Fields)

Cameras are sampled around the object, with azimuth uniform over 360° and the
elevation fixed at 30° [arxiv:2112.01455 §4.4]. Each render is composited over
a random background and scored by the negative similarity to the caption under
a frozen CLIP model [arxiv:2112.01455 §4.5] [arxiv:2112.01455 eq. 7]. A
transmittance penalty pushes the mean transmittance towards a target, set to
88% and annealed in from 40% [arxiv:2112.01455 §4.5] [arxiv:2112.01455 eq. 8].
The scene is kept inside a cube of side 2 [arxiv:2112.01455 §4.6]. The network
is a residual MLP with 280K parameters, against 494K for the NeRF baseline
[arxiv:2112.01455 §4.7]. The transmittance weight λ is not stated
[arxiv:2112.01455 eq. 9]. The authors report a CLIP R-Precision of 68.3 at 20K
iterations with 252-pixel renders [arxiv:2112.01455 tab. 1]. About 1 h 12 min
is their estimate for 10K iterations on eight TPU cores [arxiv:2112.01455 app. C].
Card: [Zero-Shot Text-Guided Object Generation with Dream
Fields](/papers/2112.01455).

### Score distillation through a voxel renderer (SJC)

A voxel radiance field is rendered from random cameras. A pretrained 2D
diffusion denoiser gives a score for each clean render, and that score is
chained through the renderer, so the 3D score is the expected 2D score times
the renderer Jacobian [arxiv:2212.00774 §4] [arxiv:2212.00774 eq. 11]. A clean
render is out of distribution for a denoiser trained on noisy inputs, so the
score is estimated by adding noise and averaging the denoiser's outputs
(NEScore) [arxiv:2212.00774 §4.1] [arxiv:2212.00774 eq. 16]. An emptiness
loss on the ray weights, with β = 10, suppresses floating density
[arxiv:2212.00774 §4.2]. The text-to-3D run uses Stable Diffusion v1.5 on a
four-channel latent feature grid [arxiv:2212.00774 §5.2]. The paper reports
about 25 minutes per object on one A6000 [arxiv:2212.00774 app. A3]. It reports
no quantitative metric; the results are shown visually [arxiv:2212.00774 §5.2]
[arxiv:2212.00774 fig. 6]. Card: [Score Jacobian Chaining: Lifting Pretrained
2D Diffusion Models for 3D Generation](/papers/2212.00774).

### Per-frame pose fit and map refinement (SplaTAM)

For each new RGB-D frame, the camera pose is fitted by rendering the Gaussian
map and minimising depth and colour error, over pixels whose silhouette is
above 0.99, while the map is held fixed [arxiv:2312.02126 §3]
[arxiv:2312.02126 eq. 8]. The Gaussians are then densified where the map is
sparse or misses geometry, and refined over a window of keyframes with the
poses fixed [arxiv:2312.02126 §3]. The paper reports an average tracking error
of 0.36 cm on Replica, against 0.52 cm for Point-SLAM, and about 1.2 cm on
ScanNet++, where Point-SLAM and ORB-SLAM3 fail to track [arxiv:2312.02126 tab. 1].
On Replica R0, one tracking iteration takes 25 ms and one mapping iteration 24 ms
on an RTX 3080 Ti [arxiv:2312.02126 tab. 6]. Replica uses 40 tracking and 60
mapping iterations per frame, and a lighter SplaTAM-S setting uses 10 and 15
[arxiv:2312.02126 §5]. The method needs known camera intrinsics and dense
depth [arxiv:2312.02126 §5]. Card: [SplaTAM: Splat, Track & Map 3D Gaussians
for Dense RGB-D SLAM](/papers/2312.02126).

### Remeshing an SMPL-X body to generated views (PSHuman)

A parametric body is fitted first. Its translation, shape and pose are
optimised so that its normal and silhouette renders match the generated body
views, weighted by per-view confidence [arxiv:2409.10141 §3.2]
[arxiv:2409.10141 eq. 7]. Vertices and face connectivity are then optimised
against the same weighted normal and silhouette mismatch, plus a Laplacian
smoothness term with weight 0.02 [arxiv:2409.10141 §3.2] [arxiv:2409.10141 eq. 8].
The six views come from a multiview diffusion model, conditioned on renders of
SMPL-X [arxiv:2409.10141 §3.1]. On THuman2.1, the paper reports a Chamfer
distance of 0.4399 against 0.5754 for the best baseline, SIFU
[arxiv:2409.10141 tab. 1]. It estimates about one minute end to end
[arxiv:2409.10141 §1]. The released code uses a different texturing method from
the one the paper describes, so the paper's texture step should not be copied
as-is [arxiv:2409.10141 §3.2]. Card: [PSHuman: Photorealistic Single-image 3D
Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit
Remeshing](/papers/2409.10141).

### Physically based shading of Gaussians at night (driving scenes)

Each Gaussian carries an albedo, a roughness, a metalness and a normal. Its
diffuse light is a Lambertian term written with a second-degree
spherical-harmonics light, which a global MLP predicts from a timestep and
camera embedding [arxiv:2602.13549 §3.4] [arxiv:2602.13549 eq. 4]. Its
specular light comes from four anisotropic spherical Gaussian lobes per
Gaussian, combined with a simplified Disney BRDF [arxiv:2602.13549 §3.5]
[arxiv:2602.13549 eq. 6] [arxiv:2602.13549 eq. 7]. The two parts are summed in
HDR and compressed with Reinhard before the standard rasteriser draws the
scene [arxiv:2602.13549 §3.6] [arxiv:2602.13549 eq. 8] [arxiv:2602.13549 eq. 9].
The loss is a weighted RGB term, D-SSIM term and depth-normal term, optimised
with Adam for 40,000 iterations per driving sequence [arxiv:2602.13549 eq. 14]
[arxiv:2602.13549 §4.2]. The authors report 31.9 dB reconstruction PSNR on 11
Waymo night scenes, against 31.0 for OmniRe, and 29.7 dB on 11 nuScenes night
scenes, against 28.7 [arxiv:2602.13549 tab. 1] [arxiv:2602.13549 tab. 2]. In
their ablation, removing the specular or the diffuse part lowers PSNR
[arxiv:2602.13549 tab. 3]. The per-sequence fit is a scalability limit the
authors name themselves [arxiv:2602.13549 §5]. Card: [Nighttime Autonomous
Driving Scene Reconstruction with Physically-Based Gaussian
Splatting](/papers/2602.13549).

### Point-based radiance field with per-scene pruning and growing (Point-NeRF)

A radiance field is stored as a set of points, each with a feature and a
confidence in [0, 1] [arxiv:2201.08845 §3]. A shaded location uses its eight
nearest points within a radius [arxiv:2201.08845 app. H]. Their features pass
through a small MLP, are blended with inverse-distance weights scaled by
confidence, and become colour and density [arxiv:2201.08845 eq. 3]
[arxiv:2201.08845 eq. 4] [arxiv:2201.08845 eq. 6]. Pixels are formed by volume
rendering along each ray [arxiv:2201.08845 eq. 1]. The starting points come
from a cross-scene network built on MVS depth, which the paper times at about
0.2 s for three views [arxiv:2201.08845 §4.1] [arxiv:2201.08845 §5]. Per scene,
the rendering loss is minimised with a sparsity term on confidences
[arxiv:2201.08845 eq. 12]. Every 10K iterations, points with confidence below
0.1 are pruned, and a point is grown where a ray's most opaque sample lies far
from every point [arxiv:2201.08845 §4.2] [arxiv:2201.08845 eq. 10]. The growing
thresholds are not stated [arxiv:2201.08845 §4.2]. On NeRF Synthetic, the
authors report 30.71 PSNR after 20K steps, about 40 minutes, and 33.31 after
200K steps, against 31.01 for NeRF [arxiv:2201.08845 tab. 2]
[arxiv:2201.08845 §6.2]. Card: [Point-NeRF: Point-based Neural Radiance
Fields](/papers/2201.08845).

### Two-stage text-to-mesh with differentiable marching tetrahedra (Magic3D)

The loss is score distillation: a diffusion prior's noise prediction is
back-propagated through the renderer to the scene parameters
[arxiv:2211.10440 §3] [arxiv:2211.10440 eq. 1]. Stage 1 optimises a hash-grid
neural field with a 64x64 base diffusion prior, for 5000 steps with up to 1024
samples per ray [arxiv:2211.10440 §4.1] [arxiv:2211.10440 §5]. Stage 2 turns
the density into a signed distance on a deformable tetrahedral grid, extracts
a surface with differentiable marching tetrahedra, and optimises the vertex
offsets and a neural texture with a 512x512 latent diffusion prior, for 3000
steps [arxiv:2211.10440 §4.2] [arxiv:2211.10440 §4.3] [arxiv:2211.10440 §5].
A differentiable rasteriser tracks the 3D point under each pixel, which is
how the texture is queried [arxiv:2211.10440 §4.3]. An angular regulariser
between adjacent faces keeps the surface smooth under the noisy gradient
[arxiv:2211.10440 §4.3]. Stage 2 samples timesteps from (0.02, 0.5) with
weight σ_t² [arxiv:2211.10440 app. B]. The authors report about 40 minutes on
8 A100 GPUs in total, and 61.7% of user-study raters prefer it to DreamFusion
for realism and detail [arxiv:2211.10440 §5] [arxiv:2211.10440 tab. 1]. Card:
[Magic3D: High-Resolution Text-to-3D Content Creation](/papers/2211.10440).

### Feed-forward point splatting from one image (SynSin)

A spatial feature network gives each pixel a 64-channel feature, and a depth
network gives each pixel a depth [arxiv:1912.08804 §3.1]. The pixels are lifted
into 3D points, moved by the relative camera pose, and splatted onto a disk of
radius r around their projected centre [arxiv:1912.08804 §3.2] [arxiv:1912.08804 eq. 1].
Each pixel keeps its K nearest points, sorted by depth, and composites them
front to back [arxiv:1912.08804 eq. 2]. Gradients reach every kept point, and
reach both features and depths [arxiv:1912.08804 §3.2]. A refinement network
fills the holes and a discriminator judges the output [arxiv:1912.08804 §3.3].
The training loss uses only image pairs and their relative poses
[arxiv:1912.08804 §1] [arxiv:1912.08804 §3.4]. The paper reports 20.91 PSNR on
Matterport3D and 22.31 on RealEstate10K for its full model, from one input
view [arxiv:1912.08804 tab. 1] [arxiv:1912.08804 tab. 2]. On RealEstate10K,
the two-view StereoMag baseline reports 25.34 PSNR, which is higher
[arxiv:1912.08804 tab. 2]. A user study over 100 trajectories prefers SynSin to
a voxel baseline in 68.7% of majority votes [arxiv:1912.08804 tab. 4]. The
renderer takes 36 ms forward and 5 ms backward for a batch of six clouds of
512² points on one V100 [arxiv:1912.08804 §3.2]. Card: [SynSin: End-to-end
View Synthesis from a Single Image](/papers/1912.08804).

### Light and material separation on a glossy object (PhySG)

The shape is a signed distance function, written as an MLP with positional
encoding and read with sphere tracing [arxiv:2104.00674 §3 "Geometry modeling"].
The diffuse albedo comes from a second MLP of the surface point, and the
specular term is one monochrome lobe [arxiv:2104.00674 §3 "Appearance modeling"].
The light is an environment map of 128 spherical Gaussians [arxiv:2104.00674 §3
"Appearance modeling"]. Shading is evaluated in closed form, so no sampling of
the light is needed [arxiv:2104.00674 §3]. The training loss has an L1 colour
term on object pixels, a penalty on rays that miss the object, a unit-norm
constraint on the SDF gradient and a smoothness term on normals inside small
patches [arxiv:2104.00674 §3 "Loss functions"]. Surface points are found by
sphere tracing, which is not differentiated; gradients reach the surface point
through the implicit function [arxiv:2104.00674 §3 "Geometry modeling"].
The authors report a synthetic novel-view PSNR of 35.93 and a relighting PSNR
of 33.25 [arxiv:2104.00674 tab. 1]. On real scenes the paper says IDR is
slightly better on the reported metrics, which the authors attribute to bias in
the physics-based model [arxiv:2104.00674 tab. 3]. The model assumes distant
light, one material per object, and no self-occlusion or indirect light
[arxiv:2104.00674 §3]. Card: [PhySG: Inverse Rendering with Spherical Gaussians
for Physics-based Material Editing and Relighting](/papers/2104.00674).

### Text-driven stylisation of a fixed mesh (Text2Mesh)

The input mesh's faces stay fixed. A neural style field maps each vertex to a
colour and to a displacement along its normal, and the displacement is bounded
[arxiv:2112.03221 §3.1]. Each vertex is encoded with Fourier features before
the field runs [arxiv:2112.03221 §3.1] [arxiv:2112.03221 eq. 1]. The objective
is a sum of cosine similarities between CLIP embeddings of the prompt and of
augmented renders from about five views around an anchor [arxiv:2112.03221 §3.2]
[arxiv:2112.03221 §3.3] [arxiv:2112.03221 eq. 5]. A second, grey render of the
displaced mesh is scored on its own, so the geometry gets a gradient of its
own [arxiv:2112.03221 §3.2] [arxiv:2112.03221 eq. 4]. Gradients come through an
interpolation-based differentiable renderer [arxiv:2112.03221 §3.1]. The
authors report that without augmentation the result follows no target text,
and that without the geometry-only term the network imitates shape with
shading [arxiv:2112.03221 §4.2] [arxiv:2112.03221 fig. 5]. Card: [Text2Mesh:
Text-Driven Neural Stylization for Meshes](/papers/2112.03221).

### Photo fit of a 3D face with a GAN texture prior (GANFIT)

The face is a linear morphable model of shape and expression, with a camera
and lighting, all optimised [arxiv:1902.05978 §2.1] [arxiv:1902.05978 eq. 2].
The skin is a generator from a 512-value latent to a UV map, trained as a GAN
on about 10,000 UV textures [arxiv:1902.05978 §3.1] [arxiv:1902.05978 eq. 5]
[arxiv:1902.05978 §4.1]. The render is compared with the photo through four
terms: a cosine distance between face-recognition embeddings, a content loss
on intermediate layers of that network, a full-resolution L1 pixel loss, and a
landmark loss [arxiv:1902.05978 eq. 8] [arxiv:1902.05978 eq. 9]
[arxiv:1902.05978 eq. 10] [arxiv:1902.05978 eq. 11]. The fit runs in two
stages: the landmark loss alone first, then all parameters together under a
weighted sum with a regulariser [arxiv:1902.05978 §3.4] [arxiv:1902.05978 eq.
12]. The optimiser is Adam with learning rate 0.01 [arxiv:1902.05978 §4.1].
On the cooperative setting of MICC Florence, the authors report a mean
point-to-plane error of 0.95 against 1.50 for the best baseline there
[arxiv:1902.05978 tab. 1]. The fit takes about 30 s per image on one GTX 1080
Ti [arxiv:1902.05978 §4.1]. Card: [GANFIT: Generative Adversarial Network
Fitting for High Fidelity 3D Face Reconstruction](/papers/1902.05978).

### Text-to-mesh from a subdivision surface with CLIP and a diffusion prior (CLIP-Mesh)

The shape is a Loop subdivision limit surface defined by a set of control
vertices, and the texture and normal map are free maps [arxiv:2203.13333 §3].
Each step renders random views, encodes them with CLIP, and minimises the mean
negative cosine similarity to the prompt [arxiv:2203.13333 eq. 1]. A uniform
Laplacian over one-ring neighbours keeps the shape smooth, with a weight that
decays during training [arxiv:2203.13333 eq. 2] [arxiv:2203.13333 §3]. A
diffusion prior conditioned on the prompt gives a second CLIP target, weighted
by α [arxiv:2203.13333 eq. 3] [arxiv:2203.13333 eq. 4]. Each view draws its
background, position offset, field of view and distance at random, so the
texture cannot carry the result on its own [arxiv:2203.13333 §3]. The authors
leave the prior loss out of the Table 1 setting, because the prior was trained
on ViT-B/32 embeddings only [arxiv:2203.13333 §4.3]. Against Dreamfields, the
paper reports CLIP R-precision of 96.7 against 93.5 with ViT-B/16 for both
generation and evaluation, and 91.4 against 86.6 with ViT-B/32
[arxiv:2203.13333 tab. 1]. The authors report about 50 minutes per shape on
one 16 GB GPU, against more than 24 hours for Dreamfields on four A100s
[arxiv:2203.13333 §4.1]. Card: [CLIP-Mesh: Generating textured meshes from
text using pretrained image-text models](/papers/2203.13333).

## When to use which

| Situation | Use | Why |
|---|---|---|
| One image, a known 3D model of the object, need its pose | Mask and depth fit, many random starts | The paper runs the fit from 1000 starts and keeps the lowest loss, because the fit can settle in local minima [arxiv:2012.09856 app. Implementation Details] [arxiv:2012.09856 app. Failure Cases] |
| Hand and object are estimated together and must not overlap | Joint objective with the rendered terms | The collision score drops from 17.6 to 7.7 when the penetration loss is added; units and dataset are not stated [arxiv:2012.09856 tab. 3] |
| Tight time budget per image | Budget for the cost the paper reports | About 1 minute per image on a GTX 2080 Ti, of which about 18 s is the object fit and about 30 s the joint step [arxiv:2012.09856 app. Implementation Details] |
| Calibrated photos of a static scene, need fast per-scene fitting | Volume rendering fit of a voxel grid | 11 minutes per synthetic scene on one Titan RTX, against 1.45 days for the JAXNeRF rerun; author-reported [arxiv:2112.05131 tab. 2] |
| Many scenes with different content, no time to retune weights | Plan to retune the regularisation per dataset | TV weights were fixed per dataset, not per scene; the paper says a per-scene sweep would likely improve results [arxiv:2112.05131 §5] [arxiv:2112.05131 app. B] |
| Text prompt on a mesh, output must stay an editable material | Material graph fit | The output is a procedural graph that can be edited; the authors report about 7 minutes per shape on average, on one A6000, with the dataset not stated [arxiv:2404.17569 §4.1] [arxiv:2404.17569 §4.4] |
| Text prompt, no 3D data, a quantitative check is wanted, and a NeRF output is acceptable | Dream Fields | Its own paper reports R-Precision 68.3; the score uses a CLIP model, so it partly measures agreement with that model [arxiv:2112.01455 tab. 1] [arxiv:2112.01455 §5.1]. CLIP-Mesh reports 93.5 for Dreamfields under its own setup, so the two numbers are not comparable (see Contradictions) [arxiv:2203.13333 tab. 1] |
| Text prompt, a textured mesh with a normal map is the output, no 3D data, and about 50 minutes per shape on one GPU is acceptable | CLIP-Mesh | Author-reported R-precision above Dreamfields in all four generation and evaluation pairings, and about 50 minutes per shape on a 16 GB GPU [arxiv:2203.13333 tab. 1] [arxiv:2203.13333 §4.1]. The prior loss is not in those numbers [arxiv:2203.13333 §4.3]. In the paper's ablation, adding the prior raises the B/32 and L/14 scores and lowers B/16 [arxiv:2203.13333 tab. 2] |
| One face photo, a textured 3D face that keeps identity is wanted, and per-image optimisation is acceptable | GANFIT | Author-reported 0.95 mean point-to-plane error on MICC cooperative against 1.50 for the best baseline there, and about 30 s per image [arxiv:1902.05978 tab. 1] [arxiv:1902.05978 §4.1]. The paper does not state the iteration counts or the initial values [arxiv:1902.05978 §3.4] |
| Text prompt, a latent diffusion model is available, visual quality is the goal | Score distillation (SJC) | Results are qualitative on Stable Diffusion, with about 25 minutes per object on one A6000 and no metric to compare against [arxiv:2212.00774 §5.2] [arxiv:2212.00774 app. A3] |
| Night or low-light driving footage, one fit per sequence is acceptable, and light must act on each Gaussian's material | Physically based shading (night driving) | Author-reported gains of about 1 dB PSNR over OmniRe on nuScenes reconstruction; the paper reports no frame rate, training time or memory [arxiv:2602.13549 tab. 2] [arxiv:2602.13549 §4.5] [arxiv:2602.13549 §1] |
| Posed RGB-D stream, online tracking and mapping needed | Gaussian-map SLAM (SplaTAM) | Per-frame tracking and mapping times are reported on Replica, with low tracking error; it needs known intrinsics and dense depth [arxiv:2312.02126 tab. 1] [arxiv:2312.02126 tab. 6] [arxiv:2312.02126 §5] |
| One full-body photo of a person, body model estimable from it | Remeshing an SMPL-X body (PSHuman) | About one minute end to end according to the paper; the body model is estimated from the input image, and the paper says the face branch depends on a clear face [arxiv:2409.10141 §1] [arxiv:2409.10141 §3.1] |
| One image, known camera for each training view, no 3D volumes for training, and foreground masks are easy to get | Silhouette projection (PTN) | The paper reports a chair test IoU of 0.5027 without volume supervision, against 0.5067 with it, from single runs [arxiv:1612.00814 tab. 1] [arxiv:1612.00814 §3.2]. The released code uses ground-truth volumes for its targets, so this does not hold as released [arxiv:1612.00814 §3.1] |
| No 3D model of the object | Not covered by the pose-fit variants | The pose-fit paper needs a 3D model per image, chosen by an annotator [arxiv:2012.09856 §5.1]. Its "without 3D supervision" claim does not remove this input |
| Loss weights must be fixed in advance | Treat the paper's weights as defaults | Weights are tuned per image during annotation [arxiv:2012.09856 §5.2]; the λ values are defaults, not tested across data [arxiv:2012.09856 app. Implementation Details] |
| Posed photos, a fast start from a feed-forward guess, and per-scene refinement is affordable | Point-based radiance field (Point-NeRF) | The feed-forward output alone is below IBRNet on DTU (23.89 against 26.04 PSNR); after 10K iterations it reaches 30.12, against 31.35 for IBRNet [arxiv:2201.08845 tab. 1] |
| Sparse or holey COLMAP points as the starting cloud | Point-NeRF with pruning and growing | On the Ship scene, pruning and growing raise PSNR from 19.35 to 30.18 for COLMAP points [arxiv:2201.08845 tab. 4]; the growing thresholds are not stated [arxiv:2201.08845 §4.2] |
| Text prompt, a textured mesh for a 3D tool is the goal, and 8 A100-class GPUs are available | Two-stage text-to-mesh (Magic3D) | About 40 minutes on 8 A100s and 61.7% preference over DreamFusion in a user study [arxiv:2211.10440 §5] [arxiv:2211.10440 tab. 1]. The 1.5 h DreamFusion figure is quoted from DreamFusion, not measured by the authors [arxiv:2211.10440 §1] |
| Reimplementing Magic3D, no official code | Fill the gaps from the paper's own choices | No official code was found. The text-guidance weight, SDF offset, tetrahedral grid resolution and angular regulariser weight are not stated [arxiv:2211.10440 §3] [arxiv:2211.10440 §4.2] [arxiv:2211.10440 §4.3] |
| One input image, a nearby new view is needed, no depth labels, and a GAN refinement step is acceptable | Feed-forward point splatting (SynSin) | 22.31 PSNR on RealEstate10K from one view, against 25.34 for the two-view StereoMag baseline [arxiv:1912.08804 tab. 2]. Training uses image pairs only [arxiv:1912.08804 §3.4] |
| Posed photos of one glossy object under fixed light, and the light or material must be editable afterwards | Light and material separation (PhySG) | Relighting PSNR 33.25 and novel-view PSNR 35.93 on synthetic data [arxiv:2104.00674 tab. 1]. On real scenes the paper says IDR is slightly better on its metrics [arxiv:2104.00674 tab. 3]. The model assumes distant static light [arxiv:2104.00674 §3] |
| Matte or Lambertian object under fixed light, and light must be recovered | Not PhySG | The paper says light and albedo cannot be separated without extra priors when the material has no specular highlights [arxiv:2104.00674 §4] |
| Text prompt to restyle an existing mesh, shape must stay, no 3D generator or stylised dataset available | Text-driven stylisation (Text2Mesh) | Author-reported user study: 3.90 for natural depiction against 2.83 for a VQGAN-CLIP baseline, from 57 raters [arxiv:2112.03221 tab. 1] [arxiv:2112.03221 §4.3]. Name the object category in the prompt, since a mismatched prompt can erase the content [arxiv:2112.03221 §4.6] |
| Need a number to rank stylisations by | Not CLIP similarity alone | The paper shows a degenerate solution that scores high on CLIP similarity, so the authors suggest judging outputs visually [arxiv:2112.03221 §4.2] |

## Implementation sketch

Our own notation for the common core. Each variant changes the renderer, the
objective and the optimiser. The objective is either a mismatch to observed
images or a frozen model's judgement of the render. The optimiser is not
stated for the object fit [arxiv:2012.09856 §3.2], so it is left as a
placeholder.

```pseudocode
# Common core: fit parameters by differentiable rendering
# [arxiv:2012.09856 §3.2] [arxiv:2112.05131 §3.5] [arxiv:2404.17569 §3.3]
def fit_by_rendering(params, target, renderer, steps):
    for step in range(steps):
        # render the current parameters into image space; differentiable
        # in params  [arxiv:2112.05131 §3.1] [arxiv:2404.17569 eq. 2]
        predicted = renderer(params)
        # objective: a mismatch to observations, or a frozen model's score
        # on the render  [arxiv:2012.09856 eq. 5] [arxiv:2112.01455 eq. 7]
        loss = objective(predicted, target)
        # gradient method; the optimiser differs per paper
        # [arxiv:2112.05131 §3.5] [arxiv:2404.17569 §4.1]
        params = gradient_step(loss, params)
    return params
```

Variant notes. The object fit wraps this core in a loop over 1000 random
rotations and keeps the lowest loss [arxiv:2012.09856 app. Implementation
Details]. The voxel fit adds a coarse-to-fine prune and split between steps
[arxiv:2112.05131 §3.4]. The material fit starts from a retrieved graph
instead of a random start [arxiv:2404.17569 §3.3]. Dream Fields replaces the
mismatch with a negative CLIP similarity and draws a random pose each step
[arxiv:2112.01455 §4.2] [arxiv:2112.01455 §4.4]. SJC has no loss to
differentiate: it feeds the denoiser score in as the gradient at the render,
so the score's scale matters. The paper's score divides by σ², while its
released code divides by σ, so the two differ by a factor σ [arxiv:2212.00774 eq. 3]
[arxiv:2212.00774 eq. 16]. SplaTAM uses the same loop per frame, with the
pose as the parameter and the Gaussians as the map [arxiv:2312.02126 §3].
PSHuman uses mesh vertices and topology as the parameters, with a
normal-and-silhouette mismatch as the objective [arxiv:2409.10141 §3.2]. PTN
uses the occupancy grid as the parameters, predicted by a network rather than
optimised per object, with a per-view squared silhouette error as the
objective [arxiv:1612.00814 §3.2] [arxiv:1612.00814 eq. 1]. The night-driving
shading is a per-Gaussian colour step before rasterisation: Reinhard is applied
to each Gaussian's HDR colour, and the results are then alpha-blended, so the
pixel is not the Reinhard value of the blended radiance [arxiv:2602.13549 §3.6]
[arxiv:2602.13549 eq. 2]. The paper does not discuss this choice
[arxiv:2602.13549 §3.6].

Point-NeRF and Magic3D add two more cases. Point-NeRF's pipeline uses the
same core loop with per-point features and confidences as parameters, and
the rendering loss as objective [arxiv:2201.08845 §4.2]. Magic3D's stage 2 uses
the mesh vertices and texture as parameters, with the diffusion score as the
gradient at the rendered pixels [arxiv:2211.10440 §4.3]. Magic3D's stage 1 uses
the hash grid as its parameters [arxiv:2211.10440 §4.2]. Its stage 2 also has
no loss to differentiate in the usual sense, so the same caveat as SJC applies.

SynSin's parameters are not fitted per scene: its feature and depth networks
output the point cloud for each input image, and the renderer's gradients
reach those outputs [arxiv:1912.08804 §3.1] [arxiv:1912.08804 §3.2]. The
objective is an image loss with an adversarial term [arxiv:1912.08804 §3.4].
PhySG fits its parameters per object: the SDF and albedo MLPs and the light
lobes, with the colour loss on object pixels as the main term
[arxiv:2104.00674 §3 "Loss functions"]. Its surface point comes from sphere
tracing, which is not differentiated, so the gradient is taken through the
implicit surface at that point [arxiv:2104.00674 §3 "Geometry modeling"].

Text2Mesh fits per input mesh. Its parameters are the style field's weights,
evaluated at each vertex, and the mesh is updated only through the bounded
normal offsets [arxiv:2112.03221 §3.1]. The objective sums three CLIP
similarities: the full colour render, a cropped full colour render, and a
cropped grey render of the displaced mesh. The grey term updates only the
shared trunk and the displacement branch [arxiv:2112.03221 eq. 5]
[arxiv:2112.03221 §3.2].

## Contradictions

Within one paper, the LPIPS for the synthetic scenes is 0.049 in the main
table and 0.050 in the ablation tables [arxiv:2112.05131 tab. 2]
[arxiv:2112.05131 tab. 3]. The per-scene mean is 0.049 [arxiv:2112.05131 app. D].
Neither value is resolved here.

The PTN paper says its silhouette targets come from image observations, which
are trivial to segment on a white background [arxiv:1612.00814 §3]. The
released code builds those targets from ground-truth volumes instead (see the
card's Paper vs code section). The two do not agree on whether the
no-volume model needs 3D data. Neither is resolved here.

Within 2602.13549, the abstract says the method outperforms the state of the
art, but on the Waymo novel-view SSIM it is second: 0.720 against 0.722 for
StreetGaussians [arxiv:2602.13549 tab. 1]. The table is the source used here.

Dream Fields reports the same value in two rows of one table, and the same
value in two rows of its ablation table; these may be copy errors, so check a
cell before citing it [arxiv:2112.01455 tab. 1] [arxiv:2112.01455 tab. 2].

Dream Fields' R-Precision for its own model is 68.3 [arxiv:2112.01455 tab. 1].
CLIP-Mesh's comparison row gives Dreamfields 93.5 on the same kind of metric
with ViT-B/16 for generation and evaluation [arxiv:2203.13333 tab. 1]. CLIP-Mesh
scores on 153 captions taken from Dreamfields, two shapes per caption, and a
held-out 45° elevation [arxiv:2203.13333 §4.3]. The paper does not say where
its Dreamfields figures come from [arxiv:2203.13333 tab. 1]. Neither value is
resolved here; check the setup behind a number before comparing.

The text-to-3D cards report different kinds of results: Dream Fields and
CLIP-Mesh give a CLIP retrieval score, and SJC gives only renders
[arxiv:2112.01455 tab. 1] [arxiv:2203.13333 tab. 1] [arxiv:2212.00774 §5.2].
Magic3D reports a user preference against DreamFusion and no automatic metric
[arxiv:2211.10440 §5] [arxiv:2211.10440 tab. 1]. The cards cover different tasks
and renderers, so their numbers are not directly comparable.

CLIP-Mesh's text says the prior was not used with the B/16 generation model
[arxiv:2203.13333 §4.3], yet its ablation table has a "Prior loss" row under a
B/16 heading [arxiv:2203.13333 tab. 2]. The B/16 column falls at that row,
while the text says the prior gives the best overall results [arxiv:2203.13333
§4.4]. Neither reading is resolved here.

Point-NeRF gives NeRF's training time as 10 hours in its table, and as 20 hours
or more in its text [arxiv:2201.08845 tab. 1] [arxiv:2201.08845 §6.2]. Neither
value is resolved here.

SynSin's text and appendix give different learning rates. The text gives 0.0001
for the generator and 0.01 for the discriminator [arxiv:1912.08804 §3.4], while
the appendix gives 0.001 for its own model [arxiv:1912.08804 app. C]. Neither
is resolved here.

SynSin's Table 1 caption says the full model beats the other rows on all
metrics. The small fine-tuning ablation has a higher PSNR on Matterport3D, 21.14
against 20.91 [arxiv:1912.08804 tab. 1], so the caption does not hold for every
cell.

Text2Mesh's paper and its released code disagree on several settings. The
paper gives 1500 iterations, averages CLIP embeddings over views, and weights
its three terms equally. The code's default is 6000 iterations with per-view
similarity averaging, and its geometry term is off unless a flag is set. The
demo scripts follow the paper, and the card's Paper vs code section lists the
rest [arxiv:2112.03221 app. D.2] [arxiv:2112.03221 §3.2] [arxiv:2112.03221 eq. 2].
Neither is resolved here.

## Papers

- [Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision](/papers/1612.00814): predicts a voxel grid from one image and trains it on silhouettes projected by a fixed perspective layer.
- [Reconstructing Hand-Object Interactions in the Wild](/papers/2012.09856): fits a known object's pose by rendering its mask and depth, and optimises hand and object jointly with contact terms.
- [Plenoxels: Radiance Fields without Neural Networks](/papers/2112.05131): fits a sparse voxel grid by volume rendering and gradient descent, with no neural network.
- [MaPa: Text-driven Photorealistic Material Painting for 3D Shapes](/papers/2404.17569): fits a procedural material graph to a painted view with a differentiable material renderer.
- [Zero-Shot Text-Guided Object Generation with Dream Fields](/papers/2112.01455): optimises a NeRF so that its renders from random cameras match a caption under a frozen CLIP model.
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](/papers/2212.00774): passes a 2D diffusion score back through a voxel radiance field renderer.
- [SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM](/papers/2312.02126): tracks camera poses and refines a Gaussian map by rendering it differentiably.
- [PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing](/papers/2409.10141): carves an SMPL-X-initialised mesh to generated normal and silhouette views by differentiable remeshing.
- [Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting](/papers/2602.13549): adds diffuse and specular light with a Reinhard tone map to composite Gaussian driving scenes, fitted to night footage.
- [Point-NeRF: Point-based Neural Radiance Fields](/papers/2201.08845): renders a cloud of neural points by differentiable ray marching, starts from a feed-forward guess, and prunes and grows points per scene.
- [Magic3D: High-Resolution Text-to-3D Content Creation](/papers/2211.10440): creates a textured mesh from text in two stages, a hash-grid field with a 64x64 prior, then a mesh with a 512x512 latent prior.
- [SynSin: End-to-end View Synthesis from a Single Image](/papers/1912.08804): predicts features and depth from one image, splats them as a point cloud into a new view, and refines the result with a GAN, with no 3D supervision.
- [PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting](/papers/2104.00674): fits an SDF shape, albedo and spherical-Gaussian light to posed photos of a glossy object, so the light and material can be edited afterwards.
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](/papers/2112.03221): restyles a fixed mesh's colour and normal offsets from a text prompt, scored by CLIP on differentiable renders.
- [GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction](/papers/1902.05978): fits a morphable face model and a GAN skin texture to one photo, with face-recognition and pixel losses through a differentiable renderer.
- [CLIP-Mesh: Generating textured meshes from text using pretrained image-text models](/papers/2203.13333): optimises a subdivision-surface mesh, its texture and normal map from a text prompt, scored by CLIP and a diffusion prior through a differentiable renderer.