Differentiable rendering

Overview

The problem is recovering scene parameters (a pose, a voxel grid, a material, a Gaussian map, a mesh) from images, or generating them from a prompt, when a rendering model that maps those parameters to pixels is available. The core idea is to make the renderer differentiable with respect to its parameters, so that a loss on the rendered image gives gradients for the parameters 2112.05131 §3.1 2012.09856 §3.2. The shared invariant is that the render is a function of the parameters, and a scalar loss is minimised by a gradient method 2012.09856 eq. 5 2112.05131 eq. 2 2404.17569 §3.3 2312.02126 §3.

The loss does not always compare the render with photos. Dream Fields scores the render with a frozen CLIP model 2112.01455 §4.2, and SJC passes a frozen diffusion model's score back through the renderer 2212.00774 §4. Magic3D uses the same kind of diffusion score in both of its stages, and passes it through a volume renderer and then a mesh rasteriser 2211.10440 §3 2211.10440 §4.3.

Text2Mesh also scores renders with CLIP, but it changes only each vertex's colour and normal offset, so the mesh keeps its shape 2112.03221 §3.1 2112.03221 §3.2.

The sixteen cards differ in what is optimised, what the renderer is, and what judges the render:

• A GAN texture prior on a 3D morphable face model, rendered and compared with a single photo through deep face-recognition features 1902.05978 §3.1 1902.05978 §3.3.
• A neural mesh renderer, compared against an instance mask and a monocular depth map, for a known object's pose 2012.09856 §3.2.
• Volume rendering of a sparse voxel grid, compared against training photos, for a radiance field 2112.05131 §3.1.
• A procedural material graph turned into texture maps and rendered with a microfacet BRDF, compared against a 2D painted image, for surface materials 2404.17569 §3.3.
• A NeRF scored by a frozen CLIP model over random camera poses, with no photos 2112.01455 §4.2.
• A voxel radiance field driven by the score of a pretrained 2D diffusion model 2212.00774 §4.
• A Gaussian map, where each frame fits the camera pose and the map is then refined on keyframes, for dense RGB-D SLAM 2312.02126 §3.
• A remeshed SMPL-X body, carved to generated normal and silhouette views, for a single photo of a person 2409.10141 §3.2.
• A voxel grid predicted from one image, projected by a fixed perspective layer and compared with silhouettes from several views, for single-view reconstruction 1612.00814 §3.2.
• A Gaussian scene whose per-Gaussian colour is shaded by a global light model and four specular lobes, compared with night driving footage, for nighttime reconstruction 2602.13549 §3.2.
• A cloud of neural points with features and confidences, rendered by ray marching, and pruned and grown per scene, for novel views of posed photos 2201.08845 §3 2201.08845 §4.2.
• A two-stage text-to-3D pipeline: a hash-grid field scored by a 64x64 prior, then a mesh scored by a 512x512 latent prior through a differentiable rasteriser 2211.10440 §4.1 2211.10440 §4.3.
• A feature point cloud predicted from one image, splatted into a new view by a soft point renderer and refined by a GAN, with no depth labels 1912.08804 §3.1 1912.08804 §3.2.
• A signed distance surface and a spherical-Gaussian light fitted to posed photos of one glossy object under fixed light, so that light and material can be changed afterwards 2104.00674 §3.
• A fixed input mesh whose vertex colours and bounded normal offsets come from a neural field, scored by CLIP on augmented multi-view renders, for text-driven stylisation 2112.03221 §3.1 2112.03221 §3.2.
• A Loop-subdivision surface started from a sphere, with a texture and a normal map, scored by CLIP and a diffusion prior through a differentiable renderer, with a Laplacian smoothness term, for a text-to-mesh output 2203.13333 §3 2203.13333 eq. 4.

Variants

Mask and depth fit of a known object's pose

Scale, rotation and translation are optimised so the rendered silhouette matches an instance mask, and the rendered depth matches a monocular depth estimate 2012.09856 §3.2 2012.09856 eq. 3 2012.09856 eq. 4. The mask comes from a class-agnostic instance model, keeping the instance that overlaps the detected hand box most 2012.09856 §3.2. The fit starts from 1000 rotations, uniform in [−π, π], with translation at zero, and the lowest-loss result is kept 2012.09856 app. Implementation Details. The paper reports about 18 s for this step per image on a GTX 2080 Ti 2012.09856 app. Implementation Details. Card: Reconstructing Hand-Object Interactions in the Wild.

Joint objective with the rendered terms kept

The rendered mask and depth losses stay in one joint objective with a symmetric chamfer term between hand and object vertices and a collision term read from a voxel signed-distance grid 2012.09856 §3.3 2012.09856 eq. 6 2012.09856 eq. 8 2012.09856 eq. 9. The paper also refines the hand afterwards with a small learned network 2012.09856 §3.4. Card: Reconstructing Hand-Object Interactions in the Wild.

Volume rendering fit of a voxel grid

Each ray colour is the NeRF volume rendering sum over samples along the ray 2112.05131 §3.1 2112.05131 eq. 1. The grid stores opacity and degree-2 spherical harmonic colour at each occupied corner, interpolated trilinearly 2112.05131 §3.2 2112.05131 §3.3. The loss is a mean squared error on ray colours plus a total-variation term, with RMSProp as the optimiser 2112.05131 §3.5 2112.05131 eq. 2. Training starts on a coarse grid, prunes empty cells, and splits the rest in half 2112.05131 §3.4. No neural network is used. The authors report 31.71 PSNR on the 8 NeRF synthetic scenes, against 31.85 for JAXNeRF, and about 11 minutes per scene on one Titan RTX 2112.05131 tab. 2. The paper also describes a slower JAX implementation 2112.05131 §3.8. Card: Plenoxels: Radiance Fields without Neural Networks.

Material graph fit to a painted view

A procedural material graph is turned by DiffMat v2 into albedo, normal and roughness maps. These are sampled with UV maps and rendered with a microfacet BRDF, under a light field initialised by InvRenderNet and a residual light that starts at zero 2404.17569 §3.3 2404.17569 eq. 2. Before optimising, a search over UV scales and orientations picks the lowest Gram loss 2404.17569 supp.: material graph optimization. The loss sums an albedo term and a render term (masked L1 plus a VGG Gram style term), each with weight 1 2404.17569 supp.: material graph optimization. The graph is first chosen by retrieval from a library, and the fit starts from that graph 2404.17569 §3.3. The optimiser is Adam, with 300 iterations in total 2404.17569 §4.1. Card: MaPa: Text-driven Photorealistic Material Painting for 3D Shapes.

Silhouette projection of a predicted voxel grid (PTN)

A network predicts a 32³ occupancy grid from one 64×64 image [arxiv:1612.00814 §3.1] 1612.00814 §4. A fixed perspective layer samples the grid onto each training camera and takes the maximum along depth, which the paper says approximates ray tracing 1612.00814 §3.2 1612.00814 eq. 4. The loss is the squared error between predicted and observed silhouettes, averaged over views 1612.00814 eq. 1. A volume loss can be added with a weight, but the paper's aim is training when ground-truth volumes are not available 1612.00814 §3.1 1612.00814 eq. 2. The encoder is pre-trained with a rotation curriculum so that its code does not depend on the viewpoint 1612.00814 §3.3 1612.00814 §4. On chairs, the projection-only model reports a test IoU of 0.5027, against 0.5067 with volume supervision and 0.4983 for a volume-only CNN 1612.00814 tab. 1. The released Torch code takes its silhouette targets from projected ground-truth volumes, not from image masks, so the no-volume model as released uses 3D data (see the card's Paper vs code section). Card: Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision.

Text-scored NeRF with no photos (Dream Fields)

Cameras are sampled around the object, with azimuth uniform over 360° and the elevation fixed at 30° 2112.01455 §4.4. Each render is composited over a random background and scored by the negative similarity to the caption under a frozen CLIP model 2112.01455 §4.5 2112.01455 eq. 7. A transmittance penalty pushes the mean transmittance towards a target, set to 88% and annealed in from 40% 2112.01455 §4.5 2112.01455 eq. 8. The scene is kept inside a cube of side 2 2112.01455 §4.6. The network is a residual MLP with 280K parameters, against 494K for the NeRF baseline 2112.01455 §4.7. The transmittance weight λ is not stated 2112.01455 eq. 9. The authors report a CLIP R-Precision of 68.3 at 20K iterations with 252-pixel renders 2112.01455 tab. 1. About 1 h 12 min is their estimate for 10K iterations on eight TPU cores 2112.01455 app. C. Card: Zero-Shot Text-Guided Object Generation with Dream Fields.

Score distillation through a voxel renderer (SJC)

A voxel radiance field is rendered from random cameras. A pretrained 2D diffusion denoiser gives a score for each clean render, and that score is chained through the renderer, so the 3D score is the expected 2D score times the renderer Jacobian 2212.00774 §4 2212.00774 eq. 11. A clean render is out of distribution for a denoiser trained on noisy inputs, so the score is estimated by adding noise and averaging the denoiser's outputs (NEScore) 2212.00774 §4.1 2212.00774 eq. 16. An emptiness loss on the ray weights, with β = 10, suppresses floating density 2212.00774 §4.2. The text-to-3D run uses Stable Diffusion v1.5 on a four-channel latent feature grid 2212.00774 §5.2. The paper reports about 25 minutes per object on one A6000 2212.00774 app. A3. It reports no quantitative metric; the results are shown visually 2212.00774 §5.2 2212.00774 fig. 6. Card: Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation.

Per-frame pose fit and map refinement (SplaTAM)

For each new RGB-D frame, the camera pose is fitted by rendering the Gaussian map and minimising depth and colour error, over pixels whose silhouette is above 0.99, while the map is held fixed 2312.02126 §3 2312.02126 eq. 8. The Gaussians are then densified where the map is sparse or misses geometry, and refined over a window of keyframes with the poses fixed 2312.02126 §3. The paper reports an average tracking error of 0.36 cm on Replica, against 0.52 cm for Point-SLAM, and about 1.2 cm on ScanNet++, where Point-SLAM and ORB-SLAM3 fail to track 2312.02126 tab. 1. On Replica R0, one tracking iteration takes 25 ms and one mapping iteration 24 ms on an RTX 3080 Ti 2312.02126 tab. 6. Replica uses 40 tracking and 60 mapping iterations per frame, and a lighter SplaTAM-S setting uses 10 and 15 2312.02126 §5. The method needs known camera intrinsics and dense depth 2312.02126 §5. Card: SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM.

Remeshing an SMPL-X body to generated views (PSHuman)

A parametric body is fitted first. Its translation, shape and pose are optimised so that its normal and silhouette renders match the generated body views, weighted by per-view confidence 2409.10141 §3.2 2409.10141 eq. 7. Vertices and face connectivity are then optimised against the same weighted normal and silhouette mismatch, plus a Laplacian smoothness term with weight 0.02 2409.10141 §3.2 2409.10141 eq. 8. The six views come from a multiview diffusion model, conditioned on renders of SMPL-X 2409.10141 §3.1. On THuman2.1, the paper reports a Chamfer distance of 0.4399 against 0.5754 for the best baseline, SIFU 2409.10141 tab. 1. It estimates about one minute end to end 2409.10141 §1. The released code uses a different texturing method from the one the paper describes, so the paper's texture step should not be copied as-is 2409.10141 §3.2. Card: PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing.

Physically based shading of Gaussians at night (driving scenes)

Each Gaussian carries an albedo, a roughness, a metalness and a normal. Its diffuse light is a Lambertian term written with a second-degree spherical-harmonics light, which a global MLP predicts from a timestep and camera embedding 2602.13549 §3.4 2602.13549 eq. 4. Its specular light comes from four anisotropic spherical Gaussian lobes per Gaussian, combined with a simplified Disney BRDF 2602.13549 §3.5 2602.13549 eq. 6 2602.13549 eq. 7. The two parts are summed in HDR and compressed with Reinhard before the standard rasteriser draws the scene 2602.13549 §3.6 2602.13549 eq. 8 2602.13549 eq. 9. The loss is a weighted RGB term, D-SSIM term and depth-normal term, optimised with Adam for 40,000 iterations per driving sequence 2602.13549 eq. 14 2602.13549 §4.2. The authors report 31.9 dB reconstruction PSNR on 11 Waymo night scenes, against 31.0 for OmniRe, and 29.7 dB on 11 nuScenes night scenes, against 28.7 2602.13549 tab. 1 2602.13549 tab. 2. In their ablation, removing the specular or the diffuse part lowers PSNR 2602.13549 tab. 3. The per-sequence fit is a scalability limit the authors name themselves 2602.13549 §5. Card: Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting.

Point-based radiance field with per-scene pruning and growing (Point-NeRF)

A radiance field is stored as a set of points, each with a feature and a confidence in [0, 1] 2201.08845 §3. A shaded location uses its eight nearest points within a radius 2201.08845 app. H. Their features pass through a small MLP, are blended with inverse-distance weights scaled by confidence, and become colour and density 2201.08845 eq. 3 2201.08845 eq. 4 2201.08845 eq. 6. Pixels are formed by volume rendering along each ray 2201.08845 eq. 1. The starting points come from a cross-scene network built on MVS depth, which the paper times at about 0.2 s for three views 2201.08845 §4.1 2201.08845 §5. Per scene, the rendering loss is minimised with a sparsity term on confidences 2201.08845 eq. 12. Every 10K iterations, points with confidence below 0.1 are pruned, and a point is grown where a ray's most opaque sample lies far from every point 2201.08845 §4.2 2201.08845 eq. 10. The growing thresholds are not stated 2201.08845 §4.2. On NeRF Synthetic, the authors report 30.71 PSNR after 20K steps, about 40 minutes, and 33.31 after 200K steps, against 31.01 for NeRF 2201.08845 tab. 2 2201.08845 §6.2. Card: Point-NeRF: Point-based Neural Radiance Fields.

Two-stage text-to-mesh with differentiable marching tetrahedra (Magic3D)

The loss is score distillation: a diffusion prior's noise prediction is back-propagated through the renderer to the scene parameters 2211.10440 §3 2211.10440 eq. 1. Stage 1 optimises a hash-grid neural field with a 64x64 base diffusion prior, for 5000 steps with up to 1024 samples per ray 2211.10440 §4.1 2211.10440 §5. Stage 2 turns the density into a signed distance on a deformable tetrahedral grid, extracts a surface with differentiable marching tetrahedra, and optimises the vertex offsets and a neural texture with a 512x512 latent diffusion prior, for 3000 steps 2211.10440 §4.2 2211.10440 §4.3 2211.10440 §5. A differentiable rasteriser tracks the 3D point under each pixel, which is how the texture is queried 2211.10440 §4.3. An angular regulariser between adjacent faces keeps the surface smooth under the noisy gradient 2211.10440 §4.3. Stage 2 samples timesteps from (0.02, 0.5) with weight σ_t² 2211.10440 app. B. The authors report about 40 minutes on 8 A100 GPUs in total, and 61.7% of user-study raters prefer it to DreamFusion for realism and detail 2211.10440 §5 2211.10440 tab. 1. Card: Magic3D: High-Resolution Text-to-3D Content Creation.

Feed-forward point splatting from one image (SynSin)

A spatial feature network gives each pixel a 64-channel feature, and a depth network gives each pixel a depth 1912.08804 §3.1. The pixels are lifted into 3D points, moved by the relative camera pose, and splatted onto a disk of radius r around their projected centre 1912.08804 §3.2 1912.08804 eq. 1. Each pixel keeps its K nearest points, sorted by depth, and composites them front to back 1912.08804 eq. 2. Gradients reach every kept point, and reach both features and depths 1912.08804 §3.2. A refinement network fills the holes and a discriminator judges the output 1912.08804 §3.3. The training loss uses only image pairs and their relative poses 1912.08804 §1 1912.08804 §3.4. The paper reports 20.91 PSNR on Matterport3D and 22.31 on RealEstate10K for its full model, from one input view 1912.08804 tab. 1 1912.08804 tab. 2. On RealEstate10K, the two-view StereoMag baseline reports 25.34 PSNR, which is higher 1912.08804 tab. 2. A user study over 100 trajectories prefers SynSin to a voxel baseline in 68.7% of majority votes 1912.08804 tab. 4. The renderer takes 36 ms forward and 5 ms backward for a batch of six clouds of 512² points on one V100 1912.08804 §3.2. Card: SynSin: End-to-end View Synthesis from a Single Image.

Light and material separation on a glossy object (PhySG)

The shape is a signed distance function, written as an MLP with positional encoding and read with sphere tracing 2104.00674 §3 "Geometry modeling". The diffuse albedo comes from a second MLP of the surface point, and the specular term is one monochrome lobe 2104.00674 §3 "Appearance modeling". The light is an environment map of 128 spherical Gaussians [arxiv:2104.00674 §3 "Appearance modeling"]. Shading is evaluated in closed form, so no sampling of the light is needed 2104.00674 §3. The training loss has an L1 colour term on object pixels, a penalty on rays that miss the object, a unit-norm constraint on the SDF gradient and a smoothness term on normals inside small patches 2104.00674 §3 "Loss functions". Surface points are found by sphere tracing, which is not differentiated; gradients reach the surface point through the implicit function 2104.00674 §3 "Geometry modeling". The authors report a synthetic novel-view PSNR of 35.93 and a relighting PSNR of 33.25 2104.00674 tab. 1. On real scenes the paper says IDR is slightly better on the reported metrics, which the authors attribute to bias in the physics-based model 2104.00674 tab. 3. The model assumes distant light, one material per object, and no self-occlusion or indirect light 2104.00674 §3. Card: PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting.

Text-driven stylisation of a fixed mesh (Text2Mesh)

The input mesh's faces stay fixed. A neural style field maps each vertex to a colour and to a displacement along its normal, and the displacement is bounded 2112.03221 §3.1. Each vertex is encoded with Fourier features before the field runs 2112.03221 §3.1 2112.03221 eq. 1. The objective is a sum of cosine similarities between CLIP embeddings of the prompt and of augmented renders from about five views around an anchor 2112.03221 §3.2 2112.03221 §3.3 2112.03221 eq. 5. A second, grey render of the displaced mesh is scored on its own, so the geometry gets a gradient of its own 2112.03221 §3.2 2112.03221 eq. 4. Gradients come through an interpolation-based differentiable renderer 2112.03221 §3.1. The authors report that without augmentation the result follows no target text, and that without the geometry-only term the network imitates shape with shading 2112.03221 §4.2 2112.03221 fig. 5. Card: Text2Mesh: Text-Driven Neural Stylization for Meshes.

Photo fit of a 3D face with a GAN texture prior (GANFIT)

The face is a linear morphable model of shape and expression, with a camera and lighting, all optimised 1902.05978 §2.1 1902.05978 eq. 2. The skin is a generator from a 512-value latent to a UV map, trained as a GAN on about 10,000 UV textures 1902.05978 §3.1 1902.05978 eq. 5 1902.05978 §4.1. The render is compared with the photo through four terms: a cosine distance between face-recognition embeddings, a content loss on intermediate layers of that network, a full-resolution L1 pixel loss, and a landmark loss 1902.05978 eq. 8 1902.05978 eq. 9 1902.05978 eq. 10 1902.05978 eq. 11. The fit runs in two stages: the landmark loss alone first, then all parameters together under a weighted sum with a regulariser 1902.05978 §3.4 [arxiv:1902.05978 eq. 12]. The optimiser is Adam with learning rate 0.01 1902.05978 §4.1. On the cooperative setting of MICC Florence, the authors report a mean point-to-plane error of 0.95 against 1.50 for the best baseline there 1902.05978 tab. 1. The fit takes about 30 s per image on one GTX 1080 Ti 1902.05978 §4.1. Card: GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction.

Text-to-mesh from a subdivision surface with CLIP and a diffusion prior (CLIP-Mesh)

The shape is a Loop subdivision limit surface defined by a set of control vertices, and the texture and normal map are free maps 2203.13333 §3. Each step renders random views, encodes them with CLIP, and minimises the mean negative cosine similarity to the prompt 2203.13333 eq. 1. A uniform Laplacian over one-ring neighbours keeps the shape smooth, with a weight that decays during training 2203.13333 eq. 2 2203.13333 §3. A diffusion prior conditioned on the prompt gives a second CLIP target, weighted by α 2203.13333 eq. 3 2203.13333 eq. 4. Each view draws its background, position offset, field of view and distance at random, so the texture cannot carry the result on its own 2203.13333 §3. The authors leave the prior loss out of the Table 1 setting, because the prior was trained on ViT-B/32 embeddings only 2203.13333 §4.3. Against Dreamfields, the paper reports CLIP R-precision of 96.7 against 93.5 with ViT-B/16 for both generation and evaluation, and 91.4 against 86.6 with ViT-B/32 2203.13333 tab. 1. The authors report about 50 minutes per shape on one 16 GB GPU, against more than 24 hours for Dreamfields on four A100s 2203.13333 §4.1. Card: CLIP-Mesh: Generating textured meshes from text using pretrained image-text models.

When to use which

Implementation sketch

Our own notation for the common core. Each variant changes the renderer, the objective and the optimiser. The objective is either a mismatch to observed images or a frozen model's judgement of the render. The optimiser is not stated for the object fit 2012.09856 §3.2, so it is left as a placeholder.

# Common core: fit parameters by differentiable rendering
# [arxiv:2012.09856 §3.2] [arxiv:2112.05131 §3.5] [arxiv:2404.17569 §3.3]
def fit_by_rendering(params, target, renderer, steps):
    for step in range(steps):
        # render the current parameters into image space; differentiable
        # in params  [arxiv:2112.05131 §3.1] [arxiv:2404.17569 eq. 2]
        predicted = renderer(params)
        # objective: a mismatch to observations, or a frozen model's score
        # on the render  [arxiv:2012.09856 eq. 5] [arxiv:2112.01455 eq. 7]
        loss = objective(predicted, target)
        # gradient method; the optimiser differs per paper
        # [arxiv:2112.05131 §3.5] [arxiv:2404.17569 §4.1]
        params = gradient_step(loss, params)
    return params

Variant notes. The object fit wraps this core in a loop over 1000 random rotations and keeps the lowest loss [arxiv:2012.09856 app. Implementation Details]. The voxel fit adds a coarse-to-fine prune and split between steps 2112.05131 §3.4. The material fit starts from a retrieved graph instead of a random start 2404.17569 §3.3. Dream Fields replaces the mismatch with a negative CLIP similarity and draws a random pose each step 2112.01455 §4.2 2112.01455 §4.4. SJC has no loss to differentiate: it feeds the denoiser score in as the gradient at the render, so the score's scale matters. The paper's score divides by σ², while its released code divides by σ, so the two differ by a factor σ 2212.00774 eq. 3 2212.00774 eq. 16. SplaTAM uses the same loop per frame, with the pose as the parameter and the Gaussians as the map 2312.02126 §3. PSHuman uses mesh vertices and topology as the parameters, with a normal-and-silhouette mismatch as the objective 2409.10141 §3.2. PTN uses the occupancy grid as the parameters, predicted by a network rather than optimised per object, with a per-view squared silhouette error as the objective 1612.00814 §3.2 1612.00814 eq. 1. The night-driving shading is a per-Gaussian colour step before rasterisation: Reinhard is applied to each Gaussian's HDR colour, and the results are then alpha-blended, so the pixel is not the Reinhard value of the blended radiance 2602.13549 §3.6 2602.13549 eq. 2. The paper does not discuss this choice 2602.13549 §3.6.

Point-NeRF and Magic3D add two more cases. Point-NeRF's pipeline uses the same core loop with per-point features and confidences as parameters, and the rendering loss as objective 2201.08845 §4.2. Magic3D's stage 2 uses the mesh vertices and texture as parameters, with the diffusion score as the gradient at the rendered pixels 2211.10440 §4.3. Magic3D's stage 1 uses the hash grid as its parameters 2211.10440 §4.2. Its stage 2 also has no loss to differentiate in the usual sense, so the same caveat as SJC applies.

SynSin's parameters are not fitted per scene: its feature and depth networks output the point cloud for each input image, and the renderer's gradients reach those outputs 1912.08804 §3.1 1912.08804 §3.2. The objective is an image loss with an adversarial term 1912.08804 §3.4. PhySG fits its parameters per object: the SDF and albedo MLPs and the light lobes, with the colour loss on object pixels as the main term 2104.00674 §3 "Loss functions". Its surface point comes from sphere tracing, which is not differentiated, so the gradient is taken through the implicit surface at that point 2104.00674 §3 "Geometry modeling".

Text2Mesh fits per input mesh. Its parameters are the style field's weights, evaluated at each vertex, and the mesh is updated only through the bounded normal offsets 2112.03221 §3.1. The objective sums three CLIP similarities: the full colour render, a cropped full colour render, and a cropped grey render of the displaced mesh. The grey term updates only the shared trunk and the displacement branch 2112.03221 eq. 5 2112.03221 §3.2.

Contradictions

Within one paper, the LPIPS for the synthetic scenes is 0.049 in the main table and 0.050 in the ablation tables 2112.05131 tab. 2 2112.05131 tab. 3. The per-scene mean is 0.049 2112.05131 app. D. Neither value is resolved here.

The PTN paper says its silhouette targets come from image observations, which are trivial to segment on a white background 1612.00814 §3. The released code builds those targets from ground-truth volumes instead (see the card's Paper vs code section). The two do not agree on whether the no-volume model needs 3D data. Neither is resolved here.

Within 2602.13549, the abstract says the method outperforms the state of the art, but on the Waymo novel-view SSIM it is second: 0.720 against 0.722 for StreetGaussians 2602.13549 tab. 1. The table is the source used here.

Dream Fields reports the same value in two rows of one table, and the same value in two rows of its ablation table; these may be copy errors, so check a cell before citing it 2112.01455 tab. 1 2112.01455 tab. 2.

Dream Fields' R-Precision for its own model is 68.3 2112.01455 tab. 1. CLIP-Mesh's comparison row gives Dreamfields 93.5 on the same kind of metric with ViT-B/16 for generation and evaluation 2203.13333 tab. 1. CLIP-Mesh scores on 153 captions taken from Dreamfields, two shapes per caption, and a held-out 45° elevation 2203.13333 §4.3. The paper does not say where its Dreamfields figures come from 2203.13333 tab. 1. Neither value is resolved here; check the setup behind a number before comparing.

The text-to-3D cards report different kinds of results: Dream Fields and CLIP-Mesh give a CLIP retrieval score, and SJC gives only renders 2112.01455 tab. 1 2203.13333 tab. 1 2212.00774 §5.2. Magic3D reports a user preference against DreamFusion and no automatic metric 2211.10440 §5 2211.10440 tab. 1. The cards cover different tasks and renderers, so their numbers are not directly comparable.

CLIP-Mesh's text says the prior was not used with the B/16 generation model 2203.13333 §4.3, yet its ablation table has a "Prior loss" row under a B/16 heading 2203.13333 tab. 2. The B/16 column falls at that row, while the text says the prior gives the best overall results [arxiv:2203.13333 §4.4]. Neither reading is resolved here.

Point-NeRF gives NeRF's training time as 10 hours in its table, and as 20 hours or more in its text 2201.08845 tab. 1 2201.08845 §6.2. Neither value is resolved here.

SynSin's text and appendix give different learning rates. The text gives 0.0001 for the generator and 0.01 for the discriminator 1912.08804 §3.4, while the appendix gives 0.001 for its own model 1912.08804 app. C. Neither is resolved here.

SynSin's Table 1 caption says the full model beats the other rows on all metrics. The small fine-tuning ablation has a higher PSNR on Matterport3D, 21.14 against 20.91 1912.08804 tab. 1, so the caption does not hold for every cell.

Text2Mesh's paper and its released code disagree on several settings. The paper gives 1500 iterations, averages CLIP embeddings over views, and weights its three terms equally. The code's default is 6000 iterations with per-view similarity averaging, and its geometry term is off unless a flag is set. The demo scripts follow the paper, and the card's Paper vs code section lists the rest 2112.03221 app. D.2 2112.03221 §3.2 2112.03221 eq. 2. Neither is resolved here.

Papers

• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision: predicts a voxel grid from one image and trains it on silhouettes projected by a fixed perspective layer.
• Reconstructing Hand-Object Interactions in the Wild: fits a known object's pose by rendering its mask and depth, and optimises hand and object jointly with contact terms.
• Plenoxels: Radiance Fields without Neural Networks: fits a sparse voxel grid by volume rendering and gradient descent, with no neural network.
• MaPa: Text-driven Photorealistic Material Painting for 3D Shapes: fits a procedural material graph to a painted view with a differentiable material renderer.
• Zero-Shot Text-Guided Object Generation with Dream Fields: optimises a NeRF so that its renders from random cameras match a caption under a frozen CLIP model.
• Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation: passes a 2D diffusion score back through a voxel radiance field renderer.
• SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM: tracks camera poses and refines a Gaussian map by rendering it differentiably.
• PSHuman: Photorealistic Single-image 3D Human Reconstruction using Cross-Scale Multiview Diffusion and Explicit Remeshing: carves an SMPL-X-initialised mesh to generated normal and silhouette views by differentiable remeshing.
• Nighttime Autonomous Driving Scene Reconstruction with Physically-Based Gaussian Splatting: adds diffuse and specular light with a Reinhard tone map to composite Gaussian driving scenes, fitted to night footage.
• Point-NeRF: Point-based Neural Radiance Fields: renders a cloud of neural points by differentiable ray marching, starts from a feed-forward guess, and prunes and grows points per scene.
• Magic3D: High-Resolution Text-to-3D Content Creation: creates a textured mesh from text in two stages, a hash-grid field with a 64x64 prior, then a mesh with a 512x512 latent prior.
• SynSin: End-to-end View Synthesis from a Single Image: predicts features and depth from one image, splats them as a point cloud into a new view, and refines the result with a GAN, with no 3D supervision.
• PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting: fits an SDF shape, albedo and spherical-Gaussian light to posed photos of a glossy object, so the light and material can be edited afterwards.
• Text2Mesh: Text-Driven Neural Stylization for Meshes: restyles a fixed mesh's colour and normal offsets from a text prompt, scored by CLIP on differentiable renders.
• GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction: fits a morphable face model and a GAN skin texture to one photo, with face-recognition and pixel losses through a differentiable renderer.
• CLIP-Mesh: Generating textured meshes from text using pretrained image-text models: optimises a subdivision-surface mesh, its texture and normal map from a text prompt, scored by CLIP and a diffusion prior through a differentiable renderer.