--- kind: concept slug: neural-implicit-surface title: Neural implicit surfaces summary: >- Neural implicit surfaces store shapes as learned scalar fields and read the surface from a level set. NeuS fits a signed distance, and UNISURF fits occupancy. PhySG adds material and light. CLIP-Forge decodes occupancy from text, iSDF fits online, One-2-3-45 predicts one from one photo, and NeuralRecon, a discrete exception, outputs a voxel TSDF per fragment. tags: [3d-reconstruction, sdf, volume-rendering, multi-view, occupancy, text-to-shape, online-mapping, robotics, single-view-3d, feed-forward, inverse-rendering, relighting, monocular-video, tsdf, unsigned-distance, human-object-interaction, body-model-fitting] papers: ["2104.00674", "2104.00681", "2104.10078", "2106.10689", "2110.02624", "2204.02296", "2204.06950", "2306.16928"] related: [neural-radiance-field, multi-view-reconstruction, gaussian-splatting, implicit-neural-representation, normalizing-flows, simultaneous-localization-and-mapping, feed-forward-reconstruction, diffusion-models, physically-based-rendering, differentiable-rendering, positional-encoding, monocular-depth-estimation, dense-correspondence] status: draft updated_at: 2026-11-01 --- ## Overview A neural implicit surface stores the shape as a network $f$ that maps a 3D point to a scalar value. In NeuS, $f$ is the signed distance from the surface: it is positive outside and negative inside, and the surface is the zero level set [arxiv:2106.10689 §3.1] [arxiv:2106.10689 eq. 1]. In multi-view methods such as NeuS, training fits $f$ and a colour network to posed photographs, and the mesh is read off the level set afterwards. UNISURF predicts occupancy instead, and its surface is the 0.5 level set [arxiv:2104.10078 §3] [arxiv:2104.10078 eq. 1]. CLIP-Forge also predicts occupancy, and its surface is a threshold on that value [arxiv:2110.02624 §3.1] [arxiv:2110.02624 §3.3]. iSDF also fits a signed distance, but to posed depth frames that arrive as a stream during robot exploration [arxiv:2204.02296 §4]. One-2-3-45 also predicts an SDF and renders it with NeuS-style weights. It needs no per-shape fitting, because one feed-forward network builds the SDF from views that an image model generates [arxiv:2306.16928 §1] [arxiv:2306.16928 §3.3]. PhySG also fits an SDF, from posed photos of one glossy object under static light. It sphere-traces that SDF to the surface and adds a material and an environment light [arxiv:2104.00674 §3]. BEHAVE is the other kind of use. It predicts unsigned distances from a human and from a known object to query points, and it fits a body model and an object template to those values rather than extracting a level set [arxiv:2204.06950 §4.1] [arxiv:2204.06950 §4.2]. What supervises $f$ differs between the variants. Surface renderers intersect one ray with the surface per pixel. They need object masks and can get stuck in local minima on occluded or thin objects [arxiv:2106.10689 §1]. PhySG is a surface renderer, and its loss separates object pixels from background pixels [arxiv:2104.00674 §3]. Volume renderers integrate along the whole ray, which trains more robustly. Their density has no surface constraint, so meshes taken from it are noisy [arxiv:2106.10689 §1]. UNISURF integrates only over a window around the first surface crossing, and the window narrows during training [arxiv:2104.10078 §4.3]. iSDF renders nothing. Each sampled point is supervised by a bound: the signed distance to the nearest measured surface point in the batch [arxiv:2204.02296 §4.3] [arxiv:2204.02296 eq. 4]. The requirement all volume-rendering variants share is about the rendering weight $w(t)$ along a ray. It should peak at the surface (unbiased), and when two surfaces lie on one ray, the nearer one should hide the farther one (occlusion-aware) [arxiv:2106.10689 §3.1]. The variants below differ in which of the two they satisfy. NeuralRecon is the exception in this set. It stores its geometry as a sparse voxel grid of truncated distances rather than as a function that can be evaluated at any point, and it is supervised on those voxels directly, with no rendering step [arxiv:2104.00681 §3.2] [arxiv:2104.00681 §3.3]. ## Variants ### NeuS: an opaque density from a logistic CDF Card: [NeuS (Wang et al., arXiv:2106.10689)](/papers/2106.10689). - **Density.** The opacity between two samples comes from the logistic CDF of the SDF, so each segment's alpha is the drop in that CDF divided by the CDF at the near end, clamped at zero [arxiv:2106.10689 eq. 13]. - **Weight.** The weight is the usual transmittance times alpha. For a locally flat surface it peaks at the surface, and it stays occlusion-aware [arxiv:2106.10689 §3.1] [arxiv:2106.10689 app. B]. - **Sharpness.** One trainable scalar $s$ sets how thick the layer around the surface is. Its inverse is expected to approach zero as training converges [arxiv:2106.10689 §3.1]. On curved surfaces the peak offset is second order in $1/s$, so the bias shrinks as $s$ grows [arxiv:2106.10689 app. C]. - **Loss.** A colour term, an Eikonal term that keeps $f$ a true distance field, and an optional mask term [arxiv:2106.10689 §3.2]. - **Mesh.** Extracted from the zero level set of $f$ [arxiv:2106.10689 eq. 1]. ### Naive SDF-to-density conversion The obvious way to reuse volume rendering takes density as the logistic density of the SDF, $\sigma = \phi_s(f)$, and the standard weight $T\sigma$. This is occlusion-aware but biased: the weight is already falling at the surface, so its peak lies in front of it [arxiv:2106.10689 §3.1] [arxiv:2106.10689 app. B]. On one DTU scene, the paper's ablation gives a Chamfer distance of 1.49, against 0.59 for NeuS [arxiv:2106.10689 §4.3]. ### Straightforward normalised S-density Normalising the logistic density along the ray gives a weight that is unbiased but not occlusion-aware. Two surfaces on one ray then blend about equally [arxiv:2106.10689 §3.1]. In the same ablation it gives a Chamfer distance of 4.45 and SDF error of 44.34, the worst of the three conversions [arxiv:2106.10689 §4.3]. ### UNISURF: occupancy with a shrinking sample window Card: [UNISURF (Oechsle et al., arXiv:2104.10078)](/papers/2104.10078). This variant models the surface as an occupancy field $o_\theta$ rather than an SDF. One rendering formula serves both surface and volume rendering, depending on how the samples are placed [arxiv:2104.10078 §4.1]. - **Weight.** The NeRF alpha blending is kept, with occupancy in place of alpha. Each sample's weight is its occupancy times the visibility in front of it, so occlusion is handled by construction [arxiv:2104.10078 §4.1] [arxiv:2104.10078 eq. 7]. - **Sample window.** Samples are stratified in a window around the first surface crossing. The half-width decays exponentially during training, from 1.0 to 0.05 in the paper's DTU setting [arxiv:2104.10078 §4.3] [arxiv:2104.10078 eq. 13] [arxiv:2104.10078 eq. 14] [arxiv:2104.10078 §4.4]. - **Limit.** The authors argue that volume rendering approaches surface rendering as the window shrinks and the sample count grows. The proof is in a supplement that is not in our source [arxiv:2104.10078 §4.1]. This is a limit argument, not the first-order weight-peak analysis that NeuS gives [arxiv:2106.10689 §3.1]. - **Loss.** An L1 colour loss over sampled rays, plus a term that ties the normal at a surface point to the normal at a nearby random point. The paper's loss has no Eikonal term [arxiv:2104.10078 eq. 9] [arxiv:2104.10078 eq. 10] [arxiv:2104.10078 eq. 11]. - **Results.** Author-reported mean DTU Chamfer 1.02 without masks, against 0.90 for IDR with masks [arxiv:2104.10078 tab. 1]. The evaluation cleans the meshes with the object masks, so "without masks" describes training only [arxiv:2104.10078 §5.2]. - **Ablation.** On the Buddha scene, surface rendering alone does not converge, and a uniform volume window gives a worse Chamfer than the shrinking window [arxiv:2104.10078 fig. 7] [arxiv:2104.10078 §5.5]. ### PhySG: a surface renderer with spherical-Gaussian shading Card: [PhySG (Zhang et al., arXiv:2104.00674)](/papers/2104.00674). This variant is for relighting and material editing. It takes posed photos of one glossy object under one static, distant light, plus object masks [arxiv:2104.00674 §3]. It fits an SDF, a spatially varying albedo, one monochrome specular lobe and a 128-lobe environment map [arxiv:2104.00674 §3]. - **Rendering.** Each ray is sphere-traced to its surface point, with no gradient through the marching. Gradients reach the surface point through implicit differentiation [arxiv:2104.00674 §3 "Geometry modeling"]. - **Normal.** The normal is the gradient of the SDF at the surface point [arxiv:2104.00674 §3]. - **Shading.** The light is a spherical-Gaussian mixture, and the rendering equation is evaluated in closed form. No Monte Carlo sampling is used for the light integral [arxiv:2104.00674 §3 "Appearance modeling"]. - **Loss.** A colour term, a term that keeps the SDF non-negative on rays that miss the object, an Eikonal term, and a smoothness term [arxiv:2104.00674 §3 "Loss functions"]. - **Assumptions.** The light is distant, each object has one shared monochrome specular term, and the rendering omits self-occlusion and indirect light [arxiv:2104.00674 §3]. - **Matte surfaces.** Under Lambertian materials, light and albedo are ambiguous, and the paper says light cannot be recovered without extra priors [arxiv:2104.00674 §4 "Robustness to material roughness"]. - **Results.** On synthetic data, novel-view PSNR is 35.93 and relighting PSNR is 33.25 [arxiv:2104.00674 tab. 1]. On the synthetic geometry table, PhySG's normal error is 2.528 degrees and IDR's is 2.207, with Chamfer L1 of 0.00142 against 0.00136 [arxiv:2104.00674 tab. 2]. On real data the paper's table shows IDR slightly ahead on the same metrics, which the authors attribute to bias in the physics-based appearance model [arxiv:2104.00674 tab. 3]. - **Compute.** The paper trains on one 12 GB GPU for 250k iterations [arxiv:2104.00674 §3 "Loss functions"]. ### CLIP-Forge: text-to-shape with a conditional flow prior Card: [CLIP-Forge (Sanghi et al., arXiv:2110.02624)](/papers/2110.02624). This variant is generative. It learns a prior over shapes rather than fitting one object from photos, and its surface comes from a threshold on occupancy. - **Shape code.** A voxel encoder maps a 32^3 grid to a 128-d embedding, and an implicit decoder predicts occupancy at query points, trained with mean squared error [arxiv:2110.02624 §3.1] [arxiv:2110.02624 app. A]. - **Prior.** A conditional RealNVP flow maps shape embeddings to a unit Gaussian. Its condition is the CLIP image feature of a rendered view [arxiv:2110.02624 §3.2]. - **Text at test time.** The CLIP text feature replaces the image feature as the condition, and the flow is inverted on Gaussian noise [arxiv:2110.02624 §3.3]. The paper does not measure how closely the two feature types match [arxiv:2110.02624 §3.3]. - **Several shapes.** One prompt gives several shapes by sampling the flow again. Nothing is optimised per request [arxiv:2110.02624 §1] [§3.3]. - **Scope.** Trained on ShapeNet's 13 classes and generalises poorly beyond them [arxiv:2110.02624 app. F] [arxiv:2110.02624 §6]. It produces geometry only [arxiv:2110.02624 app. H]. ### iSDF: online SDF from depth frames Card: [iSDF (Ortiz et al., arXiv:2204.02296)](/papers/2204.02296). This variant is online and needs no photos or ground-truth SDF. A randomly initialised MLP is trained as depth frames arrive, with no pretraining [arxiv:2204.02296 §1] [arxiv:2204.02296 §4]. - **Sampling.** Each iteration takes 200 random pixels from each of five frames: the two latest and three keyframes. Along each ray it takes 20 stratified depths, 8 Gaussian depths around the measured depth, and the measured depth itself [arxiv:2204.02296 §4.2]. - **Bound.** Each sample is bounded by the signed distance to the nearest measured surface point in the batch. The paper says this bound is used in all its experiments [arxiv:2204.02296 §4.3] [arxiv:2204.02296 eq. 4]. - **Loss.** Samples in free space get a one-sided loss that is zero between 0 and the bound. Samples near the measured surface get an L1 term to the bound. A cosine term supervises the gradient, and an Eikonal term applies beyond 10 cm from the surface [arxiv:2204.02296 eq. 6] [arxiv:2204.02296 eq. 8] [arxiv:2204.02296 eq. 10]. - **Truncation.** The paper does not state the truncation distance $t$, so an implementer must pick one [arxiv:2204.02296 §4.4]. - **Speed.** The authors report about 33 ms per training iteration on an i7-8700K with an RTX 2080. The split is about 2 ms sampling, 2 ms bounds, 10 ms forward and 19 ms backward [arxiv:2204.02296 §5.2] [arxiv:2204.02296 §6.3]. Frame rate and tracking are not evaluated. - **Memory.** About 1 MB for the network plus around 20 keyframes. The authors' comparison gives 5 MB for Voxblox and 30 MB for KinectFusion+ [arxiv:2204.02296 §6.3]. - **Accuracy.** Author-reported: SDF error below 6 cm at the end of each of the six main sequences, and lower error than Voxblox and KinectFusion+. The baseline values appear only in a figure [arxiv:2204.02296 §6.2] [arxiv:2204.02296 fig. 8]. - **Assumptions.** Poses come from an external tracker, so this is not SLAM [arxiv:2204.02296 §4]. Evaluation uses simulated ReplicaCAD and handheld ScanNet sequences [arxiv:2204.02296 §5.1]. ### One-2-3-45: feed-forward SDF from generated views Card: [One-2-3-45 (Liu et al., arXiv:2306.16928)](/papers/2306.16928). This variant takes one photo and fits nothing per shape. A frozen image model imagines the object from other angles, and one network builds the SDF from those views [arxiv:2306.16928 §1] [arxiv:2306.16928 §3]. - **Source views.** The frozen Zero123 model renders 32 views: 8 anchor views, each with 4 nearby views 10 degrees apart [arxiv:2306.16928 §4.1] [arxiv:2306.16928 §3.3]. Zero123 has no card here, so only the paper's description of it is used. - **Elevation.** The input's camera elevation is unknown, so it is estimated from the four nearby views. Features are matched with LoFTR, and the elevation is chosen by reprojection error over candidate angles [arxiv:2306.16928 §3.4] [arxiv:2306.16928 app. elevation]. On 1,700 random renders the median error is 5.4 degrees and the mean 9.7 degrees [arxiv:2306.16928 §4.3] [arxiv:2306.16928 fig. 7]. - **Network.** A cost volume over the 32 posed views, built from the variance of projected features, feeds a sparse 3D CNN. An MLP predicts the SDF, and a second predicts colour blending weights. The SDF is rendered with NeuS-style weights, and the mesh is taken from it by marching cubes [arxiv:2306.16928 §3.3]. - **Training.** Shapes come from the LVIS subset of Objaverse. One ground-truth view is the target, and the 32 predicted views are the input [arxiv:2306.16928 §3.3] [arxiv:2306.16928 §4.1]. The paper calls mixing ground-truth and predicted views in training essential [arxiv:2306.16928 §4.3]. - **Why not a per-scene fit.** Given the same 32 views with ground-truth poses, TensoRF and NeuS fitted per scene produce distortions and floaters [arxiv:2306.16928 §3.2] [arxiv:2306.16928 fig. 3]. - **Speed.** Author-reported about 45 s per shape on one A100. The reconstruction module takes about 5 s of that [arxiv:2306.16928 §4.2]. - **Accuracy.** Author-reported average F-score 83.5 at a 0.05 threshold, and average CLIP similarity 78.1. Both are on 20 Objaverse and 20 Google Scanned Objects shapes. The F-score is the best in the table, and CLIP similarity is second to Shap-E [arxiv:2306.16928 tab. 1] [arxiv:2306.16928 §4.2]. - **Ablations.** Shown only as figures. Dropping the depth loss misses fine geometry, and training on ground-truth views alone leaves regions missing at inference [arxiv:2306.16928 §4.3] [arxiv:2306.16928 fig. 8]. - **Limits.** Errors of ±30 degrees in the input elevation distort the mesh [arxiv:2306.16928 §4.3] [arxiv:2306.16928 fig. 10]. Most failures come from inconsistent views from the image model [arxiv:2306.16928 app. failure] [arxiv:2306.16928 fig. 15]. - **Text input.** A text-to-image model can supply the photo. The paper gives only figures for this route [arxiv:2306.16928 §4.4] [arxiv:2306.16928 fig. 11]. ### NeuralRecon: a voxel TSDF per fragment Card: [NeuralRecon (Sun et al., arXiv:2104.00681)](/papers/2104.00681). This variant is discrete. It predicts a sparse grid of truncated signed-distance values for a fragment of nine key frames in one network pass. The input is monocular video, and the poses come from an external SLAM system [arxiv:2104.00681 §3] [arxiv:2104.00681 §3.1]. - **Volume.** Image features from each key frame are back-projected into the fragment's bounding volume. Each voxel gets the average of the views that see it, weighted by visibility [arxiv:2104.00681 §3.2]. - **Decoding.** Three coarse-to-fine levels use sparse 3D convolutions. At each level, two heads predict an SDF value and an occupancy score. Voxels whose occupancy is below 0.5 are removed, and the rest are upsampled 2× before the next level [arxiv:2104.00681 §3.2] [arxiv:2104.00681 §3.3]. - **Fusion.** A ConvGRU carries a hidden state across fragments. It takes the place of the running-average TSDF fusion of KinectFusion [arxiv:2104.00681 §3.2] [arxiv:2104.00681 §4.3]. - **Surface.** The finest level's values are written into a global volume, and Marching Cubes runs on that volume at each time step. No rendering step is involved [arxiv:2104.00681 §3.2]. - **Loss.** Occupancy is trained with binary cross-entropy. SDF is trained with an L1 loss on a log-transformed value. Both losses apply at every level [arxiv:2104.00681 §3.2]. - **Truncation and voxels.** The truncation is 12 cm, and the finest voxel is 4 cm [arxiv:2104.00681 §3.3]. The coarse voxel sizes are not stated in the paper. - **Speed and accuracy.** Author-reported 30 ms per key frame on an RTX 2080 Ti [arxiv:2104.00681 §4.2] [arxiv:2104.00681 tab. 4]. On the ScanNet Atlas split, the single-layer F-score is 0.562, against 0.558 for COLMAP, whose time column reads 2,076 ms [arxiv:2104.00681 tab. 1]. - **Ablation.** With five key frames over the occupied-voxel area, GRU fusion gives an F-score of 0.488, against 0.462 for linear running-average fusion. Going from five to nine key frames over the fragment bounding volume raises the GRU result from 0.494 to 0.516 [arxiv:2104.00681 tab. 5]. ### BEHAVE: unsigned distances as fitting targets Card: [BEHAVE (Bhatnagar et al., arXiv:2204.06950)](/papers/2204.06950). This variant tracks a person and a known rigid object in each frame, from four calibrated RGBD cameras. It does not render, and it does not read a surface from a level set. Its implicit fields are unsigned distances, and they feed a fit [arxiv:2204.06950 §3] [arxiv:2204.06950 §4]. - **Fields.** A 3D CNN, similar to IF-Net, encodes the voxelised human and object points. At each query point, interpolated features feed three heads: unsigned distances to the human and to the object, an SMPL correspondence, and nine orientation values [arxiv:2204.06950 §4.1] [arxiv:2204.06950 §4.2] [arxiv:2204.06950 §4.4]. - **Object fit.** The object energy adds a one-sided point-to-mesh term to the sum of predicted distances at all object mesh vertices. Occluded vertices therefore still target zero distance, and the network supplies the missing surface [arxiv:2204.06950 eq. 4] [arxiv:2204.06950 §4.2]. - **Human fit.** The SMPL term asks that the distance from each query point to its predicted SMPL correspondence, after posing, matches the predicted unsigned distance [arxiv:2204.06950 eq. 3] [arxiv:2204.06950 §4.1]. Off-surface correspondences are posed with LoopReg's diffused SMPL formulation, which the paper does not describe [arxiv:2204.06950 §4.1]. - **Orientation.** The object rotation is initialised from the average of the orientation outputs of query points within 2 cm of the object, turned into a rotation by least squares and an SVD projection [arxiv:2204.06950 §4.2]. - **Contacts.** Query points are marked as contact when both predicted distances fall below a threshold. The paper does not give that threshold, and the marked object vertices are pulled onto their predicted SMPL correspondences [arxiv:2204.06950 §4.3] [arxiv:2204.06950 eq. 5]. - **Training.** All heads use an L2 loss. The targets come from the dataset's SMPL and object fits, and the orientation loss only applies to query points within the contact threshold [arxiv:2204.06950 §4.4]. - **Cost.** The paper queries 30k points per frame, against about 2M for the 128³ grid of IP-Net, and it avoids Marching Cubes. No runtime is reported, so the cost advantage is the authors' claim [arxiv:2204.06950 §5.3]. - **Results.** Author-reported on the BEHAVE recordings: SMPL vertex error 4.99 cm and object vertex error 21.20 cm. The retrained IP-Net and LoopReg reach 6.61 cm and 9.12 cm on the SMPL term, and PHOSA, a single-image method, reaches 13.73 cm and 34.73 cm [arxiv:2204.06950 tab. 2]. Removing the orientation prediction raises object error to 24.02 cm [arxiv:2204.06950 tab. 3]. - **Scope.** The object needs a rigid template mesh, and only its rotation and translation are estimated [arxiv:2204.06950 §4]. The formulation is per frame, with no motion prior or temporal smoothing [arxiv:2204.06950 §4]. ### Surface renderers (named, not covered) - **Surface rendering** (IDR and DVR, as the paper names them) needs foreground masks and is the family NeuS is compared against [arxiv:2106.10689 §1]. No card for these papers is in this repository, so nothing from them is used here. ## When to use which | Situation | Use | Why | |---|---|---| | Mesh from posed photos, with object masks | NeuS with the mask term | Mean DTU Chamfer 0.77, the lowest of the methods compared in that setting [arxiv:2106.10689 tab. 1] | | Mesh from posed photos, no masks | NeuS without masks | Mean Chamfer 0.84, against 1.02 for UNISURF [arxiv:2106.10689 tab. 1]. The paper uses a background model for points outside the unit sphere [arxiv:2106.10689 app. D.1] | | Mesh from posed photos, no masks, and an occupancy field is the target | UNISURF | Mean Chamfer 1.02 without masks, close to IDR's 0.90 with masks [arxiv:2104.10078 tab. 1]. The surface is the 0.5 occupancy level set [arxiv:2104.10078 §3] | | Only novel views are needed | Not NeuS | On held-out DTU views NeuS reaches a mean PSNR of 28.55 against 29.31 for NeRF [arxiv:2106.10689 tab. 4] | | Faster rendering | Sphere tracing, if a small quality loss is acceptable | About 320 s down to about 60 s per 1600×1200 image. Mean PSNR 31.97 down to 30.85 [arxiv:2106.10689 app. E] [arxiv:2106.10689 tab. 3] | | Limited training time | Budget for a long run | 300k iterations of 512 rays, about 14 h with masks and 16 h without, on one RTX 2080Ti [arxiv:2106.10689 §4.1] [arxiv:2106.10689 app. D.1] | | Thin parts or abrupt depth changes | NeuS over surface rendering | The paper reports this qualitatively, on its own thin-object captures and BlendedMVS scenes, with no metric [arxiv:2106.10689 §4.2] [arxiv:2106.10689 fig. 5] | | Textureless surfaces | Expect failures | Listed as a limitation and shown as a failure case [arxiv:2106.10689 §5] [arxiv:2106.10689 fig. 16] | | Scene larger than a unit sphere | Normalise the cameras first | The method assumes the region of interest lies in a unit sphere [arxiv:2106.10689 §4.1] | | Online map from a depth camera while the robot moves | iSDF | Built for a stream of posed depth frames. About 33 ms per training iteration on the authors' hardware [arxiv:2204.02296 §6.3] | | Collision costs and gradients for trajectory planning | iSDF | Author-reported lowest collision-cost error on the six main sequences, shown in a figure [arxiv:2204.02296 §6.2] [arxiv:2204.02296 fig. 8] | | Parts of a room the camera has not seen | iSDF, qualitatively | The paper reports that iSDF fills unobserved regions and the back of objects, without a numerical measure [arxiv:2204.02296 §6.1] [arxiv:2204.02296 fig. 5] | | Real-time room mesh from monocular video, poses from a SLAM system, voxel output acceptable | NeuralRecon | Predicts a fragment in one pass, about 30 ms per key frame on an RTX 2080 Ti [arxiv:2104.00681 §4.2] [arxiv:2104.00681 tab. 4]. Single-layer F-score 0.562 on ScanNet [arxiv:2104.00681 tab. 1] | | Continuous field to query at arbitrary points | Not NeuralRecon | Its output is a voxel grid, with the finest voxel at 4 cm [arxiv:2104.00681 §3.3] | | Posed RGB photos and no depth sensor | NeuS | iSDF is trained on posed depth images [arxiv:2204.02296 §4]. NeuS is trained on posed photos [arxiv:2106.10689 §4.1] | | Poses unknown or unreliable | Neither, as published | NeuS assumes posed images [arxiv:2106.10689 §4.1]. iSDF assumes an external tracker [arxiv:2204.02296 §4] | | Shape from a text prompt, with no text–shape pairs | CLIP-Forge-style conditional flow | Trained on shapes and rendered views only. Author-reported 83.33% category agreement on 224 prompts over ShapeNet's 13 classes [arxiv:2110.02624 tab. 1] | | Text prompt for a category outside ShapeNet's 13 | Not this variant | The model generalises poorly beyond its training categories [arxiv:2110.02624 app. F] | | Several shapes for one prompt | Resample the flow | Each sample is one flow inversion, with no per-prompt optimisation [arxiv:2110.02624 §1] [§3.3] | | Mesh of a real object from posed photos | NeuS, not CLIP-Forge | CLIP-Forge does not reconstruct from photos. Its inputs are rendered views of training shapes [arxiv:2110.02624 §3.2] | | One photo, a mesh in about a minute, no per-shape fitting | One-2-3-45 | Author-reported about 45 s on one A100, against about 15 min for Zero123+SD and about 90 min for RealFusion in its table [arxiv:2306.16928 tab. 1] | | One photo with an unknown camera elevation | One-2-3-45, with a check | It estimates the elevation itself, but errors of ±30 degrees distort the mesh [arxiv:2306.16928 §4.3] [arxiv:2306.16928 fig. 10] | | Several posed photos of one object | NeuS | One-2-3-45 is built for a single input. NeuS is trained on posed photos [arxiv:2306.16928 §3.2] [arxiv:2106.10689 §4.1] | | Relighting or changing the material of one object, under one static light, with masks | PhySG | It is the only variant here that outputs albedo, a specular lobe and an environment light for editing [arxiv:2104.00674 §3]. Its relighting PSNR of 33.25 is on synthetic data [arxiv:2104.00674 tab. 1] | | Matte (Lambertian) object, and relighting is needed | Not PhySG | Light and albedo are ambiguous under Lambertian materials [arxiv:2104.00674 §4 "Robustness to material roughness"] | | Self-shadowing or indirect light matters for the edit | Not PhySG as published | The model omits self-occlusion and indirect light [arxiv:2104.00674 §3] | | Person and a known rigid object tracked per frame, with contacts, from calibrated multi-view RGBD | BEHAVE | Author-reported SMPL error 4.99 cm and object error 21.20 cm on its own recordings [arxiv:2204.06950 tab. 2] | | Object with no template mesh, or a single RGB image | Not BEHAVE | It needs a rigid template and four RGBD cameras [arxiv:2204.06950 §3] [arxiv:2204.06950 §4]. PHOSA, the single-image baseline, relies on hand-made contacts [arxiv:2204.06950 §5.1] | | Temporal smoothing or a motion prior across frames | Not BEHAVE as published | The formulation is per frame [arxiv:2204.06950 §4] | ## Implementation sketch Per-ray weights from SDF samples, as in NeuS. Variants differ in the two marked lines. UNISURF uses the occupancy value directly as alpha, and draws its samples from a window around the first surface crossing [arxiv:2104.10078 §4.1] [arxiv:2104.10078 §4.3]. CLIP-Forge does not render rays: a flow inverted under the text condition gives a shape embedding, and the decoder is thresholded on a grid [arxiv:2110.02624 §3.3]. iSDF does not render rays either. It bounds each sampled point by the nearest measured surface point in the batch and minimises a per-point loss [arxiv:2204.02296 §4.3] [arxiv:2204.02296 §4.4]. One-2-3-45 uses NeuS-style weights as well, but its SDF comes from a feed-forward network over generated views [arxiv:2306.16928 §3.3]. PhySG finds one surface point per pixel by sphere tracing, and shades it with a closed-form spherical-Gaussian integral rather than by sampling the ray [arxiv:2104.00674 §3]. NeuralRecon does none of these steps. It back-projects image features into a voxel grid, decodes that grid coarse-to-fine, and fuses fragments with a GRU, with no ray sampling [arxiv:2104.00681 §3.2]. BEHAVE does not sample rays either. It evaluates unsigned distances at query points and fits a body model and a template to them [arxiv:2204.06950 §4.1] [arxiv:2204.06950 §4.2]. ```pseudocode # NeuS-style opaque weights for one ray [arxiv:2106.10689 §3.1] [eq. 13] def opaque_weights(sdf, s): # Logistic CDF of s * SDF at the section points, in (0, 1) cdf = sigmoid(s * sdf) # Alpha per segment: drop in CDF, clamped at zero [eq. 13] # Naive variant instead uses 1 - exp(-sigma * delta), sigma = phi_s(f) alpha = clamp((cdf[:-1] - cdf[1:]) / cdf[:-1], min=0) # Transmittance in front of each segment, starting at 1 [eq. 11] trans = cumprod(1 - alpha, exclusive=True) return trans * alpha # Colours are taken at segment mid-points, not section points [app. D.1] # Straightforward variant normalises phi_s over the ray instead [eq. 4] ``` Colour is then the weighted sum of per-point colours. The loss is an L1 colour term, plus an Eikonal term on $\|\nabla f\|$ and an optional mask term [arxiv:2106.10689 §3.2]. The paper trains with 64 uniform samples and then four rounds of 16 importance samples per ray [arxiv:2106.10689 app. D.1]. ## Contradictions The papers report different tasks and metrics. NeuS and UNISURF both report Chamfer distance on the 15 DTU scans [arxiv:2106.10689 tab. 1] [arxiv:2104.10078 tab. 1]. NeuS's table gives UNISURF's no-mask numbers, and they match UNISURF's own table on every scan. CLIP-Forge reports FID, MMD and accuracy on text-to-shape generation [arxiv:2110.02624 tab. 1]. iSDF reports SDF error, collision-cost error and gradient cosine distance on simulated and handheld room-scale sequences [arxiv:2204.02296 §5.1] [arxiv:2204.02296 §5.3]. One-2-3-45 reports F-score and CLIP similarity for single-photo mesh generation, on 40 Objaverse and Google Scanned Objects shapes [arxiv:2306.16928 tab. 1]. PhySG reports PSNR and normal error on synthetic inverse rendering and relighting [arxiv:2104.00674 tab. 1]. NeuralRecon reports F-score and depth metrics on ScanNet and 7-Scenes, from monocular video [arxiv:2104.00681 tab. 1] [arxiv:2104.00681 tab. 3]. BEHAVE reports vertex errors on human-object tracking, and its PHOSA numbers are after Procrustes alignment, a step the paper names for depth ambiguity [arxiv:2204.06950 §5.1]. Numbers across these tasks are not comparable, and no disagreement between the papers' results is known. Within the NeuS paper and its released code, these differ, and the NeuS card's Paper vs code section lists them. - **Importance-sampling sharpness.** The paper uses a fixed $s = 32 \cdot 2^i$ in round $i$ [arxiv:2106.10689 app. D.1]. The code uses $64 \cdot 2^i$. - **Where the SDF is evaluated for alpha.** The paper uses SDF values at both ends of each segment [arxiv:2106.10689 app. D.1]. The code evaluates the SDF once at the mid-point and extrapolates to the ends. - **Mask-free slope annealing.** The released code blends the alpha slope over the first 50k iterations of mask-free training. The paper does not describe this. CLIP-Forge has its own gaps, and the sources are the paper and the CLIP-Forge card's Paper vs code section. - **Seed reporting.** The ablations average three seeds, but the main result reports the best seed [arxiv:2110.02624 app. A]. The RealNVP-D row gives FID 2591.87 [arxiv:2110.02624 tab. 4], and the main setting gives 2425.25 [arxiv:2110.02624 tab. 1]. - **Stage 2 optimiser.** The paper gives a learning rate of 1e-4 and 100 epochs for both stages [arxiv:2110.02624 §4]. The released code uses a different flow learning rate and a longer default run. PhySG has its own gaps, and the sources are the paper and the PhySG card's Paper vs code section. - **Synthetic metric alignment.** The synthetic inverse-rendering table aligns each prediction to ground truth with per-channel median scale factors before the metrics are computed [arxiv:2104.00674 §4 "Synthetic data"]. That alignment is not in the released repository, so the numbers are hard to reproduce exactly. - **Ray start and eikonal samples.** The paper starts sphere tracing at a bounding box and does not say whether the box is a cube or a sphere [arxiv:2104.00674 §3 "Geometry modeling"]. The released code uses a sphere for ray start and a cube for the Eikonal samples, and adds the ray-hit points to the Eikonal samples [arxiv:2104.00674 §3 "Loss functions"]. iSDF has its own gaps, and the sources are the paper and the iSDF card's Paper vs code section. - **Bound method.** The paper says the batch bound is used in all experiments [arxiv:2204.02296 §4.3]. The card records that the released configs use the ray bound instead. - **Optimiser.** The paper gives Adam with weight decay [arxiv:2204.02296 §4.4]. The released code uses AdamW, which applies the decay differently. NeuralRecon has its own gaps, and the sources are the paper and the NeuralRecon card's Paper vs code section. - **Key-frame rule.** The paper requires both a translation and a rotation threshold to be exceeded [arxiv:2104.00681 §3.1]. The released code accepts either one. - **Feature volume.** The released code adds one channel, the normalised mean depth of each voxel over its views, to the averaged image features. The paper does not mention it. - **Loss weights.** The released training config sets level weights and an occupancy positive weight of 1.5. The paper states neither value. One-2-3-45 has its own gaps, and the sources are the paper and the One-2-3-45 card's Paper vs code section. - **Depth loss weight.** The paper sets the depth weight to 1 and calls depth supervision essential [arxiv:2306.16928 app. training] [arxiv:2306.16928 §4.3]. The card records that the released training config sets it to 0.0. The paper does not say which setting produced its model. - **Training length.** The paper states 300k iterations [arxiv:2306.16928 app. training]. The card records a shorter run in the released config. BEHAVE has its own gaps, and the sources are the paper and the BEHAVE card's Pitfalls section. The total energy has no weights between its terms [arxiv:2204.06950 eq. 1], and the optimiser and its stopping rule are not stated [arxiv:2204.06950 §4]. The contact threshold for network-predicted contacts is not given [arxiv:2204.06950 §4.3]. The paper's pseudo-ground-truth comes from fits and crowd-marked keypoints [arxiv:2204.06950 §3], so its errors are relative to those fits. Baseline numbers in the DTU table are partly copied from other papers and not rerun [arxiv:2106.10689 §4.2] [arxiv:2106.10689 tab. 1]. Compare them with that in mind. ## Papers - [PhySG (Zhang et al., arXiv:2104.00674)](/papers/2104.00674): an SDF sphere-traced under static light, with spherical-Gaussian shading that gives material and light for relighting and editing. - [NeuS (Wang et al., arXiv:2106.10689)](/papers/2106.10689): an SDF rendered through an opaque, occlusion-aware density whose weights peak on the surface. - [UNISURF (Oechsle et al., arXiv:2104.10078)](/papers/2104.10078): an occupancy field rendered by one formula for surface and volume rendering, with a sample window that shrinks during training. - [CLIP-Forge (Sanghi et al., arXiv:2110.02624)](/papers/2110.02624): text-to-shape generation with an occupancy decoder and a conditional flow prior, trained without text-shape pairs. - [iSDF (Ortiz et al., arXiv:2204.02296)](/papers/2204.02296): an SDF trained online from posed depth frames, with bounds from the nearest measured surface point, for robot mapping and planning. - [One-2-3-45 (Liu et al., arXiv:2306.16928)](/papers/2306.16928): a single photo lifted to a textured mesh by a feed-forward SDF network over generated views, with no per-shape fitting. - [NeuralRecon (Sun et al., arXiv:2104.00681)](/papers/2104.00681): a sparse voxel TSDF predicted per fragment of key frames from monocular video, with fragments fused by a ConvGRU for real-time reconstruction. - [BEHAVE (Bhatnagar et al., arXiv:2204.06950)](/papers/2204.06950): per-frame tracking of a person and a template object from multi-view RGBD, with unsigned distance fields and SMPL correspondences fitted jointly and contacts refined.