--- kind: concept slug: neural-radiance-field title: Neural radiance field (NeRF) and variants summary: >- A neural radiance field stores a scene in a learned function of position and view direction, rendered by volume rendering along camera rays. Variants trade render speed, training cost, surface quality and few-view robustness: KiloNeRF renders fast with tiny MLPs, Plenoxels trains fast with no network, NeuS recovers surfaces slowly, and DS-NeRF and FreeNeRF help with few views. tags: [radiance-fields, novel-view-synthesis, 3d-reconstruction, volume-rendering, few-shot, dynamic-scenes, voxel-grid, depth-supervision, text-to-3d, camera-pose, point-cloud, scene-graph, single-image, slam, real-time, sample-placement, level-of-detail, semantic-segmentation] papers: ["2103.13744", "2106.10689", "2301.10241", "2301.09632", "2301.05187", "2403.17888", "2303.07418", "2104.14786", "2210.13445", "2104.10078", "2107.02791", "2112.05131", "2012.02190", "2112.01455", "2105.06468", "2104.00677", "2104.06405", "2201.08845", "2211.10440", "2401.03890", "2403.07563", "2011.10379", "2303.14184", "2210.13641", "2302.10663", "2103.03231", "2112.05504", "2205.04334"] related: [gaussian-splatting, explicit-feature-grid, empty-space-skipping, knowledge-distillation, multi-view-reconstruction, neural-implicit-surface, novel-view-synthesis, frequency-regularization, few-shot-learning, implicit-neural-representation, positional-encoding, feed-forward-reconstruction, benchmark-validity, differentiable-rendering, vision-language-pretraining, dynamic-neural-radiance-field, multi-view-stereo, image-registration, diffusion-models, 3d-scene-graph, fourier-features, image-to-3d-generation, score-distillation-sampling, classifier-free-guidance, simultaneous-localization-and-mapping, monocular-depth-estimation, level-of-detail-rendering, curriculum-learning, meta-learning, semantic-segmentation] status: draft updated_at: 2026-11-01 --- ## Overview A neural radiance field maps a 3D point and a viewing direction to a colour and a density. To render a pixel, the renderer picks points along the camera ray, queries the field at each one, and composites the results [arxiv:2103.13744 §3.1]. The original design uses one deep MLP with 10 hidden layers, mostly 256 units wide, evaluated at every sample [arxiv:2103.13744 §1]. The original paper is [NeRF](https://arxiv.org/abs/2003.08934); it has no card in this wiki, so it is named here and not used as a source. Plain NeRF also assumes known camera poses. When those poses are noisy or missing, the poses can be optimised together with the field [arxiv:2104.06405 §3.2]. A dense monocular SLAM tracker can supply the poses and depths instead, and the field is then fitted in real time [arxiv:2210.13641 §3.3]. With few views, a semantic term from a frozen image encoder can stand in for the pixel loss at unseen poses [arxiv:2104.00677 §4.1]. The invariant all variants share is the compositing rule. The pixel colour is the sum over samples of transmittance times opacity times colour, where the transmittance is the product of (1 − opacity) over all earlier samples [arxiv:2103.13744 eq. 2–4]. Plenoxels keeps this rule with its opacity in the same role [arxiv:2112.05131 eq. 1], and UNISURF keeps it with an occupancy value in place of opacity [arxiv:2104.10078 §4.1]. Variants change six things: how the field is stored (one large MLP, many small MLPs, factorised plane features, a sparse voxel grid with no network, or a signed distance function), how many samples each ray needs, what the density means, which nonlinearity the MLP uses [arxiv:2301.05187 §3.2], whether time is an input [arxiv:2104.14786 §4.2], and which extra training signals shape the field. A volumetric density gives fog that is blended along the ray. A surface-oriented density is designed so the weight peaks on one surface [arxiv:2106.10689 §3.1]. UNISURF instead models an occupancy field whose 0.5 level set is the surface, and it blends colours over a band of samples around that surface [arxiv:2104.10078 §3] [§4.3]. With only a few input views, a plain NeRF overfits the training photos, so the training signals are what matter most [arxiv:2303.07418 §3.2]. Depth from a sparse point cloud is one such signal [arxiv:2107.02791 §3.2]. A network trained across many scenes can also predict a field from one or a few images with no per-scene fit [arxiv:2012.02190 §1]. At the other extreme, a frozen text-image model can be the only training signal, with no photographs at all [arxiv:2112.01455 §4.2]. A frozen 2D vision-language model can also give per-view features as the target for a field fitted to posed RGB-D frames, which lets a robot query the scene in words [arxiv:2403.07563 §3.2]. A single photo of an object can also be lifted with a diffusion model's views as the prior, once a token for that object is learned [arxiv:2302.10663 §3.2]. ## Variants ### Grid of tiny MLPs for fast rendering (KiloNeRF) The bounded scene's box is cut into a uniform grid of up to 16 × 16 × 16 cells, and each cell gets its own MLP with 4 hidden layers of 32 units [arxiv:2103.13744 §3.2]. Training has three stages: train a teacher NeRF, distil each cell network from the teacher, then fine-tune on the photographs [arxiv:2103.13744 §3.3]. At render time, empty space is skipped using an occupancy grid, and rays stop once their transmittance falls below 0.01 [arxiv:2103.13744 §3.4]. The author-reported result is PSNR within a few hundredths of a dB of NeRF on Synthetic-NeRF, and at or above NeRF on the other three datasets, with renders of 26–91 ms on a GTX 1080 Ti against 37–183 s for NeRF [arxiv:2103.13744 tab. 1]. The price is a long pipeline: about 2 days for the teacher, up to 8 h of distillation and 17 h of fine-tuning on one 1080 Ti [arxiv:2103.13744 §4.1]. It needs a known bounding box and is limited to bounded scenes [arxiv:2103.13744 §3.2] [arxiv:2103.13744 §5]. Card: [KiloNeRF](/papers/2103.13744). ### Sparse voxel grid with no neural network (Plenoxels) Plenoxels removes the MLP. The scene is a sparse grid of voxels, and each occupied voxel corner stores an opacity and degree-2 spherical-harmonic coefficients: 9 per colour channel, 27 per corner [arxiv:2112.05131 §3.2]. Opacity and colour at a point are trilinear interpolations of the eight nearest corners, and colour is evaluated at the view direction [arxiv:2112.05131 §3.3]. The grid starts coarse. After optimisation it is pruned by a weight threshold, and each remaining voxel is split in half along every axis [arxiv:2112.05131 §3.4]. The loss is a mean squared error plus a total-variation term, optimised with RMSProp [arxiv:2112.05131 §3.5] [eq. 2]. The author-reported result on the eight NeRF synthetic scenes is 31.71 PSNR, against 31.85 for JAXNeRF, with about 11 minutes of training per scene on one Titan RTX against about 1.45 days for JAXNeRF [arxiv:2112.05131 tab. 2]. The authors reran JAXNeRF themselves and scaled its time from four GPUs [arxiv:2112.05131 app. D]. On real forward-facing scenes it reports 26.29 PSNR against 26.71 for JAXNeRF, and on four 360° scenes 20.40 against 20.49 for NeRF++ [arxiv:2112.05131 tab. 2]. Unbounded scenes use normalised device coordinates for forward-facing data and a sphere background for 360° data [arxiv:2112.05131 §3.6]. The regularisation weights are fixed per dataset, not per scene [arxiv:2112.05131 §5] [app. B]. Card: [Plenoxels](/papers/2112.05131). See also [explicit feature grids](/concepts/explicit-feature-grid). ### Factorised plane grids for fast training (K-Planes) A scene of dimension d uses one 2D feature plane for each pair of dimensions: three planes for static 3D scenes and six for dynamic 4D scenes [arxiv:2301.10241 §3.1]. The plane values are read by bilinear interpolation and multiplied together. A small learned decoder then turns the product into colour and density, either with a linear colour basis or with a hybrid MLP [arxiv:2301.10241 §3.3]. Multiscale features and a proposal sampler are used on top [arxiv:2301.10241 §3.2] [arxiv:2301.10241 §3.4]. The author-reported training time is 33–52 minutes on one GPU for the static and monocular dynamic benchmarks [arxiv:2301.10241 tab. 3]. Quality is close to TensoRF-class methods on static scenes, while the dynamic results trail TiNeuVox and V4D [arxiv:2301.10241 tab. 3]. The card gives no render-time figures. Card: [K-Planes](/papers/2301.10241). See also [explicit feature grids](/concepts/explicit-feature-grid). ### Six planes with a time axis for dynamic scenes (HexPlane) HexPlane stores a dynamic scene in six learned planes: three spatial (XY, XZ, YZ) and three spatio-temporal (ZT, YT, XT). Each spatial plane is paired with the spatio-temporal plane on the remaining spatial axis and time, so XY pairs with ZT. A point is read from all six planes by bilinear interpolation, the two readings in each pair are multiplied, the three products are concatenated, and a learned matrix projects the result [arxiv:2301.09632 §3.3] [eq. 5–6]. A tiny MLP decodes colour, and colour is composited along each ray as in NeRF, with an MSE loss [arxiv:2301.09632 §3.4] [eq. 7]. Its basis factorisation follows TensoRF, and the paper replaces the time basis with a learned piecewise-linear function [arxiv:2301.09632 §3.1] [§3.2]. The author-reported PSNR on the flame-salmon scene of Plenoptic Video is 29.47, against 29.58 for DyNeRF, with 12 h of training on one V100 against 1344 h for DyNeRF [arxiv:2301.09632 tab. 1]. Those training times are not like-for-like: DyNeRF's figure comes from its published 8-GPU run [arxiv:2301.09632 §4.1]. On D-NeRF, with no deformation field, it reports 31.04 PSNR in about 11.5 min, while TiNeuVox-B reports 32.67 [arxiv:2301.09632 tab. 2]. The paper's claim that it "distinctly outperforms" the others there is not supported against TiNeuVox-B [arxiv:2301.09632 §4.1]. The Plenoptic Video dataset is CC BY-NC 4.0, so its numbers can be cited but the data cannot be shipped in a commercial product [arxiv:2301.09632 supp. License]. Card: [HexPlane](/papers/2301.09632). See also [explicit feature grids](/concepts/explicit-feature-grid). ### Signed distance with an opaque density for surfaces (NeuS) The geometry is a signed distance function, and the surface is its zero level set [arxiv:2106.10689 §3.1]. Naively turning the distance into NeRF-style density biases the rendered surface, because the weight peaks before the surface is reached [arxiv:2106.10689 §3.1] [arxiv:2106.10689 app. B]. NeuS instead defines an opaque density from a logistic CDF of the distance, so that the weight peaks on the surface and the nearest surface still occludes the ones behind it [arxiv:2106.10689 eq. 5–13]. Training adds an Eikonal term that keeps the field a true distance function [arxiv:2106.10689 eq. 16]. The author-reported mean Chamfer distance on 15 DTU scenes is 0.77 with masks and 0.84 without, against 0.90 for IDR with masks and 1.02 for UNISURF without [arxiv:2106.10689 tab. 1]. Costs are high: 14 h (with masks) and 16 h (without) per scene on one 2080 Ti [arxiv:2106.10689 §4.1], and about 320 s per 1600 × 1200 image at default sampling, or about 60 s with sphere tracing at about 1.1 dB lower mean PSNR [arxiv:2106.10689 app. E] [arxiv:2106.10689 tab. 3]. Cameras must be normalised to a unit sphere [arxiv:2106.10689 §4.1]. Card: [NeuS](/papers/2106.10689). See also [multi-view reconstruction](/concepts/multi-view-reconstruction). ### Occupancy surface without object masks (UNISURF) UNISURF models the shape as an occupancy field, and the surface is the set of points where occupancy is 0.5 [arxiv:2104.10078 §3] [eq. 1]. One rendering formula covers both modes. Surface rendering evaluates colour once at the first crossing of the ray with the surface. Volume rendering sums colours over samples in a window around that crossing [arxiv:2104.10078 §4.1] [eq. 7–8]. The window's half-width shrinks exponentially during training, from a wide start to a narrow end [arxiv:2104.10078 §4.3] [eq. 14]. The wide early window lets volume rendering shape the model without object masks, and the narrow late window sharpens the surface. On 15 DTU scans the author-reported mean Chamfer distance is 1.02 with no masks and no trimming, against 0.90 for IDR with masks and 1.49 for NeRF-derived meshes [arxiv:2104.10078 tab. 1]. The authors target solid objects. With occupancy restricted to 0 and 1, fog and smoke are out of scope [arxiv:2104.10078 §4.1] [§6]. Rendering one view took 178 s in volume mode and 84 s in surface mode on one V100 [arxiv:2104.10078 fig. 5]. Card: [UNISURF](/papers/2104.10078). See [neural implicit surfaces](/concepts/neural-implicit-surface). ### Explicit Gaussian primitives instead of ray marching (2DGS) 2D Gaussian splatting is not ray marching. It draws flat, oriented Gaussian disks by exact ray–splat intersection, and it sorts them and alpha-blends them front to back [arxiv:2403.17888 §4.2] [arxiv:2403.17888 eq. 12]. Depth and normal come from the splats themselves, and a depth-distortion loss and a normal-consistency loss push the splats onto a surface [arxiv:2403.17888 §5]. Meshes come from TSDF fusion of the rendered depth [arxiv:2403.17888 §6.1]. The author-reported DTU Chamfer distance is 0.80 at 10.9 min per scene on one RTX 3090, against 0.84 for NeuS and 1.96 for 3DGS [arxiv:2403.17888 tab. 1]. Image quality is slightly below 3DGS: 24.34 against 24.64 PSNR outdoors on Mip-NeRF 360 [arxiv:2403.17888 tab. 4]. Card: [2DGS](/papers/2403.17888). See [Gaussian splatting](/concepts/gaussian-splatting) for the wider family. A 2024 survey groups the 3DGS follow-ups by theme, and it compiles numbers from their original papers. It proposes no method, so use it as a map and not as a source of settings [arxiv:2401.03890 §4] [§6]. The official code is non-commercial, so reimplement from the paper and do not copy the code. ### Training-time regularisers for few input views (FreeNeRF) FreeNeRF keeps a plain NeRF and changes only the training. It applies a frequency mask to the encoded inputs, so that the sine and cosine bands are revealed one band of three entries at a time, from the coarsest upward [arxiv:2303.07418 §3.2] [eq. 4–5]. It also penalises the density of the samples nearest the camera, for the first M samples of each ray [arxiv:2303.07418 §3.3] [eq. 6]. The author-reported result, with 3, 6 or 9 views, is the highest PSNR in its DTU and LLFF tables [arxiv:2303.07418 tab. 2] [tab. 3]. For 3 views on DTU, it reports 19.92 PSNR against 18.50 for a reproduced RegNeRF baseline [arxiv:2303.07418 tab. 2]. The price is a training overhead of 1.02–1.04× over its baselines [arxiv:2303.07418 tab. 4]. The paper's experiments use 8 views on Blender and 3–9 views on DTU and LLFF [arxiv:2303.07418 §4.1]. Card: [FreeNeRF](/papers/2303.07418). See [frequency regularisation](/concepts/frequency-regularization) and [few-shot learning](/concepts/few-shot-learning). ### Semantic consistency from a frozen CLIP encoder (DietNeRF) DietNeRF keeps the pixel loss and adds a term that is computed at poses that have no photo. Every K = 10 iterations it renders a random pose at reduced resolution, embeds that render and a random training image with a frozen CLIP ViT, and pulls the two embeddings together [arxiv:2104.00677 §4.1] [§4.4] [alg. 1]. No pixels are compared at the new pose. With eight random views of each Realistic Synthetic scene, the author-reported PSNR is 23.15 against 14.93 for NeRF. A further 50K steps of pixel-only fine-tuning raise it to 23.59 [arxiv:2104.00677 tab. 1]. Changing the encoder moves the Lego result from 22.06 (ImageNet ViT-B/32) to 23.90 (CLIP ViT-B/32) [arxiv:2104.00677 tab. 4]. On DTU single-view fine-tuning of pixelNeRF, the semantic term lowers PSNR (14.24 against 16.05) and lowers FID (190.7 against 265.2) [arxiv:2104.00677 tab. 2]. The paper gives no weight for the term, and the released code uses 0.1 where its argument default is 0.2. The paper's equation reads as a dot product that the text says is minimised, while the code minimises its negative. The released repository has no licence file, so reimplement from the paper. Card: [DietNeRF](/papers/2104.00677). See also [vision-language pre-training](/concepts/vision-language-pretraining). ### Joint pose and field fitting with coarse-to-fine encoding (BARF) BARF treats each camera's six-degree pose as a parameter and optimises it together with the network, through the rendering loss [arxiv:2104.06405 §3.2] [eq. 8]. Starting from random weights and imperfect poses, it must avoid a bad local fit. It does so by opening the sinusoidal positional-encoding bands one at a time: band k is scaled by a window that is zero until the schedule reaches k, then rises along a half cosine over one unit, then stays at one. The raw coordinates are never masked [arxiv:2104.06405 §3.4] [eq. 10] [eq. 13–14]. For NeRF the schedule runs from iteration 20K to 100K of 200K [arxiv:2104.06405 §4.2] [§4.3]. The author-reported mean rotation error on eight perturbed Blender scenes falls from 6.17 to 0.19 degrees, and PSNR rises from 22.12 to 27.50 dB, against 29.40 dB for NeRF with true poses [arxiv:2104.06405 tab. 2]. On eight LLFF scenes started from identity poses, the rotation error is 0.57 degrees, but the no-encoding baseline has the smaller translation error [arxiv:2104.06405 tab. 3] [tab. 5]. The paper says the best per-scene schedule is unknown and out of scope [arxiv:2104.06405 app. B.2]. Pose errors are defined only up to a similarity transform, so they are reported after Procrustes alignment of the camera centres [arxiv:2104.06405 §4.2] [app. B.1]. Card: [BARF](/papers/2104.06405). See [image registration](/concepts/image-registration) for the classical analogy the paper draws. ### Feed-forward field from one or few images (pixelNeRF) pixelNeRF trains one network across many scenes, so a new scene needs no optimisation at test time. A fully convolutional ResNet34 encodes each input image into a pixel-aligned feature grid [arxiv:2012.02190 §4] [app. B.1]. For a query point, the point is projected into the input image, and the feature at that pixel is interpolated and given to the MLP [arxiv:2012.02190 §4.1]. The point and the view direction are expressed in each input camera's frame, not in a canonical frame [arxiv:2012.02190 §4] [§1]. With several views, each view passes through the early layers separately, the results are averaged, and the later layers predict density and colour from the average [arxiv:2012.02190 §4.2] [eq. 4–6]. The author-reported PSNR on ShapeNet chairs is 23.72 at one view and 26.20 at two, against 22.89 and 24.48 for SRN, under SRN's protocol [arxiv:2012.02190 tab. 2]. Over 13 categories at 64 × 64 with one input view, it reports 26.80 against 23.28 for SRN [arxiv:2012.02190 tab. 4]. On unseen categories it reports 22.71 against 18.71 for SRN [arxiv:2012.02190 tab. 5]. The DTU results appear only as plotted quantiles, with no numeric table [arxiv:2012.02190 fig. 10]. Training takes about 5 to 6 days per model on one GPU, and rendering is slow as in NeRF, growing with the number of input views [arxiv:2012.02190 app. B.2] [§6]. The method needs known intrinsics and poses [arxiv:2012.02190 §4.1] [§6]. Card: [pixelNeRF](/papers/2012.02190). See [feed-forward reconstruction](/concepts/feed-forward-reconstruction). ### Feature field for robot navigation and grasping (GeFF) GeFF is a scene-level generalizable NeRF. Its encoder reads posed RGB-D frames and outputs sparse latent points in one forward pass, with no per-scene fit [arxiv:2403.07563 §1] [§3.1]. Two decoders read those latents at any 3D point: a signed distance for geometry, and a semantic feature for language [arxiv:2403.07563 §3.2] [eq. 2] [eq. 4]. Opacity along a ray comes from the signed distance through a sigmoid with a learnable sharpness, not from a density [arxiv:2403.07563 §3.2] [eq. 4]. Pre-training is on ScanNet for 50 epochs, which the paper reports as 6 days on eight RTX 3090 GPUs [arxiv:2403.07563 §4.1]. The semantic features are distilled from a frozen teacher, and the total loss is a weighted sum of five terms [eq. 6]. The variant is evaluated on a quadruped with an arm, and the features are turned into a point cloud and a 2D grid for planning and grasping [arxiv:2403.07563 §3.3]. The author-reported overall success is 62.4%, against 43.2% for LERF and 41.9% for ConceptGraphs [arxiv:2403.07563 tab. 1]. The two baselines ran offline on batched frames, so the timing comparison is not like-for-like. Dropping the auxiliary teacher input lowers the overall object-level count from 41 of 66 to 25 of 66 [arxiv:2403.07563 tab. 2]. Geometry from 512 latent points has a depth L2 error of 0.012, against 0.040 for bilinear upsampling of a downsampled grid [arxiv:2403.07563 tab. 4]. The paper says it plans to release code, and none was found, so reimplement from the paper. Card: [GeFF](/papers/2403.07563). See also [implicit neural representation](/concepts/implicit-neural-representation) and [knowledge distillation](/concepts/knowledge-distillation). ### Neural point cloud with per-scene pruning and growth (Point-NeRF) Point-NeRF replaces the global field with a cloud of points. Each point has a position, a feature vector and a confidence in [0, 1] [arxiv:2201.08845 §3]. A query point is shaded from its eight nearest points, blended with inverse-distance weights times confidence, and the result drives the colour and density heads [arxiv:2201.08845 §3] [app. H] [eq. 4–7]. An MVS-style network predicts the initial points from three posed images in about 0.2 s, and it is trained with the rendering loss only, after its depth part is pre-trained on ground-truth depth [arxiv:2201.08845 §4.1] [§5]. Each scene is then refined. Every 10K iterations the method removes points below confidence 0.1, and it adds a point where a ray's most opaque sample is far from every existing point. The paper does not state the thresholds for that growth step [arxiv:2201.08845 §4.2] [eq. 10]. The author-reported DTU PSNR is 30.12 after 10K refinement steps, about 20 min, against 27.01 for NeRF after 200K steps, about 10 h [arxiv:2201.08845 tab. 1]. On NeRF Synthetic it reports 30.71 PSNR after 20K steps, about 40 min, and 33.31 after 200K steps, against 31.01 for NeRF [arxiv:2201.08845 tab. 2] [§6.2]. Without per-scene refinement the DTU result is 23.89, below IBRNet's 26.04 and MVSNeRF's 26.63 [arxiv:2201.08845 tab. 1]. The method has no background model, so unbounded scenes are outside its reach [arxiv:2201.08845 app. J]. The official code is licensed for non-commercial use, so reimplement from the paper. Card: [Point-NeRF](/papers/2201.08845). See also [multi-view stereo](/concepts/multi-view-stereo). ### Depth from sparse structure-from-motion points (DS-NeRF) DS-NeRF keeps the NeRF model and adds a depth term to the loss. The sparse 3D keypoints that structure-from-motion (SfM) produces while estimating the camera poses give each keypoint a depth in every camera that sees it. The loss pulls the ray-termination distribution toward a Gaussian centred on that depth, with width set by the keypoint's average reprojection error [arxiv:2107.02791 §3.2]. The headline uses the negative log-likelihood (KL) form of this loss [arxiv:2107.02791 tab. 1]. On NeRF Real with 2 views, the author-reported PSNR is 20.2 for the KL form and 19.5 for a plain MSE on depth, against 13.5 for NeRF [arxiv:2107.02791 tab. 1]. The paper reports 2–3× fewer iterations to reach NeRF's peak PSNR, at 362.4 ms per iteration against 359.8 ms for NeRF [arxiv:2107.02791 §4.5]. On DTU at 3 views it trails DTU-pretrained pixelNeRF, which the paper attributes to DTU priors [arxiv:2107.02791 §4.3] [tab. 2]. The method inherits SfM's depth error, and it needs SfM on every training set [arxiv:2107.02791 §3.2] [app. A]. Card: [DS-NeRF](/papers/2107.02791). The card records that the authors' released configs use the MSE form by default, not the KL form, so state which one you implement. ### Text-guided generation with no 3D data (Dream Fields) Dream Fields keeps a NeRF but trains it from text. Renders from random azimuths are scored against a caption by a frozen CLIP image-text model, and the loss is the negative similarity averaged over the sampled poses. No 3D or multi-view training data is used [arxiv:2112.01455 §4.2] [eq. 7]. Without extra priors the field fills the view with haze. Three priors keep it to one object: a transmittance loss that rewards sparsity, random backgrounds during training, and a fixed scene box [arxiv:2112.01455 §4.5] [§4.6] [eq. 8]. The paper also swaps the NeRF MLP for a residual MLP with 280K parameters, against 494K [arxiv:2112.01455 §4.7]. The author-reported CLIP R-Precision is 68.3 at 20K iterations with 252-pixel renders, against 31.37 for a simplified NeRF at 10K iterations [arxiv:2112.01455 tab. 1]. The score is a retrieval measure on object-centric COCO captions with a different CLIP model at evaluation, not a geometry or quality measure [arxiv:2112.01455 §5.1]. Quantitative results stop at 10K iterations, and validation scores fall after 5–10K [arxiv:2112.01455 app. F]. One object takes about 1 h 12 min for 10K iterations on 8 TPU cores [arxiv:2112.01455 app. C]. The released code anneals the transmittance target from 50% to 90%, where the paper's text gives 40% to 88%, and it sets the loss weight λ to 0.5, a value the paper does not state; see the [Dream Fields card](/papers/2112.01455) for the paper-versus-code list. Card: [Dream Fields](/papers/2112.01455). See also [vision-language pre-training](/concepts/vision-language-pretraining). ### Two-stage text-to-3D with a diffusion prior (Magic3D) Magic3D also generates from a prompt, but its signal is a text-to-image diffusion prior, not a CLIP score. Score distillation sampling sends the prior's noise prediction for a random render back to the scene parameters [arxiv:2211.10440 §3] [eq. 1]. Stage one fits a hash-grid field, with a base diffusion prior on 64 × 64 renders for 5000 steps, about 15 min on eight A100 GPUs [arxiv:2211.10440 §4.1] [§5]. Stage two converts the field to a textured mesh and refines it with a latent diffusion prior on 512 × 512 renders for 3000 steps [arxiv:2211.10440 §4.2] [§4.3] [§5]. The stage-two timestep range and weight are not those of stage one [arxiv:2211.10440 app. B]. In a user study on DreamFusion's prompts, raters chose Magic3D's canonical-view videos for realism and detail 61.7% of the time [arxiv:2211.10440 tab. 1]. The paper reports no automatic metric, and the text guidance weight is not stated. No official code was found. Card: [Magic3D](/papers/2211.10440). See also [diffusion models](/concepts/diffusion-models). ### Static and dynamic fields for one monocular video (Dynamic NeRF) Dynamic NeRF splits a monocular video into a still background and a moving part, and gives each its own field. The static NeRF is trained only on pixels that a per-frame mask marks as static [arxiv:2105.06468 §3.2] [eq. 4]. The dynamic field takes a point and a time, and outputs forward and backward 3D scene flow, density, colour and a blend weight [arxiv:2105.06468 §3.3] [eq. 5]. The same field is queried at the flowed points at times t+1 and t-1, so each ray gets a photometric loss at three times [arxiv:2105.06468 §3.3] [eq. 6–9]. Most of the work is in the regularisers. The scene flow must reproduce RAFT optical flow, the flow should be slow and smooth, forward and backward flow should cancel, and depth should match a monocular depth estimate [arxiv:2105.06468 §3.4]. The method needs a binary mask of the moving objects in every frame, which the paper says can come from segmentation or rotoscoping [arxiv:2105.06468 §3.1]. On seven sequences of the Dynamic Scene dataset, the author-reported average is 24.74 PSNR and 0.118 LPIPS, against 23.99 and 0.205 for Li et al. [arxiv:2105.06468 tab. 1]. On the Playground sequence, removing the motion loss drops PSNR from 23.65 to 22.61 [arxiv:2105.06468 tab. 2]. The benchmark uses 12-frame sequences, each frame from a different camera, and the training takes about 22 h per sequence on one V100 [arxiv:2105.06468 §4.1] [arxiv:2105.06468 supp. Implementation Details]. Card: [Dynamic View Synthesis from Dynamic Monocular Video](/papers/2105.06468). See [dynamic NeRF](/concepts/dynamic-neural-radiance-field). ### Evaluating monocular dynamic fields (reality check) This paper changes how dynamic variants are scored, not the field. Monocular training videos are often built by alternating frames from several cameras, which the authors call camera teleportation. That gives the model multi-view information a single phone could not record [arxiv:2210.13445 §1] [§4.1]. The authors retrain on rectified Nerfies-HyperNeRF sequences with only one camera, and report a 1–2 dB masked PSNR drop without teleportation [arxiv:2210.13445 §5.1] [§1]. Their iPhone benchmark has seven multi-camera and seven single-camera sequences, and all methods score lower on it. Nerfies loses 4.4 dB masked PSNR from the Nerfies-HyperNeRF numbers [arxiv:2210.13445 §5.2]. The paper scores images only on co-visible pixels, and it checks keypoint correspondence (PCK-T) as well [arxiv:2210.13445 §4.2] [app. B] [app. C]. Its effective multi-view factors measure how much multi-view signal a sequence carries [arxiv:2210.13445 §3]. Card: [Monocular Dynamic View Synthesis: A Reality Check](/papers/2210.13445). ### Scene graph of a background and per-object fields (Neural Scene Graphs) Neural Scene Graphs splits a driving scene into a graph. One NeRF-style field holds the static background. Each object that moves is a node with its own field, defined in the object's local frame [arxiv:2011.10379 §3] [§3.2]. Objects of one class share weights and differ through a 256-dimensional latent code per object [arxiv:2011.10379 §3.2] [eq. 6]. A world point is mapped into an object's frame using the object's bounding-box size [arxiv:2011.10379 §3.2] [eq. 8]. The object's density depends only on that local point and its latent code, and its colour also depends on the view direction and the object's position, which keeps density consistent across poses [arxiv:2011.10379 §3.2] [eq. 7] [eq. 9]. The background is sampled on six planes instead of a dense ray march, and each object a ray hits gets seven points between its box entry and exit [arxiv:2011.10379 §4.1] [§5]. All samples are sorted by depth before compositing [arxiv:2011.10379 eq. 12–13]. Because the objects are separate nodes, editing changes the graph's edges, so objects can be moved, rotated, removed or recombined [arxiv:2011.10379 §5.1]. Training needs the tracked object boxes, which the paper obtains from labels or from a tracker [arxiv:2011.10379 §5.1] [fig. 9]. On three KITTI scenes the author-reported reconstruction PSNR is 26.66, against 23.34 for NeRF and 24.18 for NeRF with a time input [arxiv:2011.10379 tab. 1]. On novel compositions, with one frame held out, it reports 25.11 against 18.25 for NeRF [arxiv:2011.10379 tab. 1]. Training takes about one day per scene on one TITAN Xp [arxiv:2011.10379 §5]. The official repository has no licence file, so reimplement from the paper. Card: [Neural Scene Graphs](/papers/2011.10379). See also [dynamic NeRF](/concepts/dynamic-neural-radiance-field) and [3D scene graphs](/concepts/3d-scene-graph). ### Single image to 3D with a diffusion prior (Make-It-3D) Make-It-3D builds a textured 3D object from one reference photo, with no multi-view training data [arxiv:2303.14184 §1]. Its coarse stage fits a NeRF under several signals at once: a masked L1 loss at the reference view, score distillation sampling (SDS) from Stable Diffusion at novel views, a CLIP-based diffusion loss at novel views, and a depth prior that the rendered depth must correlate with [arxiv:2303.14184 §3.2] [§3.1] [eq. 2–5]. The two diffusion losses never apply to the same step. The CLIP-based loss is used at small timesteps and SDS at large ones, with the switch at timestep 400 [arxiv:2303.14184 §3.2] [app. C.1]. The field uses a multi-scale hash encoding and a small MLP [arxiv:2303.14184 app. B.1]. The refine stage exports the coarse NeRF as a coloured point cloud. Points that the reference photo does not cover are coloured from the NeRF render. A U-Net renderer, fed by rasterised point descriptors at three scales, then fills in the hidden texture, while the geometry stays fixed [arxiv:2303.14184 §3.3] [app. B.2]. The paper states no loss weights [arxiv:2303.14184 §3.2]. On DTU, the author-reported LPIPS of the enhanced model is 0.0908, against 0.4075 for DreamFusion+ [arxiv:2303.14184 tab. 1]. The coarse stage takes about two hours on one 32 GB V100 [arxiv:2303.14184 §4.1]. Card: [Make-It-3D](/papers/2303.14184). See also [score distillation sampling](/concepts/score-distillation-sampling) and [image-to-3D generation](/concepts/image-to-3d-generation). ### Covariance-weighted depth from a dense SLAM tracker (NeRF-SLAM) NeRF-SLAM runs two threads on a monocular video with no depth sensor. A dense tracker (DROID-SLAM) estimates keyframe poses and a depth per pixel, and it also gives the marginal covariance of each depth and pose [arxiv:2210.13641 §3.1] [eq. 1–2]. A hash-grid radiance field (Instant-NGP) is fitted to the keyframes as they arrive, in a second thread [arxiv:2210.13641 §3.2] [§3.3]. The mapping loss is a colour term plus a depth term, and the depth term is weighted by each pixel's depth covariance [arxiv:2210.13641 eq. 3–4]. The paper's argument is that this weighting is what makes noisy monocular depth usable. In its synthetic ablation, raw unweighted depth lowers the depth error but makes PSNR about 3 dB worse than using no depth, while the weighted version keeps PSNR near the no-depth case [arxiv:2210.13641 §4.4] [fig. 6]. The poses are refined jointly with the field [arxiv:2210.13641 §3.2]. The paper does not define the norm in its weighted depth term [arxiv:2210.13641 eq. 4], so the form of the weighting has to be checked against the released code, which is not fully in the repository [card](/papers/2210.13641). The author-reported Replica results use 2000 frames per scene from a random trajectory. The average PSNR is 41.40 dB, against 17.76 dB for Nice-SLAM without depth and 24.61 dB for Nice-SLAM with ground-truth depth [arxiv:2210.13641 tab. 1]. The average depth L1 is 4.49 cm, which is higher than the 4.08 cm of Nice-SLAM with ground-truth depth [arxiv:2210.13641 tab. 1]. The system runs at about 10 frames per second on 640 × 480 images, on one 11 GB GPU [arxiv:2210.13641 §4.5] [§3.4]. The paper names dense correlation volumes and hierarchical grids as the memory costs [arxiv:2210.13641 §5]. Card: [NeRF-SLAM](/papers/2210.13641). See also [simultaneous localisation and mapping](/concepts/simultaneous-localization-and-mapping) and [monocular depth estimation](/concepts/monocular-depth-estimation). ### Single photo with a learned token and score distillation (RealFusion) RealFusion builds a 360° radiance field of an object from one RGBA photo, with no 3D training data [arxiv:2302.10663 §1]. The photo supervises the field from its own camera, through an image term and a mask term. Random prior cameras supervise it through score distillation sampling (SDS) with Stable Diffusion [arxiv:2302.10663 §3.2] [eq. 4] [eq. 6]. Only one view exists, so the prompt is built from single-image textual inversion: a new token is learned from heavily augmented copies of the photo, with the diffusion model frozen [arxiv:2302.10663 §3.2] [app. A]. The paper uses a guidance weight of 100 and says values below 30 gave many geometric deformities [arxiv:2302.10663 app. A]. Without the learned token, the back often becomes a generic member of the object's category [arxiv:2302.10663 §4.4] [fig. 7]. On seven CO3D categories, with three images each, the author-reported mean F-score is 9.58, against 8.24 for Shelf-Supervised Mesh Prediction, which trains one model per category. The F-score is lower for skateboard and teddybear [arxiv:2302.10663 tab. 1]. On the synthetic lego test set, which the paper says favours depth-supervised methods, it reports 16.5 PSNR, against 16.6 for DS-NeRF and 21.0 for SinNeRF [arxiv:2302.10663 tab. 2] [app. D]. Its failures include floating parts, a second face on the back (the Janus problem), and a back that copies the front [arxiv:2302.10663 app. F] [§4.3]. Reconstruction takes about 45 minutes on one V100 [arxiv:2302.10663 app. A], and the token is a separate stage before it. Card: [RealFusion](/papers/2302.10663). See also [score distillation sampling](/concepts/score-distillation-sampling) and [image-to-3D generation](/concepts/image-to-3d-generation). ### One depth oracle per ray for few samples (DONeRF) DONeRF keeps a NeRF-style shading network but evaluates it at very few points per ray. A depth oracle MLP runs once per ray and scores log-spaced depth classes. Its scores are read as a piecewise-constant density, and the samples are placed at uniform steps of that density's inverse CDF [arxiv:2103.03231 §5.1] [fig. 8]. The oracle's targets are filtered in image and depth space during training only [arxiv:2103.03231 §5.1]. The shading network has eight layers of 256 units, and the oracle has eight hidden layers of 256 units with no skip connection [arxiv:2103.03231 app. D]. The author-reported mean PSNR over six Blender scenes is 31.14 at 4 samples per ray, against 30.52 for NeRF at 256 [arxiv:2103.03231 tab. 2]. At 2 samples it is 29.50, below NeRF [arxiv:2103.03231 tab. 2]. Training uses a reference depth per pixel. A dense NeRF can supply it instead, and the averages then fall by 0.8 to 1.5 dB across the four sample counts [arxiv:2103.03231 tab. 2]. The paper times 51.3 ms per 800 × 800 frame at 2 samples on an RTX 3090 with TensorRT in half precision, and shading is the main cost, growing roughly linearly with the sample count [arxiv:2103.03231 §6.1]. Its speed-up figures are FLOP ratios, not measured wall-clock ratios, and the paper reports no NeRF wall-clock time [arxiv:2103.03231 tab. 2] [§6.3]. The scenes are static, with poses limited to 30° pitch and 20° yaw from the initial direction [arxiv:2103.03231 §6.2]. The authors' code is CC BY-NC, so this page does not use it. Card: [DONeRF](/papers/2103.03231). See also [empty-space skipping](/concepts/empty-space-skipping), which removes empty samples rather than predicting where the surface is. ### Multi-scale growing network for city-scale ranges (BungeeNeRF) BungeeNeRF targets scenes seen from very different distances, from satellite to street. Training images are split into L_max = 4 scales by camera distance, with a new scale at each 2× zoom-out [arxiv:2112.05504 §3.3]. The network starts with a base block of four layers of width 256, with no skip connection. Each later stage adds one two-layer residual block with its own output head, and head L is the base output plus the residuals of blocks 2 to L [arxiv:2112.05504 §3.3] [eq. 2]. At stage L, every head is supervised on the rays of scales up to its own, so each head gives one level of detail [arxiv:2112.05504 eq. 3] [§3.3]. The encoding is Mip-NeRF's integrated positional encoding with M = 10 [arxiv:2112.05504 §3.1] [§4]. Each stage runs 100k iterations with Adam at a learning rate of 5e-4, and the optimiser is reset at each stage [arxiv:2112.05504 §4]. The author-reported average PSNR on two Google Earth Studio city scenes is 23.48 and 23.61 at the baselines' iteration budget, and 24.51 and 24.42 with longer training [arxiv:2112.05504 tab. 2]. Mip-NeRF-full, the strongest baseline shown, reports 22.31 and 22.83 [arxiv:2112.05504 tab. 2]. The text's "0.5 to 5 dB" gain is not reached by any pair in the table, whose gaps run from about 0.65 to 3.9 dB [arxiv:2112.05504 §4.1] [tab. 2]. Removing the growing schedule lowers the 56 Leonard average to 22.14 [arxiv:2112.05504 tab. 3]. The number of stages is a tuning choice that the paper does not sweep, and the scale of each image set needs its camera distance, or a texture-size proxy if depth is unknown [arxiv:2112.05504 §3.3]. The paper gives no training time or hardware [arxiv:2112.05504 §4]. The released code differs from the paper in settings such as samples per ray, activations and loss normalisation, so use the paper's values only with that gap in mind. The imagery is Google Earth Studio material, its reuse is governed by Google's terms, and the data are not in the repository. Card: [BungeeNeRF](/papers/2112.05504). See also [level of detail](/concepts/level-of-detail-rendering) and [curriculum learning](/concepts/curriculum-learning). ### Object MLPs in tracked boxes with a semantic head (Panoptic Neural Fields) Panoptic Neural Fields models a driving scene as a background MLP and one MLP per tracked object [arxiv:2205.04334 §3.1]. Each object's MLP lives inside an oriented box whose rotation, translation and size are given at discrete times and interpolated between them [arxiv:2205.04334 §3.1]. A point takes the sum of the contributions of the boxes it hits, and the background output if it hits none [arxiv:2205.04334 §3.1] [eq. 1]. Every point also outputs a semantic vector, and the training loss adds a per-pixel cross-entropy on predicted segmentation to the colour loss [arxiv:2205.04334 §3.3] [eq. 3]. The boxes and track parameters are optimised with the MLPs, and each rotation is projected back onto SO(3) after each step [arxiv:2205.04334 §3.1] [app. A]. Object MLPs for cars can start from a FedAvg-learned prior trained on ShapeNet car renders, and other objects get a density-bias start [arxiv:2205.04334 §3.4] [alg. 1]. The author-reported mIoU on KITTI-360 novel views is 74.28, with 21.91 PSNR, against 53.01 mIoU and 21.18 PSNR for NeRF with PSPNet on the same task [arxiv:2205.04334 tab. 2]. On the NSG dynamic-KITTI protocol it reports 27.48 PSNR against 26.66 for Neural Scene Graphs [arxiv:2205.04334 tab. 3]. The paper gives no training time or hardware [arxiv:2205.04334 §5]. The rigid objects it handles must come with detections, tracks, poses and 2D segmentation from other tools, and their settings are not stated [arxiv:2205.04334 §1] [§3.1]. The paper does not say what an object MLP returns outside its box [arxiv:2205.04334 eq. 1], so mask it to the box. No official code was found. Card: [Panoptic Neural Fields](/papers/2205.04334). Compare [Neural Scene Graphs](/papers/2011.10379), which uses one latent code per object and no semantic head. ## When to use which | Situation | Use | Why | |---|---|---| | Bounded scene, rendering speed matters most, and you can afford a long multi-stage training run with custom kernels | KiloNeRF | Author-reported 26–91 ms renders against 37–183 s for NeRF [arxiv:2103.13744 tab. 1]. Needs a teacher, distillation and fine-tuning [arxiv:2103.13744 §4.1] and fused kernels for variable batch sizes [arxiv:2103.13744 §3.5] | | Static, bounded or 360° scene, a fit on one GPU in minutes, and you can run CUDA code and tune the regularisers per dataset | Plenoxels | Author-reported 11 min per synthetic scene against about 1.45 days for JAXNeRF, at 31.71 against 31.85 PSNR [arxiv:2112.05131 tab. 2]. Weights are fixed per dataset [arxiv:2112.05131 §5]. The released code needs CUDA (see its card) | | Static or video scene, short training on one GPU, and you want a model whose parts you can inspect | K-Planes | 33–52 min of training in the paper's static and monocular dynamic settings [arxiv:2301.10241 tab. 3]. Each plane is a separate learned array with its own regulariser [arxiv:2301.10241 §3.2]. K-Planes, HexPlane and ST-NeRF take time as an input [arxiv:2301.10241 §3.1] [arxiv:2301.09632 §3.3] [arxiv:2104.14786 §4.2]. The other method cards do not | | Dynamic multi-view video, and you want a six-plane model with a fast training run rather than an MLP per point | HexPlane | Author-reported 12 h on one V100, against 1344 h for DyNeRF, at 29.47 against 29.58 PSNR on flame salmon [arxiv:2301.09632 tab. 1]. The time ratio is not like-for-like [arxiv:2301.09632 §4.1]. The planes are a factorised form of a 4D volume, not a dense one [arxiv:2301.09632 §3.1]. The Plenoptic Video data is CC BY-NC, so do not ship it [arxiv:2301.09632 supp. License] | | Text prompt, no 3D or multi-view data, and an optimisation of about an hour per object is acceptable | Dream Fields | 68.3 CLIP R-Precision at 20K iterations [arxiv:2112.01455 tab. 1]. The metric is CLIP-based retrieval, so it does not measure geometry [arxiv:2112.01455 §5.1]. About 1 h 12 min per 10K iterations on 8 TPU cores [arxiv:2112.01455 app. C] | | Text prompt that should become a textured mesh at higher resolution than a 64 × 64 prior, and about 40 min on eight A100 GPUs is acceptable | Magic3D | Author-reported: 61.7% of raters prefer its videos to DreamFusion's for realism and detail [arxiv:2211.10440 tab. 1]. The 40 min is not like-for-like with DreamFusion's 1.5 h, which is quoted from DreamFusion [arxiv:2211.10440 §1] [§5]. No official code was found | | Camera poses are noisy or unknown, and you accept a hand-set coarse-to-fine schedule that you must tune for each dataset | BARF | Author-reported rotation error of 0.19 degrees on perturbed Blender poses, from 6.17 [arxiv:2104.06405 tab. 2]. The paper uses one schedule for all LLFF scenes and says the best one per scene is unknown [arxiv:2104.06405 app. B.2]. On LLFF, the no-encoding baseline has the smaller translation error [arxiv:2104.06405 tab. 5] | | Per-scene fit must be fast, you have posed images and can run an MVS-style network, and a point-based model with no background model is acceptable | Point-NeRF | Author-reported DTU PSNR 30.12 after about 20 min, against 27.01 for NeRF after about 10 h [arxiv:2201.08845 tab. 1]. Without refinement it is below IBRNet on DTU [arxiv:2201.08845 tab. 1]. No background model [arxiv:2201.08845 app. J]. Growth thresholds not stated [arxiv:2201.08845 §4.2] | | Monocular video of a moving scene, a moving-object mask for every frame, and about 22 h of GPU time per 12-frame clip is acceptable | Dynamic NeRF | Author-reported 24.74 PSNR average on seven sequences, against 23.99 for Li et al. [arxiv:2105.06468 tab. 1]. Needs a mask for every frame [arxiv:2105.06468 §3.1]. The released code's README numbers differ from the paper's (see Contradictions) | | Surface mesh from posed photos, and hours per scene are acceptable | NeuS | Best mean DTU Chamfer among the methods it compares, with and without masks [arxiv:2106.10689 tab. 1]. Needs normalised cameras [arxiv:2106.10689 §4.1] | | Surface from posed photos with no object masks, for solid objects | UNISURF | Mean DTU Chamfer 1.02 with no masks [arxiv:2104.10078 tab. 1]. NeuS's table lists its no-mask mean at 0.84, but each number comes from its own paper's protocol [arxiv:2106.10689 tab. 1] [arxiv:2104.10078 tab. 1]. Solid objects only [arxiv:2104.10078 §4.1] | | Surface from many views, with fast rasterisation and about 11 min of training | 2DGS | DTU Chamfer 0.80 at 10.9 min [arxiv:2403.17888 tab. 1]. But its mean Tanks and Temples F1 is 0.32 against 0.38 for NeuS [arxiv:2403.17888 tab. 2] | | Unbounded scene for view synthesis | K-Planes, 2DGS or Plenoxels | K-Planes uses an ℓ∞ contraction for unbounded scenes [arxiv:2301.10241 §3.4]. 2DGS reports Mip-NeRF 360 view synthesis, slightly below 3DGS [arxiv:2403.17888 tab. 4]. Plenoxels reports 20.40 PSNR on four 360° scenes, against 20.49 for NeRF++ [arxiv:2112.05131 tab. 2]. KiloNeRF needs a known box [arxiv:2103.13744 §3.2] and NeuS a unit sphere [arxiv:2106.10689 §4.1] | | Only 3–9 posed views per scene, and you want to keep a plain NeRF with no pre-training | FreeNeRF | Training-only regularisers with 1.02–1.04× training overhead [arxiv:2303.07418 tab. 4]. Reported DTU 3-view PSNR 19.92 against 18.50 for a reproduced RegNeRF baseline [arxiv:2303.07418 tab. 2]. Tested on 8 Blender views, and 3–9 DTU or LLFF views only [arxiv:2303.07418 §4.1] | | 8 or fewer posed views, you can afford a frozen-encoder forward pass on one low-resolution render every 10 iterations, and you accept a semantic loss whose weight the paper does not give | DietNeRF | Author-reported 23.15 PSNR at 8 views against 14.93 for NeRF [arxiv:2104.00677 tab. 1]. Simplified NeRF, tuned by hand per scene, reaches 20.09 [arxiv:2104.00677 tab. 1]. Loss weight not stated [arxiv:2104.00677 §4.4] | | One or a few posed images, and you need a new view in one forward pass with no per-scene fit | pixelNeRF | 23.72 PSNR at one chair view against 22.89 for SRN, and 26.80 mean over 13 categories against 23.28 for SRN [arxiv:2012.02190 tab. 2] [tab. 4]. Results depend on the input views chosen [arxiv:2012.02190 app. B.2]. Rendering is slow and grows with the number of input views [arxiv:2012.02190 §6] | | A mobile robot must navigate and also grasp by open-vocabulary part queries, from posed RGB-D frames that arrive during operation | GeFF | Author-reported 62.4% overall success on a quadruped, against 43.2% for LERF [arxiv:2403.07563 tab. 1]. Both baselines ran offline, so the latency gap is not like-for-like. Six days of pre-training on eight RTX 3090 GPUs [arxiv:2403.07563 §4.1]. Several training settings are not stated, including loss weights and latent width [arxiv:2403.07563 eq. 6] [§3.1] | | 2–10 posed views, a structure-from-motion run gives you sparse keypoints, and you want a plain NeRF with a depth term | DS-NeRF | 20.2 PSNR at 2 views against 13.5 for NeRF on NeRF Real, with the KL form [arxiv:2107.02791 tab. 1]. Needs SfM keypoints and inherits their depth error [arxiv:2107.02791 §3.2] [app. A]. Trails DTU-pretrained pixelNeRF at 3 views on DTU [arxiv:2107.02791 tab. 2] | | You can change only the MLP's activation, keep the standard NeRF pipeline, and need no speed or surface result | WIRE | Reported +0.1 to +0.4 dB PSNR over SIREN, and +0.2 dB over the other nonlinearities at 25 views [arxiv:2301.05187 app. A.5] [§4.3]. One scene only, and no NeRF timing [arxiv:2301.05187 §4.3] | | Free-viewpoint video of several moving people, with 16 synchronised cameras, and you need to move, copy or retime a person afterwards | ST-NeRF | Reported 33.22 PSNR on its captured data against 28.26 for NeRF-T [arxiv:2104.14786 tab. 1]. Editing needs no retraining [arxiv:2104.14786 §5]. Training takes 12 to 36 h per scene on one RTX 3090 [arxiv:2104.14786 §6], and the method needs the 16-camera rig [arxiv:2104.14786 §6.4] | | Driving or other multi-object video where objects must be moved, removed or recombined, and tracked object boxes are available | Neural Scene Graphs | Author-reported reconstruction PSNR 26.66 against 23.34 for NeRF on three KITTI scenes [arxiv:2011.10379 tab. 1]. Needs tracked object boxes, from labels or a tracker [arxiv:2011.10379 §5.1]. About one day per scene on one TITAN Xp [arxiv:2011.10379 §5]. The official repository has no licence file [card](/papers/2011.10379) | | One reference photo, the unseen sides must be plausible, and you can run a diffusion model, a captioner and a depth model, with about two hours on a 32 GB V100 | Make-It-3D | Author-reported DTU LPIPS 0.0908 (enhanced) against 0.4075 for DreamFusion+ [arxiv:2303.14184 tab. 1]. The paper states no loss weights [arxiv:2303.14184 §3.2], and the released optimiser differs (see Contradictions). The 400-image benchmark is also the authors' own [arxiv:2303.14184 tab. 2] | | Real-time dense map from a monocular video on one GPU with about 11 GB, and you can run a dense SLAM tracker that also returns depth covariances | NeRF-SLAM | Author-reported Replica average PSNR 41.40 against 17.76 for Nice-SLAM without depth, at about 10 fps [arxiv:2210.13641 tab. 1] [§4.5]. Its average depth L1 (4.49 cm) is above that of Nice-SLAM with ground-truth depth (4.08 cm) [arxiv:2210.13641 tab. 1]. Needs the tracker's covariances [arxiv:2210.13641 §3.1] | | One RGBA photo of an object, a 360° model is needed, and you can run a text-to-image diffusion model and a per-object token fit | RealFusion | Author-reported mean CO3D F-score 9.58 against 8.24 for Shelf-Supervised Mesh Prediction [arxiv:2302.10663 tab. 1], with lower scores on two of seven categories. Back sides are a guess, and extra faces can appear [arxiv:2302.10663 app. F] | | Real-time or interactive rendering with only a handful of samples per ray, and you can train a depth oracle, with reference depth or a dense NeRF's depth | DONeRF | Author-reported 31.14 dB average at 4 samples against 30.52 dB for NeRF at 256 [arxiv:2103.03231 tab. 2]. At 2 samples it is below NeRF [arxiv:2103.03231 tab. 2]. Measured timing is on one RTX 3090 at 800 × 800 [arxiv:2103.03231 §6.1]. Its code is CC BY-NC [card](/papers/2103.03231) | | Scene seen from satellite to street level, one model with several detail levels, and camera distances known per image | BungeeNeRF | Author-reported 24.51 and 24.42 average PSNR on two city scenes with longer training, against 22.31 and 22.83 for Mip-NeRF-full [arxiv:2112.05504 tab. 2]. The number of scales is a tuning choice [arxiv:2112.05504 §3.3] | | Driving scene with rigid tracked objects that must be segmented, removed or moved, with tracked boxes and 2D segmentation from other tools | Panoptic Neural Fields | Author-reported 74.28 mIoU and 21.91 PSNR on KITTI-360 novel views [arxiv:2205.04334 tab. 2]. Eq. 1 leaves the out-of-box output open [arxiv:2205.04334 eq. 1]. No official code was found [card](/papers/2205.04334) | | Scoring a monocular dynamic NeRF on your own video | Co-visible masked metrics and PCK-T, from the reality-check paper | Full-image PSNR can reward multi-view cues that a single camera cannot supply. The masked metrics and the keypoint test measure the pixels and points the training video saw [arxiv:2210.13445 §4.2] [§1] | ## Implementation sketch Common core for the ray-marching variants: sample the ray, query the field, convert densities to opacities, and composite. The scene-specific parts are in `field` and in the sample selection. ```pseudocode # Pixel colour from samples along one ray [arxiv:2103.13744 §3.1] [eq. 2-4] # Input: origin (3,), direction (3,), sample depths t (K,) # Output: pixel colour (3,) def render_ray(field, origin, direction, t): # query the field at each sample point [arxiv:2103.13744 eq. 1] points = origin + t[:, None] * direction colours, sigmas = field(points, direction) # spacing to the next sample; last one is a constant [arxiv:2103.13744 §3.1] deltas = append(diff(t), last_spacing) # opacity per sample; the minus sign is required [arxiv:2103.13744 eq. 4] alphas = 1 - exp(-relu(sigmas) * deltas) # light that survives all earlier samples [arxiv:2103.13744 eq. 3] transmittance = exclusive_cumprod(1 - alphas) weights = transmittance * alphas # weighted sum of colours is the pixel [arxiv:2103.13744 eq. 2] return sum(weights[:, None] * colours, axis=0) ``` Where the variants differ: - **Field.** `field` is one large MLP (NeRF), a grid of small MLPs routed by cell index [arxiv:2103.13744 §3.2], or a product of plane lookups [arxiv:2301.10241 §3.1]. In HexPlane the lookups are paired and multiplied, then concatenated and projected, and time enters as one of the six planes [arxiv:2301.09632 eq. 5–6]. For NeuS, `field` returns a signed distance and the opacity is computed from the distance at two neighbouring sample points, not from one density per sample [arxiv:2106.10689 eq. 13]. - **Field, no network.** In Plenoxels, `field` trilinearly interpolates opacity and 27 spherical-harmonic coefficients from the eight nearest stored corners, then evaluates the harmonics at the view direction [arxiv:2112.05131 §3.2] [§3.3]. The loop above is unchanged, and the `sigmas` and `deltas` enter eq. 1 as written. - **Field, occupancy.** In UNISURF, `field` returns occupancy in [0, 1]. The occupancy takes the place of `alphas` in the sum, so there is no exponential step. Surface rendering instead evaluates colour once at the first crossing [arxiv:2104.10078 §4.1] [eq. 6–8]. - **Field, image-conditioned.** In pixelNeRF, `field` also takes a feature from the input image at the projected point, and the feature enters the MLP as a residual. Several views are averaged inside the MLP, and the loop above is unchanged [arxiv:2012.02190 §4.1] [§4.2] [§5]. - **Layers.** In ST-NeRF there is one `field` per layer. Each layer's rays are clipped to its box, sampled and deformed with that layer's own networks, and the samples of all layers are merged before the sum above [arxiv:2104.14786 §4.2] [§4.3] [eq. 5]. - **Sample selection, one oracle.** DONeRF replaces the coarse pass with one per-ray oracle whose class scores set the sample depths, and its shading network is evaluated only at those depths [arxiv:2103.03231 §5.1]. - **Sample selection.** KiloNeRF drops samples in empty occupancy cells and stops rays early at inference [arxiv:2103.13744 §3.4]. Both are speed measures; training may skip the early stop. UNISURF instead chooses samples in a window around the first surface crossing, and its window width decays with training [arxiv:2104.10078 §4.3] [eq. 13–14]. - **Sign of the opacity.** KiloNeRF prints `1 - exp(σδ)` as eq. 4, but positive density needs the minus sign shown above [arxiv:2103.13744 §3.1]. - **No ray marching in 2DGS.** It sorts splats by centre depth and blends them front to back [arxiv:2403.17888 §4.2]. The ray-marching loop above does not apply. - **Activation in WIRE.** The loop above is unchanged. The MLP's nonlinearity is the complex Gabor wavelet of [arxiv:2301.05187 eq. 2], and the output is its real part [arxiv:2301.05187 §3.2]. - **Training-only changes in FreeNeRF.** The loop above is unchanged. The encoded inputs are multiplied by a frequency mask that grows over training, and the loss gets a penalty on the densities of the first M samples [arxiv:2303.07418 eq. 4–6]. Both terms are applied during training only [arxiv:2303.07418 §3.2] [§3.3]. - **Loss in DS-NeRF.** The colour loop is unchanged. The loss adds a depth term over keypoint rays, each rendered with the same termination weights, and it is weighted by a hyperparameter the paper does not give [arxiv:2107.02791 §3.2]. - **Loss in Dream Fields.** The colour loop is unchanged, but the photometric loss is replaced. Each step renders one random pose, composites it over a random background, and takes the negative CLIP similarity to the caption, plus the transmittance term [arxiv:2112.01455 §4.2] [§4.5] [eq. 7–9]. The weight λ is not stated in the paper, so this loop leaves it open. - **Field, point-based.** In Point-NeRF, `field` gathers the nearest points to each sample, blends their features and confidences with inverse-distance weights, and then decodes colour and density. The sum over samples is unchanged [arxiv:2201.08845 eq. 1] [eq. 4–7]. The neighbour radius and the shading-sample count are not stated [arxiv:2201.08845 §3]. - **Encoding under unknown poses.** In BARF, `encoding` is replaced by one whose bands are scaled by the window of the schedule, and the poses are parameters that get gradients from the same loss. The loop is otherwise unchanged [arxiv:2104.06405 eq. 10–15]. The schedule is driven by a fraction of the total iterations, so changing the run length moves the band openings [arxiv:2104.06405 §4.2] [card](/papers/2104.06405). - **Loss, semantic.** In DietNeRF the pixel loss is unchanged. Every K iterations, one extra term is added: the negative cosine between frozen-encoder embeddings of an unseen-pose render and a training image [arxiv:2104.00677 eq. 5] [§4.4]. Minimise the negative cosine, as the released code does. The equation as printed, read as a positive dot product, would push the embeddings apart (see the variant). - **Loss, diffusion prior.** In Magic3D, the photometric loss is replaced by score distillation sampling. Stage one samples the timestep uniformly on (0, 1) with weight one. Stage two samples it on (0.02, 0.5) and weights the gradient by the squared noise level [arxiv:2211.10440 app. B] [eq. 1–2]. - **Scene graph.** In Neural Scene Graphs, each sample takes its density and colour from the node that owns it, and density uses the object-frame point and the latent code [arxiv:2011.10379 §3.2] [eq. 7]. Background samples lie on equispaced planes rather than at free depths [arxiv:2011.10379 §4.1]. The samples from all nodes are then sorted by depth and composited with the same sum [arxiv:2011.10379 eq. 12–13]. - **Loss, single image.** In Make-It-3D's coarse loop, the reference view gets a masked L1 loss. A novel view gets the CLIP-based loss at timesteps up to 400 and SDS above that [arxiv:2303.14184 §3.2] [app. C.1]. The refine stage does not ray-march. It rasterises a coloured point cloud and decodes it with a learned renderer [arxiv:2303.14184 §3.3]. - **Loss, covariance-weighted depth.** In NeRF-SLAM the colour loop is unchanged. The depth residual is weighted by a per-pixel depth covariance from the tracker, with λ_D = 1 [arxiv:2210.13641 eq. 3–4] [§3.2]. The depth itself is the expected termination distance from the same sample weights [eq. 5–6]. The paper leaves the norm of the weighted term undefined, so the weighting is open here (see the [card](/papers/2210.13641)). - **Loss, single image with a learned token.** In RealFusion the reconstruction term is a masked image loss at the fixed camera, plus a mask loss on opacity. Each step also renders one random prior view and adds SDS on it, with a view-dependent prompt built from the learned token and a guidance weight of 100 [arxiv:2302.10663 eq. 6] [app. A]. The token is trained first, as a separate step [arxiv:2302.10663 app. A]. ## Contradictions - KiloNeRF's NeRF baseline on NSVF's four datasets scores higher than NSVF reports for NeRF. The authors attribute this to longer training [arxiv:2103.13744 §4.1]. The NSVF paper is not carded, so this is not resolved here. - The numbers are not like-for-like across rows. The 2DGS table reports NeuS at 0.84, which matches NeuS's no-mask column [arxiv:2403.17888 tab. 1] [arxiv:2106.10689 tab. 1]. Its NeRF column (1.49) also matches NeuS's no-mask NeRF column. The mask setting changes the comparison. - Speed figures use different hardware, image sizes and baselines. KiloNeRF's NeRF times are for NeRF with hierarchical sampling [arxiv:2103.13744 §4.1]. NeuS's rendering time is for NeuS itself at 1600 × 1200 [arxiv:2106.10689 app. E]. Plenoxels' JAXNeRF time is scaled from four GPUs [arxiv:2112.05131 app. D]. - NeRF-SLAM gives two frame rates for its full pipeline: about 12 fps in one sentence and 10 fps in its summary [arxiv:2210.13641 §4.5]. This page quotes the 10 fps figure and does not pick between them. - NeRF-SLAM's headline gains (up to 179% PSNR and 86% lower depth L1) are measured against the Nice-SLAM row without depth, but the text does not name that row. Against Nice-SLAM with ground-truth depth, the gains are smaller [arxiv:2210.13641 §4.3] [tab. 1]. - RealFusion's paper puts the coarse-to-fine switch at the half-way point of training, about step 2500 of 5000 [arxiv:2302.10663 §3.2] [app. A]. The released code lifts the masking after step 3000 (see the [RealFusion card](/papers/2302.10663)). This page gives the paper's value. - Plenoxels gives two NeRF numbers for the synthetic 100-image setting: 31.85 PSNR for JAXNeRF in its main table, and 31.01 for NeRF in its view-count ablation [arxiv:2112.05131 tab. 2] [tab. 3]. The two rows use different baselines, and this page does not pick one. - UNISURF's paper gives a sample window of 1.0 to 0.05 for DTU. The released DTU configs use 2.0 to 0.1 [card](/papers/2104.10078), and the paper does not say which produced its table [arxiv:2104.10078 §4.4]. Use the paper's values only with that gap in mind. - DS-NeRF's text calls its keypoint uncertainty a variance, but its formula uses twice the square of that value [arxiv:2107.02791 §3.2]. The paper's headline uses the KL form, and its own MSE variant scores 19.5 at 2 views against 20.2 for KL [arxiv:2107.02791 tab. 1]. - pixelNeRF's text says the image feature enters at each ResNet block [arxiv:2012.02190 §4.1] [§5]. The authors' released code adds it at the first three of five blocks (see the [pixelNeRF card](/papers/2012.02190)). This page does not pick one reading, and reproductions should say which they use. - FreeNeRF gives the frequency-schedule end for 9 views as 20% of training in its main text and 50% in its appendix [arxiv:2303.07418 §4.1] [arxiv:2303.07418 app. A.1]. The two cannot both be followed, and this page does not pick one. - FreeNeRF's default DTU 3-view number is 19.92 in its main table and 20.20 in its ablation table, and the paper does not explain the gap [arxiv:2303.07418 tab. 2] [arxiv:2303.07418 tab. 5]. Compare only numbers taken from the same table. - WIRE gives its NeRF width two ways. The main text says 128 features for all nonlinearities [arxiv:2301.05187 §4.3]. The appendix says 182 features for the other nonlinearities and 128 for WIRE [arxiv:2301.05187 app. A.5]. Under the appendix reading the baselines are wider than WIRE, so the +0.2 dB comparison is not width-matched. This page does not pick one reading. - ST-NeRF's Table 1 caption says the method is best on all four metrics, but the text says it beats every baseline on PSNR, SSIM and MAE, and the table shows NeRF-T with the higher SSIM (0.9243 against 0.9203) and ST-NeRF with the higher MAE (0.1178 against 0.0219) [arxiv:2104.14786 tab. 1] [§6.2]. Those are the same paper's claims, and they disagree with its own table. - ST-NeRF gives its synthetic camera count as 36 in the dataset section and 17 in the view ablation [arxiv:2104.14786 §6] [§6.3]. This page does not pick one. - Dynamic NeRF's paper reports 24.74 PSNR and 0.118 LPIPS on seven sequences [arxiv:2105.06468 tab. 1]. The card records that the released PyTorch README reports 26.10 and 0.082 for the same sequences, and that the README attributes the paper's numbers to the TensorFlow code [card](/papers/2105.06468). This page does not pick one; the paper's table is the citable figure. - The reality-check paper defines mPSNR as the average of per-pixel PSNR over the kept pixels [arxiv:2210.13445 app. B]. Its code computes one PSNR from the mean squared error over those pixels [code: dycheck/core/metrics/image.py#L46]. Numbers from the code are therefore not the paper's metric. - Point-NeRF's Table 1 lists NeRF at 10 h on DTU, while its text gives 20 h or more for the same comparison [arxiv:2201.08845 tab. 1] [§6.2] [§7]. The speed-up factor depends on which one is used, and the paper gives both. - BARF's paper gives a single MLP with 128 hidden units for the synthetic scenes [arxiv:2104.06405 §4.2]. The released configs use eight feature layers of 256 units, for both synthetic and real scenes (see the [BARF card](/papers/2104.06405)). This page does not pick one, and a reproduction should state which it uses. - Magic3D's 40 min on eight A100 GPUs and DreamFusion's 1.5 h on TPUv4 are measured on different hardware, and the second is quoted from DreamFusion [arxiv:2211.10440 §1] [§5]. Treat "faster" as author-reported. - GeFF names MaskCLIP as its teacher [arxiv:2403.07563 §3.2], but its training details say the teacher is a ViT-L CLIP model [arxiv:2403.07563 §4.1]. The paper does not say whether MaskCLIP runs on a ViT-L backbone, so this page does not pick one. - GeFF's total loss includes a colour term [arxiv:2403.07563 eq. 6], but the text gives no colour decoder or colour rendering for GeFF, whose features are driven by the SDF and depth [arxiv:2403.07563 §3.2]. The term's meaning is left open here. - Make-It-3D's paper says Adam with learning rate 0.001 for both stages [arxiv:2303.14184 §4.1]. The released code's default optimiser is Adan with a learning rate of 5e-3, and Adam is used only when it is selected [card](/papers/2303.14184). The paper's novel-view camera ranges are 0.8 to 1.2 for distance and 40 to 80 degrees for field of view [arxiv:2303.14184 app. B.1], where the code's defaults are 1.0 to 1.5 and 15 to 25 [card](/papers/2303.14184). This page gives the paper's values. - Make-It-3D's refine stage is described as a coarse-style objective plus a regulariser [arxiv:2303.14184 §3.3]. The released code adds SDS, a CLIP image loss and a contextual loss there, none of which the paper mentions [card](/papers/2303.14184). The two sources cannot both be followed, and this page does not pick one. - Neural Scene Graphs' paper sums the squared colour error over the ray batch and trains with a linear learning-rate decay [arxiv:2011.10379 eq. 14] [§5]. The released code takes a mean over the batch, decays the rate exponentially, and runs to 1 000 000 steps, where the paper says about 350k [card](/papers/2011.10379). Reproductions should say which they follow. - Neural Scene Graphs' novel-composition block gives the SRN row the same reconstruction values as the SRN row above it [arxiv:2011.10379 tab. 1]. The paper does not say whether they were re-measured, so check the supplement before quoting that row. ## Papers - [KiloNeRF](/papers/2103.13744): a grid of tiny per-cell MLPs, distilled from a NeRF teacher, for fast rendering of bounded scenes. - [Plenoxels](/papers/2112.05131): a sparse voxel grid of opacity and spherical-harmonic coefficients, fitted with NeRF's rendering and no neural network. - [NeuS](/papers/2106.10689): a signed distance field rendered with an opaque, occlusion-aware density, for surface reconstruction. - [UNISURF](/papers/2104.10078): an occupancy field rendered by one formula for surfaces and volumes, for mask-free multi-view reconstruction. - [K-Planes](/papers/2301.10241): factorised 2D planes whose products are decoded to colour, for fast training and static or dynamic scenes. - [HexPlane](/papers/2301.09632): six spatial and spatio-temporal planes whose paired products are decoded to colour, for dynamic multi-view scenes. - [2DGS](/papers/2403.17888): flat Gaussian disks rendered by ray–splat intersection, for accurate surfaces from rasterised primitives. - [FreeNeRF](/papers/2303.07418): a frequency schedule and a near-camera density penalty added to NeRF training, for few input views. - [DS-NeRF](/papers/2107.02791): a depth term from SfM keypoints added to NeRF training, for few views and faster convergence. - [pixelNeRF](/papers/2012.02190): a NeRF conditioned on pixel-aligned features of one or a few posed images, trained across many scenes, with no per-scene fit. - [WIRE](/papers/2301.05187): a complex Gabor wavelet activation for implicit networks, with a NeRF experiment on the drums scene. - [ST-NeRF](/papers/2104.14786): one space-time radiance field per performer and per background, composited for editable free-viewpoint video. - [Monocular Dynamic View Synthesis: A Reality Check](/papers/2210.13445): a non-teleporting protocol, co-visibility masked metrics and an iPhone benchmark for monocular dynamic fields. - [Dream Fields](/papers/2112.01455): a NeRF optimised against a frozen CLIP caption score over random views, for text-to-3D with no 3D data. - [Dynamic View Synthesis from Dynamic Monocular Video](/papers/2105.06468): a static NeRF plus a time-conditioned scene-flow field, with flow regularisers, for one moving video with masks. - [Putting NeRF on a Diet](/papers/2104.00677): a frozen CLIP embedding of renders at unseen poses adds a semantic term to NeRF's pixel loss, for few views. - [BARF](/papers/2104.06405): a NeRF whose camera poses are optimised jointly, with positional-encoding bands opened from low to high frequency. - [GeFF](/papers/2403.07563): a generalizable NeRF with SDF and CLIP-distilled features, encoded from posed RGB-D frames in one pass, for robot navigation and grasping. - [Point-NeRF](/papers/2201.08845): a cloud of feature points with confidences, initialised by a pre-trained MVS-style network and refined per scene. - [Magic3D](/papers/2211.10440): a coarse hash-grid field under a 64 × 64 diffusion prior, then a textured mesh under a 512 × 512 latent prior, for text-to-3D. - [A Survey on 3D Gaussian Splatting](/papers/2401.03890): a survey that maps 3DGS follow-ups by theme and collects their reported numbers; it proposes no method. - [Neural Scene Graphs](/papers/2011.10379): a static background field plus one latent-coded field per tracked object, composed by a scene graph for editable driving scenes. - [Make-It-3D](/papers/2303.14184): a NeRF coarse stage under SDS, CLIP and depth priors, then a point-cloud refine stage with a neural renderer, for one-photo 3D. - [NeRF-SLAM](/papers/2210.13641): a hash-grid field fitted in parallel to a dense monocular SLAM tracker, with its depth term weighted by the tracker's covariances, for real-time mapping. - [RealFusion](/papers/2302.10663): a hash-grid field fitted to one RGBA photo, with SDS on random views under a token learned from augmented copies, for 360° single-object reconstruction. - [DONeRF](/papers/2103.03231): a depth oracle evaluated once per ray places 2 to 16 shading samples, for real-time rendering of compact NeRFs. - [BungeeNeRF](/papers/2112.05504): a base block plus one residual block and output head per scale, trained stage by stage, for city scenes from satellite to street level. - [Panoptic Neural Fields](/papers/2205.04334): one MLP per tracked object inside its box, plus a background MLP with a semantic head, for driving scenes.