Paper · 2023-03-28 · cs.CV, cs.GR
F²-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories
F²-NeRF trains a grid-based NeRF on unbounded scenes with arbitrary camera paths. Space is warped per region by PCA over projected image coordinates of nearby cameras ("perspective warping"), regions come from an octree, and each region gets its own hash functions. Author-reported: best PSNR among fast-training NeRFs on the authors' Free dataset (26.32 in about 12 min on a 2080 Ti).
Neural radiance field (NeRF) and variantsNovel view synthesisExplicit feature grids (dense voxels and factorised planes)Space warpingnerfhash-gridunbounded-scenesnovel-view-synthesis
arXivPDFCodeMarkdown14 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A neural radiance field (NeRF) learns a scene so that new camera views can be rendered. Grid-based versions learn fast, but a fixed grid of boxes wastes space on empty air and cannot cover a huge scene evenly. Scenes are often "unbounded": the camera walks far away, and the far background needs far less detail than the objects close to the path.
The paper's idea is to bend space before it is put into the grid. The bend is chosen so that grid cells line up with the rays of the cameras that see each area. It is found by a principal-component step over the positions where a point lands in the nearby photos. The scene is split into regions, each with its own bend, and regions are finer where cameras are close. The method works for any camera path, not only walks around an object or straight forward-facing shots.
TL;DR
- Grid NeRF for unbounded scenes with arbitrary camera trajectories; author-reported training time is about 12 to 14 minutes on one 2080 Ti.
- The core is "perspective warping": a 3D-to-3D map built by PCA on the projected 2D coordinates of selected cameras, applied per octree leaf.
- Each leaf has its own hash functions into one shared hash grid (16 levels, 2^19 entries per level), followed by a small MLP and a view-dependent colour MLP.
- The paper reports the method as the best PSNR among fast-training NeRFs on its new Free dataset, and comparable results on LLFF and NeRF-360-V2.
- The official code is C++ with LibTorch and CUDA kernels; it differs from the paper in several hyperparameters and in how subdivision and step sizes work (see Paper vs code).
Problem
Grid NeRFs that train in minutes §1 store one value per grid cell. Their memory grows with the cube of the scene size, so they are normally used for bounded scenes §1. Existing ways to handle unbounded scenes are the NDC warp for forward-facing paths and the inverse-sphere warp for 360-degree object-centred paths §1. Both assume a particular path shape. On long, narrow "free" trajectories with several foreground objects, the authors report that rendering quality drops, because empty space still takes grid cells and foreground objects get too few of them §1 §4.2 tab. 1. The method assumes that the poses of all input images are known §3.
Method
Overview. The scene is warped into a bounded space. The warped space is divided into an octree; each leaf is mapped by its own warp; a hash grid is built on the common warp space; points are sampled along rays with a sampling rule that follows the warp; colour and density are composited with standard volume rendering eq. 1 §3.1. The volume rendering sum is ĉ = Σ_i T_i α_i c_i with α_i = 1 − exp(−δ_i σ_i) and T_i the accumulated transmittance eq. 1 §3.1.
Proper warping (definition). A warp F is "proper" for a region S and its visible cameras if the squared distance between two points in warp space equals the sum of their squared distances in the cameras' image coordinates, for all point pairs in S §3.2, Definition 1. In 2D, two cameras that each project to a 1D image give such a warp directly §3.2 fig. 2.
Perspective warping. For 3D, the authors stack the image coordinates of all chosen cameras into a vector G(x), which is a proper warp but has too many dimensions §3.2. They then ask for a linear map M from that vector to 3D that best preserves distances between sampled points (Problem 1) §3.2. The solution is PCA: M is built from the first three eigenvectors of the covariance of the projected points §3.2 app. A.2. The warp is F(x) = M G(x), followed by a post-normalisation that puts the region near the origin §3.2 fig. 3. For a forward-facing set of cameras, the warp reduces to the NDC warp; for a 360-degree set, the inner sphere is left unwarped and the outer region is squashed like NDC, so inverse-sphere warping appears as a special case §3.2 fig. 4 fig. 5 app. B.
Space subdivision. The root is a cube whose side is 512 times the size of the bounding box of all camera centres §3.3. A node is split into 8 children when the test below holds; the paper states it as "any visible camera centre within λ·s of the node centre", with λ = 3 and s the node side §3.3. Each leaf is a region S. For regions with more than n_c = 4 visible cameras, the paper keeps n_c cameras by maximising their minimum pairwise distance §3.3 app. A.1. Each chosen camera is then rectified: it is rotated to look at the region centre, and it is moved to a common distance r, the mean distance of the nearest quarter of cameras to the centre app. A.1.
Per-leaf warp construction. Inside a leaf: sample n = 32³ points, project them into the chosen cameras, stack the 2D coordinates, take the top three eigenvectors of their covariance, then scale each warp axis so that a unit step in warp space moves the image by about one pixel app. A.1 app. A.2 eq. 4. The scale is the average over the sampled points app. A.2.
Scene representation. A hash grid with 16 levels and 2 features per entry; each level has 2^19 entries §3.4 app. A.5. To avoid one table per leaf, all leaves share one table, but each leaf uses different hash functions: integer primes per leaf, level and axis, plus an offset Δ, combined by XOR and reduced modulo the level's table size §3.4 eq. 2. The 8 corner features are trilinearly interpolated. The 32 values (16 features per level × 2 channels) feed a tiny MLP with one hidden layer of width 64 that outputs density and scene features app. A.5. The scene features and the spherical-harmonic encoding of the view direction feed a second MLP with two hidden layers of width 64 that outputs RGB app. A.5 fig. 6.
Perspective sampling. Points are placed so that a fixed step in warp space corresponds to a fixed warp-space distance l, rather than a fixed distance in the world §3.5. The next point is found by a first-order approximation of the warp along the ray: the step is l divided by the norm of J·d, where J is the Jacobian of the warp at the current point and d the ray direction §3.5 app. A.3. The paper sets l to √3, the diagonal of a unit cube in warp space app. A.3.
Rendering. Preparation: subdivide, then build a warp for each region. Rendering: march along each ray through the octree leaves it crosses, sample with perspective sampling, look up the hash grid, and composite §3.6.
Training loss eq. 3 eq. 6 §3.7 app. A.4:
- colour term: a Charbonnier-style term sqrt(diff² + ε) with ε = 10⁻⁴ eq. 3 app. A.4;
- disparity penalty on the mean of squared per-ray disparity, where disparity is Σ w_i / t_i eq. 7 app. A.4;
- total-variation term on features at the borders of neighbouring leaves, 8192 random pairs per iteration eq. 8 app. A.4.
# Perspective warp for one leaf region S [§3.2] [app. A.1] [app. A.2]
# Input: visible cameras (positions, axes, focal length); region centre c
# Output: 3 x 2n_c matrix M, plus a per-leaf sample-spacing scale
def build_leaf_warp(visible_cams, centre):
# Choose n_c cameras spread out in direction [§3.3] [app. A.1]
chosen = farthest_point_sample(visible_cams, n_c)
# Turn each chosen camera to face the centre, at a common distance r
# [app. A.1]
rectified = rectify(chosen, centre)
# Sample 32^3 points inside the region [app. A.2]
points = sample_in_region(centre, count=32 ** 3)
# Stack each point's 2D image coordinates over the chosen cameras
# [§3.2] [app. A.2]
coords = stack_projections(rectified, points) # (n, 2 * n_c)
# M' = top-3 eigenvectors of the coordinate covariance [§3.2] [app. A.2]
M_prime = top_eigenvectors(covariance(coords), k=3)
# Per-axis scale so one warp unit is about one pixel [app. A.2]
scales = axis_scales(M_prime, rectified, points) # (3,)
return diag(scales) @ M_prime# Perspective sampling along one ray [§3.5] [app. A.3]
# Input: origin, direction, near and far depth inside one leaf
# Output: sample depths along the ray
def sample_ray(origin, direction, t_near, t_far):
# Warp-space step length l = sqrt(3) [app. A.3]
step_l = sqrt(3)
t = t_near
depths = []
while t < t_far:
# Jacobian of the warp at the current point [app. A.3]
jac = warp_jacobian(origin + t * direction) # (3, 3)
# Distance along the ray that moves the warp by step_l [app. A.3]
t = t + step_l / norm(jac @ direction)
depths.append(t)
return depthsHyperparameters
Values are as stated in the paper unless marked "code". Where the paper and the official code disagree, both are shown in "Paper vs code".
| Name | Value | Where | Notes |
|---|---|---|---|
| Training steps | 20k | app. A.5 | same for all scenes; 50k runs in the Supplement |
| Points per batch | 256k | app. A.5 | fixed number of point samples; ray count varies |
| Optimizer | Adam | app. A.5 | betas and weight decay not stated |
| Learning rate | linear 0 → 1e-1 over 1k steps, cosine to 1e-2 | app. A.5 | conflicts with code, see Paper vs code |
| Hash levels | 16 | §3.4 app. A.5 | |
| Features per level | 2 | app. A.5 | |
| Entries per level | 2^19 | app. A.5 | one hash function per leaf, level and axis |
| Per-level resolution | not stated | — | code: 2^3 to 2^10 per warp unit, see Pitfalls |
| Density MLP | 1 hidden layer, width 64 | app. A.5 | output width not stated |
| Colour MLP | 2 hidden layers, width 64 | app. A.5 | input: scene features + view encoding |
| View encoding | spherical harmonics | app. A.5 | degree not stated (code: 4) |
| Colour loss ε | 1e-4 | eq. 3 app. A.4 | |
| λ_TV | 1e-1 | app. A.4 | all datasets |
| λ_Disp, LLFF | 2.5e-4 | app. A.4 | |
| λ_Disp, Free and NeRF-360-V2 | 1e-3 | app. A.4 | |
| TV samples per iteration | 8192 border pairs | app. A.4 | |
| Octree root side | 512 × camera-centre extent | §3.3 | |
| Split threshold λ | 3 | §3.3 | code uses 1.5, see Paper vs code |
| Cameras per region n_c | 4 | §3.3 app. A.1 | code uses 6, see Paper vs code |
| Warp sampling step l | √3 | app. A.3 | code uses a different value, see Paper vs code |
| PCA point count | 32³ | app. A.2 | |
| Rectified distance r | mean of the nearest 1/4 cameras | app. A.1 | |
| Training time | 12 min (Free), 14 min (NeRF-360-V2), 13 min (LLFF) | tab. 1 tab. 2 tab. 3 | author-reported, 2080 Ti |
Results
Author-reported. Setup: Free dataset (seven new scenes with long, narrow trajectories) §4.1; LLFF (eight forward-facing scenes) §4.1; NeRF-360-V2 (seven 360-degree scenes) §4.1; every eighth image is a test view and PSNR, SSIM and LPIPS (VGG) are reported §4.1. Training times are on one 2080 Ti §4.1 tab. 1.
Table 1 (Free dataset) tab. 1
| Method | Train time | PSNR ↑ | SSIM ↑ | LPIPS (VGG) ↓ |
|---|---|---|---|---|
| NeRF++ | hours | 23.47 | 0.603 | 0.499 |
| mip-NeRF-360 | hours | 27.01 | 0.766 | 0.295 |
| mip-NeRF-360 (stopped at 30 min) | 30m | 22.04 | 0.537 | 0.586 |
| Plenoxels | 25m | 19.13 | 0.507 | 0.543 |
| DVGO | 21m | 23.90 | 0.651 | 0.455 |
| Instant-NGP | 6m | 24.43 | 0.677 | 0.413 |
| F²-NeRF | 12m | 26.32 | 0.779 | 0.276 |
Table 2 (NeRF-360-V2) tab. 2
| Method | Train time | PSNR ↑ | SSIM ↑ | LPIPS (VGG) ↓ |
|---|---|---|---|---|
| NeRF++ | hours | 26.21 | 0.729 | 0.348 |
| mip-NeRF-360 | hours | 28.94 | 0.837 | 0.208 |
| Plenoxels | 22m | 23.35 | 0.651 | 0.471 |
| DVGO | 16m | 25.42 | 0.695 | 0.429 |
| Instant-NGP | 6m | 26.24 | 0.716 | 0.404 |
| F²-NeRF | 14m | 26.39 | 0.746 | 0.361 |
Table 3 (LLFF) tab. 3
| Method | Train time | PSNR ↑ | SSIM ↑ | LPIPS (VGG) ↓ |
|---|---|---|---|---|
| NeRF | hours | 26.50 | 0.811 | 0.250 |
| mip-NeRF | hours | 26.60 | 0.814 | 0.246 |
| Plenoxels | 17m | 26.29 | 0.839 | 0.210 |
| DVGO | 11m | 26.34 | 0.838 | 0.197 |
| TensoRF | 48m | 26.73 | 0.839 | 0.204 |
| Instant-NGP | 6m | 25.09 | 0.758 | 0.267 |
| F²-NeRF | 13m | 26.54 | 0.844 | 0.189 |
Ablation on the Free "pillar" scene tab. 4 §4.3
| Setting | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| A: no warp, disparity sampling | 26.81 | 0.693 | 0.385 |
| B: inverse-sphere warp, disparity sampling | 27.09 | 0.711 | 0.358 |
| C: inverse-sphere warp, exponential sampling | 27.86 | 0.751 | 0.289 |
| D: perspective warp, exponential sampling | 28.65 | 0.796 | 0.221 |
| E: perspective warp, perspective sampling | 28.76 | 0.798 | 0.219 |
Longer training (Supplement) app. C.1: on the Free dataset, Instant-NGP at 50k steps (15 min) reaches 25.07 PSNR, below F²-NeRF at 20k steps (26.32); F²-NeRF at 50k steps reaches 26.85 (30 min) and with a 2^20 table 27.19 (36 min). On NeRF-360-V2, F²-NeRF at 50k steps (33 min) reaches 26.92 PSNR. app. C.1
Compatibility with MLP-based NeRF app. C.2: on LLFF with an 8-layer MLP, mean PSNR is 26.31 for NDC warp, 26.02 for inverse-sphere warp and 26.29 for perspective warp (250k steps; the authors note NDC is below its published figure because of fewer steps).
Extrapolation app. C.3: on the NeRF synthetic "Lego" scene with elevation above 30° held out, perspective warp and no warp are close: 37.63 vs 37.15 PSNR for elevations 0–30°, and 27.35 vs 27.03 for 60–90°.
Paper vs code
Official code: https://github.com/totoro97/f2-nerf, Apache-2.0, at commit
98f0daacb80e76724eb91519742c30fb35d0f72d. It is C++ with LibTorch and custom
CUDA kernels, with Python scripts for camera poses. The project page links it. The LICENSE and the source and config files cited
below were read; the code was not built or run.
Disagreements.
- Cameras per region: paper n_c = 4 §3.3 app. A.1; code keeps N_PROS = 12 coordinates, i.e. 6 virtual cameras PersSampler.h PersSampler.cpp.
- Split rule: paper splits when a visible camera centre lies within λ·s with λ = 3 §3.3. Code splits when at least 6 cameras are visible and a soft-minimum of their distances (geometric mean of the nearest quarter, in log space) is below 1.5·s; the threshold is 1.5 in the config PersSampler.cpp PersSampler.cpp perspective.yaml.
- Subdivision schedule: paper describes one subdivision during preparation §3.6. Code subdivides during training at milestones 2000, 4000, 6000, 8000, 10000 (LLFF: 1000 to 10000) perspective.yaml llff.yaml PersSampler.cu.
- Pruning: not in the paper. Code marks octree nodes invalid when their weight or alpha stays below thresholds (absolute 0.01 / 0.02, relative 0.1 of the per-ray maximum), and compacts every 1000 steps PersSampler.cu PersSampler.cu.
- Warp sampling step: paper l = √3 in warp space app. A.3. Code uses sample_l = 1/256 (Free, NeRF-360) or 1/512 (LLFF), times a per-sample factor drawn uniformly from [0.5, 1.5] (training), times a fineness that decays geometrically from 4 to 1 over 10k steps perspective.yaml llff.yaml PersSampler.cu PersSampler.cu PersSampler.cu ExpRunner.cpp. The first sample on each ray is not jittered PersSampler.cu.
- Distance alignment: paper sets r to the mean distance of the nearest quarter of cameras app. A.1. Code uses a log-space geometric mean of the lower quartile and pushes cameras nearer than that to that distance PersSampler.cpp PersSampler.cpp.
- Scale normalisation: paper averages the per-point solved scales app. A.2. Code computes a per-point step 1/max|J| for each warp axis, and divides the warp matrix by the mean of those steps PersSampler.cpp. These are not obviously the same quantity; check numerically before relying on the paper's formula.
- PCA sample points: paper says the 32³ points are evenly sampled app. A.2. Code draws uniform random points PersSampler.cpp.
- Learning rate: paper gives a peak of 1e-1 and a final value of 1e-2 app. A.5. Code peaks at 1e-2 and decays by cosine to 1e-3 (alpha = 0.1 times peak) 20k.yaml ExpRunner.cpp.
- Disparity weight: paper λ_Disp = 1e-3 for Free and NeRF-360-V2, 2.5e-4 for LLFF app. A.4. Code default is 0 20k.yaml; Free and NeRF-360 configs do not override it free.yaml nerf-360.yaml; LLFF sets 5e-2 llff.yaml.
- Variance loss: not in the paper. Code adds a term sqrt(variance of weights + 0.01), with weight 0.01 ramped from zero at 5000 to full at 10000 steps ExpRunner.cpp 20k.yaml.
- Hash offset: paper's Δ is added to the hash argument before the modulo, with prime multipliers eq. 2. Code adds a random real-valued offset (100 to 1100 per level and leaf) to the warp coordinates before taking the voxel corner, and the primes are integers in [2^28, 2^30) Hash3DAnchored.cpp Hash3DAnchored.cpp Hash3DAnchored.cu.
Agreements checked. 16 levels and 2 channels Hash3DAnchored.h; 2^19 entries per level Hash3DAnchored.cpp; 256k points per batch and 20k steps 20k.yaml; color loss with ε = 1e-4 ExpRunner.cpp; disparity as the sum of weight over t Renderer.cpp; 8192 border pairs for the TV term Renderer.cpp; λ_TV = 0.1 20k.yaml; root side 512 PersSampler.cpp; shader MLP with two hidden layers of width 64 and input 32 sh_shader.yaml; tiny MLP width 64 with 16 outputs hash3d_anchored.yaml.
Code-only details the paper does not state. Feature resolution per level runs from 2^3 to 2^10 per warp unit Hash3DAnchored.cu Hash3DAnchored.h. Image downsampling is 2 for Free, 2 or 4 for NeRF-360 (per scene) and 4 for LLFF free.yaml nerf-360.yaml llff.yaml. Background colour during training is random noise per ray by default Renderer.cpp Renderer.cpp. Gradients through the hash table are scaled by 128 and stored in fp16 Hash3DAnchored.cu Hash3DAnchored.cu.
Pitfalls
- Learning-rate schedule is ambiguous. The paper's peak of 1e-1 and final value of 1e-2 differ from the code (1e-2 peak, 1e-3 final). The paper text may be a typo; there is no evidence either way. Any reimplementation should test both and say which it uses.
- Disparity weight depends on the config. The paper's values are not what the Free or NeRF-360 configs use (0). An implementer who copies the paper gets a different regulariser from the released runs.
- Unstated but numerically sensitive values. Feature init (not stated in paper; code uses (rand × 0.2 − 1) × 1e-4 Hash3DAnchored.cpp, which draws only from [−1e-4, −0.8e-4] and is probably not what was intended), hash resolution range, grad scaling, and the density head width are all absent from the paper.
- Step length is a unit change. √3 in warp space (paper) and 1/256 with jitter and a decaying fineness (code) give very different sampling densities. The code's warp scale is set by the normalisation step, so the two numbers are not comparable without that scale.
- Subdivision and pruning change the model during training. Leaves appear at milestones and nodes can be deleted by the occupancy statistics. A reimplementation that subdivides once, as the paper's figure suggests, will not reproduce the code's behaviour.
- Abstract understates cost. The abstract says "a few minutes"; the tables give 12 to 14 minutes on a 2080 Ti for F²-NeRF, against 6 minutes for Instant-NGP, which uses a CUDA implementation the authors call faster than theirs §4.1. Longer runs narrow the gap app. C.1.
- Baselines are not all like-for-like. The mip-NeRF-360 row with the best PSNR trains for hours; the fast row is a 30-minute early stop tab. 1. Baseline warps also differ by dataset: NDC on LLFF, inverse sphere on Free and NeRF-360, Instant-NGP with tuned scales §4.1.
- Camera poses are assumed known §3.1. The code uses COLMAP or hloc scripts for poses; their accuracy is not evaluated here.
- Table size is a cost lever. Doubling the entries per level to 2^20 raises Free PSNR from 26.85 to 27.19 at 36 min instead of 30 min, and the paper gives no memory figures for either size app. C.1.
Lineage
- Builds on: NeRF (arXiv:2003.08934), for volume rendering and the NDC warp §1 §3.2; NeRF++ (arXiv:2010.07492), for the inverse-sphere warp §3.2; mip-NeRF 360 for the inverse-sphere warp and the robust colour loss §3.2 §3.7 app. A.4; Instant-NGP for the hash grid and the MLP layout §3.4 app. A.5.
- Compared against: Plenoxels (2112.05131), DVGO, TensoRF, Instant-NGP, mip-NeRF and mip-NeRF 360 on the three datasets tab. 1 tab. 2 tab. 3.
- Superseded by: not stated.
Source and license
Adapted from “F²-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories” by Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, Wenping Wang (arXiv:2303.15951), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/totoro97/f2-nerf at commit 98f0daacb80e (Apache-2.0). Any quoted code is shown under that license.