Self-Calibrating Neural Radiance Fields

In plain words

A camera never matches the textbook model exactly: its lens bends light in ways a simple formula misses. The usual fix is to photograph a printed checkerboard and solve for the camera's settings. This paper asks whether the camera can be worked out from ordinary photos of a scene instead.

The idea is to learn the 3D scene and the camera at the same time. A neural network stores the scene's geometry and colour, and the camera model has adjustable settings. Both are tuned until renders from the current camera guess match the real photos. A second check uses matching points between two photos: the rays from the camera through a matched pair should meet in space, and the gap between them is penalised, which pushes the camera settings toward a consistent geometry.

Training starts with a simple camera and adds correction terms one stage at a time, which keeps learning stable. The authors report that the learned cameras give better renders than a neural scene model that is handed camera settings from a standard structure-from-motion tool.

TL;DR

• Learns a NeRF scene and a differentiable camera model jointly from images, with no calibration target §1.
• Camera model: pinhole intrinsics and extrinsics as residuals on fixed initial values, a 4th-order radial distortion, and per-pixel ray origin and direction offsets §4.
• Loss: NeRF photometric loss plus a projected ray distance (PRD) loss on matched pixels, switched on in stages (curriculum) §5, §6.
• Author-reported: higher PSNR than NeRF both without and with COLMAP cameras on LLFF, and than NeRF++ on Tanks and Temples §7.
• Official MIT-licensed code exists, but several defaults and run scripts differ from the paper. See Paper vs code.

Problem

Classical self-calibration assumes a linear pinhole camera. It cannot express the generic non-linear distortion of real lenses, and it relies on sparse correspondences and on geometric constraints computed from a non-differentiable pipeline §1. NeRF-based pipelines in this paper start from COLMAP camera poses. When the camera information is wrong, the matcher can find no inlier correspondences, and the PRD metric is undefined (reported as "nan") tab. 1.

The method assumes enough reliable feature matches between views. On fish-eye images the authors found conventional matching unreliable, so they dropped the PRD loss for those scenes §7.5.

Method

The renderer is NeRF volume rendering. A colour is the sum over samples along a ray of accumulated transmittance times the sample's alpha and colour §3, eq. 1.

1. Intrinsics as residuals §4.1, eq. 2. The intrinsic matrix is K = K_0 + \Delta K, where K_0 is an initial matrix and \Delta K holds learnable residuals on f_x, f_y, c_x, c_y eq. 2. The paper says the norm of \Delta K is bounded, but gives no bound.

2. Extrinsics via a 6D rotation §4.1, eq. 3. Rotation is parameterised by six numbers a_1, a_2 (the first two columns of a matrix). Let b_1 = N(a_1), b_2 = N(a_2 - (b_1 \cdot a_2) b_1) and b_3 = b_1 \times b_2, where N normalises to unit length eq. 3. The learned rotation is R = f(a_0 + \Delta a) and the translation t = t_0 + \Delta t; R_0, t_0 and K_0 stay fixed §4.1.

3. Pinhole rays §4.1, eq. 4. A pixel p gives a direction r_d = R K^{-1} p and origin r_o = t eq. 4. Gradients flow from the rays into the residuals.

4. Radial distortion §4.1, eq. 5–8. Let n = ((p_x - c_x)/c_x, (p_y - c_y)/c_y, 1) eq. 5. Per-axis factors are d_x = 1 + k_1 n_x^2 + k_2 n_x^4 and d_y = 1 + k_1 n_y^2 + k_2 n_y^4 eq. 6. The distorted pixel is p' = (p_x d_x, p_y d_y, 1) eq. 7, and rays are r_d = R K^{-1} p', r_o = t eq. 8. The coefficients are k = (k_1 + z_{k_1}, k_2 + z_{k_2}) with learnable residuals. The paper follows the COLMAP radial fisheye model and drops higher orders §4.1.

5. Generic per-ray offsets §4.2, eq. 9. Per-pixel residuals z_d (direction) and z_o (origin) are learned on a discrete grid and bilinearly interpolated at p. The corrected rays are r_d' = r_d + z_d and r_o' = r_o + z_o §4.2. Eq. 9 is printed with a typo; see Pitfalls.

6. Projected ray distance §5.1, eq. 10–13. For a match p_A \leftrightarrow p_B with rays r_A and r_B, the closest points \hat x_A and \hat x_B on the two rays are found in closed form eq. 10–11. Their 3D gap \hat d eq. 12 is not normalised for how far the points are from the cameras, so the paper instead reprojects each closest point into the other image and averages the two pixel distances eq. 13.

7. Chirality §5.2, eq. 14. A match is kept only if both closest points lie in front of their cameras, expressed as positive z-depth in the camera frame eq. 14.

8. Photometric loss §5.3, eq. 15. \mathcal{L} = \sum_{p \in I} \lVert C(p) - \hat C(r(p)) \rVert_2^2, summed over pixels of an image eq. 15. Gradients reach the camera residuals through the rays.

9. Curriculum and joint loop §6.1, §6.2, alg. 1. Geometry is learned first with the initial camera. Then camera terms are added in order: linear intrinsics and extrinsics, radial distortion, then per-ray offsets §6.1. The PRD loss is added on top of the photometric loss every n iterations once the PRD step n_{prd} is reached. The target view is chosen at random among views within 30° of the source, and only pairs with enough correspondences are used §6.2. Algorithm 1 gives the loop:

# Joint optimization with curriculum learning  [alg. 1]
# theta: NeRF weights; z: camera residuals (K, extrinsics, ray offsets)
def train(images, pose_init, K0, n, n_prd, lam):
    # Initialise NeRF weights and camera residuals  [alg. 1]
    theta = init_nerf()
    z = init_camera_residuals(pose_init, K0)
    for iteration in count(1):
        # Curriculum: choose which parameter groups learn now  [§6.1]
        active = get_params(iteration, z, theta)
        # Rays from the current camera model  [§4.1] [§4.2]
        rays = camera_model(K0, z)
        # Photometric loss from volume rendering  [eq. 1] [eq. 15]
        loss = photometric_loss(rays, images, theta)
        # Every n steps once n_prd is reached, add the PRD loss  [alg. 1]
        if iteration % n == 0 and iteration >= n_prd:
            # Random target view within 30 degrees of the source  [§6.2]
            other = sample_view_within(30, images)
            matches = correspondences(images, other)
            # lam is the PRD weight; not stated in the paper  [alg. 1]
            loss = loss + lam * prd_loss(matches, z)
        # Gradient step on active parameters  [alg. 1]
        # Paper writes s + grad L; a minimiser steps the other way
        for param in active:
            param = adam_update(param, grad(loss, param))

# Projected ray distance for one match  [§5.1] [eq. 11-13]
# A camera maps 3D points to pixels; a ray has origin and direction
def projected_ray_distance(ray_a, ray_b, pixel_a, pixel_b, cam_a, cam_b):
    # Closest point on each ray to the other ray  [eq. 11]
    point_a, point_b = closest_points(ray_a, ray_b)
    # Chirality: both points must lie in front of their cameras  [eq. 14]
    if depth(cam_a, point_b) <= 0 or depth(cam_b, point_a) <= 0:
        return None
    # Reproject each closest point into the other image  [eq. 13]
    reproj_b = project(cam_a, point_b)
    reproj_a = project(cam_b, point_a)
    # Average the two pixel distances  [eq. 13]
    dist_a = norm(reproj_b - pixel_a)
    dist_b = norm(reproj_a - pixel_b)
    # Code squares these and thresholds at eta = 5.0; see Paper vs code
    return 0.5 * (dist_a + dist_b)

Key equations, as written in the paper:

d_\pi = \frac{\lVert \pi_A(\hat{\mathbf{x}}_B) - \mathbf{p}_A \rVert + \lVert \pi_B(\hat{\mathbf{x}}_A) - \mathbf{p}_B \rVert}{2} eq. 13

\mathcal{L} = \sum_{\mathbf{p} \in \mathcal{I}} \lVert C(\mathbf{p}) - \hat{C}(\mathbf{r}(\mathbf{p})) \rVert_2^2 eq. 15

Hyperparameters

Results

Author-reported. Hardware not stated. Metrics are PSNR / SSIM / LPIPS, plus PRD (lower is better) where given. The main text says Tables 1 and 3 report training-set metrics §7.2 §7.4; the split for Table 2 is not stated.

Datasets: LLFF (8 scenes; Tables 1 and 2 use Flower, Fortress, Leaves and Trex, and Table 5 also uses Room) and Tanks and Temples (4 scenes), both with COLMAP-estimated cameras §7.1. The fish-eye scenes (Globe, Cube) were captured by the authors with a fish-eye camera §7.1 §7.5.

Table 1, no calibrated camera information, LLFF tab. 1

Table 2, COLMAP initialisation, LLFF tab. 2

Table 3, NeRF++ vs ours, Tanks and Temples, training set tab. 3

Table 4, fish-eye scenes, NeRF++ with radial distortion tab. 4 (no PRD column)

Table 5, ablation on Fortress and Room tab. 5. IE = learnable intrinsics and extrinsics; OD = learnable non-linear ray offsets; PRD = projected ray distance loss. Each phase trained for 200K iterations §7.6.

Reading across tables (my comparison, not the paper's claim): Table 1 PSNR for all four LLFF scenes is above the COLMAP-initialised NeRF rows of Table 2. The paper says "for some scenes" §7.2. In Table 5 the PRD term adds only small PSNR gains over IE and OD (+0.2 and +0.3 dB). The paper also says PRD can raise the overall PRD on some scenes §7.6.

Paper vs code

Code: official repository POSTECH-CVLab/SCNeRF at commit 94001e5 (the cloned HEAD). Licence: MIT, root LICENSE (copyright Yoonwoo Jeong); LICENSE-NeRF (MIT, bmild) covers NeRF-derived parts. The code has NeRF (NeRF/) and NeRF++ (nerfplusplus/) variants, and the NeRF++ variant is what carries the radial distortion.

• PRD distance. Paper eq. 13 averages the pixel norms eq. 13. The code sums the squared pixel errors and averages the two directions with 0.5 ray_dist_loss.py. The threshold η = 5.0 is then compared with squared pixels, about 2.2 px.
• Chirality. Paper eq. 14 tests positive z-depth of the reprojected closest points eq. 14. The code instead keeps a match only when the closest-point ray parameters t0 and t1 are positive ray_dist_loss.py. These tests are related but not identical.
• Closest-point algebra. The code computes t0 and t1 with dot products over the unit directions ray_dist_loss.py. I checked that its t0 matches the standard closest-point solution for two lines. I did not check it symbol for symbol against paper eq. 11.
• Radial distortion. Paper eq. 7 scales the raw pixel coordinate, p_x d_x eq. 7. The NeRF++ sampler scales the offset from the principal point, c + (p - c) \cdot d, with r = (p - c)/c nerf_sample_ray_split.py. In the NeRF path, the distortion class defines distortion_noise camera_model.py, but ray generation in NeRF/get_rays.py never applies it get_rays.py.
• Radial enabled per config. The T&T config does not enable radial distortion (no add_radial, default -1) config_argparser.py. The fish-eye config sets it globe_ours.txt.
• Photometric loss. Paper eq. 15 is a sum over pixels eq. 15. The code uses the mean over rays and channels run_nerf_helpers.py.
• PRD weight. The paper gives no λ. The NeRF argument default is 1.0 config_argparse.py, and the NeRF++ default and the flower script use 1e-4 config_argparser.py main1_flower_ours.sh.
• NeRF++ learning-rate decay. Paper: 7.5M steps app. Implementation Details. Code default: 750K steps config_argparser.py ddp_train_nerf.py. The T&T config does not override it.
• Ray batch. Paper: 512 rays for NeRF++ and 1024 for fish-eye app. Implementation Details. Configs: 256 for both NeRF++ T&T and fish-eye tat_intermediate_M60_ours.txt globe_ours.txt.
• NeRF samples. Paper: 64 coarse and 128 fine app. Implementation Details. Flower config: N_importance = 64 flower.txt.
• Focal initialisation. Paper §6.1 says half the width and height; §7.2 says the width and height. The code uses fx=W, fy=H create_nerf.py, matching §7.2.
• Update sign. Algorithm 1 writes s \leftarrow s + \nabla_s \mathcal{L} alg. 1. The code minimises the loss with Adam create_nerf.py.
• Skipped step. In NeRF/run_nerf.py, the backward pass and optimizer step on PRD-scheduled iterations sit inside if img_i in image_pairs.keys() run_nerf.py run_nerf.py. By my reading of the indentation, a PRD-scheduled iteration for an image with no pair does no update. I did not run this.
• Fish-eye PRD. The paper drops the PRD loss on fish-eye scenes §7.5. The Globe config sets IE, radial, OD and N_iters but no add_prd globe_ours.txt; the NeRF++ parser default is -1 config_argparser.py. I did not check how the Cube config or the training code treats that default.
• Matcher. The paper does not name a matcher. The code defaults to SuperGlue and offers SIFT config_argparse.py. The PRD loss depends directly on this choice.
• NeRF sign convention. The PRD code negates the x focal term for the NeRF coordinate system ray_dist_loss.py. The paper does not mention this.
• Evaluation filter. Validation and test PRD keeps only matches whose ground-truth-projected squared error is below 1.0, about 1 px prd_evaluation.py, applied only when mode != "train" prd_evaluation.py. Train-set PRD does not use it. The paper does not describe the filter.

The 6D rotation construction matches paper eq. 3 camera_utils.py. I did not review the volume renderer itself, so no check of eq. 1 against code is claimed.

Pitfalls

• Eq. 9 is misprinted. The bilinear sum is written as z_d[x,y]\,z_d[x,y], with the weights in a separate, unnumbered row. Read it as bilinear weights times the offset at each grid corner. The code instead upsamples the offset grid bilinearly to the full image camera_model.py. The paper gives no grid size.
• Eq. 7 and the principal point. Eq. 7 scales the raw pixel coordinate, while the NeRF++ code scales the offset from the principal point. These give different pixels, so an implementation that follows eq. 7 literally will not match the code.
• Stages. The paper says stages are added sequentially: linear camera, then radial, then non-linear ray offsets §6.1. The T&T config adds IE, then OD, then PRD, with no radial stage tat_intermediate_M60_ours.txt. The order of stages therefore depends on the config.
• Initialisation. Without COLMAP, the code uses identity rotation and zero translation load_llff.py. Focal lengths start at W and H, and the principal point at the image centre run_nerf.py. The paper's §6.1 focal wording differs.
• Threshold scale. η = 5.0 is applied to squared pixel error in the code, which is about 2.2 px. An implementation using the paper's norm form would need a different η.
• Costs. The flower script runs 800K iterations with 1024 rays per step, and PRD matching runs every 10 steps once it starts at 400K main1_flower_ours.sh. NeRF++ runs 1.5M iterations tat_intermediate_M60_ours.txt. Hardware is not stated.
• Undefined metric. Table 1 reports PRD as "nan" when no inlier matches are found with the wrong camera information tab. 1. Treat these cells as missing, not as zero.
• Appendix numbering. The supplementary sections have no letters in the LaTeX source, so app. ... cites them by title.

Lineage

• Builds on: NeRF arXiv:2003.08934 for the volumetric renderer §3; NeRF++ arXiv:2010.07492 for the unbounded-scene variant used on Tanks and Temples §3 §7.4. The generic camera model follows Grossberg et al. and Schöps et al. §4.2. The 6D rotation representation follows Zhou et al. §4.1. Initial COLMAP cameras come from Schönberger et al. §7.1.
• Compared against: NeRF with COLMAP cameras tab. 2; NeRF++ on Tanks and Temples tab. 3; NeRF++ with radial distortion on fish-eye scenes tab. 4.
• Superseded by: not known from content in this wiki.