FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Views

In plain words

Given a handful of photos of a scene taken from unknown positions, FLARE works out where each camera was, what the 3D shape looks like, and what the scene looks like from new viewpoints. It does this in one pass through a neural network, with no per-scene optimisation. The network first guesses the camera positions, because knowing roughly where each photo was taken makes the rest easier. It then builds a 3D point map for each photo, first in that photo's own frame and then in one shared frame. Finally it places small coloured blobs (3D Gaussians) on those points, so new views can be drawn quickly. The authors aim at the case of few photos, where classic structure-from-motion tends to fail.

TL;DR

• FLARE is a single transformer-based network. It predicts camera poses, then camera-centric point maps, then a learned global point map, then 3D Gaussians that can be rendered from new views §3.
• Author-reported: on RealEstate10K with 5 input images, AUC@30° is 79.9, against 51.6 for PoseDiffusion and 61.1 for MASt3R tab. 1.
• Author-reported: on DL3DV, 8 input views give 23.33 dB PSNR, against 22.08 dB for MVSplat with its two closest views tab. 3.
• Training uses 8 views per sample on eight public datasets, for 200 epochs on 64 GPUs over about 14 days §4. Several loss weights and the batch size are not stated.
• The official code repository is CC BY-NC-SA 4.0. This card does not use it, so every statement below comes from the paper.

Problem

• Classic structure-from-motion and multi-view stereo rely on handcrafted feature matching, and they are not differentiable §1.
• Traditional structure-from-motion struggles with sparse views and limited viewpoints §1.
• Optimisation-based joint pose and geometry methods (BARF, NeRF--) need a good initialisation and generalise poorly to new scenes §1.
• Deep camera regression methods treat sparse-view structure-from-motion as regressing camera parameters, but they are less accurate and generalise less well §1.
• Two-view point-map methods (DUSt3R, MASt3R) need a slow global registration step after prediction §1.
• The paper targets 2 to 8 uncalibrated input images, with inference under 0.5 s [abstract] §1.
• Assumptions: the inputs are uncalibrated images [abstract]. For novel-view synthesis on RealEstate10K, the model is fine-tuned with intrinsics as a condition §4.3, so that setting uses calibration. Whether intrinsics are given in the other evaluations is not stated.

Method

The pipeline runs in four stages, with camera poses as an intermediate representation §3, pipeline figure. Stage 1 is a neural pose predictor. Stage 2 predicts camera-centric point maps and refines the poses in the same network. Stage 3 maps the local point tokens into one global frame. Stage 4 regresses 3D Gaussians whose centres are the global points.

1. Coarse poses. Each image is split into non-overlapping patches. Learnable camera latents are concatenated with the patch tokens and fed to a small decoder-only transformer F_p. Section 3.1 describes one latent per image; the appendix describes one shared latent for the source views (see Pitfalls). Each pose is a 7-vector: translation plus a normalised quaternion §3.1 eq. 1.
2. Camera-centric geometry. A transformer F_l receives the image tokens, camera tokens built from the coarse poses, and learnable pose tokens Q_f. It outputs local point tokens and refined poses §3.2 eq. 2. A DPT-based decoder D_l turns the tokens into local point maps and confidence maps §3.2 eq. 3.
3. Global projection. Instead of geometric reprojection, a learned transformer F_g takes the local point tokens and the refined poses and outputs global tokens §3.2 eq. 4. The paper says F_g has the same architecture as F_l with different inputs. A second DPT decoder D_g gives global point maps and confidence maps §3.2 eq. 5.
4. Pose augmentation (training only). Gaussian noise is added to the predicted poses so that the network learns to cope with noisy poses at inference §3.2.
5. Gaussian head. Gaussian centres are the global point maps §3.3. A DPT head on the output of F_g gives an appearance feature. This feature is fused with VGG features of the input images and passed through a shallow CNN decoder F_a, which regresses opacity, rotation, scale and spherical-harmonic coefficients per Gaussian §3.3.
6. Scale normalisation and rendering. Predicted and ground-truth point maps are each averaged to a scale factor (s from predicted, s_gt from ground truth). Gaussian centres, Gaussian scales and the novel camera position are divided by these factors, and a differentiable Gaussian rasteriser renders the image §3.3 eq. 6.

Forward pass, one pass over N views:

# FLARE forward pass  [§3.1-§3.3]
# Input:  images (N, 3, H, W), uncalibrated; N = 8 in training
# Output: refined poses (N, 7), global point maps, Gaussian parameters

def flare_forward(images):
    # Split each image into non-overlapping patch tokens  [§3.1]
    image_tokens = patchify(images)                # (N, P, d)

    # Coarse poses from learnable camera latents  [§3.1] [eq. 1]
    coarse_poses = pose_predictor(camera_latents, image_tokens)

    # Camera-centric tokens; refined poses from pose tokens  [§3.2] [eq. 2]
    local_tokens, refined_poses = local_transformer(
        image_tokens, coarse_poses, pose_tokens)

    # Local point maps and confidences from a DPT decoder  [§3.2] [eq. 3]
    local_points, local_conf = local_dpt(local_tokens)

    # Learned projection of local tokens into one global frame  [§3.2] [eq. 4]
    global_tokens = global_transformer(local_tokens, refined_poses)

    # Global point maps and confidences from a DPT decoder  [§3.2] [eq. 5]
    global_points, global_conf = global_dpt(global_tokens)

    # Appearance feature; fused with VGG features of the input views  [§3.3]
    appearance = appearance_dpt(global_tokens)
    fused = fuse(appearance, vgg_features(images))

    # Per-Gaussian opacity, rotation, scale and SH from a shallow CNN  [§3.3]
    gaussians = shallow_cnn(fused)

    return refined_poses, global_points, gaussians

Training loss. The total is a weighted sum of three terms §3.4 eq. 12:

• Pose loss eq. 7: Huber loss over the pose parametrisation, summed over the N views, applied to both the coarse poses and the refined poses against ground truth. The paper follows the VGGSfM pose loss §3.4.
• Geometry loss eq. 8–9: a DUSt3R-style confidence-weighted regression, with one term for camera-frame maps and one for global-frame maps. Each term is the Euclidean distance between normalised predicted and ground-truth points, weighted by confidence C and penalised by −α log C §3.4.
• Splatting loss eq. 10–11: L2 between rendered and ground-truth images, plus 0.5 times a VGG perceptual loss, plus 0.1 times a depth term. The depth target comes from a monocular depth estimator (the bibliography key is depthanything). The depth term uses a per-image scale W and shift Q to align the predicted depth §3.4.

Hyperparameters

Results

Author-reported. Setup is given in each row. No independent check was made.

Multi-view pose on RealEstate10K, 5 input images, no fine-tuning tab. 1 §4.1. Metrics: RRA@5°, RTA@5°, AUC@30°.

Sparse-view reconstruction, 8 views tab. 2 §4.2. Accuracy (ACC) and completeness (COMP) are the point-cloud metrics of §4.2 (lower is better). Overall is their mean in the table values. AUC@30° is the pose accuracy.

Novel-view synthesis on DL3DV, 8 input views for FLARE tab. 3 §4.3. Baselines use the two input views closest to the target app. A.

Novel-view synthesis on RealEstate10K, 2 views tab. 4 §4.3. FLARE is fine-tuned on this set; intrinsics are a condition.

Ablation on BlendedMVS held-out scenes, geometry only, no rendering loss tab. 5 §4.4. Lower is better for all three columns.

Supplementary: number of views on DTU tab. 6 app. B. The paper says gains plateau as views increase.

Supplementary: dense-view DTU tab. 7 app. B. Accuracy and completion are reported with lower-is-better values.

The appendix plots MVSplat's PSNR, SSIM and LPIPS against the number of input views app. B, figure "Relationship between MVSplat Performance and Input Views". The values in the figure source fall with more views: PSNR is 22.08 dB at 2 views and 17.39 dB at 8. The appendix says MVSplat is therefore reported at 2 views app. B.

Paper vs code

No comparison made. The official repository, linked from the project page named in the abstract, is licensed CC BY-NC-SA 4.0. Its code is non-commercial, so this card does not use it and cites no code lines. The code, code_commit and code_license fields record what was found. The paper's stated values above are the only source for this card.

Pitfalls

• Pose metrics. The text defines the accuracy at threshold τ as the minimum of RRA@τ and RTA@τ, and AUC as the area under that curve §4.1. Table 1 gives RRA and RTA only at 5°, so the AUC@30° cannot be recomputed from the table.
• Scale at test time. Predicted and ground-truth scale factors (s and s_gt) normalise the Gaussians and the novel camera position §3.3 eq. 6. The paper does not say how s_gt is obtained when no ground truth exists, so novel-view rendering on unseen scenes needs a different choice.
• Camera latents. Section 3.1 says each image has its own learnable camera latent §3.1. The appendix says there are two: one for the reference view and one shared by all source views, duplicated N−1 times app. A. The paper does not say which one the reported results use.
• Pose augmentation. The noise distribution and magnitude are not stated §3.2. It is applied to predicted poses, not to ground truth.
• Loss constants. λ_pose, λ_geo, λ_splat, the confidence penalty α and the Huber parameter are not stated §3.4 eq. 7–12. The values cannot be reproduced from the paper alone.
• Encoder. The paper says the model is trained from scratch except for the encoder §4 but does not name the encoder, and it does not say whether the encoder is frozen.
• VGG features. The appearance branch uses a pretrained VGG network §3.3. Which VGG variant and whether it is frozen are not stated.
• Depth alignment. The depth loss uses a scale W and shift Q §3.4. How they are computed (least squares, per image or per batch) is not stated.
• Batch size and schedule. Batch size is not stated. The learning rate decays from 1e-4 to 1e-5 without a schedule shape §4.
• Compute. About 14 days on 64 GPUs §4 is roughly 896 GPU-days, a derived figure. The paper gives no per-scene memory use. It says the number of views is limited only by GPU memory §3.2.
• Claimed input range. The abstract says 2 to 8 inputs. Section 4 says the model works up to 25 views, but the 25-view evidence is a single DTU table tab. 6.
• Calibration. The novel-view model on RealEstate10K takes intrinsics as a condition §4.3. The authors say it "doesn't require camera extrinsic information" §4.3, which is narrower than uncalibrated input. Whether the other evaluations use intrinsics is not stated.
• Unequal view counts. FLARE uses 8 views on DL3DV while the pose-required baselines use 2 closest views §4.3 app. A. The comparison is not matched on input count. For LPIPS on DL3DV, the 2-view FLARE result (0.182) is better than its own 8-view result (0.237) tab. 3.
• Reconstruction claims. The text says FLARE "achieves higher performance" than DUSt3R, MASt3R and Spann3R §4.2, but on ETH3D overall error MASt3R is lower (0.3522 vs 0.4978) tab. 2.
• Missing baseline. The text says COLMAP is compared in the reconstruction section §4.2, but its row is commented out in the table source, so no COLMAP number is reported there.
• Table 7 header. The "Overall" column is marked with an up arrow, but the bold best entry is the lowest value. The lower-is-better reading is the consistent one.
• Ablation scope. The ablation is geometry only, on one dataset (BlendedMVS held-out), with rendering loss excluded §4.4. It does not test appearance. The variant with rendering loss has a slightly lower overall error (0.0282 vs 0.0288) tab. 5.
• Training data quality. DL3DV depth is produced by COLMAP multi-view stereo, not measured app. A. Its noise propagates into the geometry targets.
• Timing. "Under 0.5 s" appears in the abstract and introduction. The timing hardware and whether it includes Gaussian rendering are not stated.

Lineage

• Builds on: DUSt3R and MASt3R (point maps, confidence-weighted regression) §1, §3.2; PoseDiffusion and VGGSfM (camera regression, pose loss) §3.1, §3.4; DPT decoders §3.2. PF-LRM is cited for the benefit of joint pose and geometry supervision §3.2.
• Compared against: COLMAP, PixSfM, RelPose, PoseDiffusion (poses); DUSt3R, MASt3R, Spann3R (reconstruction); CoPoNeRF, Splatt3R, MVSplat, pixelSplat, pixelNeRF, AttnRend (novel-view synthesis) §4.
• Superseded by: not stated.