Continuous 3D Perception Model with Persistent State

In plain words

Most systems that rebuild a 3D scene from photos need all the photos up front, or they fit a 3D model to each scene over many minutes. CUT3R works the other way round. It reads photos one after another, the way a person walks into a room and gradually builds a mental map of it.

The model keeps a small, fixed-size memory of what it has seen so far. Each new photo changes that memory and is also explained using it. From one photo the model outputs a 3D point for every pixel and an estimate of where the camera was. Stacking those outputs gives a growing 3D reconstruction. The model can also be asked about a camera position it has never photographed, and it answers with a guess of what is there, based on what it learned from real scenes.

The paper reports that this single approach covers several 3D tasks, works on both video and loose photo collections, and copes with moving people and objects. It is trained on a large mix of datasets, and the authors note that it can drift over very long sequences.

TL;DR

CUT3R is a recurrent transformer. It keeps 768 learned state tokens, updates them with each image, and reads context back from them. Each frame yields a world-frame pointmap, a camera pose and confidence maps, all in metric scale §3.1. A virtual camera described by a raymap can probe the state to predict an unseen view §3.2. The paper reports competitive or best results among online methods on video depth, camera pose and 3D reconstruction; all numbers are author-reported §4. The official code is non-commercial (CC BY-NC-SA 4.0).

Problem

Offline reconstruction methods that align many pairwise predictions with global alignment assume a static scene and cannot run as frames arrive §4.1. Online methods such as Spann3R handle streams, but the paper states they assume a static scene §4.1. Several baselines predict relative (not metric) pointmaps §4 Baselines, and most calibration-free pose methods work through test-time optimisation per sequence §4.2. The paper targets one model that is online, works without camera calibration or poses, handles dynamic content, and can infer unobserved parts of a scene §1.

Method

Inputs and state. The model takes a stream of images with no camera information §3. Each image is encoded into patch tokens by a ViT encoder §3.1, eq. 1. The state is a set of tokens that starts as learnable tokens shared by all scenes, so nothing is known before the first image §3.1.

State-input interaction (state-update and state-readout). Two interconnected transformer decoders run on the image tokens and the state tokens together §3.1, eq. 2. A learnable pose token z is placed in front of the image tokens. After each decoder block the two sides cross-attend, so the state takes in the new image (update) and the image takes in the stored context (readout). The output is the enriched image tokens, the pose-token output, and the updated state.

Heads. Three heads read the outputs §3.1, eqs. 3–5:

• a pointmap in the image's own frame, with confidence, from the image tokens (head_self, a DPT or linear head);
• a pointmap in the world frame, with confidence, from the image tokens and the pose token (head_cross, same family, with pose modulation inside its layer norms app. B);
• a 6-DoF pose, the rigid transform from the current frame to the world frame, from the pose token (head_pose, an MLP) §3.1. The world frame is defined as the frame of the first image, and all outputs are in meters §3.1. The paper argues that predicting the same quantity twice simplifies training and lets partially labelled datasets supervise one head or the other §3.1.

Querying unseen views. A virtual camera is encoded as a raymap, a six-channel image with a ray origin and direction per pixel §3.2. A separate small transformer encodes it. The raymap then goes through the same decoder with shared weights, but the state is not written to, so the query only reads §3.2. The same heads predict a pointmap and a colour head predicts the RGB of each ray §3.2.

Training. Pointmaps use a confidence-weighted L2 regression loss that follows MASt3R, with a penalty on log confidence §3.3, eq. 7. Pose is penalised by L2 on quaternion and translation §3.3, eq. 8. When the query is a raymap, an MSE colour loss is added §3.3. Metric data use the true scale; data with unknown scale do not use raymap queries §3.3.

Curriculum. Four stages §3.4 app. B:

1. 4-view sequences, mostly static datasets, 224x224 images, linear heads.
2. Adds dynamic datasets and partially annotated data, same resolution and heads.
3. Higher resolution with varied aspect ratios, longest side 512 px, DPT heads.
4. Encoder frozen; only decoder and heads are trained on longer sequences of 4 to 64 views.

The overview diagram is fig. 3; the teaser shows online reconstruction fig. 1 and the raymap query idea fig. 2.

Data. The text says 32 datasets, synthetic and real, static and dynamic, indoor and outdoor, object-centric and scene-level §3.4 app. A; the table lists 33 tab. 6. Some datasets carry only camera parameters or single views; single views are stacked with state resets to the initial state app. A.

Pseudocode (own wording, following §3 and the official implementation's structure; differences are listed in Paper vs code):

# Online loop over a stream of images  [§3.1] [eq. 1-5]
# Input:  frames, a list of (3, H, W) images without camera information
# Output: per frame, a world-frame pointmap, a pointmap in the image frame,
#         their confidences and a camera pose

def run_stream(frames):
    # Learnable state tokens shared by all scenes  [§3.1]
    state = learned_state_tokens                     # (768, 768)
    for i, image in enumerate(frames):
        # Encode the image into patch tokens  [§3.1] [eq. 1]
        tokens = image_encoder(image)                # (P, 1024)
        # Pose token: learned for the first frame  [§3.1]
        pose = learned_pose_token                    # (1, 768)
        # State and image tokens interact in both directions  [§3.1] [eq. 2]
        state, image_out, pose_out = decoders(
            state, tokens, pose
        )
        # Heads read the decoder features  [§3.1] [eq. 3-5]
        pts_self, conf_self = head_self(image_out)
        pts_world, conf_world = head_cross(image_out, pose_out)
        camera_pose = head_pose(pose_out)
        yield pts_self, conf_self, pts_world, conf_world, camera_pose

# Probing the state with an unseen view  [§3.2] [eq. 6]
# Input:  state from a run, raymap (6, H, W) of the virtual camera
# Output: pointmap and RGB for the unseen view; state is NOT changed

def query_state(state, raymap):
    # Encode the raymap with its own small transformer  [§3.2] [eq. 6]
    ray_tokens = raymap_encoder(raymap)          # (P, 1024)
    # Read the state through the shared decoders, no state write  [§3.2]
    _, query_out, pose_out = decoders(state, ray_tokens, None)
    pts, conf = head_cross(query_out, pose_out)
    rgb = head_color(query_out)                  # RGB per ray  [§3.2]
    return pts, conf, rgb

Hyperparameters

Values from the paper unless marked "code". The paper's supplementary material defers to the code's configs in several places app. B.

Results

Author-reported. Each table gives the setup from its caption and text. Bold and underline in the paper mark best and second-best per column; the values below are as printed. Baselines are the paper's own choices §4 Baselines.

Monocular depth, zero-shot, per-frame median scaling (Abs Rel lower is better; δ<1.25 higher is better) tab. 1 §4.1:

Video depth, per-sequence scale alignment (Abs Rel, δ<1.25, FPS on KITTI at 512x144 on A100) tab. 2 §4.1:

Video depth, metric scale (no alignment) tab. 2 §4.1: MASt3R-GA gives Sintel Abs Rel 1.022 (δ 14.3), Bonn 0.272 (70.6), KITTI 0.467 (15.2). This paper gives 1.029 (23.8), 0.103 (88.5), 0.122 (85.5).

Camera pose, Sim(3)-aligned ATE and RPE (lower is better) tab. 3 §4.2:

Among online methods the paper reports the best overall result, with Spann3R better on ScanNet ATE tab. 3 §4.2.

Sparse-view 3D reconstruction, 7-Scenes and NRGBD, means only (Acc and Comp in metres, lower is better; NC higher is better) tab. 4 §4.3:

Medians are in tab. 4. The paper also reports 25 times the speed of DUSt3R-GA §4.3.

Qualitative comparisons with Spann3R and MonST3R on Internet videos are in fig. 4; probing unseen regions is shown in fig. 6 (MapFree and ARKitScenes validation sets, not seen in training §4.4).

State update (revisiting), same setup as above tab. 5 fig. 5 §4.4: running the state over the whole set once and then re-running it on the same frames improves the means from 7-Scenes Acc 0.126 to 0.113 and NRGBD Acc 0.099 to 0.094. "Revisit" is not an online result, because it needs a second pass over the frames.

Dataset coverage for training tab. 6 app. A: the table has 33 rows, with flags for metric scale, real or synthetic, static or dynamic, camera-only and single-view. The text says 32 datasets §3.4.

Paper vs code

Repository: the official code (https://github.com/CUT3R/CUT3R) is linked from the authors' project page (https://cut3r.github.io/); it is the only repository we found and it is not a third-party copy. Its LICENSE is CC BY-NC-SA 4.0 (non-commercial), so this card cites the code but quotes none of it. Checkpoints are linked from the repository README on Google Drive README.md; their license was not checked. The paper text was checked against the configs in config/, src/train.py, src/dust3r/model.py, src/dust3r/losses.py and the dataset loader at code_commit.

1. Pose token. The paper describes one learnable pose token prepended to every image §3.1, eq. 2. In the code it is used for the first frame only; later frames read their pose token from a learned memory through a read block, and the memory is written after each frame model.py model.py model.py. The paper does not describe this memory.
2. Which decoder layers feed the heads. The paper does not say which decoder layers feed the heads. The code passes the input features and the outputs of decoder layers 6, 9 and 12 of 12; the pose token is removed from the three layer outputs model.py.
3. Confidence loss normalisation. Paper eq. 7 sums over all pointmaps §3.3. The code averages the per-view confidence losses and multiplies by 2.0 losses.py losses.py.
4. Pose loss. Paper eq. 8 sums the L2 terms over frames t §3.3. The code takes the mean over frames and batch, and adds the pose loss to the total without a weight; a commented-out clip and weight of 5 is present losses.py losses.py. In the pose function I did not see a division of the translation by the scale factor s from eq. 8; I did not trace the call site to confirm.
5. Raymap sampling. The paper says each non-first metric frame is replaced by its raymap with probability 20% app. B. The code uses 80% image only, 15% raymap only and 5% image plus raymap base_multiview_dataset.py. The first frame is never replaced, as the paper says.
6. Frame-interval sampling. The paper says fixed and variable intervals each make up about half of video samples app. B. The code has two samplers, one with a fixed-interval probability of 0.8 base_multiview_dataset.py and one with 0.5 base_multiview_dataset.py. I did not trace which one each dataset uses.
7. Learning rate. The paper gives one initial rate, 1e-4 §3.4. The code uses 1e-4 only in stage 1; stage 2 uses 5e-6, stage 3 uses 1e-5 and stage 4 uses 1e-6 stage2.yaml stage3.yaml dpt_512_vary_4_64.yaml.
8. Stage 4 configuration. The paper describes stage 4 as a frozen encoder with 4 to 64 views §3.4. The config dpt_512_vary_4_64.yaml matches this, and the README names its checkpoint as the final model README.md. The other stage-4 config, stage4.yaml, uses a fixed 32 views stage4.yaml and was not described in the paper.
9. Epochs, batch and effective batch size. The paper gives no epoch counts and no batch size. The configs use 100, 35, 40 and 10 epochs for the four stages stage1.yaml stage2.yaml stage3.yaml stage4.yaml. Stages 1 to 4 set batch size 16 with accumulation 1, but the 16-view and 64-view configs use 6 with accumulation 2 and 4 with accumulation 4 linear_224_fixed_16.yaml dpt_512_vary_4_64.yaml. The paper uses eight GPUs; how the config batch maps onto GPUs was not traced.
10. Undocumented training details. Gradient clipping at 1.0 and AdamW betas of (0.9, 0.95) are in the code but not in the paper train.py train.py.
11. Head activations. The configs use conf_mode ('exp', 1, inf) and pose_mode ('exp', -inf, inf) stage1.yaml. The paper does not give output activations for confidence or pose.
12. Heads by stage. The paper says the linear heads of stages 1 and 2 switch to DPT in stages 3 and 4 app. B. The configs agree: stage 1 uses linear and stage 3 uses dpt stage1.yaml stage3.yaml.
13. Alpha. The paper's eq. 7 has a weight alpha with no numerical value. The configs set alpha to 0.2 stage1.yaml.

Pitfalls

• Non-commercial code. The official code is CC BY-NC-SA 4.0. Do not copy code from it into a commercial system. Implement from the paper and this card, and check the weights' license before use; we did not check it.
• Numbers that are not in the paper. Alpha in eq. 7, epochs, effective batch size, weight decay, gradient clipping and output activations must be taken from the configs or chosen by you; the paper does not state them.
• Memory is an unpublished detail. The learned memory and its read and write blocks model.py are needed to match the code's pose tokens for frames after the first. Following only the paper's single pose token gives a different model.
• Raymap queries need metric data. The raymap branch is enabled only for training data with metric 3D annotation; otherwise the scale of the state and the query can disagree §3.3 app. B.
• State resets. Single-view datasets are fed as sequences of independent views, and the state is reset to the initial state after each view app. A. Forgetting the reset makes single-view batches leak context between views.
• Drift and reset are part of the design. The authors state that long sequences can drift because there is no global alignment, and that the structure generation is deterministic, so extrapolations far from the input views can be blurry §5 Limitations.
• "Revisit" is not online. The better numbers in tab. 5 come from a second pass over the same frames §4.4. Do not compare them with online rows.
• Speed claims depend on setup. The tab. 2 FPS values are for 512x144 images on one A100, except Spann3R, which only supports 224x224 inputs tab. 2 caption, so its row is not like-for-like. The paper's "nearly 50 times" and "25 times" comparisons come from the FPS columns of tab. 2 and tab. 4 §4.1 §4.3. Memory use and training time are not reported; the authors say recurrent training is time-consuming §5.
• Baselines are a mix. Several baselines are optimised per sequence (global alignment), so their numbers are not online. Table 4's MonST3R row carries the citation key of MASt3R; the label is the reliable part.
• Some cells are not wins. Sintel Abs Rel in tab. 1 and tab. 2 favours MASt3R or MonST3R; the metric-scale Sintel row favours MASt3R; ScanNet ATE in tab. 3 favours Spann3R. The paper's wording "competitive or state-of-the-art" §1 does not mean first place in every column.
• Sampling probabilities. Two frame-interval samplers appear in the code with different defaults (0.8 and 0.5) base_multiview_dataset.py. Check which one your dataset uses before comparing sampling statistics.

Lineage

• Builds on: DUSt3R, the encoder weights for stage 1 and the decoder design, which the paper cites as a base and as its baseline §3.4 §4; CroCo, the decoder family the paper cites §3.1; MASt3R, the confidence-aware regression loss §3.3.
• Compared against: DUSt3R, MASt3R, MonST3R and Spann3R, as both offline and online baselines; the offline ones with global alignment §4.
• Superseded by: not known from this wiki.