iLRM: An Iterative Large 3D Reconstruction Model

In plain words

Some systems can look at a few photos of a scene and build a 3D model in one pass, with no slow per-scene fitting. Many of them make one 3D blob for every pixel of every photo, so the model grows with the image resolution and the number of photos, and it spends a lot of computation comparing all photos with all others.

iLRM works differently. It starts from a small set of "viewpoint" slots, one per camera, and keeps refining them over several rounds. In each round, each slot looks at the detailed pixels of its own photo, and then all slots talk to each other to share what they learned. This is loosely like how classic optimisation fits a 3D model: look at the error, correct, and repeat. Because the slots are fewer than the pixels, the 3D model stays compact and the computation grows more slowly with the number of photos.

TL;DR

iLRM is a transformer that outputs 3D Gaussians from posed images. Its scene tokens are updated layer by layer by cross-attention to full-resolution image tokens, with self-attention across views in between §3.2. The code released with the paper implements only the 2-view, full-resolution setting. Author-reported results are on RealEstate10K and DL3DV, and the paper claims a 0.5 s inference time for 32 views at 540x960 §1.

Problem

Pixel-aligned feed-forward Gaussian models create one Gaussian per pixel, so the Gaussian count grows with resolution and views. The paper's example is 200 views at 1K resolution, which gives about 200 million Gaussians §1. Full attention across all image tokens of all views makes the cost quadratic in the number of views and the image resolution §1 §3.1. Reducing the resolution or the view count loses detail §1. The paper also argues that one-shot generation lacks the feedback loop that optimisation-based reconstruction uses §1.

Method

The model maps N posed images to a set of 3D Gaussians, one set per viewpoint slot eq. 1 §3.1. The steps are:

1. Viewpoint tokens. For each view, a Plücker ray embedding is computed per pixel from the intrinsics and pose. Rays are split into p x p patches, flattened and linearly projected to tokens. No positional embedding is added, because the Plücker coordinates already differ between patches §3.2.
2. Image tokens. RGB patches and Plücker patches are concatenated, then linearly projected to tokens §3.2 eq. 2.
3. Update layers. The model has 12 update layers. Each layer applies a cross-attention block, in which each viewpoint token attends only to the image tokens of its own view, and then a self-attention block over all viewpoint tokens of all views §3.2 eq. 3 eq. 4. Each layer has its own weights §3.2.
4. Token uplifting (described for the mismatched-resolution case). If the viewpoint tokens are at lower resolution than the image tokens, each viewpoint token is expanded by a factor k = 2 into finer query tokens before cross-attention. After attention, the result is projected back to the original width §3.2.
5. Decoding. After the last layer, a single linear layer maps each viewpoint token to Gaussian parameters, followed by post-activations §3.2 app. A.
6. Mini-batch cross-attention (optional, not used in the 2-view setting). Only a structured subset of image tokens and viewpoint tokens is attended in each cross-attention. The paper's main variant is "quarter" sampling §3.2 fig. 4.

Interpretation. The paper frames each update layer as a feedback correction step: viewpoint tokens are changed by an amount that depends on the fixed image tokens, analogous to a gradient step §3.2. This framing is motivational; the paper does not derive it from a specific objective.

Training objective. Gaussians are rasterised into held-out target views and supervised with MSE plus a perceptual term, weighted by lambda = 0.5 eq. 5 §3.3.

# iLRM forward pass  [§3.2] [eq. 1]-[eq. 4]  [app. A]
# Input:  images (B, V, 3, H, W); poses (B, V, 4, 4); intrinsics (B, V, 3, 3)
# Output: per-view Gaussian parameters, H*W Gaussians per view

def ilrm_forward(images, poses, intrinsics):
    # Plücker ray for every pixel of every view  [§3.2]
    rays = plucker_rays(poses, intrinsics)          # (B, V, H*W, 6)
    # Viewpoint tokens from rays only, no learned queries  [§3.2]
    viewpoint_tokens = layer_norm(
        linear_view(patchify(rays, p)))             # (B, V*HW/p^2, d)
    # Image tokens from RGB and rays, full resolution  [§3.2] [eq. 2]
    image_tokens = layer_norm(
        linear_image(patchify(concat(images, rays), p)))  # (B, V*HW/p^2, d)
    # Twelve update layers, each a cross-attention then a self-attention  [§3.2]
    for layer in range(12):
        # Per-view cross-attention, each view attends to its own image  [eq. 3]
        viewpoint_tokens = viewpoint_tokens + cross_attn[layer](
            viewpoint_tokens, image_tokens)
        viewpoint_tokens = viewpoint_tokens + mlp[layer](viewpoint_tokens)
        # Global self-attention over all viewpoint tokens of all views  [eq. 4]
        viewpoint_tokens = viewpoint_tokens + self_attn[layer](
            viewpoint_tokens)
        viewpoint_tokens = viewpoint_tokens + mlp[layer](viewpoint_tokens)
    # Decode each token to 16 values in the paper  [app. A]
    raw = linear_out(layer_norm(viewpoint_tokens))
    return activate(raw)

# Gaussian parameters after decoding  [app. A] [§4]
# Input:  raw values per Gaussian; ray origin o and direction r per pixel
# Output: means, opacity, covariance, colour

def activate(raw, origin, direction):
    # Depth is the mean of the three depth channels  [app. A]
    # The released code also squashes it: sigmoid * 500  [see Paper vs code]
    depth = mean(raw_depth_channels)
    # Position is origin plus depth along the ray  [app. A]
    means = origin + depth * direction
    # Opacity, scale and rotation use GS-LRM-style activations  [app. A]
    opacity = sigmoid(raw_opacity)
    scale = exp(raw_scale)
    rotation = normalize(raw_rotation)
    # Colour: degree-0 spherical-harmonic term only  [app. A]
    colour = raw_colour
    return means, opacity, scale, rotation, colour

# Training loss  [§3.3] [eq. 5]
# Input:  rendered target views and ground truth for each target index t
# Output: scalar loss

def training_loss(rendered, targets, lam=0.5):
    # MSE and perceptual loss, summed over target views  [eq. 5]
    loss = 0.0
    for t in target_indices:
        loss = loss + mse(rendered[t], targets[t])
        loss = loss + lam * perceptual(rendered[t], targets[t])
    return loss

Hyperparameters

Results

Author-reported. Setup: RE10K at 256x256, unless noted. Timing is on an RTX 4090 for the RE10K row tab. 1. The top block is the paper's (2, F, F) setting, which is the released checkpoint README.md.

Ablations (RE10K, (4, H, F), batch 16 on RTX 4090, author-reported):

Model size (layers 12, 9, 6, 3) gives PSNR 29.24, 29.01, 28.68, 28.04 tab. 6. Training cost at (8, H, F), batch 16: iteration time measured on RTX 4090, memory on H100 without gradient checkpointing. Baseline 1.51 s and 62.5 GB; quarter cross-attention 0.94 s and 39.0 GB tab. 8.

Paper vs code

Code: the repository Gynjn/iLRM at commit 9699c3521ade20dd0a7cf99783f620801423aa3a, linked from the paper's project page and owned by the first author. There is no LICENSE file, so code is cited by path and line only, with no quoted code.

• Only the 2-view, full-resolution setting is implemented. Viewpoint tokens are built on the same grid as image tokens, so the code has no half or quarter viewpoint resolution scaler.py scale.yaml. The paper's (2, F, F) row is the one that does not use token uplifting §4. The README checkpoint (re10k_2view) reports the same numbers as that row README.md.
• No token uplifting, no mini-batch cross-attention. Searches of the source for "uplift", "quarter", "mini" and "group" found nothing outside visualization code. The cross-attention is a plain per-view attention with the paper's QK norm transformer.py. The paper's Figure 4 variants are therefore not reproducible from this code.
• Gaussian output is 14 values, not 16. The token decoder outputs 3 + 3 + 3 + 4 + 1 channels per patch pixel: xyz, degree-0 colour, scale, rotation, opacity scaler.py scaler.py. The paper's 2-channel xy offset is absent. Position comes from a ray through the pixel centre scaler.py scaler.py.
• Depth scaling is not in the paper. Depth is sigmoid of the mean of the three channels, times 500 scaler.py.
• Activations are offset in code. Opacity is shifted by -2.0 before sigmoid, scale is shifted by -6.9 and clamped at -1.2 before exp, and rotation is normalised in the decoder encoder_scale.py cuda_splatting.py. The paper says these follow GS-LRM, without the constants.
• Loss is MSE only in the shipped RE10K experiment. The experiment config overrides the loss to MSE re10k.yaml. LPIPS exists with weight 0.05 applied after step 150,000 lpips.yaml. The paper specifies lambda = 0.5 from the start §3.3 eq. 5.
• Learning rate and schedule differ. Code default peak is 4e-4 main.yaml, not 2e-4. With cosine_lr set, the code uses a one-cycle schedule with warm-up of 1% of steps model_wrapper.py, so the 2500-step warm-up in app. A is not used.
• Training length and batch differ. The RE10K config trains at batch 32 for 100,001 steps re10k.yaml. The paper gives 200k steps for (2, F, F) on 8 H100s, with the batch size not stated app. A.
• Gradient clipping at 1.0 is in the code main.yaml and not in the paper.
• Weight decay exclusions differ. The code excludes all 1-D parameters (including RMSNorm weights) from decay, not only LayerNorm weights model_wrapper.py.
• Image values are remapped. Support RGB is mapped to [-1, 1] before tokenisation scaler.py. The paper does not state this.
• Background is white in the rasteriser cuda_splatting.py. The paper does not state the background.
• Rendering uses gsplat with a near plane of 0.01 and a far plane of 500 cuda_splatting.py.
• Viewpoint token initialisation. The default path builds viewpoint tokens from Plücker rays only. The appendix's learnable-embedding variant is not in the default path scaler.py app. D.
• Matches the paper: 12 update layers (as 12 pairs), hidden dim 768, 12 heads, patch size 8, AdamW betas 0.9 and 0.95, weight decay 0.05, init std 0.02.

Pitfalls

• Gaussian count. With full-resolution viewpoints, the output has H x W Gaussians per view. For 256x256 and two views this gives 131,072, matching tab. 1. The compactness claim depends on the decoupled settings that the released code does not implement.
• Attention cost. Self-attention in the code runs over all viewpoint tokens of all views together, so cost grows with the number of views times the number of tokens per view scaler.py. With full-resolution viewpoints this is the same token count as image-based attention, so the efficiency argument in §3.2 applies only to the decoupled settings.
• Pose sensitivity. The model needs accurate poses. The paper reports degraded PSNR under translation noise: 24.30 to 21.49 at sigma = 0.005 for iLRM against 23.97 to 20.49 for Long-LRM app. F.
• Unstated values. Background colour, image value remapping, the ray normalisation and the depth scale are not in the paper. An implementer needs the code for them, cited above.
• Batch and step counts for (2, F, F) are not stated in the paper. The code default is batch 32 for 100k steps.
• Ablation numbers differ from the main table. The 4-view ablation baseline is 29.24 PSNR, against 30.37 for (4, H, F) in tab. 1. The ablations use a different training setup (batch 16, RTX 4090).
• Quarter cross-attention FLOPs are estimated from a formula, not measured. The formula is given in app. E. Measured iteration times are in tab. 8.
• Gradient checkpointing is on by default in the processor scaler.py. The paper's training memory numbers were taken without it tab. 8.

Lineage

• Builds on: GS-LRM for the Plücker tokenisation, the activation design and the training setup §3.2 app. A; LVSM and LGM for training setup and Plücker embedding §3.2 app. A.
• Compared against: pixelSplat, MVSplat, DepthSplat, Gen-Den, Long-LRM and optimisation-based 3D-GS §4.1 tab. 5.
• Superseded by: not stated in this card.