Camera pose encoding

Overview

A model that must respect a camera needs a per-token signal for that camera. Feeding the raw pose as one vector per clip is too coarse, so the usual route is to attach a camera signal to each token. The design question is what that per-token signal depends on: the world frame, the relation between two cameras, or the viewing ray of each pixel together with its lens model 2512.07237 §1 2512.07237 §2.

Two cards cover this concept, and they use the viewing ray in different roles. The video-generation card builds a local ray frame for each token, so lens distortion and field of view enter the attention through the ray geometry, not through a pinhole matrix. An optional gravity-aligned map adds absolute pitch and roll 2512.07237 §3.2. The 3D-reconstruction card uses per-pixel Plücker rays as the only input of its viewpoint tokens, with no learned query embedding and no positional embedding added 2507.23277 §3.2. The reported results come from different tasks and different settings, so neither card's numbers can be read against the other's 2512.07237 tab. 1 2507.23277 tab. 1.

Variants

Per-token ray frame with a lens model (UCPE)

Each token is one viewing ray with an origin and a unit direction. The local z-axis is the ray direction, the local x-axis comes from the camera's downward axis crossed with z, and y completes a right-handed frame. The token then uses the inverse of the ray-to-world transform as its geometric operator 2512.07237 eq. 5 2512.07237 eq. 6 2512.07237 eq. 7.

Inside attention, this transform acts on half of each head's feature dimensions, and RoPE acts on the other half 2512.07237 eq. 10. The values are transformed too, following the GTA-style scheme 2512.07237 eq. 4. The branch runs in parallel with each self-attention block of the diffusion transformer, not in sequence, and its output projection starts at zero, so the pretrained model is unchanged at step 0 2512.07237 §3.3.

The optional Lat-Up map gives two absolute signals per pixel: the ray's elevation above the horizon, and an "up" direction found by nudging each ray by a small fixed angle and reprojecting it through the lens model 2512.07237 eq. 8 2512.07237 eq. 9 2512.07237 app. B. It enters as a bias on the tokens 2512.07237 §3.3.

Card: Unified Camera Positional Encoding for Controlled Video Generation.

Plücker ray viewpoint tokens (iLRM)

For each view, a Plücker ray embedding is computed for every pixel from the intrinsics and the pose. The rays are cut into p x p patches, flattened and linearly projected to tokens. No positional embedding is added, because the Plücker values already differ between patches 2507.23277 §3.2. These viewpoint tokens are refined over 12 update layers. Each layer runs a cross-attention in which a view's tokens attend only to that view's image tokens, then a self-attention over the tokens of all views 2507.23277 §3.2 2507.23277 eq. 3 2507.23277 eq. 4.

The model needs accurate poses. The paper reports PSNR falling from 24.30 to 21.49 under translation noise with sigma = 0.005, against 23.97 to 20.49 for Long-LRM 2507.23277 app. F.

The released code implements only the two-view, full-resolution setting. Its viewpoint tokens sit on the same grid as the image tokens, so the output is one Gaussian per pixel per view. The paper's decoupled settings, which give a more compact output, are not in the code [see the card's Paper vs code section].

Card: iLRM: An Iterative Large 3D Reconstruction Model.

Pose-only relative encodings and other baselines

The UCPE card describes these as the alternatives the ray method is set against. Their details are the UCPE card's description only 2512.07237 §3.1. None of them has a card in this wiki:

• Plücker coordinates: a per-pixel 6-vector that stacks the ray direction and its moment (origin cross direction). It is absolute and depends on the world frame 2512.07237 §3.1 2512.07237 fig. 2b. The iLRM card uses this encoding as the viewpoint-token input 2507.23277 §3.2.
• CaPE: a relative query-key transform built from pairwise poses.
• GTA: also transforms values and the output.
• PRoPE: a projective matrix that includes intrinsics, limited to pinhole cameras.

When to use which

Run time is close to the baselines: 184 s for one clip on a single A800 against 179 s and 177 s 2512.07237 app. D.

Implementation sketch

Two cores share one ingredient: a per-pixel viewing ray. The UCPE core is the per-token ray transform, applied in attention next to RoPE. The iLRM core is the Plücker ray per pixel, patchified and projected to viewpoint tokens. The Lat-Up map is optional in UCPE. Lines are shortened from the cited equations; check the exact definitions in the cited equations and the cards.

# Per-token world-to-ray transform  [arxiv:2512.07237 eq. 5-7]
# Input:  c2w (4, 4) pose; ray_dir (H, W, 3) unit rays in world frame
# Output: t_rw (H, W, 4, 4) one transform per token

def ray_transform(c2w, ray_dir):
    # Local z-axis is the ray itself  [eq. 5-6]
    z_axis = normalize(ray_dir)                       # (H, W, 3)
    # Local x-axis from the camera's downward axis  [eq. 6]
    x_axis = normalize(cross(c2w[:3, 1], z_axis))     # (H, W, 3)
    # Local y completes a right-handed frame  [eq. 6]
    y_axis = cross(z_axis, x_axis)                    # (H, W, 3)
    # Ray-to-world; invert for world-to-ray  [eq. 7]
    r_wr = stack_columns(x_axis, y_axis, z_axis)      # (H, W, 3, 3)
    return invert_rigid(r_wr, c2w[:3, 3])             # (H, W, 4, 4)

# Attention with the ray transform on half the dims  [arxiv:2512.07237 eq. 10]
# Input:  q, k, v (B, N, d); t_rw (B, N, 4, 4)
# Output: (B, N, d)

def ray_attention(q, k, v, t_rw):
    # Split each head: first half gets the ray transform, second half RoPE  [eq. 10]
    q_geo, q_rope = split_half(q)
    k_geo, k_rope = split_half(k)
    # Apply the ray transform to the geometric half  [eq. 10]
    q_geo = apply_block_transform(t_rw, q_geo)
    k_geo = apply_block_transform(t_rw, k_geo)
    # Rotary embedding on the other half  [§3.3]
    q_rope, k_rope = rope(q_rope), rope(k_rope)
    # Value transform, as in the GTA-style scheme  [eq. 4]
    v = apply_block_transform(t_rw, v)
    out = attention(concat(q_geo, q_rope), concat(k_geo, k_rope), v)
    # Output transform undoes the value transform  [eq. 4]
    return apply_block_transform(invert(t_rw), out)

# Plücker ray per pixel, the iLRM viewpoint input  [arxiv:2507.23277 §3.2]
# Input:  origin o (3,), unit direction d (H, W, 3)
# Output: (H, W, 6)

def plucker_ray(o, d):
    # Moment is origin cross direction  [arxiv:2512.07237 §3.1]
    m = cross(o, d)                                   # (H, W, 3)
    return concat(d, m)                               # (H, W, 6)

Where the UCPE variants differ: the transform itself (per-lens ray mapping), the split between geometric and RoPE dimensions (the paper uses half, matching its GTA and PRoPE baselines 2512.07237 §3.3), and whether the Lat-Up bias is added 2512.07237 §3.2. Where the iLRM variant differs: the ray is used as a token input with no positional embedding, and the viewpoint tokens are at the same resolution as the image tokens in the released code [see the card's Paper vs code section].

Implementation notes from the UCPE card's code check:

• Build the ray transforms on the token grid, not the pixel grid. The card's check reads the token grid as frame size divided by the VAE factor times two [see the card's Paper vs code section].
• Use one pose per latent frame. The released code keeps every fourth pose, which matches the video VAE's temporal compression [see the card's Paper vs code section]. The paper does not describe this step.
• Replace undefined rays (outside the lens image circle) with the identity transform and zero the Lat-Up channels. The paper does not say what happens there [see the card's Pitfalls section].

Contradictions

• Absolute encodings across scenes. The UCPE card says absolute encodings such as Plücker coordinates depend on the world frame and so generalise poorly across scenes and viewpoints 2512.07237 §3.1. The iLRM card feeds Plücker rays and normalises the cameras first, to an average pose scaled so the largest camera distance is 1 2507.23277 app. A. It reports good results in that setting. Neither card tests the other's setting, so this disagreement is stated here and not resolved.
• Base model size. The paper calls Wan2.1 T2V 1.3B a "7.3 billion parameter" model in its introduction and appendix D 2512.07237 §1 2512.07237 app. D. Its experiments and released scripts use the 1.3B model 2512.07237 app. D. The "<1%" trainable share is computed against 7.3B. Against 1.3B, the 35.6M adapter is about 2.7%.
• Trainable parameters. The text gives 35.5M for UCPE 2512.07237 §4.2. The tables give 35.6M 2512.07237 tab. 1.
• Lat-Up channel order. The text lists latitude first, then the two up components 2512.07237 §3.2. The released code concatenates the up components first and latitude last (card's code check). Weights trained one way will not load correctly the other way.
• Batch size. Appendix D gives a batch of 8 on 8 A800 GPUs 2512.07237 app. D. The code's default is one sample per device, with no gradient accumulation (card's code check).
• Baseline learning rates. Appendix D gives 1e-5 for the Wan CameraCtrl baseline and does not state one for ReCamMaster 2512.07237 app. D. The card's code check finds 1e-5 in the training script for the Plücker baseline and the default 1e-4 for the ReCamMaster command.
• Inference defaults. Neither the number of steps nor the guidance scale is stated in the paper 2512.07237 app. D. The released code sets 50 steps and a guidance scale of 5.0 (card's code check).

Papers

• Unified Camera Positional Encoding for Controlled Video Generation: per-token ray frames with a lens model, a gravity-aligned up/latitude map, and a zero-initialised parallel adapter on Wan2.1 T2V 1.3B; the video-generation card on this concept.
• iLRM: An Iterative Large 3D Reconstruction Model: per-pixel Plücker rays as viewpoint tokens with no positional embedding, refined over 12 cross- and self-attention layers in a feed-forward 3D Gaussian model; the reconstruction card on this concept.