--- kind: concept slug: camera-pose-encoding title: Camera pose encoding summary: >- How a transformer is told where each camera is and how its lens bends light. A per-token ray frame handles any lens shape and, with a gravity-aligned map, gives absolute pitch and roll for video. Plücker ray tokens serve 3D reconstruction. Each card reports numbers only in its own setting, so the two are not directly comparable. tags: [video-generation, camera-control, positional-encoding, 3d-reconstruction] papers: ["2507.23277", "2512.07237"] related: [positional-encoding, relative-positional-encoding, diffusion-models, parameter-efficient-fine-tuning, feed-forward-reconstruction, gaussian-splatting] status: draft updated_at: 2026-11-01 --- ## Overview A model that must respect a camera needs a per-token signal for that camera. Feeding the raw pose as one vector per clip is too coarse, so the usual route is to attach a camera signal to each token. The design question is what that per-token signal depends on: the world frame, the relation between two cameras, or the viewing ray of each pixel together with its lens model [arxiv:2512.07237 §1] [arxiv:2512.07237 §2]. Two cards cover this concept, and they use the viewing ray in different roles. The video-generation card builds a local ray frame for each token, so lens distortion and field of view enter the attention through the ray geometry, not through a pinhole matrix. An optional gravity-aligned map adds absolute pitch and roll [arxiv:2512.07237 §3.2]. The 3D-reconstruction card uses per-pixel Plücker rays as the only input of its viewpoint tokens, with no learned query embedding and no positional embedding added [arxiv:2507.23277 §3.2]. The reported results come from different tasks and different settings, so neither card's numbers can be read against the other's [arxiv:2512.07237 tab. 1] [arxiv:2507.23277 tab. 1]. ## Variants ### Per-token ray frame with a lens model (UCPE) Each token is one viewing ray with an origin and a unit direction. The local z-axis is the ray direction, the local x-axis comes from the camera's downward axis crossed with z, and y completes a right-handed frame. The token then uses the inverse of the ray-to-world transform as its geometric operator [arxiv:2512.07237 eq. 5] [arxiv:2512.07237 eq. 6] [arxiv:2512.07237 eq. 7]. Inside attention, this transform acts on half of each head's feature dimensions, and RoPE acts on the other half [arxiv:2512.07237 eq. 10]. The values are transformed too, following the GTA-style scheme [arxiv:2512.07237 eq. 4]. The branch runs in parallel with each self-attention block of the diffusion transformer, not in sequence, and its output projection starts at zero, so the pretrained model is unchanged at step 0 [arxiv:2512.07237 §3.3]. The optional Lat-Up map gives two absolute signals per pixel: the ray's elevation above the horizon, and an "up" direction found by nudging each ray by a small fixed angle and reprojecting it through the lens model [arxiv:2512.07237 eq. 8] [arxiv:2512.07237 eq. 9] [arxiv:2512.07237 app. B]. It enters as a bias on the tokens [arxiv:2512.07237 §3.3]. Card: [Unified Camera Positional Encoding for Controlled Video Generation](/papers/2512.07237). ### Plücker ray viewpoint tokens (iLRM) For each view, a Plücker ray embedding is computed for every pixel from the intrinsics and the pose. The rays are cut into p x p patches, flattened and linearly projected to tokens. No positional embedding is added, because the Plücker values already differ between patches [arxiv:2507.23277 §3.2]. These viewpoint tokens are refined over 12 update layers. Each layer runs a cross-attention in which a view's tokens attend only to that view's image tokens, then a self-attention over the tokens of all views [arxiv:2507.23277 §3.2] [arxiv:2507.23277 eq. 3] [arxiv:2507.23277 eq. 4]. The model needs accurate poses. The paper reports PSNR falling from 24.30 to 21.49 under translation noise with sigma = 0.005, against 23.97 to 20.49 for Long-LRM [arxiv:2507.23277 app. F]. The released code implements only the two-view, full-resolution setting. Its viewpoint tokens sit on the same grid as the image tokens, so the output is one Gaussian per pixel per view. The paper's decoupled settings, which give a more compact output, are not in the code [see the card's Paper vs code section]. Card: [iLRM: An Iterative Large 3D Reconstruction Model](/papers/2507.23277). ### Pose-only relative encodings and other baselines The UCPE card describes these as the alternatives the ray method is set against. Their details are the UCPE card's description only [arxiv:2512.07237 §3.1]. None of them has a card in this wiki: - Plücker coordinates: a per-pixel 6-vector that stacks the ray direction and its moment (origin cross direction). It is absolute and depends on the world frame [arxiv:2512.07237 §3.1] [arxiv:2512.07237 fig. 2b]. The iLRM card uses this encoding as the viewpoint-token input [arxiv:2507.23277 §3.2]. - CaPE: a relative query-key transform built from pairwise poses. - GTA: also transforms values and the output. - PRoPE: a projective matrix that includes intrinsics, limited to pinhole cameras. ## When to use which | Situation | Use | Why | |---|---|---| | Lens is wide-angle, fisheye or has unknown distortion | Per-token ray frame (UCPE) | The ray mapping is swapped per lens; the paper reports the lowest FoV and distortion errors (k1, k2) among its compared methods [arxiv:2512.07237 tab. 1]. Alternative lens models are reported in the appendix [arxiv:2512.07237 app. E]. | | Pinhole camera and only relative pose is needed | Pose-only relative encodings may suffice | The paper says PRoPE is limited to pinhole cameras [arxiv:2512.07237 §3.1]. Its own ablation shows PRoPE or GTA in place of the ray encoding has higher rotation error than the default [arxiv:2512.07237 tab. 1]. | | Pitch and roll must be set in absolute terms | Enable the Lat-Up map | Rows with absolute orientation report pitch and roll; the rows without it do not [arxiv:2512.07237 tab. 1]. UCPE reports 4.35° pitch and 3.74° roll against 6.62° and 5.29° for ReCamMaster [arxiv:2512.07237 tab. 1]. | | Small trainable budget is required | Adapter of about 36M parameters | The paper reports 35.6M trainable parameters against 354M for ReCamMaster and 1.5B for Wan CameraCtrl [arxiv:2512.07237 tab. 1]. The base model is frozen. See the contradictions on the base size. | | Tune the width of the adapter | Compression 1/8 (default) | It is the paper's chosen point. Other ratios win single metrics: 1/2 and 1/4 have lower rotation error, 1/12 has lower FVD [arxiv:2512.07237 tab. 1]. The wider fractions cost more trainable parameters: 141M at 1/2 and 71.0M at 1/4 [arxiv:2512.07237 tab. 1]. | | Only real-world pose transfer, no training on it | Zero-shot ray encoding | On RealEstate10K with a 100° pinhole, the paper reports the lowest rotation, translation and camera-motion errors among the listed methods, including two trained on that dataset [arxiv:2512.07237 tab. 2]. Image quality is not the best [arxiv:2512.07237 tab. 2]. | | Camera motion without absolute orientation, lowest video FVD | Check the table before choosing | Without the Lat-Up map, UCPE has higher FVD than ReCamMaster (569.31 against 555.54) while its rotation error is lower [arxiv:2512.07237 tab. 1]. | | Multi-view 3D reconstruction from posed photos in one forward pass | Plücker viewpoint tokens (iLRM) | On RE10K with two views at 256x256, the paper reports 28.65 PSNR against 27.47 for DepthSplat, and 0.025 s against 0.065 s on an RTX 4090 [arxiv:2507.23277 tab. 1]. The 32-view row is compared with optimisation-based 3D-GS, not with a feed-forward baseline [arxiv:2507.23277 tab. 5]. | | Per-view tokens with no learned view embedding | Plücker viewpoint tokens (iLRM) | The default path drops the positional and learned embedding. A learnable-embedding variant is described in the paper but is not in the default code path [arxiv:2507.23277 §3.2] [arxiv:2507.23277 app. D]. | | Output must be more compact than one Gaussian per pixel | Not available from the released iLRM code | The paper's decoupled viewpoint resolution gives fewer Gaussians, but the released code builds viewpoint tokens at full resolution [arxiv:2507.23277 §3.2] [see the card's Paper vs code section]. | Run time is close to the baselines: 184 s for one clip on a single A800 against 179 s and 177 s [arxiv:2512.07237 app. D]. ## Implementation sketch Two cores share one ingredient: a per-pixel viewing ray. The UCPE core is the per-token ray transform, applied in attention next to RoPE. The iLRM core is the Plücker ray per pixel, patchified and projected to viewpoint tokens. The Lat-Up map is optional in UCPE. Lines are shortened from the cited equations; check the exact definitions in the cited equations and the cards. ```pseudocode # Per-token world-to-ray transform [arxiv:2512.07237 eq. 5-7] # Input: c2w (4, 4) pose; ray_dir (H, W, 3) unit rays in world frame # Output: t_rw (H, W, 4, 4) one transform per token def ray_transform(c2w, ray_dir): # Local z-axis is the ray itself [eq. 5-6] z_axis = normalize(ray_dir) # (H, W, 3) # Local x-axis from the camera's downward axis [eq. 6] x_axis = normalize(cross(c2w[:3, 1], z_axis)) # (H, W, 3) # Local y completes a right-handed frame [eq. 6] y_axis = cross(z_axis, x_axis) # (H, W, 3) # Ray-to-world; invert for world-to-ray [eq. 7] r_wr = stack_columns(x_axis, y_axis, z_axis) # (H, W, 3, 3) return invert_rigid(r_wr, c2w[:3, 3]) # (H, W, 4, 4) ``` ```pseudocode # Attention with the ray transform on half the dims [arxiv:2512.07237 eq. 10] # Input: q, k, v (B, N, d); t_rw (B, N, 4, 4) # Output: (B, N, d) def ray_attention(q, k, v, t_rw): # Split each head: first half gets the ray transform, second half RoPE [eq. 10] q_geo, q_rope = split_half(q) k_geo, k_rope = split_half(k) # Apply the ray transform to the geometric half [eq. 10] q_geo = apply_block_transform(t_rw, q_geo) k_geo = apply_block_transform(t_rw, k_geo) # Rotary embedding on the other half [§3.3] q_rope, k_rope = rope(q_rope), rope(k_rope) # Value transform, as in the GTA-style scheme [eq. 4] v = apply_block_transform(t_rw, v) out = attention(concat(q_geo, q_rope), concat(k_geo, k_rope), v) # Output transform undoes the value transform [eq. 4] return apply_block_transform(invert(t_rw), out) ``` ```pseudocode # Plücker ray per pixel, the iLRM viewpoint input [arxiv:2507.23277 §3.2] # Input: origin o (3,), unit direction d (H, W, 3) # Output: (H, W, 6) def plucker_ray(o, d): # Moment is origin cross direction [arxiv:2512.07237 §3.1] m = cross(o, d) # (H, W, 3) return concat(d, m) # (H, W, 6) ``` Where the UCPE variants differ: the transform itself (per-lens ray mapping), the split between geometric and RoPE dimensions (the paper uses half, matching its GTA and PRoPE baselines [arxiv:2512.07237 §3.3]), and whether the Lat-Up bias is added [arxiv:2512.07237 §3.2]. Where the iLRM variant differs: the ray is used as a token input with no positional embedding, and the viewpoint tokens are at the same resolution as the image tokens in the released code [see the card's Paper vs code section]. Implementation notes from the UCPE card's code check: - Build the ray transforms on the token grid, not the pixel grid. The card's check reads the token grid as frame size divided by the VAE factor times two [see the card's Paper vs code section]. - Use one pose per latent frame. The released code keeps every fourth pose, which matches the video VAE's temporal compression [see the card's Paper vs code section]. The paper does not describe this step. - Replace undefined rays (outside the lens image circle) with the identity transform and zero the Lat-Up channels. The paper does not say what happens there [see the card's Pitfalls section]. ## Contradictions - **Absolute encodings across scenes.** The UCPE card says absolute encodings such as Plücker coordinates depend on the world frame and so generalise poorly across scenes and viewpoints [arxiv:2512.07237 §3.1]. The iLRM card feeds Plücker rays and normalises the cameras first, to an average pose scaled so the largest camera distance is 1 [arxiv:2507.23277 app. A]. It reports good results in that setting. Neither card tests the other's setting, so this disagreement is stated here and not resolved. - **Base model size.** The paper calls Wan2.1 T2V 1.3B a "7.3 billion parameter" model in its introduction and appendix D [arxiv:2512.07237 §1] [arxiv:2512.07237 app. D]. Its experiments and released scripts use the 1.3B model [arxiv:2512.07237 app. D]. The "<1%" trainable share is computed against 7.3B. Against 1.3B, the 35.6M adapter is about 2.7%. - **Trainable parameters.** The text gives 35.5M for UCPE [arxiv:2512.07237 §4.2]. The tables give 35.6M [arxiv:2512.07237 tab. 1]. - **Lat-Up channel order.** The text lists latitude first, then the two up components [arxiv:2512.07237 §3.2]. The released code concatenates the up components first and latitude last (card's code check). Weights trained one way will not load correctly the other way. - **Batch size.** Appendix D gives a batch of 8 on 8 A800 GPUs [arxiv:2512.07237 app. D]. The code's default is one sample per device, with no gradient accumulation (card's code check). - **Baseline learning rates.** Appendix D gives 1e-5 for the Wan CameraCtrl baseline and does not state one for ReCamMaster [arxiv:2512.07237 app. D]. The card's code check finds 1e-5 in the training script for the Plücker baseline and the default 1e-4 for the ReCamMaster command. - **Inference defaults.** Neither the number of steps nor the guidance scale is stated in the paper [arxiv:2512.07237 app. D]. The released code sets 50 steps and a guidance scale of 5.0 (card's code check). ## Papers - [Unified Camera Positional Encoding for Controlled Video Generation](/papers/2512.07237): per-token ray frames with a lens model, a gravity-aligned up/latitude map, and a zero-initialised parallel adapter on Wan2.1 T2V 1.3B; the video-generation card on this concept. - [iLRM: An Iterative Large 3D Reconstruction Model](/papers/2507.23277): per-pixel Plücker rays as viewpoint tokens with no positional embedding, refined over 12 cross- and self-attention layers in a feed-forward 3D Gaussian model; the reconstruction card on this concept.