Unified Camera Positional Encoding for Controlled Video Generation

In plain words

Text-to-video models can be told where the camera should go, but most of them treat the camera as if it were a simple pinhole, the idealised lens with no distortion. Real lenses bend the light: wide-angle and fisheye cameras look different even when they point the same way. Existing methods also leave the camera's tilt and roll unanchored, because they only define motion relative to the first frame.

The idea here is to give every pixel of the video its own viewing ray, the straight line along which that pixel sees the scene. Each ray is described in its own local coordinate frame, so the attention layers can reason about geometry ray by ray and handle any lens shape. A second signal shows the model which way is "up" in the image, so the camera's pitch and roll can be set directly. The combined encoding is added to a pretrained video generator through a small side branch, which leaves the original model's behaviour intact at the start of training.

TL;DR

• UCPE (Unified Camera Positional Encoding) gives each token a ray-based geometric frame that covers pose, intrinsics and lens distortion §3.2.
• A gravity-aligned latitude-up map gives absolute pitch and roll control §3.2.
• It is injected through a parallel, zero-initialised spatial attention adapter on Wan2.1 T2V 1.3B §3.3 app. D.
• Author-reported: best camera control and video-quality numbers among the four baselines on a synthetic UCM dataset tab. 1, and the best zero-shot pose numbers on RealEstate10K tab. 2.
• Code is public under MIT. The repository's train.sh matches the paper's main configuration; several paper statements disagree with each other or with the code (see Paper vs code).

Problem

Camera-conditioned attention methods fall into two groups, and both fall short for diverse lenses §1 §2:

• Absolute encodings feed raw camera parameters or Plücker coordinates into the network. They depend on the world frame, so they generalise poorly across scenes and viewpoints §3.1 fig. 2.
• Relative encodings (CaPE, GTA, PRoPE) inject pairwise camera relations into attention. They assume one linear projection per camera, so they cannot model distortion that varies across the image §3.1 §4.3.
• Text-to-video camera control defines poses relative to the first frame. Pitch and roll of the initial view are then ambiguous §1.

The paper's claim is that a method must handle non-linear projection per token and anchor absolute orientation, without retraining a whole video model.

Method

3.1 Preliminaries

A camera is a ray-mapping function from pixel coordinates to a ray origin and a unit direction in camera coordinates. Central cameras share one origin; the paper derives its equations for central cameras and uses the Unified Camera Model (UCM) as the example lens §3.1 app. A.

Given the camera-to-world pose, the world-space ray is the rotated camera ray, and its origin is the translation eq. 1.

Existing encodings, in the paper's notation:

• Plücker encoding stacks the ray direction and its moment (origin cross direction) into a 6-vector per pixel. It is absolute and coordinate-dependent §3.1 fig. 2b.
• CaPE replaces each query-key product by a term that depends on the relative pose, through a block-diagonal transform D built from world-to-camera poses eq. 2 eq. 3.
• GTA also transforms the values and the output, which keeps the attention consistent under the change of frame eq. 4.
• PRoPE replaces the pose with a projective matrix that includes the intrinsics. It is limited to pinhole cameras §3.1 fig. 2c.

3.2 Unified Camera Positional Encoding

Relative ray encoding. Each token is one viewing ray with origin o_t and direction d_t eq. 5. The paper builds a local ray frame per token:

• the local z-axis is the ray direction;
• the local x-axis is the cross product of the camera's downward axis with z;
• the local y-axis completes the right-handed frame eq. 6.

The ray-to-world transform stacks these axes with the origin as translation eq. 7. The token uses its inverse, the world-to-ray transform, which is the geometric operator applied in attention §3.2.

Absolute orientation (Lat-Up map). Each pixel gets two signals:

• a latitude, the elevation of the ray above the horizon, positive when looking up eq. 8;
• an up vector: the ray is rotated by a small angle δ around the axis formed by the ray and world up; the rotated ray is projected back to the image with the lens model, and the normalised pixel displacement is the up vector eq. 9 app. B. The paper uses δ = 0.1 rad as its example value app. B.

The Lat-Up map is optional. When present it is a three-channel input (one latitude, two up components) §3.2 §3.3.

3.3 Spatial attention adapter

• The ray transform takes half of the feature dimension and RoPE takes the other half, giving one block-diagonal operator per token eq. 10 §3.2. The paper uses the same split as its GTA and PRoPE baselines, which it says is their best-performing configuration §3.3.
• The adapter runs in parallel with each DiT self-attention block, not in sequence §3.3 fig. 3.
• Inputs are projected to 1/C of the width with proportionally fewer heads; the output goes through a linear layer with zero-initialised weights, so the pretrained model is unchanged at initialisation §3.3.
• The Lat-Up features pass through a linear layer to the token width and are added to the input tokens as a bias §3.3.
• The base model is Wan §3.3 app. D.

# Relative ray frame per token  [§3.2] [eq. 5-7]
# Input:  c2w (4, 4) camera pose; d_cam (H, W, 3) unit UCM rays  [app. A]
# Output: t_rw (H, W, 4, 4) world-to-ray transform per token

def world_to_ray(c2w, d_cam):
    # Rotate each pixel ray into world space  [eq. 1]
    d_world = rotate(c2w[:3, :3], d_cam)               # (H, W, 3)
    # Central camera: every ray starts at the camera centre  [§3.1]
    origin = c2w[:3, 3]                                # (3,)
    # Local z-axis is the ray direction  [eq. 5] [eq. 6]
    z_axis = normalize(d_world)                        # (H, W, 3)
    # Local x-axis from the camera's downward axis  [eq. 6]
    x_axis = normalize(cross(c2w[:3, 1], z_axis))      # (H, W, 3)
    # Local y-axis completes the frame  [eq. 6]
    y_axis = cross(z_axis, x_axis)                     # (H, W, 3)
    # Ray-to-world rotation; the origin is the translation  [eq. 7]
    r_wr = stack_columns(x_axis, y_axis, z_axis)       # (H, W, 3, 3)
    # Invert the rigid transform to get world-to-ray  [§3.2]
    r_rw = transpose(r_wr)                             # (H, W, 3, 3)
    t_rw = -r_rw @ origin                              # (H, W, 3)
    return assemble(r_rw, t_rw)                        # (H, W, 4, 4)

# Lat-Up absolute orientation map  [§3.2] [eq. 8-9] [app. B]
# Input:  d_world (H, W, 3) unit rays; c2w_rot (3, 3); delta = 0.1 rad
# Output: lat (H, W, 1), up (H, W, 2)

def lat_up_map(d_world, c2w_rot, delta=0.1):
    # Latitude: elevation of each ray, positive when looking up  [eq. 8]
    lat = atan2(-d_world[..., 1], norm_xz(d_world))    # (H, W, 1)
    # Rotation axis towards world up  [app. B]
    axis = normalize(cross(d_world, world_up))         # (H, W, 3)
    # Rodrigues rotation of each ray by delta  [app. B] [eq. 8-9]
    rotated = rodrigues(axis, delta, d_world)          # (H, W, 3)
    # Back to camera frame, then project with the same lens model  [app. B]
    shifted = project_ucm(transpose(c2w_rot) @ rotated)  # (H, W, 2)
    # Up: normalised pixel displacement from the original pixel  [eq. 9]
    up = normalize(shifted - pixel_centre)             # (H, W, 2)
    return lat, up

# Spatial attention adapter, parallel to each DiT self-attention  [§3.3]
# Input:  tokens (B, N, d); t_rw (B, N, 4, 4); latup (B, N, 3) or None
# Output: (B, N, d) branch added to the block output

def ucpe_adapter(tokens, t_rw, latup):
    # Absolute orientation enters as a bias on the tokens  [§3.2] [§3.3]
    if latup is not None:
        tokens = tokens + latup_linear(latup)          # (B, N, d)
    # Project to 1/C of the width with fewer heads  [§3.3] [tab. 1]
    queries = project_q(tokens)                        # (B, N, d / C)
    keys = project_k(tokens)                           # (B, N, d / C)
    values = project_v(tokens)                         # (B, N, d / C)
    # Ray transform on half the dims, RoPE on the other half  [eq. 10]
    queries, keys, values = apply_ucpe(
        queries, keys, values, t_rw)
    # Attention with values transformed as in GTA; output mapped back  [eq. 4]
    out = attention(queries, keys, values)
    # Zero-initialised output projection  [§3.3]
    return zero_init_linear(out)                       # (B, N, d)

Training data

• Source: a corpus of 24.1k 360° YouTube videos app. A.
• Curation: scene-cut detection, a panoramic SLAM pass that splits clips where tracking is lost, a second localisation pass for poses, and a rotation score that rejects clips with large drift app. A.
• Quality filters: Q-Align quality score, a watermark detector and a vision- language filter for non-equirectangular, low-quality or overlay content app. A.
• Trajectory scale is normalised per clip with a near-plane depth statistic app. A.
• Rotation transfer: about 1k CameraBench clips with text-described motion, with zoom clips removed; poses from ViPE, the most jittery 20% dropped; matched by translation and assigned greedily under diversity caps app. A.
• Augmentation: consistent yaw, yaw/pitch, and smooth yaw-pitch-roll panning, giving 48k unique camera motions app. A.
• Lens sampling: four UCM categories (pinhole, wide-angle, fisheye, extreme fisheye) with xFoV and ξ drawn uniformly per category app. A.
• Output: about 48k clips of 81 frames at 480 × 832 and 16 fps, captioned with a vision-language model; the paper reports about three days on one A800 node app. A.
• Evaluation split: 272 clips matched to the CameraBench test split app. A.

Hyperparameters

Results

Author-reported. Setup: synthetic UCM dataset (about 48k clips, 81 frames at 480 × 832), Wan2.1 T2V 1.3B base, and four camera-conditioned baselines fine- tuned under the same setting, except where noted §4.1 §4.2 app. D. Pitch and roll cells for the "w/o" rows are shaded as not applicable in the paper.

Main comparison, with absolute orientation control tab. 1:

Without absolute orientation control tab. 1:

Ablations, all with absolute orientation control tab. 1:

Zero-shot on RealEstate10K, 100 clips, 100° pinhole, no fine-tuning on that dataset tab. 2 §4.2:

Cost app. D: one-clip inference on a single A800 takes 184 s for UCPE, against 179 s for ReCamMaster and 177 s for Wan CameraCtrl (81 frames, 480 × 832).

The paper also reports qualitative comparisons fig. 4 fig. 5 and robustness checks: a text prompt that asks for telephoto while the camera is set to a fisheye lens follows the camera app. E; a Brown-Conrady lens is used at inference by swapping the ray-mapping function app. E; an image-to-video variant fine-tunes Wan2.1 I2V 14B without the Lat-Up map app. E.

Paper vs code

Code: https://github.com/chengzhag/UCPE, MIT, checked at commit d992f1807803ba99331e807e8f018ed552886afd (shallow clone of the default branch). The paper's abstract points to this repository.

• Ray frame and world-to-ray transform match. The paper's axes (eq. 6) and transform (eq. 7) are implemented as in the paper: local x from the camera's y axis, y from z and x, translation -R·t camera_control.py eq. 6 eq. 7.
• Lat and Up maps match eq. 8 and eq. 9. The latitude uses the same atan2; the Up map rotates the ray by δ = 0.1 by Rodrigues' formula, projects it with UCM and normalises the displacement camera_control.py app. B.
• Lat-Up channel order differs. The text lists latitude then up §3.2. The code concatenates up then latitude wan_video_panshot.py. Weights trained with one order will not load correctly with the other.
• Output projection zero-init matches; Lat-Up projection init is not stated. The output projection is zero-initialised camera_control.py, matching §3.3. The Lat-Up linear layer keeps PyTorch's default initialisation camera_control.py; the paper says nothing about it.
• Width and heads match the table. The adapter uses dim / C and heads / C camera_control.py. At C = 8 this gives one head of width 192, as in tab. 1.
• Intrinsics enter only through the rays. For the ray method the code passes no intrinsic matrix to the attention; it sets only the ray transforms as the view matrices wan_video_panshot.py camera_control.py. This matches the paper's claim that lens effects live in the ray frames §3.2.
• Rays are computed on the token grid. Ray transforms are built at the token resolution, which is the frame size divided by the VAE factor times two camera_control.py wan_video_panshot.py. The paper says "per image token" §3.2 but does not give the grid size.
• Poses are subsampled by 4 in time. The code uses every fourth pose wan_video_panshot.py. The paper does not describe this step for UCPE; it is consistent with the 4× temporal compression of the video VAE that the appendix mentions for ReCamMaster app. D.
• Training schedule matches the paper's 10k steps. AdamW at 1e-4 [code: src/main.py#L126] main.py; 10 epochs with 1000 batches each main.py. Batch size is 1 per device with automatic device count, and no gradient accumulation is enabled main.py main.py. The paper's batch of 8 holds only if 8 GPUs are used, which the README states.
• Baseline learning rates. scripts/train.sh uses 1e-5 for the Plücker baseline, as the appendix says train.sh. The ReCamMaster command uses the default 1e-4 set_Wan2.1-T2V-1.3B.sh.
• Base model size. The code and the scripts load Wan2.1 T2V 1.3B set_Wan2.1-T2V-1.3B.sh. The paper calls the same base model "7.3 billion parameters" app. D.
• Trainable-parameter count. The text says 35.5M §4.2 and the tables say 35.6M tab. 1. The code was not used to count parameters; the number is the paper's.
• Inference defaults. Code: 50 steps main.py and CFG 5.0 in the pipeline default wan_video_panshot.py. The paper does not state either value. The evaluation settings may differ.
• Not checked. The evaluation scripts (src/evaluate.py) were not compared against §4.1 and app. C. The PRoPE attention module in thirdparty/prope/ was not read; only its call site was. The paper's ray-plus-RoPE split is therefore only checked at the call site.
• Ray degeneracy is not handled in the paper. The code sets a ray matrix to identity where rays are NaN, and zeroes Plücker features there camera_control.py camera_control.py. The paper does not say what happens outside the lens's valid image region.

Pitfalls

• Parameter-share claim depends on the wrong base size. The paper's "less than 1%" and "0.5%" figures divide by 7.3B §1 app. D. The base is the 1.3B model. 35.6M over 1.3B is about 2.7%. Do not repeat the "<1%" figure without this caveat.
• Lat-Up order. Implement the order the code uses (up, then latitude) if you want to load released weights. The paper's text order (latitude, up) is the other choice.
• Zero init covers only the output projection. The Lat-Up projection starts from random weights in the code, and the paper does not say how it should start. Only the output projection is guaranteed to leave the pretrained model unchanged at step 0.
• Degenerate ray frames. The x-axis is the cross product of the camera's down axis and the ray. If a ray is parallel to the camera's down axis, the cross product is near zero and the frame is undefined. The code normalises with a small epsilon, which yields a zero vector rather than an error. The paper does not discuss this. This is from reading the code; it was not run.
• Invalid pixels. Outside the valid lens region the ray is NaN; the code replaces its transform with identity and zeroes the Lat-Up channels camera_control.py camera_control.py. Test this path if you use a lens model with a limited image circle.
• Pixel-centre convention. The Up map depends on the pixel-centre grid. The code builds it with a helper from the equilib package camera_control.py; its half-pixel convention was not checked.
• Frame-to-pose mapping. The paper's poses are per frame; the code uses poses at every fourth frame to match the latent frame count [code: diffsynth/pipelines/wan_video_panshot.py#L884-L885]. An implementation that uses every frame will mismatch the token count.
• Batch size. The paper's effective batch of 8 requires 8 GPUs with the code's default of 1 per device. Set gradient accumulation if you train on fewer GPUs.
• Baseline fairness. ReCamMaster and Wan CameraCtrl were modified by the authors (added lens inputs, a lower learning rate for full fine-tuning). The "w/o absolute orientation" rows use the paper's own adaptation, not the original methods §4.2 app. D.
• Pose metrics depend on a rectification step. Frames are rectified to pinhole with ground-truth UCM parameters, capped at 100° FoV, before ViPE estimates poses app. C. Comparisons with other pose estimators will differ.
• Synthetic training data. All UCM training clips are synthetic, built from 360° footage with a constructed rotation. The real-world result is the zero-shot RealEstate10K test only, which uses a fixed 100° pinhole and short prompts §4.2.
• Several numbers are estimates. Pitch, roll and lens parameters are read off generated frames with GeoCalib, not measured app. C.
• Single-run results. The paper reports one table per setting; no seed variance or confidence intervals are given.

Lineage

• Builds on: CaPE §3.1; GTA §3.1; PRoPE, arXiv:2507.10496 §3.1; Wan, arXiv:2503.20314, as the base video model §3.3 app. D.
• Compared against: ReCamMaster and a Wan adaptation of CameraCtrl §4.2; AC3D and CameraCtrl, evaluated only on RealEstate10K with released weights §4.2.
• Superseded by: not known from content already in this wiki.