WorldWander: Bridging Egocentric and Exocentric Worlds in Video Generation

In plain words

Video generators can now make a camera move freely through a scene. A useful next step is switching viewpoint: take a clip filmed from a person's own eyes (first-person, egocentric) and produce the same moment seen from outside (third-person, exocentric), or the reverse. Earlier work mostly handled one direction and assumed a fixed outside camera.

WorldWander takes a pretrained video generator and teaches it, with a small add-on, to do this from example pairs instead of from camera geometry. The source clip and the clip being generated go through the same model as one combined input, and the model is told that matching moments in the two clips share a time position. For the reverse direction, a picture of the character is added to the input so the character stays the same. The training pairs were recorded in a video game and with a camera rig carried by a person.

This matters for games, embodied AI and VR, where a viewer or an agent needs to move between these two viewpoints.

TL;DR

• WorldWander is a LoRA fine-tune of the Wan2.2-5B video diffusion transformer for egocentric and exocentric video translation §4 §5.1.
• It takes a source video and, for the exo-view direction, a reference image of the character. It uses no camera poses and no depth §4.1.
• Two mechanisms do the work: token concatenation with shared projections, and shared temporal positions for source and target §4.2 §4.3.
• Training data is EgoExo-8K: about 4,000 synthetic (GTA-5) and about 4,000 real-world triplets §4.5 app. B.
• The authors report better scores than four baselines on 50-triplet test sets per domain §5.2 tab. 1. The code has no license file (cite only).

Problem

Existing ego-to-exo or exo-to-ego methods work in one direction only, and mostly with static cameras, so they struggle to keep appearance, motion and environment in sync when the camera moves §1. Generating an exocentric view from an egocentric clip plus a reference image is largely unexplored §1. Three obstacles are named: synchronising the two views, keeping the character consistent across views, and the lack of paired data with moving cameras §1.

Method

The task takes triplets of an egocentric video, an exocentric video and a reference image §4.1. Two directions are trained:

• Exo-to-ego: the exocentric video is the condition; the egocentric video is generated §4.1.
• Ego-to-exo: the egocentric video and the reference image are the conditions; the exocentric video is generated §4.1.

No camera pose or depth input is used §4.1.

Backbone. The model is Wan2.2-5B, fine-tuned with LoRA in attention layers §3 §4.4 §5.1. The flow-matching loss in eq. 2 is used, with the forward path of eq. 1 and velocity target v = ε − z₀ eq. 2 §3.

In-Context Perspective Alignment §4.2. Three parts:

1. Shared latent space. One VAE encodes the ego video, the exo video and the reference image into one latent space, so no condition encoder is added eq. 5 §4.2.
2. Different noise levels. The target latent is noised along the flow path. Clean conditions are not noised. The timestep of a clean condition is 0, and the target keeps its sampled t §4.2.
3. Collaborative attention. The clean condition(s) and the noisy target are concatenated along the token dimension into one sequence eq. 6. Self-attention over this sequence uses one set of projection matrices for all parts eq. 7 §4.2. The paper reports that this converges faster than concatenating the latents along the channel dimension fig. 3 §4.2.

Collaborative Position Encoding §4.3. Rotary positions are applied to Q and K per token eq. 4. Uniform encoding over the whole sequence breaks the frame-by-frame alignment between conditions and target. Instead, the conditional and target latents are encoded separately and given the same temporal positions eq. 8 §4.3. The paper's ablation finds this better than uniform encoding tab. 2 §5.4.

Reference image. In ego-to-exo, the reference image is encoded like a single frame and kept clean. It occupies its own slot in the sequence before the egocentric condition eq. 6 §4.2.

Training objective. The flow-matching loss eq. 2 is applied to the target tokens. Only LoRA parameters are updated §4.4.

EgoExo-8K §4.5 app. B. Synthetic: GTA-5 with two players. One plays the character from the first-person view; the other records the same scene in spectator mode. A character photo serves as the reference. More than 400 paired sequences of over one minute were cut into 5-second clips of 300 frames, giving 4,000 triplets. Real-world: a participant carried a head-mounted action camera (egocentric) and a rear-mounted panoramic camera (exocentric). The reference image is a frame sampled from the exocentric video. More than 10 hours of paired video yielded 4,000 triplets.

Fine-tuning per domain. The authors fine-tune separately for the synthetic and real-world domains, holding out 50 triplets from each as the test set §5.1.

Training step (paper form)

# Training step for one triplet  [§4.2] [eq. 1] [eq. 2] [eq. 5]
# Input:  source_video (B, C, F, H, W), target_video (B, C, F, H, W),
#         ref_image (B, C, 1, H, W) for ego-to-exo only
# Output: scalar loss

def training_step(source_video, target_video, ref_image, prompt):
    # Encode all inputs with one shared VAE  [§4.2] [eq. 5]
    source = vae_encode(source_video)
    target = vae_encode(target_video)
    ref = vae_encode(ref_image)
    # Sample noise and a timestep for the target only  [§4.2] [eq. 1]
    noise = sample_gaussian_like(target)
    t = sample_timestep()
    noisy_target = (1 - t) * target + t * noise
    # Velocity regression target  [eq. 2]
    velocity = noise - target
    # Clean inputs use timestep 0; the noisy target uses t  [§4.2]
    timesteps = concat(zeros(source), zeros(ref), t_tokens(noisy_target))
    # Token concatenation, one sequence  [§4.2] [eq. 6]
    tokens = token_concat(ref, source, noisy_target)
    # Shared temporal positions for source and target  [§4.3] [eq. 8]
    positions = collaborative_positions(ref, source, target)
    # One backbone pass; LoRA layers are the only trainable weights  [§4.4]
    velocity_pred = transformer(tokens, timesteps, positions, prompt)
    # Keep only the target tokens  [§4.2]
    velocity_pred = select_target_tokens(velocity_pred)
    # Mean squared error against the velocity target  [eq. 2]
    loss = mean((velocity_pred - velocity) ** 2)
    return loss

Sampling (paper form, sampler details not stated)

# Inference for one source clip  [§4.2] [§5.1]
# Input:  source_video (B, C, F, H, W); ref_image for ego-to-exo only
# Output: generated target video

def translate(source_video, ref_image, prompt):
    # Encode the clean conditions  [§4.2] [eq. 5]
    source = vae_encode(source_video)
    ref = vae_encode(ref_image)
    # Start the target latent from Gaussian noise  [§3]
    target = sample_gaussian((C, f, h, w))
    # Denoise along the flow with the sampler of your choice; the paper
    # does not name one. The code uses 50 steps (see Paper vs code).
    for t in sampler_timesteps():
        # Conditions stay clean at timestep 0  [§4.2]
        velocity = transformer(concat(ref, source, target), t, prompt)
        # Keep the target part of the prediction  [§4.2]
        target = sampler_step(target, select_target_tokens(velocity), t)
    return vae_decode(target)

Hyperparameters

Results

Author-reported. Setup: Wan2.2-5B fine-tuned separately for synthetic and real-world data, 50 held-out triplets per domain, comparisons against four baselines §5.1 §5.2.

Baselines are AnyV2V, I2VEdit, TrajectoryCrafter and ReCamMaster §5.1. An asterisk means the baseline was fine-tuned on EgoExo-8K tab. 1. Zero-shot TrajectoryCrafter has no CLIP-I score because it cannot generate a customised character tab. 1.

Synthetic scenarios tab. 1.

The first four metric columns are exocentric-to-egocentric, and the next five are egocentric-to-exocentric, with CLIP-I used only in that direction tab. 1.

Real-world scenarios tab. 1.

Other reported results:

• A user study with 630 valid responses prefers WorldWander across the subjective criteria. Only the figure is available; the numbers are not in the text §5.2 fig. 6.
• Ablation (synthetic only, because of limited resources) tab. 2 §5.4:

The first four metric columns are exocentric-to-egocentric, as in the main table.

• In the ablation, rank 80 is picked as the best overall, but rank 64 and rank 128 each win on some metrics tab. 2 §5.4.

Paper vs code

Official repository: showlab/WorldWander, commit c9f12e85730d3976f1e89df5ee8248f7f3575a7a (cloned with --depth 1). The paper's author block links this repository. The repository has no LICENSE file.

• Loss weighting. Paper: plain flow-matching MSE eq. 2. Code: the per-sample MSE is multiplied by a timestep weight, a bell-shaped curve centred mid-schedule and normalised to mean 1 wan2_trainer.py my_schedule.py.
• Timestep distribution. Paper: t drawn from U(0, 1) eq. 2. Code: a uniform index over 1000 discrete steps of a sigma schedule warped with shift 5, then multiplied by 1000 wan2_trainer.py wan2_trainer.py my_schedule.py my_schedule.py.
• Learning rate. Paper: 1e-4 §5.1. Code: base 5e-5 scaled by √(GPUs × accumulation × nodes) wan2-2_lora_one2three_synthetic.yaml wan2_trainer.py. The two agree only at 4 GPUs with no gradient accumulation.
• LoRA targets. Paper: attention layers §5.1. Code: query, key, value, output projections and both feed-forward projections wan2_trainer.py.
• Concatenation axis. Paper: concatenation along the token dimension §4.2. Code: the frame axis is concatenated before patch embedding, giving the order reference, source, target transformer_wan.py transformer_wan.py.
• Reference preprocessing. Code: the reference image is resized straight to 704 × 1280 with no crop, so the aspect ratio is not kept custom_dataset.py. Video frames use a long-side resize followed by a centre crop custom_dataset.py. The paper does not say how the reference is resized.
• Reference positions. Code: the reference takes temporal position 0, and the source and target then share positions 1 to F transformer_wan.py. The paper's position equation includes a term for the reference but does not say which index it uses eq. 8.
• Exo-to-ego positions. Code: source and target each start at temporal position 0 transformer_wan.py. This matches the paper's shared-position claim.
• Text dropout. Code: 10% of prompts are replaced with empty text during training wan2_trainer.py. Not in the paper.
• Sampler and guidance. Code: UniPC multistep, flow shift 5, 50 steps by default, guidance 5.0 with an empty negative prompt wan2_inference.py wan2_inference.py custom_pipeline.py custom_pipeline.py. The paper states none of these.
• Test-time preprocessing. The resize and crop lines in the test dataset are commented out, so test frames and references are used at native resolution custom_dataset.py custom_dataset.py. Test stride is hard-coded to 1 wan2_inference.py; training uses stride 2 wan2-2_lora_one2three_synthetic.yaml.
• Clip sampling. Paper: 5-second, 300-frame clips, with F = 49 used for training §4.5 §5.1. Code: a random start and 49 frames taken every second frame, so one sample spans about 97 source frames custom_dataset.py.
• Ablation variants. Channel concatenation and uniform timestep are not in the repository. Only uniform position encoding is a switch transformer_wan.py transformer_wan.py. The LoRA rank configs contain only rank 80 wan2-2_lora_one2three_synthetic.yaml.
• Released data. The README links the synthetic dataset (EgoExo-Synthetic). It does not link the real-world subset README.md.
• Baseline inputs. The appendix says TrajectoryCrafter received camera poses from the training set. WorldWander uses none app. A §4.1.

Pitfalls

• Direction names. In the code, one2three means ego-to-exo with a reference image. three2one means exo-to-ego wan2-2_lora_one2three_synthetic.yaml wan2-2_lora_three2one_synthetic.yaml.
• Clean-token timesteps. The code gives clean tokens timestep 0 and noisy target tokens a per-token timestep. The per-token timestep comes from the mask subsampled by 2 on height and width, which assumes a 2×2 spatial patch wan2_trainer.py.
• Target slicing. The model returns the full sequence. Only the trailing target tokens are the prediction. Dropping this slice gives a shape error or wrong loss transformer_wan.py.
• Position trick. Reusing the condition's rotary embedding for the target depends on the reference occupying exactly one temporal slot. The index arithmetic is fragile transformer_wan.py.
• Sequence length. The source (or reference plus source) is concatenated with the target, so the attention sequence is about twice the target's token count. This is derived from the architecture, not reported. The paper gives only the wall-clock inference time §5.1.
• Paper inconsistency on resolution. The main text uses 704 × 1280 §5.1. The baseline appendix resizes all methods to 704 × 1024 app. A. The code uses 704 × 1280 for WorldWander.
• Figure reference mismatch. The LoRA-rank results are cited to the ablation figure, but the numbers are in the ablation table §5.4 tab. 2.
• "Scene" means domain. The paper says the model is fine-tuned per scene. The code's configs show one model per domain (synthetic or real) and direction §5.1 wan2-2_lora_one2three_realworld.yaml.
• Fair-comparison caveats. Only the baselines marked * were fine-tuned. TrajectoryCrafter was zero-shot and given camera poses. AnyV2V and I2VEdit edit the first frame with Qwen-image-edit first app. A. The test sets are 50 triplets per domain and no split file is in the repository.
• Numbering. Figure, table and equation numbers in this card were counted from the LaTeX source. The compiled PDF was not checked.
• Scale. The paper states 4 H200 GPUs for training and about 3 minutes for inference, with no memory figure. The README recommends an 80 GB GPU for inference README.md.

Lineage

• Builds on: Wan2.2-5B as the backbone §3 §5.1, cited as the Wan model paper, arXiv:2503.20314 [bib]. LoRA and flow matching are cited but not carded.
• Compared against: AnyV2V (arXiv:2403.14468 [bib]), I2VEdit, TrajectoryCrafter and ReCamMaster. None have cards in this wiki.
• Superseded by: not known.