Qwen-RobotWorld Technical Report: Unifying Embodied World Modeling through Language-Conditioned Video Generation

In plain words

The paper builds a video model that imagines what happens next in a scene. You give it a picture or a short clip of the current situation and a sentence describing what should happen, such as picking up a cup, and it generates the video of that action unfolding. The idea is that a plain sentence works as a common way to describe actions for very different machines: a robot arm, a car, or a person walking through a building. One system can therefore learn from all of them.

The model is split into two parts that talk to each other. A language model reads the sentence and turns it into guidance, while a video generator works on the frames. Training happens in two steps. First the model learns general visual knowledge from ordinary images and videos. Then it is specialized on robot, driving, navigation and human-demonstration data, while still seeing general data throughout, so the specialization does not erase the general knowledge.

The authors say such a model could be useful in three ways: making extra training data for robot policies, providing test environments for robot policies, and giving a planner a predicted future to check a plan against.

TL;DR

• A language-conditioned video diffusion model (60 double-stream MMDiT blocks, about 20B parameters) that predicts the future video from a current observation and a text action §3.1.
• Training data is an 8.6M video-text corpus with five-layer captions across manipulation, driving, indoor navigation and human-to-robot transfer §2.
• Two-stage curriculum: general pretraining, then a supervised stage that mixes 70% embodied and 30% general data §4.1.
• Author-reported: 1st overall on EWMBench (4.60) and DreamGen Bench (4.952), best open-source score on WorldModelBench and PBench §5.1.
• No official code was found. Many hyperparameters (learning rate, optimizer, batch size, steps) are not stated.

Problem

General video generators learn visual priors from internet video but miss embodied physics such as contact and rigid-body constraints §1. Embodied world models are usually tied to one scenario and use robot-specific action formats (joint angles, waypoints), so they do not transfer across embodiments or tasks §1. The paper's premise is that a shared natural-language action interface lets different embodied domains train jointly §1 §2.1.

Method

Overview. The model has three parts §3.1 fig. 4:

1. Action encoder. A frozen Qwen2.5-VL reads the input text and gives the last-layer hidden states as the action condition. A trainable connector projects them into the transformer §3.1.
2. State encoder/decoder. A Wan-VAE encodes frames into latents and decodes predicted latents back to frames §3.1.
3. Transition function. An MMDiT with two streams: an understanding stream (text tokens) and a generation stream (noisy video latents). Both streams attend jointly at every block §3.1.

Positional encoding. 3D RoPE splits the 128-dimensional head into 16 dimensions for time and 56 each for height and width. The authors also apply Scalable RoPE for other resolutions and durations at inference §3.2.

Conditioning (Scene2Robot). For standard first-frame image-to-video, the first frame's latents get timestep 0 and are excluded from the loss, and the text is read through the MLLM §3.3. For human-to-robot transfer, the input has three contiguous segments of equal length F: the human demonstration with hands masked, a simulated robot rendered in MuJoCo, and the noisy generation target. The first two get timestep 0 and no loss, so only the generation segment is trained. Each segment gets its own temporal index range through 3D RoPE §3.3 fig. 5.

Training objective. Flow matching on video latents. Noise is drawn from a standard normal. Timesteps are sampled from a log-normal distribution, shifted according to sequence length. For TI2V the first-frame timestep is fixed at 0 §4.2.

Captions. A five-layer annotation chain (task goal, action detail with mandatory viewpoint declaration, physical feedback, then a comprehensive caption of 50 to 100 words and a concise caption of 15 to 30 words). The two caption lengths are sampled with equal probability during training §2.1 §2.3.3. A four-stage processing pipeline (raw collection, video preprocessing, annotation, quality filtering with an LLM judge and human review) produces the corpus §2.3.

# Flow-matching training step for one batch  [§4.2] [§3.1]
# Input:  video (B, F, 3, H, W); captions (B,)
# Output: scalar loss

def training_step(video, captions):
    # 1. Encode the caption with the frozen MLLM  [§3.1]
    text_states = frozen_mllm(captions)            # (B, L, d)
    text_tokens = connector(text_states)           # trainable  [§3.1]

    # 2. Encode frames into VAE latents  [§3.1]
    clean = vae_encode(video)                      # (B, N, C)

    # 3. Sample noise and one timestep per sample  [§4.2]
    noise = random_normal_like(clean)              # standard normal
    t = sample_lognormal_timestep(batch_size, N)   # shift formula not stated

    # 4. Build the noisy input and the flow target  [§4.2]
    noisy = interpolate_path(clean, noise, t)      # path form not stated
    target = flow_target(clean, noise)             # form not stated

    # 5. Predict and apply the loss  [§4.2]
    # First-frame (TI2V) latents get t = 0 and are masked out
    pred = mmdit(noisy, text_tokens, t)
    loss = masked_loss(pred, target, loss_mask)    # loss form not stated
    return loss

# Build the three-segment input for human-to-robot transfer  [§3.3] [fig. 5]
# Input:  human_video (F, ...); robot_video (F, ...)
# Output: tokens, timesteps, loss_mask

def build_h2r_sequence(human_video, robot_video, F):
    # 1. Scene condition: human demo, hands masked, VAE-encoded  [§3.3]
    scene = vae_encode(masked(human_video))        # F frames

    # 2. Robot reference: MuJoCo render, VAE-encoded  [§3.3]
    robot = vae_encode(robot_video)                # F frames

    # 3. Generation segment: noisy latents to denoise  [§3.3]
    gen = random_normal_like(F_latents)

    # Conditions get t = 0 and no loss; only gen is trained  [§3.3]
    tokens = concat_temporal(scene, robot, gen)    # 3D RoPE per segment
    timesteps = concat(zeros(2 * F), sample_t(F))
    loss_mask = concat(zeros(2 * F), ones(F))
    return tokens, timesteps, loss_mask

Pretraining (general). Joint training on text-to-image, text-to-video and text-image-to-video over general data. The ratio shifts from pure T2I to the full three-task mix. Real-world general data comes from 14 video platforms, plus first-person hand-manipulation video (Ego4D, EPIC-Kitchens) §4.1.1.

SFT (embodied). Embodied data is added in four phases: single-view manipulation, multi-view expansion, multi-view concatenated generation, then complex tasks and cross-domain data. General data stays in every batch. The mix is 70% embodied and 30% general. Within embodied data, manipulation gets about 90% of sampling weight, while multi-view concatenation and navigation/driving get about 5% each §4.1.2.

Data composition (EWK corpus, about 8.6M pairs, over 200M frames):

The MANO pipeline extracts 3D hand keypoints, retargets them to end-effector trajectories, inpaints out hands, and renders robot arms by MuJoCo inverse kinematics. Each episode yields four aligned streams §2.2.5.

Hyperparameters

Results

Author-reported. The setup is the authors' own evaluation on four public benchmarks, with the baselines named below. Each benchmark's own evaluator is used as defined in its paper; the paper does not give evaluator versions beyond naming Qwen2.5-VL for DreamGen and PBench's domain QA §5.1.

Headline numbers from the paper:

• EWMBench overall 4.60, 1st, over LVP at 4.05 §5.1.1 tab. 2.
• DreamGen Bench total 4.952, 1st, over Wow at 4.728 §5.1.2 tab. 3.
• WorldModelBench total 8.99, 3rd overall, 1st among open-source models; trails only Wan2.6 and Veo3, which the paper calls closed-source §5.1.4 tab. 5.
• PBench overall 0.804, best among open-source models; domain 0.857 (3rd overall) §5.1.3 tab. 4.

Baselines: general video models Sora2, Veo3, Wan2.6, Kling, LTX-2; embodied models Cosmos, WoW, LVP, Vidar, GigaWorld §5.1.

EWMBench (21 samples across 7 tasks) §5.1.1 tab. 2

The paper does not define how the Overall column is computed from the sub-scores.

DreamGen Bench (GR1 embodiment, instruction following IF and physics alignment PA) §5.1.2 tab. 3

The Total equals the sum of the six sub-scores, checked for Ours and Cosmos-sft.

PBench (quality is the mean of eight VBench metrics; overall is the mean of quality and domain) §5.1.3 tab. 4

The Overall = (Qual. + Domain) / 2 relation holds for every row I checked, including Ours and Veo3.

WorldModelBench (instruction 0-3, common sense, physics) §5.1.4 tab. 5

Total = Instr. + CS overall + Phys. overall, checked for Ours and Veo3.

Qualitative and zero-shot results (no numbers beyond the text):

• Fine-grained grounding: pairs of prompts that differ in one keyword (object, destination, or action type) from the same first frame produce visibly different videos §5.2.1 fig. 6.
• Cross-embodiment: the same text prompt produces motion on four robot types, covering a gripper arm, two arms, a humanoid and a hand with fingers; multi-view outputs stay consistent across main and wrist cameras §5.2.2 fig. 7.
• Zero-shot comparison on four Unitree G1 tasks against LVP and Cosmos2.5-14B: the paper describes LVP as more often incomplete and Cosmos2.5-14B as weaker in instruction alignment in complex cases §5.2.3 fig. 8.
• RoboTwin-IF (a benchmark the paper introduces, built on RoboTwin): strong zero-shot results are reported in text and in qualitative cases; no table of numbers is given §5.2.3 fig. 9.
• Human-to-robot transfer across eight target embodiments §5.3 fig. 10; autonomous driving and indoor navigation generation §5.3 fig. 11.

Paper vs code

No official code.

Searches: the paper, its bibliography and the abstract contain no repository or project-page link for this model. The GitHub repository search returned nothing for "qwen-robotworld", "RobotWorld" with the org filter for QwenLM, or the title keyword "world org:QwenLM" (which lists other QwenLM world-model repos, such as Qwen-AgentWorld, BlenderWorld and WebWorld, none described as this paper's code). Those are different projects, so they are not used.

Pitfalls

• Driving sizes disagree. The driving subsection gives 1,744,405 clips (2,405 h) §2.2.3. The corpus statistics give "about 200K" navigation and driving samples §2.3.4, and the introduction gives about 200K driving samples §1. The corpus mix cannot be recomputed from these numbers without knowing which count is right.
• Pretraining size unit is unclear. Pretraining uses "over 200M real-world observation samples" §4.1.1, while the EWK corpus is described as "over 200M observation frames" §2. Whether these are frames or clips is not stated.
• Corpus total. The introduction gives the manipulation share as about 5.9M §1; the corpus statistics give about 4.3M single-view plus about 1.6M multi-view, which sums to about 5.9M §2.3.4. The overall 8.6M includes the 30% general share §2.
• Embodied sampling weights are approximate. "About 90%" and "about 5% each" §4.1.2 are the only weights stated. The paper gives no per-phase step counts, so how long each phase lasts is unknown.
• Human-to-robot counts. The data section cites 14 arm models §2.2.5; the qualitative figure says eight target embodiments §5.3 fig. 10. These describe different sets (training vs display), and the number of H2R pairs in the MANO-based set is not stated.
• Caption sampling. The two caption lengths are sampled 50/50 §2.1. The LLM judge and the annotation model are not named, and its thresholds and the human-review share are not stated §2.3.4.
• Open-source labels are not in the tables. The text calls Wan2.6 and Veo3 closed-source §5.1.4. The tables do not label other models open or closed, so "best open-source" claims (WorldModelBench, PBench) depend on a classification the paper does not show in the tables.
• EWMBench is small. It has 21 samples across 7 tasks §5.1.1. Its Overall score is not defined in the paper, and the model does not lead every column (Dyn is below Sora2 at 0.3494 tab. 2; Diversity is below Cosmos tab. 2).
• Resolution. The paper says its output resolution is lower than general generators, which lowers VBench image-quality scores §5.1.3, but gives no resolution value.
• Figure numbers here follow TeX order (the teaser in the abstract is Figure 1) and were not checked against a compiled PDF.
• Missing for reproduction: learning rate, optimizer, batch size, total steps, GPU count, sampler and steps, guidance scale, VAE latent size, value of F, the timestep shift formula, the connector's architecture, and the RoboTwin-IF data release.
• Equations are unnumbered in the source; the state-transition formula appears in prose §1 §2.1. Cite by section.
• Release status: the paper names an internal robot dataset (Qwen-Aloha) tab. 1 and does not say which parts of the corpus, the annotation prompts' models, or the model weights are released.

Lineage

• Builds on: the MMDiT transformer [esser2024scaling], Wan-VAE [wan2025wan], Qwen2.5-VL as frozen encoder [Qwen2.5-VL], 3D RoPE [su2024roformer, heo2024rotary], flow matching [lipman2023flow, liu2022flow], Megatron-LM [shoeybi2019megatron]. Scene2Robot extends first-frame image-to-video conditioning §3.3.
• Compared against: general video models Sora2, Veo3, Wan2.6, Kling, LTX-2; embodied world models Cosmos, WoW, LVP, Vidar, GigaWorld §5.1.
• Evaluated on: EWMBench [yue2025ewmbench], DreamGen Bench [dreamgen], PBench [nvidia2025pbench], WorldModelBench [worldmodelbench], and the authors' RoboTwin-IF.