Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model

In plain words

This paper builds a video model that plays the role of a game engine. You give it a single starting image and an instruction such as "draw a torch" or "open the door", or you press keys or move the mouse, and it generates the video that follows, with the scene and the action staying consistent over time.

Earlier game-video models mostly understood a fixed set of keys, such as forward, back, left and right. This work also reads free-form text, so a player can ask for something new to happen in the scene. The authors also argue that the training videos need to show a clear before-and-after change, so the model can learn cause and effect. They build such videos from gameplay footage and from synthetic clips.

To make the model fast enough to respond in real time, it is trained to produce frames one block after another and to remember the first frame. The authors report about 16 frames per second. The model still drifts on long sequences, and it handles one immediate action at a time, not multi-step plans.

TL;DR

• A 14B image-to-video mixture-of-experts model gets keyboard and mouse input as camera-motion signals and free-form text as an extra interaction signal §4.1.
• Self-Forcing-style distillation turns it into a few-step, causal, block-wise generator with a sink-token KV cache, followed by randomized long-video tuning §4.2.
• Author-reported: 16 FPS real-time generation §4.4, and the highest InterBench scores against Wan2.2 A14B, LongCat-Video and HunyuanVideo in all three categories tab. 5.
• Evaluation is in-house: InterBench uses a VLM as the judge, with an unnamed evaluator model §5.2.2.

Problem

Game world models that respond to input are usually limited to a closed set of keyboard or mouse actions §1 tab. 1. Free-form interactions, such as an object appearing or a state changing, need text conditioning that the base image-to-video model handles poorly. The authors state that base models struggle with interaction verbs, because interaction text is more complex than scene text and is tied to particular objects §4.1.

Two data problems are named. Interactive video data (clips with a clear start state, a transition and an end state) is scarce, and manual collection and labelling is costly §3. Long autoregressive rollouts drift and accumulate error §2 §4.2.4.

The paper assumes a strong pre-trained 14B image-to-video MoE base model §4.1 and a large gameplay corpus §3, tab. 2. Hardware for training and inference is not stated.

Method

The method has three parts: data construction (§3), model and training (§4.1 to §4.2), and inference (§4.3 to §4.4). Figure numbers below follow the order in which figures appear in the compiled paper.

Interactive video data §3. The authors define interactive data as a clip with a causally driven change from an initial to a final state, with a visible subject or a scene change §3. Two pipelines produce it:

1. Curation from gameplay footage from more than 150 AAA games §3, app. B. The pipeline cuts video into 6-second scene clips with PySceneDetect, locates action boundaries with RAFT optical flow, filters frames by quality, luminance and a VLM semantic check, reconstructs 6-DoF camera trajectories with VIPE, and writes two captions per clip: a standard caption and an interaction caption describing the change between neighbouring clips §3 fig. 2 fig. 3 fig. 4.
2. Synthesis from an initial frame §3. A VLM writes a scene-specific prompt from the frame and a high-level instruction. Then either an image-editing model produces a target end frame (for stationary scenes such as snowfall), or the video generator works from the first frame alone (for camera-moving actions such as opening a door). Start frames can come from a text-to-image model §3 fig. 2.

Model architecture §4.1. The base is a 14B image-to-video MoE model cited as Wan §4.1. Keyboard and mouse inputs are mapped to continuous camera-control parameters, following GameCraft-1 §4.1. During training, camera parameters are encoded as Plücker embeddings and added to the model as tokens; at inference, user inputs are converted into camera trajectories to derive those parameters §4.1. Text interaction information is produced by a multimodal LLM that extracts and reasons about the interaction, and is injected into the main model as extra guidance §4.1 fig. 5.

Training stages §4.2, tab. 2. The method has four stages:

1. Action-injected training §4.2.1. Full finetuning with a flow-matching objective, adapting the base model to the camera-conditioned architecture. Training uses three phases of 45, 81 and 149 frames at 480p. Long and short captions are sampled at random, and interaction captions are concatenated for interaction learning §4.2.1.
2. Instruction-oriented supervised finetuning §4.2.2. A 150K-sample set of real and synthetic interaction clips. The camera encoder is frozen; only the MoE experts are finetuned §4.2.2.
3. Autoregressive generator distillation §4.2.3. The bidirectional teacher is distilled into a few-step causal generator using Self-Forcing §4.2.3. The initial frame is kept permanently as a sink token in the KV cache, and the local attention uses block-sparse attention over preceding blocks §4.2.3 fig. 6. Distinct learning rates are assigned to the high- and low-noise experts, and the distillation timesteps are re-defined around the noise boundary that separates the two experts §4.2.3.
4. Randomized long-video extension tuning §4.2.4, alg. 1. Gameplay clips longer than 10 seconds are rolled out autoregressively with a random length. A random K-frame window is taken from the rollout. The student is conditioned on its own predicted frame before the window, and a fake-score model on the same frame; the real-score model is conditioned on the ground-truth frame before the window. The loss is a DMD loss. Self-forcing and teacher-forcing alternate, to teach recovery from many states §4.2.4.

Inference §4.3, fig. 7. Generation runs block by block. The KV cache is a fixed-length rolling window that keeps the sink tokens at the front plus a local attention window over the recent blocks §4.3. When a new prompt arrives, the ReCache step recomputes the last generated block with the new interaction embedding, updating both the self-attention and cross-attention KV caches §4.3.

Acceleration §4.4. FP8 quantization, parallelized VAE decoding, SageAttention in place of FlashAttention, and sequence parallelism across GPUs. Together these give 16 FPS §4.4 §1.

# Randomized Extended Long-Video Tuning  [alg. 1]
# Input:  generator (student) with parameters theta; a teacher pair
#         (real-score model, fake-score model); clip dataset; window
#         size K; max rollout length N_max; list of timesteps
# Output: updated generator parameters theta

def long_video_tuning_step(dataset):
    # Sample a ground-truth clip  [alg. 1]
    gt_video = sample(dataset)
    # Randomize the rollout length N, uniform in [K, N_max]  [alg. 1]
    rollout_length = sample_uniform(window, max_length)
    # Start from the first ground-truth frame, empty KV cache  [alg. 1]
    pred_video = [gt_video[0]]
    kv_cache = empty()
    # Step 1: autoregressive rollout, one K-frame chunk at a time  [alg. 1]
    for chunk_index in range(rollout_length // window):
        # Condition each chunk on the last K predicted frames  [alg. 1]
        prev_frames = last_frames(pred_video, window)
        # The sink token is always passed to the generator  [alg. 1] [§4.2.3]
        chunk, kv_cache = generator(prev_frames, kv_cache, sink_token)
        pred_video.append(chunk)
    # Step 2: pick a K-frame window uniformly from the rollout  [alg. 1]
    start = sample_uniform(1, rollout_length - window + 1)
    window_frames = pred_video[start : start + window - 1]
    # Step 3: student condition from the rollout, teacher condition from
    # the ground truth, both at the frame before the window  [alg. 1] [§4.2.4]
    student_cond = pred_video[start - 1]
    teacher_cond = gt_video[start - 1]
    # Step 4: add noise at a timestep drawn from the schedule  [alg. 1]
    t = sample_choice(timesteps)
    eps = normal(0, I)
    noisy = noise_schedule(window_frames, t, eps)
    # DMD loss: fake score on the student condition, real score on the
    # teacher condition  [alg. 1]
    loss = dmd(fake_score(noisy, t, student_cond),
               real_score(noisy, t, teacher_cond))
    # Gradient step on the generator with learning rate eta  [alg. 1]
    theta = theta - eta * grad(loss, theta)
    return theta

# ReCache on a new interaction prompt  [§4.3] [fig. 7]
# Input:  kv_cache (sink tokens + local window), prompt text
# Output: kv_cache with the last block recomputed

def on_new_prompt(prompt, kv_cache):
    # Extract the interaction embedding from the new prompt  [§4.3]
    interaction = encode_interaction(prompt)
    # Recompute the last autoregressive block with it, updating both
    # the self-attention and cross-attention caches  [§4.3]
    kv_cache = recompute_last_block(kv_cache, interaction)
    return kv_cache

Hyperparameters

Values are as stated in the paper. Anything not listed was not found in the paper's text, tables or appendix.

Baseline settings, as the authors ran them §5.1:

Test set composition §5.1, tab. 3:

Data pipeline settings §3: 6-second clips (PySceneDetect); RAFT optical flow for action boundaries; VIPE for 6-DoF camera trajectories; HunyuanImage-3.0 as the text-to-image source for start frames §3. Frame-filtering thresholds and the VLM and MLLM names beyond the citations are not stated.

Results

Author-reported. Setup: test set and metrics from §5.1 and §5.2; baselines are HunyuanVideo, Wan2.2 A14B and LongCat-Video, each run under the settings above. The VLM used as the InterBench judge is not named §5.2.2.

InterBench §5.3, tab. 5. Six ordinal dimensions per category (Trigger is binary). Values as reported:

The paper defines an Overall score as (5 × Trigger + Align + Fluency + Scope + EndState + Physics) / 6 §5.3. No Overall values are listed in the table.

General video quality §5.2.1, tab. 4. Compared against GameCraft, GameCraft-PCM, Matrix-Game and Matrix-Game-2.0. Table caption: "strong overall balance" tab. 4. Values as reported:

Hardware and resolution for these FPS values are not stated tab. 4.

Related-work comparison §2, tab. 1. Qualitative table of interactive world models, with resolution, training data, action type and capability columns. GameCraft-2 is listed at 480p, with gameplay, rendered and interaction data, key+mouse plus prompt-based instruction actions, continuous action space, open-ended action generalization, and scene dynamics, scene memory and real-time marked as supported tab. 1. Other rows are as reported in that table.

Qualitative §5, fig. 8 to fig. 16. Qualitative comparisons with the three baselines, in environment, actor and entity categories, plus generalization to unseen entities such as a dragon, a phone and a person appearing §5.3 fig. 15. Long-video tuning and cache settings are shown as an ablation fig. 16.

Paper vs code

No official code.

Searches, in the order given in the skill: (1) the arXiv abstract and the paper's own links give only the project page https://hunyuan-gamecraft-2.github.io/ [main.tex]; (2) the project page, read with WebFetch, links no code repository, weights or model hub entries; (3) GitHub search for "Hunyuan-GameCraft-2" returned only Hunyuan-GameCraft-2/Hunyuan-GameCraft-2.github.io, a GitHub Pages site repo with no license, which was not cloned; a search for "GameCraft" in organisation Tencent-Hunyuan returned only Tencent-Hunyuan/Hunyuan-GameCraft-1.0, the code of the predecessor paper GameCraft-1 §4.1, not this paper, so it was not used.

Pitfalls

• Training hyperparameters are missing. Optimizer, learning rate, batch size, per-expert learning rates, the number of denoising steps in the distilled model, CFG scale, DMD loss weights and the randomized-rollout constants K, N_max, L and timesteps are not stated §4.2 alg. 1. An implementer must choose these, and the reported results depend on them.
• Final cache sizes are not stated. The ablation compares sink 1 / local 6 against sink 3 / local 9 fig. 16, but the unit of "local attention size" (frames or blocks) and the block size of block-sparse attention are not given §4.2.3.
• Algorithm symbols disagree with the text. The text uses T for window length §4.2.4, while the algorithm uses K alg. 1. The algorithm's Require line lists a cache size L that is never used alg. 1. The window index i and the slice V_pred[i:i+K-1] are written 1-based and inclusive; the paper does not say how an implementation should index.
• "Interleaved" forcing is ambiguous. The text says self-forcing and teacher-forcing are interleaved §4.2.4, but does not say whether this alternates per iteration, per sample, or per window. The algorithm uses both conditions within a single step alg. 1.
• Test-set size is inconsistent. The text describes 100 images plus 20 door images §5.1. Table 3 lists per-subset prompt counts that sum to more than that (100 per environmental subset alone) tab. 3. Whether subsets share images is not stated.
• Resolution differs between training and evaluation. Training is at 480p §4.2.1 and Table 1 lists 480p for GameCraft-2 tab. 1. Evaluation uses 832×448 §5.1. The paper does not discuss this difference.
• Metric caveats. Table 4 does not show GameCraft-2 best on FVD (GameCraft 1554.2 is lower than 1856.3) or on Dynamic Average (GameCraft 67.2 is higher than 45.2) or Image Quality (Matrix-Game 0.72) tab. 4. The caption's "strong overall balance" is the authors' framing.
• The judge is not named and not validated here. InterBench uses a VLM judge §5.2.2 whose identity, prompts and agreement with human raters are not stated. The Overall score weights Trigger 5× §5.3. The Trigger value is binary; other dimensions are on a 0-1-3-5 ordinal scale app. D. A failed Trigger zeroes the other dimensions app. D.
• Baselines are run by the authors under their own settings §5.1. The FPS values of baselines and their hardware are not stated tab. 4.
• Long-horizon limits. The authors report semantic drift beyond about 500 frames, no explicit long-term memory beyond the KV cache, single-step immediate-effect actions only, and that 16 FPS is not yet low-latency enough for highly reactive play §6.
• Cost hidden in the speed claim. The speed-up uses FP8, a different attention kernel, parallel VAE decoding and sequence parallelism across GPUs §4.4. The GPU model and count behind "16 FPS" are not stated, so the speed cannot be compared to other systems.
• Figure cross-reference. The method figure caption refers to "Figure 7.(a)" for inference fig. 7, which matches the compiled figure order used in this card.

Lineage

• Builds on: Wan image-to-video MoE base model §4.1; GameCraft-1 for the keyboard/mouse-to-camera mapping §4.1; Self-Forcing for autoregressive distillation §4.2.3; DMD for the long-video tuning loss §4.2.4; LongLive for the KV-recache idea §1; Block-Sparse Attention and SageAttention for kernels §4.2.3 §4.4.
• Compared against: Table 1 lists GameNGen, Oasis, GameGen-X, Matrix, Matrix-Game, Genie 2, GameFactory, GameCraft and Genie 3 tab. 1. Quantitative comparisons are against GameCraft, GameCraft-PCM, Matrix-Game and Matrix-Game-2.0 tab. 4, and against Wan2.2 A14B, LongCat-Video and HunyuanVideo on InterBench tab. 5.
• Superseded by: not known.