GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

In plain words

Can a coding agent make a small video game from a written description, in a real game engine, so that a person could actually play it? This paper builds a test for that. It gives agents 140 game briefs, each for a small game in the Godot engine, and asks for a full game project back.

Judging a game is harder than judging ordinary code. The test starts the game, plays back short recorded sequences of mouse and keyboard input, and records the screen. A model that looks at the recordings then checks each requirement from a hidden checklist, such as whether the core mechanic works and whether the screen is readable.

The best agent tested scores about 41%. Agents often build the basic rules of a game, but the results are usually thin: little content, weak feedback on screen, and a rough presentation.

TL;DR

• GameCraft-Bench has 140 Godot tasks across 15 game families, each with a game specification, a hidden rubric and a Godot environment §3.1 §3.4.
• Submissions must launch, and must include replayable input traces that the verifier plays back and scores with a multimodal judge §3.2.
• Seven agent configurations were run; the best, Claude Code with Opus-4.7 high, reaches 41.46% overall. Author-reported tab. 4.
• The judge is GPT-5.5 by default. The repository includes a verifier that implements the scoring pipeline as described, with a few differences (see Paper vs code).

Problem

Coding-agent benchmarks usually judge code correctness, unit tests or patches §6. The authors argue that a game can only be judged by playing it, and that three things are needed: an engine-grounded environment, a complete launchable artifact, and interactive verification §2.2. Existing benchmarks each cover only part of this: OpenGame-Bench targets web games without gameplay judging, GameDevBench uses Godot but only for localized edits, and WebGameBench uses browser interaction but not an engine §1 tab. 1.

Method

The paper defines the task as a triple of specification, environment and hidden rubric §3.1. The agent observes the specification and the environment, but not the rubric. It must deliver a pair (G, Π): a complete Godot project G and a set Π of replayable traces. The verifier replays Π against G, producing observations O, and scores O against the rubric §3.1. The paper gives these definitions as unnumbered equations.

The implementation follows a five-stage pipeline §3.2 fig. 5:

1. Task packaging. Each task holds a game specification, a Godot workspace with assets and docs, and a rubric. The rubric's items fall into four groups, abbreviated M, D, V and A: the core rules, depth of content, readability and feedback on screen, and presentation style §3.2 Stage 1.
2. Agent generation. The agent may create scenes, write scripts, configure input, run the project, take screenshots and iterate. It must place the project and its traces in the required locations §3.2 Stage 2 app. C.
3. Build gate. The verifier launches the project in headless Godot. If the project fails to launch, the final score is zero §3.2 Stage 3.
4. Replay. For each trace, a fresh Godot instance replays the mouse and keyboard events at 1280×720. The verifier records a video and samples frames two per second §3.2 Stage 4.
5. Scoring. A multimodal judge scores each demo independently against the rubric items. Items are aggregated across demos: scenario-specific items take the maximum, persistent items take the mean. The four category scores are combined into the final score, which is zero if the build fails [§3.2 Stage 5] fig. 5.

The final score is:

\mathrm{Score} = \mathrm{BUILD} \times (0.15\,M + 0.35\,D + 0.15\,V + 0.35\,A) §3.2 Stage 5

where M, D, V and A are the Core Mechanics, Content Depth, Functional Visuals and Art and Presentation scores. The paper gives these weights as defaults.

# Score one submission  [§3.2]
# [code: gamecraft_bench/verifier/score.py#L85-L268]
# Input:  project (Godot files), traces (*.json), rubric (requirements)
# Output: reward in [0, 1]

def score_submission(project, traces, rubric, judge):
    # Build gate: a failed launch scores zero  [§3.2 Stage 3]
    build = 1.0 if launches(project, rubric.build_cmd) else 0.0
    if build == 0.0:
        return 0.0
    # Keep at most max_demos traces, first by filename  [app. A]
    chosen = sorted(traces)[: rubric.max_demos]
    scores = {}
    for trace in chosen:
        # Replay at 1280x720, 30 fps, in a fresh Godot instance  [app. A]
        video = replay(project, trace, viewport=(1280, 720), fps=30)
        # 2 fps from one window of at most 20 s  [§3.2 Stage 4]
        frames = sample_frames(video, every=0.5, max_seconds=20)
        # One judge call per demo scores every requirement  [§3.2 Stage 5]
        scores[trace] = judge.score(frames, rubric.requirements)
    item = {}
    for req in rubric.requirements:
        # max items: best demo; others: mean over demos  [§3.2 Stage 5]
        values = [scores[t][req.id] for t in chosen]
        item[req.id] = max(values) if req.agg == "max" else mean(values)
    total = 0.0
    for category in rubric.categories:
        # Category score: plain mean of its items  [rubric score_formula]
        cat = mean(item[i] for i in category.items)
        total = total + category.weight * cat
    return clamp(build * total, 0.0, 1.0)

Trace format. A trace is a JSON file with an optional scenario id, a total length in frames, and a time-ordered list of events: clicks, key presses, mouse moves and waits. Up to ten traces per submission are evaluated, first by filename app. A app. C.

Annotation. Twelve annotators wrote each task as a public specification and a hidden rubric, using the Harbor framework §3.4. Each rubric has at most 24 items §3.4. An annotator also wrote a minimal oracle solution to check that the task is implementable and that every rubric item can be seen in a replay §3.4.

Judge prompt. The judge gets the system instruction and a list of requirements, and must reply with strict JSON with one score per requirement on a 0 to 1 scale app. A.

Hyperparameters

Values as stated in the paper unless marked as code. "not stated" means the paper does not give a value.

Results

Author-reported. Setup: seven agent configurations on all 140 tasks, scored with GPT-5.5 as judge §4 tab. 4. Scores are in percent. The overall score is the final score; the four columns are category scores.

Family-level scores for all seven configurations are in tab. 6 app. B.

Other reported results:

• Build and trace rates. Most strong agents pass the build gate and produce valid traces. DeepSeek-V4-Pro often ignores the trace requirement §4 fig. 6.
• Judge stability. Ten repeated GPT-5.5 runs on fixed evidence give standard deviations of 0.0037 and 0.0038 for Kimi-K2.6 (Card Game and Simulation), and 0.0050 and 0.0036 for Opus-4.7 high on the same two families §5.2 fig. 9.
• Human calibration (preliminary). On Kimi-K2.6 submissions from three families (Card Game, Idle, Racing), human overall score 22.69 vs judge 26.02. The authors call it a calibration check, not an agreement study §5.2 tab. 5.
• Tool use. MiMo-V2.5-Pro spent 56.3% of its tool calls on shell commands; tool-call volume and score correlate at r = +0.016 §5.1 fig. 8.
• Category correlations. For Kimi-K2.6, Core Mechanics correlates with Content Depth at r = 0.61 and with Functional Visuals at r = 0.53; Art and Presentation correlates with Functional Visuals at r = 0.11 §5.3 fig. 10.

Paper vs code

Repository: tongxuluo/gamecraft-bench (Apache-2.0), linked from the abstract through the project page, owned by the first author. Commit d64cc5d817bd331a840a378e6e42c873b1139055. The repo contains the verifier (gamecraft_bench/verifier/), the judge backends, 141 task folders (140 plus an example), and the run scripts. Nothing in the repo checks the judge-stability or human-calibration analyses; searching it for calibration, stability and repeat-run code found none.

Differences:

• Build gate and traces. The paper says the gate also fails (BUILD = 0) when no submitted trace can be parsed §3.2 Stage 3. The code only lists *.json files in demo_outputs/ and checks the build command. A trace that fails to replay is logged as an error and skipped, and BUILD stays 1 score.py [code: gamecraft_bench/verifier/score.py#L145-L149] [code: gamecraft_bench/verifier/score.py#L173-L175] [code: gamecraft_bench/verifier/score.py#L260]. The reward is still zero, because no requirement gets a score, but BUILD is not zero.
• What the judge sees. The paper says the judge gets "the resulting videos and frames" §3.2 Stage 4. The default OpenAI backend sends sampled frames only openai_gpt.py; the Claude backend also sends frames [code: gamecraft_bench/verifier/judges/claude.py#L1-L4]. The Gemini backend sends the whole mp4 gemini.py.
• Frame cap. The judge backends keep at most 40 frames, thinned evenly openai_gpt.py [code: gamecraft_bench/verifier/judges/openai_gpt.py#L59-L63] [code: gamecraft_bench/verifier/judges/claude.py#L21] [code: gamecraft_bench/verifier/judges/kimi.py#L17]. The paper does not mention a cap. At 2 fps over a 20 s window the count is exactly 40, so the cap does not thin any demo in the shipped tasks, but it would for a longer window.
• Category weights. The paper gives 0.15, 0.35, 0.15, 0.35 as the default weights §3.2 Stage 5. Four rubrics use 0.20 for Core Mechanics and 0.30 for Content Depth, for example [code: tasks/visualnovel-arcaneacademy/tests/rubric.json#L2]. Each rubric carries its own formula; all other rubrics use the paper's weights.
• Category score. The paper does not say how items combine into a category score. The rubric formulas use the plain mean of the items in each category rubric.json.
• Missing aggregation field. A requirement without an agg field is scored as max score.py. The paper states the rule only by item type §3.2 Stage 5.
• Judge failure. After five failed attempts, all requirements of that demo score zero, and the failure is only recorded as an error [code: gamecraft_bench/verifier/score.py#L205-L225]. The paper does not describe this case.
• Replay length. The recording runs for the trace length plus 2 s [code: gamecraft_bench/verifier/replay.py#L218]. Not in the paper.
• Judge prompt. The system instruction matches app. A [code: gamecraft_bench/verifier/judges/_common.py#L24-L32]. The requirement list is formatted as - id: description lines [code: gamecraft_bench/verifier/judges/_common.py#L47-L48].
• Task suite. The paper's family counts (for example 19 platformer and 17 strategy) match the task folders. No discrepancy found.

Pitfalls

• Replay timing is wall-clock. The paper says replay is deterministic app. C §3.2 Stage 4. The code posts each input at a real-time offset from the start of recording replay.py. Same-trace runs can differ if the machine is loaded. The paper's stability test covers the judge, not the replay.
• Judge dependence. Scores come from one judge model (GPT-5.5 in the paper) with no stated sampling settings app. A. Changing the judge backend changes the input the judge sees (frames for OpenAI and Claude, full video for Gemini) and so may change the scores. The stability test covers two families only §5.2.
• Judge errors score zero. A failed judge call yields zeros, not a missing value score.py. Check the breakdown.json errors before trusting a low score.
• Window start is random. The 20 s window is drawn from a seed based on the demo id, so the sampled frames change if a trace is renamed [code: gamecraft_bench/verifier/score.py#L368-L374].
• Only the first 10 traces count. Extra traces are dropped by filename, not by quality app. A score.py.
• Visual evidence only. Audio is not evaluated. Rhythm, horror and sports games are judged on visual behaviour [Limitations].
• Not a fun measure. The benchmark measures whether the requested features appear in the replay; it does not measure whether the game is fun [Limitations].
• Human calibration is small. It covers three families and one agent §5.2 tab. 5.
• No variance for agent scores. The paper does not state how many runs each configuration got, and reports no variance for the agent scores §4 tab. 4.
• Judge only sees sampled frames. Under the default backend, a flicker or timing bug shorter than 0.5 s can be missed; the authors note that visual interpretation is the main source of residual noise §5.2.

Lineage

• Builds on: not stated. Related work is in §6.
• Compared against: OpenGame-Bench §1, GameDevBench §1, WebGameBench (described as concurrent work) §1 tab. 1. Their arXiv ids are not given in the paper text, so they are not linked.
• Uses: Harbor as the task-authoring and execution framework §3.4 app. A.
• Superseded by: not known from this wiki.