SIMA 2: A Generalist Embodied Agent for Virtual Worlds

In plain words

SIMA 2 is an AI agent that plays 3D video games the way a person does: it looks at the screen and presses keys and moves the mouse. It is built on Google's Gemini model, so it can also talk to the player, explain its plan, and follow longer or less direct instructions, such as a sketch on the screen or a list of steps.

Its training mixes recorded human play with text that explains what the player was doing and why. Its predecessor, SIMA 1, could only follow short, direct commands. The authors report that SIMA 2 does much better in the games it was trained on, and that it also works in games it has never seen. In one experiment it proposed its own tasks and scored its own attempts with a Gemini model, and it improved without new human demonstrations.

The paper is a report from a large lab, with no official code for this card. An engineer can learn the ideas from it, but cannot rebuild the system from this paper alone.

TL;DR

• A Gemini Flash-Lite agent that acts in 3D games through the same inputs a human uses: 720p frames in, keyboard and mouse out §3.2.
• Trained by supervised fine-tuning on human gameplay plus Gemini-written reasoning and dialogue, then by online RL on verifiable tasks §3.3.
• Author-reported: about twice SIMA 1's average success on training environments fig. 6; over 10% gain on ASKA and MineDojo held-out tasks §4.2.2.
• Self-improvement: Gemini sets tasks and scores 0 to 100 with a rubric; 50 or more counts as success §4.5.
• Not reproducible from the paper: no official code, and the training data, base model size and RL details are not stated.

Problem

Foundation models trained on static text, images and video are passive and do not learn from the consequences of actions §1. SIMA 1 followed short, direct instructions, could not produce text, could not take multimodal prompts, and often failed to generalise to new situations §1 §4.1. The paper asks how an agent can combine a general model's reasoning with embodied control, and how it can keep improving without task and reward design by hand §1 §2.

Method

The agent is a single policy. It takes a stream of frames and text, and it emits text and keyboard-and-mouse actions, and it chooses which of these to produce at each step §3.2 fig. 3. The base is a Gemini Flash-Lite model §3.3. Training has two stages: supervised fine-tuning on one mixed dataset of human gameplay, bridge data and Gemini pre-training data §3.3, then online RL §3.3.3. The steps below follow the paper's order.

1. Interface §3.2 fig. 3.

1. Periodically, the agent receives the latest RGB frame at 720p and adds it to its history §3.2.
2. The history also holds earlier natural-language inputs and the agent's own reasoning and responses §3.2.
3. The agent outputs chunks of actions, which are applied to the environment §3.2.
4. Output is structured text that parses deterministically into key, mouse-click and mouse-move commands, plus natural language for dialogue or reasoning §3.2.
5. The action space is 96 standard keyboard keys, mouse clicks, and discretised relative (x, y) mouse moves §3.2.
6. The agent gets no privileged state from the game §3.2.

# SIMA 2 interaction loop  [§3.2] [fig. 3]
# Input:  instruction (text), RGB frames (720p), history (list)
# Output: text (dialogue or reasoning) and low-level action chunks

def interact(env, agent, instruction):
    # History holds frames, earlier text and the agent's own text  [§3.2]
    history = []
    # The stop rule for an episode is not stated  [§3.2]
    while episode_running(env):
        # Add the latest frame; the sampling interval is not stated  [§3.2]
        history.append(env.latest_frame())
        # One structured text output, from instruction and history  [§3.2]
        output = agent.generate(instruction, history)
        # Parse into text and into keyboard and mouse commands  [§3.2]
        text, commands = parse_output(output)
        # The agent's own text is kept in its history  [§3.2]
        history.append(text)
        # Apply the action chunk to the environment  [§3.2]
        env.apply(commands)

2. Human data §3.3.1. This is most of the training data by volume. Each trajectory holds instructions, frames, and keyboard-and-mouse actions at each step §3.3.1.

1. Single-person post-hoc annotation: a player plays freely and then describes the actions, aligned to frames. The paper says these labels are not causally tied to intent §3.3.1.
2. Setter-Solver pairs: a Setter gives live instructions to a Solver who controls the avatar. The instruction always comes before the action §3.3.1.
3. Game-tasks: a player gets an instruction from a predefined game state and ends the episode when done or at a time limit §3.3.1.
4. Human ratings: binary success labels and side-by-side comparisons of trajectories §3.3.1.
5. Offline processing: resizing, heuristic and score-based filtering, remixing and weighting by environment and dataset §3.3.1.
6. Trajectories are cut into "spans", each with one task instruction §3.3.1.
7. Gemini models add synthetic language and reasoning labels offline §3.3.1.

3. Bridge data §3.3.2. Human play has no reasoning or dialogue, so this set adds them.

1. Select a small set of high-quality examples covering all training environments. Each has one instruction, and frames and actions that complete it §3.3.2.
2. Gemini Pro writes internal reasoning and dialogue for each example. The text must match the egocentric view and the behaviour §3.3.2.
3. Vary the training prompt within examples for robustness §3.3.2.
4. Include no-op steps so the agent stays still after finishing a task §3.3.2.
5. Examples cover error correction, instruction chaining, visual question answering, memory use and long-horizon behaviour §3.3.2.

4. Supervised fine-tuning §3.3. Start from a pretrained Gemini Flash-Lite checkpoint and train it to output keyboard-and-mouse responses from frames and an instruction. Human data, bridge data and Gemini pre-training data are mixed §3.3. The paper says the non-gameplay data keeps vision, dialogue, reasoning and promptability §3.3. Output is trained as text, not as discrete action tokens §3.2.

5. Reinforcement learning §3.3.3. Applied after supervised training, and limited to training environments.

1. A task is a tuple of initial game state, text instruction and verification function §3.3.3.
2. Reward comes from completing the task or from a correct answer to a grounded question. Some tasks add shaped rewards §3.3.3.
3. Tasks come from three sources: contractor-written suggestions from random game states; verifier functions applied to human trajectories, paired with an earlier state; and dialogue tasks that pair screenshots with human-written question-answer pairs §3.3.3.
4. Tasks are kept only if a human could finish them within a time limit §3.3.3.
5. The RL algorithm, its settings and the reward shaping are not stated §3.3.3.

6. Hierarchical variant §4.4 app. B. A Gemini Pro model reads the recent video history every k steps and issues natural-language instructions to SIMA 2. Pro also writes a text summary that it receives on its next call, as a recurrent memory §4.4. The value of k is not stated §4.4.

7. Self-improvement §4.5 fig. 16a. The loop uses Gemini in two roles.

1. The task setter proposes instructions that are likely achievable from the current state, and can be steered by downstream results §4.5.
2. The agent attempts the task and produces a trajectory §4.5.
3. The reward model rates the video from 0 to 100 with a rubric covering task completion and directedness. The rubric was calibrated against human preference pairs over a small dataset §4.5.
4. A score of 50 or more counts as success §4.5.
5. The agent is trained on this self-generated, scored experience §4.5.

# Self-improvement round  [§4.5] [fig. 16a]
# Input:  agent, task_setter (Gemini), reward_model (Gemini), env
# Output: scored trajectories for training

def self_improvement_round(agent, task_setter, reward_model, env):
    # Proposed instruction, achievable from the current state  [§4.5]
    task = task_setter.propose(env)
    # The agent runs the task and produces a trajectory  [§4.5]
    trajectory = interact(env, agent, task)
    # Rubric score from 0 to 100; 50 or more is success  [§4.5]
    score = reward_model.rate(trajectory, task)
    # Returns the scored trajectory for training. How scored data becomes
    # a training update is not stated  [§4.5]
    return trajectory, score

Hyperparameters

Only the values the paper states are listed. The paper gives no optimiser, learning rate, batch size, step count, or hardware for any training stage.

Results

Author-reported. Setup: Google DeepMind, Gemini-based agent, research and commercial game environments, with one held-out set of games and one held-out Minecraft benchmark (MineDojo) §3.1 §4.2. The paper gives most numbers only in bar charts. The text states relative claims and a few percentages, listed below.

Qualitative only, with no counts given: The Gunk, first 15 to 20 minutes §4.2.2 fig. 12; Genie 3 photorealistic navigation §4.2.2 fig. 13; the hierarchy examples in app. B fig. 18 fig. 19; and the 21-questions game app. B.

Paper vs code

No official code.

The paper links no repository, and the arXiv page lists no code, project page or comments. Searches for "sima" within the google-deepmind GitHub organisation and for "SIMA embodied agent" on GitHub returned zero repositories. Nothing from a third-party reimplementation was used.

Pitfalls

• Not reproducible. The base model is a closed Gemini model, the data comes from human play under licensed agreements, and no code is released. Treat the card as a description of the approach, not an implementable spec.
• "Minor regression" versus Table 1. The text calls the capability loss "only a minor regression" §4.3. Table 1 shows -25.5% on AIME after SFT and -19.5% on GPQA after SFT+RL tab. 1. The caption calls the math and STEM reductions "modest" tab. 1. Report the table values, not the adjectives.
• Which base Gemini? The Table 1 baseline is "the baseline Gemini model" and its size is not stated tab. 1 §4.3.
• Exact success rates are not in the text. The headline "doubles" and "more than 10%" claims are stated without the underlying numbers. The bars in fig. 6, fig. 8, fig. 9 and fig. 10 carry the values, and the card does not transcribe them. Read the PDF if a number is needed.
• Human baselines differ. The training-environment baseline uses experienced players, with and without the agent's time limits §4.2.1. The held-out baseline uses naive players §4.2.2. The authors say human and agent failures differ in kind, and caution against a direct comparison §4.2.2. Some timeouts were as short as three seconds §4.2.1.
• Held-out MineDojo sizes differ. The benchmark subset is 50 tasks §3.1, but the human baseline uses 16 MineDojo tasks §4.2.2. The 26-of-50 claim refers to categories, not to the human subset §4.2.2.
• Reward model is opaque. The rubric text is not given, and the calibration set is "small" with no size §4.5. The 50-point threshold is the paper's choice, and no sensitivity analysis is reported §4.5.
• Self-improvement training is unspecified. The paper does not say how scored trajectories become a gradient update, whether failures are used, or how many rounds ran §4.5. The pseudocode above stops at the scored trajectory for this reason.
• Task setter and reward model sizes are not stated. The text says only "Gemini-based" §4.5. Using Gemini Pro for bridge data is the only model size given §3.3.2.
• Hierarchy cadence. The value k and the summary format are not stated §4.4. Results are qualitative and single-example app. B.
• Data mixture. The ratio of human to bridge to pre-training data is not stated §3.3. The Setter-Solver and single-person splits are not quantified §3.3.1.
• Task time limits. Per-task limits are not given except as "some as short as three seconds" §4.2.1. Use the environment-specific limits from the evaluation suite, which the paper does not list.
• Credit and affiliations. The arXiv metadata lists "SIMA team" as one author entry; the card keeps it as one entry with spacing trimmed. The paper's own author block is "SIMA Team, Google DeepMind".
• Figure numbers. Figure numbers here follow LaTeX source order, including wrapped figures and the two subfigures of Figure 16. Check them against the PDF before citing externally.
• Generalisation is shown in few cases. The Gunk and Genie 3 results are qualitative examples §4.2.2 fig. 12 fig. 13. The authors frame them as proof of concept §4.2.2.

Lineage

• Builds on: SIMA 1, cited as Scalable Instructable Multiworld Agent §2. Its changes are described in §1 and §4.1. Gemini is the base model §3.3. Genie 3 is the world model used for held-out photorealistic environments §2 §4.2.2. No arXiv id for these is recorded in this wiki.
• Compared against: SIMA 1 §4.2; baseline Gemini Flash-Lite and Pro without SIMA training §4.3; and naive and experienced human players §4.2.
• Superseded by: not stated.