MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

In plain words

Teaching a game character to do a task from a sentence usually means someone writes a score for each task by hand, for example a score that rewards getting closer to an animal. This paper tries to skip that step. It takes videos of Minecraft players with their spoken narration or subtitles and learns which short clip matches which sentence. That learned matcher then scores what the character sees while it plays, and the character improves from those scores with ordinary trial-and-error learning.

The approach is aimed at tasks with no clear finish line, such as building a house or finding an ocean. For those, the same matcher also judges finished attempts, and its judgements line up closely with human judges on the four tasks the paper tests.

It is a research proof of concept. The matcher is large and slow to query, and the video it learns from looks different from the game's own graphics, which the paper notes as a gap it did not close with special techniques.

TL;DR

MineDojo is a Minecraft framework with a simulator, a task suite and a knowledge base built from videos, wiki pages and forum posts §1 §2 §3. This card covers its agent-learning part §4: MineCLIP, a video-text contrastive model trained on 640K video-transcript pairs §4.1, gives a dense reward to a PPO agent with self-imitation §4.2. Author-reported results are competitive with hand-made rewards on 8 Programmatic tasks and well above a CLIP-guided baseline on 4 Creative tasks §5 tab. 1. The released repository contains the model and reward wrappers, but not the pre-training loop or the RL trainer.

Problem

Embodied agents are usually trained from scratch in narrow environments and generalise poorly across tasks §1. The paper's aim is an agent that follows open-ended language tasks in Minecraft §1 §4.

For many of these tasks a dense reward is hard or impossible to write by hand §4. Creative tasks (for example "find a nether portal" or "dig a hole") have no programmatic success test, so the reward must also serve as an evaluator §5 app. C app. G.1. The card's scope is this reward-learning and RL method §4; the benchmark and knowledge base are described in §2 and §3.

Assumed setting: one language-conditioned policy trained with reinforcement learning on several tasks at once, with raw pixels as input and discrete control actions as output §4. Each task prompt is a short English goal string app. G.1.

Method

The method has two parts: a video-text model pre-trained on YouTube data (MineCLIP), and an RL loop that uses the model's score as the reward §4.

MineCLIP pre-training §4.1

• Data. The authors take 640K pairs of video snippets and time-aligned English transcripts, after a keyword filter, following the VideoCLIP procedure §4.1. The steps are: build a keyword list of Minecraft entities, blocks and items; string-match the transcripts; grow each matched transcript to 16–77 tokens; pick a random timestamp inside the match as the clip centre; grow the clip to 8–16 seconds app. E.1. The main text calls the clips "16-second" snippets §4.1; see Pitfalls.
• Video encoder. A frame encoder φ_I (ViT-B/16, 512-d embedding per frame) runs on each frame, then a temporal aggregator φ_a summarises the frame sequence, then two residual CLIP-adapter layers are applied §4.1 app. E.2. The frame encoder starts from OpenAI CLIP's public weights, and only its last two layers are fine-tuned §4.1 app. E.2. Inputs are 160 × 256 pixels, and the positional embeddings are bicubically interpolated to that size app. E.2.
• Aggregator variants. MineCLIP-avg uses average pooling over time, which is fast but ignores frame order. MineCLIP-attn uses a 2-layer transformer encoder with 512-d embeddings, 8 heads and a GLU variant with Swish activation §4.1 app. E.2. The paper says MineCLIP-attn gives a better reward in general §4.1; Table 1 does not show that for every task (see Paper vs code).
• Text encoder. Initialised from CLIP's 12-layer, 512-wide text encoder (8 heads, byte-pair vocabulary of 49,152, at most 77 tokens); only its last two layers are fine-tuned app. E.2.
• Objective. An InfoNCE contrastive loss pulls each clip towards its own transcript §4.1. The paper does not give the exact form (for example whether it is symmetric or what temperature is used).

# Pre-train the video and text encoders on matched pairs  [§4.1] [app. E.2]
def pretrain_step(videos, transcripts):
    # videos: (B, 16, 3, 160, 256) snippets; transcripts: B strings
    frame_features = frame_encoder(videos)          # (B, 16, 512)
    video_embeddings = aggregator(frame_features)   # (B, 512)
    text_embeddings = text_encoder(transcripts)     # (B, 512)
    # Pull each clip towards its own transcript, push others away  [§4.1]
    return info_nce_loss(video_embeddings, text_embeddings)

RL with the MineCLIP reward §4.2

• Reward. The learned reward is a function of a goal string G and a clip V that returns one real number §4.1. During RL, the snippet is the agent's latest 16 egocentric RGB frames, and each task prompt is used zero-shot with no further fine-tuning §4.2.
• Two reward forms. "Direct" turns the probability of the goal among the goal and its negative prompts into a reward: the chance level 1/N_T is subtracted and the result is clipped at zero. A variant without the clip is called Direct-Naive. "Delta" uses the change in goal probability since the previous step app. F.1. Direct is used for Animal-Zoo and Mob-Combat, and Delta for Creative app. G.3.
• Speed-ups. The model has about 150M parameters, so the authors (1) compute the goal's text features once per task, (2) reuse the frozen frame encoder as the agent's RGB encoder, (3) cache frame features shared by overlapping windows, and (4) add self-imitation learning §4.2.
• Policy network. Inputs are the frozen frame feature (512-d), the goal text feature, sine and cosine of yaw and pitch through an MLP, GPS through an MLP, the 3×3×3 voxel block names embedded and flattened, and the previous action app. F.2. The fused features feed a policy head and a value head, each a 3-layer MLP with 256 hidden units app. F.2 app. G.3.
• Actions. 89 discrete choices: 81 camera moves (a 9 × 9 grid of yaw and pitch from −60° to 60° in 15° steps), 6 movement actions, and "use" and "attack"; no-op is merged into the camera set app. G.2.
• PPO. The policy is trained with PPO and GAE advantages; the clip parameter is ε = 0.2 §4.2 app. G.3 eq. 1 eq. 2.
• Self-imitation. Each task has a buffer. A trajectory is kept if it succeeded, or if its return is above a threshold based on the buffer's mean and standard deviation. Successful trajectories are sampled with equal probability, and the others with a probability that grows with return. Training alternates PPO updates with supervised updates on the buffers §4.2 app. F.3 alg. 1.
• Action smoothing. An extra KL term over a sliding window of action distributions is added to the PPO objective app. G.3 eq. 3.
• Multi-stage training. Animal-Zoo and Creative are each split into two stages of two tasks, sharing one policy. The buffers carry over, and the policy head is reset at stage 2. Mob-Combat is trained in one stage app. G.3.

# Direct reward: goal probability above chance, clipped at zero  [app. F.1]
def direct_reward(goal_probability, num_prompts):
    # 1 / N_T is the chance level among the prompts  [app. F.1]
    chance = 1.0 / num_prompts
    return max(goal_probability - chance, 0.0)

# Delta reward: change in goal probability since the last step  [app. F.1]
def delta_reward(goal_probability, previous_probability):
    return goal_probability - previous_probability

# Interleave PPO with self-imitation on good trajectories  [alg. 1]
def train_ppo_si(policy, value_fn, tasks, threshold, frequency):
    # one self-imitation buffer per task  [app. F.3]
    buffers = {task: [] for task in tasks}
    env_steps = 0
    while not finished():
        # roll out the current policy on every task  [alg. 1]
        rollouts = collect_rollouts(policy, tasks)
        for task, trajectory in rollouts.items():
            # keep successes, and returns above mean + threshold * std  [alg. 1]
            if is_good(trajectory, buffers[task], threshold):
                buffers[task].append(trajectory)
        env_steps += steps_taken(rollouts)
        # clipped PPO update, then value regression  [eq. 1] [eq. 2]
        ppo_update(policy, value_fn, rollouts)
        fit_value_function(value_fn, rollouts)
        # every `frequency` env steps, imitate the buffers  [alg. 1]
        if env_steps % frequency == 0:
            # use the smallest buffer's size from every buffer  [alg. 1]
            count = min(len(buffer) for buffer in buffers.values())
            # successes equally likely; others weighted by return  [app. F.3]
            batch = prioritized_sample(buffers, count)
            self_imitation_update(policy, batch)

Hyperparameters

MineCLIP pre-training app. E.3 app. E.2 §4.1:

RL training app. G.3 app. F.2 (table values for each task group):

Other values from the text:

• Manual navigation reward for Milk Cow: λ_nav 10, λ_success 200; the navigation term is based on geodesic distance from privileged LIDAR app. G.1.
• Action smoothing loss: L_smooth = (1/|W|) Σ_{i=1}^{|W|−1} KL(π_t ‖ π_{t−|W|+i}) app. G.3 eq. 3.
• Creative-task evaluation: K-means with K = 2 on trajectory scores, threshold set at the mean of the two centroids app. G.4.
• Hardware for RL, the number of parallel simulators, and the total environment budget are not stated.

Results

Author-reported. Setup: 12 tasks from the MineDojo suite: 8 Programmatic tasks in two groups (Animal-Zoo, Mob-Combat) and 4 Creative tasks §5 app. G.1. Each entry is a success rate in percent, averaged over 3 seeds with 200 test episodes per seed tab. 1 §5. Baselines: a manual dense reward using privileged simulator state app. G.1, a sparse success-only reward, and OpenAI CLIP as the reward model §5. For Creative tasks, success is judged by MineCLIP itself tab. 1.

Table 1: success rate (%), mean ± std over 3 seeds tab. 1

Table 2: F1 of MineCLIP as a success judge on Creative tasks, against human labels (%) tab. 2

Each task has 100 successful and 100 failed trajectories labelled by humans app. G.4.

Table 3: zero-shot generalisation to unseen weather, lighting and terrain (27 scenarios) tab. 3

Numbers are success rates in %, averaged over 3 seeds with 200 episodes each. Brackets give the relative change from the training scenarios. The OpenAI CLIP baseline here was trained by imitation learning, not RL §5.

Table 4: one agent for all 12 tasks tab. 4

Success rate in %, mean ± std over 3 seeds with 200 episodes each. The "original" column repeats the Ours (Attn) column of Table 1. The 12-task agent is trained by self-imitating successful trajectories from the self-imitation pipeline §5.

Table 5: novel tasks, zero-shot and after RL fine-tuning tab. 5

Success rate in %, mean ± std over 3 seeds with 200 episodes each. The baseline was trained from scratch with RL using the same MineCLIP encoders and reward. Fine-tuning used about 5% of the total environment samples §5.

Statistical tests reported by the authors: a paired Student's t-test over the 8 Programmatic tasks gives p = 0.3991 for Ours (Attn) against the manual reward, which the authors read as "not significantly different" §5. A second paired test gives p = 0.3720 for the 12-task agent against the group-trained agents §5. The introduction's "up to 73%" improvement is a relative gain on Hunt Cow (83.5 against 48.3) §1 tab. 1.

Paper vs code

Repository: https://github.com/MineDojo/MineCLIP at commit e6c06a0245fac63dceb38bc9bd4fecd033dae735, MIT licence license. The MineDojo repository's README (https://github.com/MineDojo/MineDojo, the paper's codebase app. B) links this repository as its MineCLIP reward model and agent code, which ties it to the authors.

What the repository contains: the MineCLIP model and its CLIP-style components, a checkpoint loader (pretrained weights are hosted externally), the MineAgent policy network, reward wrappers for the Animal-Zoo and Mob-Combat tasks, and small demo scripts. The README says the in-loop demo runs a random policy README.md.

What is not in the repository: the pre-training loop and loss, the optimiser and schedule, the PPO and self-imitation trainer, the Direct and Delta reward code, and the action-smoothing loss. The only imitation loss is a helper on the actor distribution distribution.py; it is not the self-imitation loop. So the training hyperparameters come only from the paper.

Discrepancies found:

• Action space. The paper uses 89 flat discrete actions app. G.2. The demo policy uses a multi-categorical actor with head sizes [3, 3, 4, 25, 25, 8] run_env_in_loop.py. The two do not match.
• Hunt Cow reward values. The paper gives λ_attack 5, λ_nav 1, λ_success 200 app. G.1. The demo script uses nav_reward_scale 0.1, attack_reward 1 and success_reward 10 run.py.
• Mob-Combat reward values. The paper gives λ_attack 5 and λ_success 200 app. G.1. The demo uses attack_reward 1 and success_reward 10 run.py.
• Navigation reset after an attack. The paper says d_min is reset every time the agent hits the target app. G.1. The code resets the running minimum after a valid attack dense_reward.py, but it does not clear the two-entry distance history, so the first navigation term after a hit compares against the pre-hit distance.
• Demo action mapping. In the demo's transform_action, the condition action[-1] != 0 or action[-1] != 1 or action[-1] != 3 is always true, so the last action component is always set to 0 run_env_in_loop.py.
• Reward form. The paper's Direct form is max(P_G − 1/N_T, 0) app. F.1. The released code exposes a softmax over the prompt scores in the reward head base.py, but not the clip or the Delta form.
• Input normalisation (not in paper). The released code uses fixed per-channel mean (0.3331, 0.3245, 0.3051) and standard deviation (0.2439, 0.2493, 0.2873) computed from 21K clips base.py. The paper does not state these values.
• Logit scale (not in paper). The released code initialises the logit scale at log(1/0.07) and clamps it to at most 100 clip.py clip.py. The trained value is not in the repository.

Items that agree:

• Aggregator: attn.d2.nh8.glusw means 2 layers, 8 heads, GLU with Swish, matching §4.1 and app. E.2 attn.yaml transformer.py.
• Adapters: "v0-2.t0" means no image adapter before pooling, 2 video adapter layers after pooling and no text adapter conf.yaml. The residual weight starts at 4.0, so its sigmoid is about 0.98, close to identity head.py, matching app. E.2.
• Input resolution 160 × 256 and ViT-B/16 conf.yaml conf.yaml.
• Policy and value heads: 256 hidden units, depth 3 conf.yaml, matching app. G.3.

No discrepancies were checked for the feature-fusion depth: the paper lists one fusion layer app. G.3 and the code sets hidden_depth 0 with output 512 conf.yaml; these may describe the same layer but this was not verified.

Pitfalls

• Training is not reproducible from the repository. The pre-training loss, optimiser, data loader and PPO trainer are absent. An implementer must choose the InfoNCE form, the temperature, the optimiser (not stated), and how layer-wise decay groups layers (not stated) from scratch.
• Snippet length is inconsistent inside the paper. The main text describes "16-second" snippets §4.1, while the appendix grows clips to 8–16 seconds around a centre point app. E.1.
• Frame timing is not stated. Training samples 16 frames uniformly per video app. E.3. At RL time the reward uses the latest 16 frames from the simulator §4.2. The frame rate and spacing are not stated for either case. Training uses no special domain adaptation for the gap between YouTube frames and simulator frames §4.2; zero-shot visual robustness is reported in tab. 3.
• Negative prompts are not specified. Direct reward depends on the set of negative goals. The appendix gives one example ("milk a cow", "hunt a sheep", "hunt a cow" for "shear a sheep") app. F.1, and does not say how negatives are chosen for each task.
• Prompt wording is part of the reward. The Mob-Combat prompt template is "Combat {monster}" app. G.1. The paper does not examine sensitivity to wording.
• Self-imitation threshold. The buffer threshold is 2σ for two groups and 0.5σ for Creative app. G.3. Whether μ and σ are computed over the whole buffer or a recent window is not stated.
• Self-imitation schedule depends on counting. The update fires when the step counter is a multiple of the frequency alg. 1. If the counter is advanced in chunks, the check may fire late or skip. The paper says only that the counter is "increased accordingly".
• Action-smoothing loss is ambiguous. The formula averages over |W| but sums only |W| − 1 terms, and it is written as KL(π_t ‖ past). Whether it is summed or averaged over the batch is not stated. The weight is 1e-7 app. G.3 eq. 3, so its effect is small.
• Multi-stage training resets part of the policy. Stage 2 resets the policy head, and buffers carry over app. G.3. The appendix ablations cover only the reward form and self-imitation app. F.1 app. F.3, so the effect of the reset is not measured.
• Compute cost. The reward model has about 150M parameters and is queried at every step §4.2. The caching tricks in §4.2 reduce, but do not remove, this cost. Hardware and environment counts are not stated.
• Variant choice is mixed. The paper says MineCLIP-attn gives a better reward "in general" §4.1. In Table 1, MineCLIP-avg is higher on 4 of the 12 tasks: Combat Zombie, Find Nether Portal, Find Ocean and Lay Carpet.
• "Outperforms manual reward" applies to different variants. The text says the agents outperform the hand-engineered reward on three tasks §5. In Table 1, Ours (Attn) is above the manual reward on Milk Cow (64.5 against 62.8, well within the spread) and Hunt Cow, and Ours (Avg) is above it on Combat Zombie. The manual reward uses privileged simulator state and LIDAR app. G.1, so this is not a like-for-like comparison.
• Statistical power is low. The paired t-test compares 8 task means and gives p = 0.3991. A non-significant result does not show equivalence. Standard deviations are large (for example 64.5 ± 37.1), and there are only 3 seeds.
• Table 3 baseline is not comparable with Table 1. The OpenAI CLIP numbers in Table 3 come from imitation-trained agents, while the Table 1 CLIP column comes from RL §5. The two columns are not interchangeable.
• Table 4 uses a different recipe. The 12-task agent is trained mainly by self-imitating successful trajectories §5, not with the per-group RL recipe.
• Creative evaluation is narrow. Table 2 covers only 4 tasks, each with 200 labelled trajectories. Five more tasks in the appendix use 50 segments each, and the authors call the agreement "not perfect" app. G.4.
• Zero-shot success on new tasks is near zero. Hunt Pig and Harvest Spider String reach 1.3% and 1.6% without fine-tuning tab. 5.

Lineage

• Builds on: CLIP, for the contrastive objective and the initial encoder weights §4.1; CLIP4Clip, for the video architecture §4.1; VideoCLIP, for the sampling procedure §4.1 app. E.1; CLIP Adapter app. E.2; PPO §4.2; and self-imitation learning §4.2. No arXiv identifiers are recorded here.
• Compared against: a manual dense reward, a sparse success-only reward, and OpenAI CLIP as a reward §5 tab. 1.
• Superseded by: not stated in this wiki yet.