---
kind: concept
slug: model-based-reinforcement-learning
title: Model-based reinforcement learning
summary: >-
  Choosing actions with a model of the environment: plan over a model you estimate or learn, or train a policy on imagined rollouts or evolved controllers. Exact planning is cheap only for small state spaces. Learned world models cover several domains; one reported setup needs up to 16 GPU days per benchmark.
tags: [reinforcement-learning, planning, world-models]
papers: ["1701.02490", "1803.00101", "1803.10122", "1809.01999", "2203.01387", "2301.04104", "2405.18418", "2411.02446", "2411.14499", "2512.02417", "2512.03556", "2512.18850"]
related: [actor-critic, policy-gradient, model-predictive-control, multi-layer-perceptron, neuroevolution, value-iteration, learned-simulator, world-model, variational-autoencoder, hierarchical-reinforcement-learning, reward-design, intrinsic-motivation, goal-conditioned-reinforcement-learning, farthest-point-sampling, offline-reinforcement-learning]
status: draft
updated_at: 2026-11-01
---

## Overview

A model-based agent predicts what its actions will do and uses those
predictions to choose actions, instead of learning only from real trial and
error. The agent cards on this page use a model in different ways, and they
differ in what the model is and how the policy is found. In
[Real-Time Bidding by Reinforcement Learning in Display Advertising](/papers/1701.02490)
the model is the estimated distribution of market prices, and the state space
is small enough to plan exactly [arxiv:1701.02490 §2, §4]. In
[Mastering Diverse Domains through World Models](/papers/2301.04104) the model
is a learned recurrent world model, and the actor and critic are trained on
rollouts the model imagines [arxiv:2301.04104 tab. 4]. In
[Recurrent World Models Facilitate Policy Evolution](/papers/1809.01999) a
convolutional VAE and an LSTM form the model, and a single linear controller is
searched by evolution against the real reward, which the model never sees
[arxiv:1809.01999 §2, §3]. [World Models](/papers/1803.10122) describes the
same three-part agent, with the same Car Racing score [arxiv:1803.10122 tab. 2].
[Hierarchical World Models as Visual Whole-Body Humanoid Controllers](/papers/2405.18418)
is the one card here that plans at decision time: it runs sampling-based model
predictive control over a learned model, with a frozen low-level tracker
underneath [arxiv:2405.18418 §2, §3].

[Learning World Models for Unconstrained Goal Navigation](/papers/2411.02446)
keeps a Dreamer-style model and changes the data it learns from. The model is
trained on episodes that travel between subgoals taken from the agent's own
replay buffer, in both directions and across trajectories, which are the
transitions a model otherwise generalises to poorly [arxiv:2411.02446 §1,
§3.1].

[Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101)
uses the model in a narrower place. It never trains the policy on imagined
rollouts. The actor is updated on real transitions, and the model only builds
the target that the critic is trained toward, by looking a fixed number of
steps ahead [arxiv:1803.00101 §3, alg. 1 lines 9–12].

The survey [Understanding World or Predicting Future?](/papers/2411.14499)
gives a common frame. It treats the world model as the transition dynamics of
a Markov decision process, learned by supervised fitting of one-step
predictions, and names planning uses such as model predictive control, which
picks the action sequence with the highest predicted return over a horizon
[arxiv:2411.14499 §3.1, eq. 1, eq. 3]. The survey reports no experiments of its
own [arxiv:2411.14499 §1–§7].

When the data is a fixed log, the model can be exploited where the log is
thin. The offline RL survey [A Survey on Offline Reinforcement Learning](/papers/2203.01387)
files model-based methods under one class, which penalises model rollouts by
an uncertainty measure [arxiv:2203.01387 §III-E, §IV-E]. See the offline
variant below.

The shared structure is a loop: fit or estimate the model, use it to evaluate
future outcomes, and update the decision rule. The variants differ in how each
step is done. Most learned-model agents on this page do not search at decision
time. They train a policy on model rollouts, and the policy then acts directly:
[arxiv:2301.04104 §Critic learning], [arxiv:1809.01999 §2],
[arxiv:2512.03556 §3.2], [arxiv:2512.02417 §4.2.1],
[arxiv:2512.18850 §3.3] and [arxiv:2411.02446 §3.1]. Only the humanoid card
plans with the model at each step [arxiv:2405.18418 §2].

## Variants

### Exact planning over an estimated model

Fill a value table backwards over (steps left, budget left), then bid by a
threshold on the value gained from each price. The paper treats one ad
campaign as a Markov decision process whose state is the auctions left, the
budget left and the features of the current request [arxiv:1701.02490 §3.1].
Market prices are estimated from data by counting [arxiv:1701.02490 §2]. Card:
[Real-Time Bidding by Reinforcement Learning in Display Advertising](/papers/1701.02490).

The table costs O(TB) time and memory, where T is the episode length and B the
budget [arxiv:1701.02490 §4.1]. The paper replaces the table with a small
network that fits the value differences, with two tanh hidden layers of 30 and
15 units [arxiv:1701.02490 §4.1, §6.2]. The network is only checked inside the
range it was trained on [arxiv:1701.02490 tab. 4]. Reported gain: 16.67% more
clicks than the Lin baseline at budget ratio 1/8 on iPinYou, averaged over nine
campaigns [arxiv:1701.02490 tab. 2].

### Learned world model with imagined rollouts

Train a recurrent world model on replayed experience, then train an actor and a
critic on rollouts of 15 steps imagined from the model [arxiv:2301.04104 tab. 4].
The agent acts by sampling from its actor, with no lookahead search at decision
time [arxiv:2301.04104 §Critic learning]. Card:
[Mastering Diverse Domains through World Models](/papers/2301.04104).

The paper keeps one configuration for every benchmark. Its main devices are
symlog inputs, a categorical critic over 255 exponentially spaced bins trained
on twohot targets, a return scale from the 5th to 95th percentile, and free
bits on the KL terms [arxiv:2301.04104 tab. 4] [arxiv:2301.04104 §Robust
predictions]. Compute per benchmark is 0.1 to 16.1 A100 GPU days, depending on
the suite and the model size [arxiv:2301.04104 tab. 2]. Reported results
include a Minecraft Diamond return of 9.1 against 7.1 for IMPALA at 100M steps
[arxiv:2301.04104 tab. 5] and an Atari100K gamer mean of 125 against 105 for
IRIS [arxiv:2301.04104 tab. 9]. The paper's own comparison notes that the
Atari100K baselines do not all follow one protocol.

### Goal-conditioned model trained on subgoal episodes

Keep a goal-conditioned Dreamer-style agent, and change the data it learns
from. Besides the usual episodes toward environment goals, each iteration
collects episodes that visit two subgoals drawn from a set of key states. The
set is picked by farthest-point sampling over the actions in recent episodes
[arxiv:2411.02446 §3.1, §3.2] [alg. 1] [alg. 2]. The model is trained on both
kinds of episode, and the policy is trained only in imagination, with reward
equal to minus a learned temporal distance [arxiv:2411.02446 eq. 7] [alg. 1].
Card: [Learning World Models for Unconstrained Goal Navigation](/papers/2411.02446).

The paper reports the lowest one-step prediction error among the methods it
compares, on all six tasks. Compounding error is lowest on five of the six. On
Fetch Slide the ablation without subgoal sampling is slightly lower
[arxiv:2411.02446 tab. 6] [arxiv:2411.02446 tab. 7]. On 3-Block navigation
across 225 start-goal pairs, MUN reaches 95% against 56% for GC-Dreamer, but
the goals there are chosen by hand [arxiv:2411.02446 tab. 3] [arxiv:2411.02446
§4.5]. Stacking success is above 95% against about 60% for the baselines
[arxiv:2411.02446 §4.3]. One run takes 36 to 70 GPU hours on an A100
[arxiv:2411.02446 tab. 1]. The released code does not follow the paper's
sampler on every task: its 3-Block, Walker and Block Rotation configs pick
subgoal states at random instead, which the card records as a discrepancy
[code: Config/configs.yaml#L262]. Results on those tasks may not come from the
method as written.

### Learned world model with an evolved linear controller

Train a convolutional VAE to encode each frame into a latent vector, then an
LSTM whose mixture-density output predicts the next latent. The controller is
one linear layer on the latent and the LSTM state, with 867 parameters for car
racing, and it is searched with CMA-ES against the real return
[arxiv:1809.01999 §2, §3] [arxiv:1809.01999 app. A.1–A.5]. Card:
[Recurrent World Models Facilitate Policy Evolution](/papers/1809.01999).

The world model is trained on frames and actions from random rollouts, and the
reward is used only in the controller search [arxiv:1809.01999 §3]
[arxiv:1803.10122 §3.1]. The parameter counts are 867 for the car controller
and 1,088 for Take Cover [arxiv:1803.10122 tab. 1] [arxiv:1803.10122 tab. 3].
The card [World Models](/papers/1803.10122) gives the same setup and the same
Take Cover temperature table, with τ = 0.10 at 2086 virtual and 193 real
[arxiv:1803.10122 tab. 4]. For car
racing the controller is trained in the real environment, and the full model
reaches 906 ± 21, against 838 ± 11 for the Gym Leader baseline
[arxiv:1809.01999 tab. 1] [arxiv:1809.01999 §3.1]. For DoomTakeCover the
controller is trained only inside the learned model at sampling temperature
1.15, and it scores 1092 ± 556 in the real game [arxiv:1809.01999 tab. 2]
[arxiv:1809.01999 §4.2]. The standard deviation is large. The authors report
that the temperature controls how well the controller transfers: at 0.10 the
virtual score is 2086 but the real score is 193 [arxiv:1809.01999 tab. 2].

### Planning over a learned model at each step

Two TD-MPC2 world models are trained separately. A low-level tracker, pretrained
once on 836 motion-capture clips and then frozen, follows end-effector targets
on a 56-joint simulated humanoid. A visual high-level planner, trained from
scratch for each task on the task reward, outputs those targets from 64×64
camera images [arxiv:2405.18418 §3, §3.1, §3.2, §4.1]. Each action is chosen by
sampling candidate action sequences, scoring them with the model's predicted
return over a short horizon, and refitting the sampling distribution over a few
iterations [arxiv:2405.18418 §2, tab. 5]. Card:
[Hierarchical World Models as Visual Whole-Body Humanoid Controllers](/papers/2405.18418).

On eight tasks the planner's return is comparable to TD-MPC2 except on stairs,
where TD-MPC2 scores higher [arxiv:2405.18418 §4.2] [fig. 5]. SAC and DreamerV3
did not learn within 3M steps in the same setting [arxiv:2405.18418 §4.2]. In a
51-person study on the five visual tasks, 97.8% of judgements preferred the
planner's motion, against 0.4% for TD-MPC2 [arxiv:2405.18418 tab. 6]. The cost
is inference time: 88.2 ms per step against 50.8 ms for TD-MPC2
[arxiv:2405.18418 tab. 3]. The authors report that planning at both levels is
critical [arxiv:2405.18418 §4.3]. The benchmark numbers are author-reported.

### Learned world model as the RL environment, with a learned reward

Train a world model that predicts the next camera frame from the robot's
action, and a second one that imagines the finished scene from an instruction.
The policy is then trained with a standard RL algorithm inside these models,
with a reward built from the imagined scene [arxiv:2512.03556 §3.1, §3.2].
The reward has two parts: a sparse term that fires when the predicted done
signal crosses a threshold, and a dense term that measures the perceptual
distance between the current frame and the imagined goal frame
[arxiv:2512.03556 eq. 5, eq. 7, eq. 8]. The paper uses PPO
[arxiv:2512.03556 §4.3]. Card:
[RoboScape-R](/papers/2512.03556).

The reported gain is on unseen scenes. With an OpenVLA policy, out-of-domain
pick-and-place reaches 84.1% with the model reward, against 54.5% for RL with
the simulator's reward and 47.5% for supervised fine-tuning [arxiv:2512.03556
tab. 1]. In-domain differences are small, at most 3.0 points by the card's
arithmetic on the same table [arxiv:2512.03556 tab. 1]. Training the world
model took about 96 hours on an H20 cluster [arxiv:2512.03556 §4.2]. All
evaluation is in simulation [arxiv:2512.03556 §5].

### Imagined rollouts with a separate vehicle model

A Dreamer-style agent for driving splits the ego vehicle's dynamics from the
rest of the environment. A parameterised vehicle model predicts the car's own
state from its control input, and a separate environment model conditions on
that state [arxiv:2512.02417 §4.1.1, eq. 9]. The actor outputs a goal rather
than a raw action, and a predefined controller turns the goal into an action
[arxiv:2512.02417 §4.2.1]. Two mechanisms handle changed car parameters. At
deployment, the original policy's action is mapped through the inverse of the
new vehicle model, so the new car reaches the states the old car would reach.
During training, imagined rollouts are generated under both the nominal and
the shifted parameters [arxiv:2512.02417 §4.3, §4.4]. The first mechanism
cannot help when the new car cannot reach those states, for example with a
smaller maximum steering angle [arxiv:2512.02417 §4.3]. Card:
[Vehicle Dynamics Embedded World Models for Autonomous Driving](/papers/2512.02417).

On MetaDrive's roundabout the success rate is 0.70 against 0.53 for DreamerV3.
On the intersection it is 0.88 against 0.91 for DreamerV3, so the baseline is
slightly better there [arxiv:2512.02417 tab. 1]. In CARLA's unprotected left
turn the success rates are 0.94 and 0.91 [arxiv:2512.02417 tab. 1]. The
robustness results under changed mass and steering are shown only as figures
[arxiv:2512.02417 fig. 9]. The paper states that real-world performance is open
[arxiv:2512.02417 §6].

### Reward-free pretraining of the world model

Pretrain a world model and an exploration policy with no task reward, using
the disagreement of an ensemble of next-state predictors as an intrinsic
reward. Then test the frozen explorer in a town it has not seen, and fine-tune
a task policy with 10k steps of task reward [arxiv:2512.18850 §3.2.1, §3.4,
§4.3]. The ensemble has eight predictors [arxiv:2512.18850 tab. 2]. Card:
[InDRiVE](/papers/2512.18850).

Zero-shot success on Town01 (seen) is 75.65% for this method, against 62.58%
for ICM and 68.58% for RND. On Town02 (unseen) it is 73.20%, against 48.61%
and 55.83% [arxiv:2512.18850 tab. 6]. After fine-tuning, collision avoidance
reaches 96.43% on Town01 and 94.78% on Town02 [arxiv:2512.18850 tab. 6]. For
few-shot lane following, RND has the smaller gap between the two towns
(0.73 against 3.92) [arxiv:2512.18850 tab. 6]. The setting is discrete
steering and throttle on semantic-segmentation input, with transfer tested only
from Town01 to Town02 [arxiv:2512.18850 §4.1] [arxiv:2512.18850 §6].

### Model rollouts only for critic targets

Fit a one-step dynamics model on replayed transitions. At each update, roll the
model forward H steps from a replayed state under the target policy, add up the
discounted rewards along the way, and bootstrap from the critic at the end. The
actor is trained with its usual loss on the real transition, so imagined states
enter only the critic's targets [arxiv:1803.00101 §3, alg. 1 lines 9–12]. The
critic is also trained on imagined states, with each imagined step given a
target of horizon H minus t, which the paper calls the TD-k trick [arxiv:1803.00101
§3.2]. The imagined rewards come from the true reward function, not from a
learned one, so the method needs that function [arxiv:1803.00101 §3]. The
horizon is chosen by hand per task: H = 10 for cheetah and swimmer, and H = 3
for walker, where the figure caption says the model class is inadequate at
H = 10 [arxiv:1803.00101 fig. 3 caption]. Card:
[Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101).

The paper reports learning curves only, with no numeric returns in the text
[arxiv:1803.00101 §4.1, fig. 3]. In the authors' comparison, MVE with the TD-k
trick learns fastest on cheetah, swimmer and walker, against DDPG, MVE without
TD-k and an imagination-buffer baseline. On walker the gain over DDPG is
smaller at H = 3 [arxiv:1803.00101 fig. 3]. Without the TD-k trick, large
horizons hurt on oracle dynamics [arxiv:1803.00101 fig. 6]. The learned-dynamics
runs use 4 seeds [arxiv:1803.00101 app. A]. Unlike the Dreamer-style variant
above, this method does not learn its policy from imagined rollouts.

### Offline model-based penalties (survey view)

With a fixed dataset, a learned model can be right on the logged states and
wrong elsewhere, and a policy that exploits those errors does badly. The survey
groups the methods that guard against this under the model-based class. MOReL
penalises state-action pairs where an ensemble of models disagrees beyond a
threshold. MOPO uses the Frobenius norm of the model covariance as its
uncertainty. COMBO adds a conservative value penalty over model rollouts and
dataset points, mixed by a fraction f. BREMEN trains a policy inside an ensemble
of models under a trust-region constraint [arxiv:2203.01387 §III-E, §IV-E]
[eq. 31–38]. The survey gives no numeric results for these methods, and it
states no value for the MOReL threshold, the COMBO α or the BREMEN trust-region
ε [arxiv:2203.01387 §IV-E] [eq. 32] [eq. 35] [eq. 36]. It notes that the COMBO
data fraction changes how conservative the Q-function is [arxiv:2203.01387
§IV-E] [eq. 38]. Card: [A Survey on Offline Reinforcement Learning](/papers/2203.01387).

For off-policy evaluation rather than control, the survey reports that model-based
and fitted Q evaluation performed best across most metrics on the DOPE benchmark,
but that no method was best in every setting [arxiv:2203.01387 §VI-B1]. This is a
statement about scoring a policy, not about the learned policy.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Small state space you can enumerate, and a model you can estimate from logs | Exact planning over the estimated model | Exact values under the model, at O(TB) time and memory [arxiv:1701.02490 §4.1] |
| State too large for a table, but a training run is affordable | Network approximation of the value table, or a learned world model | The table is replaced by a network, but its error is reported only inside the trained range [arxiv:1701.02490 §6.2, tab. 4] |
| Many domains, no time to tune per task | Learned world model with the fixed configuration | One set of values across benchmarks [arxiv:2301.04104 tab. 4] |
| Tight budget of environment steps (400K) | Learned world model | Higher Atari100K mean than the listed baselines, with the protocol caveat [arxiv:2301.04104 tab. 9] |
| Control tasks with little compute | Learned world model at the 12M size | Proprioceptive control at 0.3 and visual control at 0.1 GPU days [arxiv:2301.04104 tab. 2] |
| Ad campaign bidding with a fixed budget | Exact planning, or its network variant | The setting of the only bidding paper on this page [arxiv:1701.02490 §3.1] |
| Low-dimensional linear policy, and you can run many rollouts in a simulator or learned model | Learned world model with an evolved linear controller | 867 parameters searched with a population of 64 and 16 rollouts per candidate [arxiv:1809.01999 app. A.5] [arxiv:1803.10122 §A.4] |
| Policy must work in the real environment, but real trials for training are limited | Controller trained inside the learned model, with the temperature tuned on the real score | Real score 1092 ± 556 at τ = 1.15, but 193 ± 58 at τ = 0.10 [arxiv:1809.01999 tab. 2] |
| Decision-time lookahead wanted over a learned model, with continuous actions and a shared low-level controller | Sampling-based planning over the learned model, with a frozen tracker below | Preferred by 97.8% of judges against TD-MPC2's 0.4% on five visual tasks, at about 1.7 times the inference time per step [arxiv:2405.18418 tab. 3, tab. 6] |
| Robot task where a hand-written reward is hard to specify across tasks, and many demonstrations and scenes exist | Policy trained in a learned world model with a model-built reward | Gains reported on unseen scenes only, all in simulation, with the world-model training cost [arxiv:2512.03556 tab. 1, §4.2] |
| Driving with changing car mass or steering limits | Vehicle-aware world model with the inverse-map deployment step | Robust in the reported mass and steering grid, but only in simulation [arxiv:2512.02417 §4.3, fig. 9] |
| Goal-reaching tasks where the model must predict well backward and across trajectories, and 36 to 70 GPU hours per run are affordable | Goal-conditioned model trained on subgoal episodes | Lowest one-step error among the compared methods on six tasks [arxiv:2411.02446 tab. 6]; navigation gains use hand-picked goals [arxiv:2411.02446 tab. 3] [arxiv:2411.02446 §4.5] |
| Reward design is costly, and a town change must not break the policy | Reward-free pretraining, then a short task fine-tune | Zero-shot success on the unseen town is higher than with ICM or RND, with one town pair [arxiv:2512.18850 tab. 6] |
| Continuous control with an actor-critic, a known reward function, and a model you trust only for a few steps | Model rollouts only for critic targets, with a short horizon | Learning curves that favour the method over DDPG on three MuJoCo tasks, with H chosen by hand per task [arxiv:1803.00101 fig. 3] [fig. 3 caption] |
| Offline data only, and the model may be wrong where the log is thin | Model-based penalty, as in the survey's model-based class | The survey names the penalty types but gives no numbers, and leaves several thresholds unstated [arxiv:2203.01387 §IV-E] [eq. 32] |

The headline numbers are author-reported, and the papers use different
baselines and metrics. Choose on the constraints above, not on the headline
numbers alone.

## Implementation sketch

Six sketches, one per variant that has a clear core loop. None is a full
program; the comments give the source of each step.

```pseudocode
# Exact planning over an estimated model (sketch)  [alg. 1] [arxiv:1701.02490 §4]
# m[d]: probability that the market price equals d  [arxiv:1701.02490 §2]

def value_table(m, T, B, expected_value):
    # With no steps left, no further value  [arxiv:1701.02490 §4]
    V = zeros(T + 1, B + 1)
    for t in range(1, T + 1):
        for b in range(B + 1):
            # Best expected value over the bids this budget allows  [alg. 1]
            V[t][b] = max(
                expected_value(m, V, t, b, a) for a in range(b + 1))
    return V
```

```pseudocode
# Learned world model with imagined rollouts (sketch)  [arxiv:2301.04104 tab. 4]

def train_step(model, actor, critic, replay):
    # Sample stored real experience  [arxiv:2301.04104 §Replay]
    batch = replay.sample()
    # Fit the world model to real transitions  [arxiv:2301.04104 eq. 2]
    model.update(batch)
    # Imagine 15 steps from each real start state  [arxiv:2301.04104 tab. 4]
    imagined = rollout(model, actor, batch.states, horizon=15)
    # Bootstrapped returns over the imagined rewards and critic values  [eq. 5]
    returns = lambda_returns(imagined, critic, gamma=0.997, lam=0.95)
    # Train the critic on those returns, then the actor on the advantage  [eq. 6]
    critic.update(returns)
    actor.update(returns - critic(imagined.states))
```

```pseudocode
# Learned world model with an evolved linear controller (sketch)
# [arxiv:1809.01999 §2] [arxiv:1809.01999 app. A.5]
# env: the real game (car racing) or the learned model (DoomTakeCover)

def train_world_model(random_frames, random_actions):
    # Fit the frame encoder on frames from random rollouts, no reward  [app. A.2]
    vae = train_vae(random_frames)
    # Fit the recurrent model to predict the next latent  [app. A.3]
    mdn_rnn = train_mdn_rnn(vae.encode(random_frames), random_actions)
    return vae, mdn_rnn

def evolve_controller(env, vae, mdn_rnn, generations):
    # Search 867 linear weights with CMA-ES  [app. A.5]
    search = cma_es(num_params=867, population=64)
    for generation in range(generations):
        candidates = search.ask()
        # Score each candidate by its mean return over 16 rollouts  [app. A.5]
        scores = [mean_return(env, vae, mdn_rnn, w, rollouts=16)
                  for w in candidates]
        search.tell(scores)
    return search.best()
```

```pseudocode
# Sampling-based planning over a learned model (sketch)  [arxiv:2405.18418 §2]
# [tab. 5]: 512 candidates, horizon 3, 64 elites, clip to [-1, 1]

def plan(latent, previous_mean):
    # Start from the previous plan, shifted by one step  [§2]
    mean = shift(previous_mean)
    std = full((horizon, action_dim), 2.0)
    for iteration in range(iterations):
        # Sample candidate action sequences and score them with the model  [§2]
        actions = clip(mean + std * noise, -1, 1)
        values = estimate_value(latent, actions)
        # Keep the best candidates and refit the distribution to them  [tab. 5]
        elites = top_k(values, 64)
        weights = softmax(0.5 * elites.values)
        mean = weighted_mean(weights, elites.actions)
        std = clip(weighted_std(weights, elites.actions), 0.05, 2.0)
    # Execute the first step of the refitted plan  [§2]
    return clip(mean[0], -1, 1)
```

```pseudocode
# Goal-conditioned model with subgoal episodes (sketch)  [alg. 1] [§3.1]
# subgoals: key states from farthest-point sampling over actions  [alg. 2]

def subgoal_episode(policy, env, subgoals, n_s, t_s):
    # Visit N_s random subgoals in turn, each with its own step budget  [alg. 1]
    trajectory = []
    for _ in range(n_s):
        target = random_choice(subgoals)
        # Act toward the target until it is reached or t_s steps pass  [alg. 1]
        trajectory += reach(policy, env, target, max_steps=t_s)
    # Stored with the environment-goal episodes for world-model training  [§3.1]
    return trajectory
```

```pseudocode
# Model rollouts only for critic targets (sketch)  [§3, eq. 2] [alg. 1]
# The model changes the critic's target; the actor is trained on real states

def mve_target(s0, actor_target, critic_target, model, reward, H, gamma):
    # Roll the learned model H steps under the target policy  [§3]
    state = s0
    imagined_return = 0.0
    for t in range(H):
        action = actor_target(state)
        # Imagined reward is the true reward function, not a learned one  [§3]
        imagined_return = imagined_return + gamma**t * reward(state, action)
        state = model(state, action)
    # Bootstrap from the target critic at the imagined end state  [eq. 2]
    last_action = actor_target(state)
    return imagined_return + gamma**H * critic_target(state, last_action)
```

Where the variants differ: the first sketch fills the whole table in one pass
and needs no sampling. The second is a loop over replayed data, and every
update depends on what the model imagined in the same step. The third trains
the model once, then searches a small policy, and the choice of `env` decides
whether the reported score is real or virtual. The fourth searches over action
sequences at every control step, so its cost is paid at decision time, not at
training time. The fifth reuses the learned-model loop of the second sketch,
and changes only which episodes it collects. The sixth also uses the model,
but only to form one critic target per update, so the policy never trains on
imagined states. The sampling details here follow the paper's description, and
the warm start is a simplification of the card's notes on the official code.

## Contradictions

- **Imagination horizon of DreamerV3.** The text gives the critic's prediction
  horizon as T = 16, and the table gives H = 15 [arxiv:2301.04104 §Critic
  learning] [arxiv:2301.04104 tab. 4]. This page uses 15, the table's value.
- **Reward head of DreamerV3.** One section trains the reward predictor with
  the symlog squared loss, and a later section uses symexp twohot for reward
  and critic outputs [arxiv:2301.04104 §Learning algorithm] [arxiv:2301.04104
  §Robust predictions].
- **DMLab comparison of DreamerV3.** The text says the method exceeds IMPALA and
  R2D2+ at 1B steps. The table lists R2D2+ only at 10B steps, where it scores
  85.4 against 71.4 for the method [arxiv:2301.04104 §Results] [arxiv:2301.04104
  tab. 8].
- **MUN paper and its released code.** The paper applies farthest-point
  subgoal selection to every task [arxiv:2411.02446 alg. 2]. The released
  3-Block, Walker and Block Rotation configs sample subgoals at random instead
  [code: Config/configs.yaml#L262]. This is a paper-to-code gap, not a
  paper-to-paper one, and the reported results on those tasks may not come
  from the method as written.
- **Between the two evolved-controller cards.** None known. [World
  Models](/papers/1803.10122) and [arxiv:1809.01999] report the same Car Racing
  score of 906 ± 21 [arxiv:1803.10122 tab. 2] [arxiv:1809.01999 tab. 1], and the
  same Take Cover temperature results [arxiv:1803.10122 tab. 4] [arxiv:1809.01999
  tab. 2].
- **DreamerV3 as a reference across cards.** DreamerV3 is the baseline in
  [arxiv:2405.18418 §4.1], [arxiv:2512.02417 tab. 1] and [arxiv:2512.18850
  §4.6]. Each uses its own tasks, so these numbers do not compare with one
  another. The humanoid card reports DreamerV3 not learning within 3M steps,
  and the driving card reports 0.53 success on the roundabout and 0.91 on the
  intersection [arxiv:2512.02417 tab. 1]. None known to conflict on a shared
  benchmark.
- **Between the other papers.** None known. The remaining cards use different
  domains and metrics and do not compare with each other on any shared
  benchmark. The survey reports no results to compare.

## Papers

- [Real-Time Bidding by Reinforcement Learning in Display Advertising](/papers/1701.02490):
  exact value iteration over (steps left, budget left) for bidding, plus a small
  network for larger campaigns.
- [World Models](/papers/1803.10122): a VAE, an MDN-RNN and a linear controller
  evolved with CMA-ES, with the Car Racing and Take Cover results this page
  cites.
- [Recurrent World Models Facilitate Policy Evolution](/papers/1809.01999):
  a VAE and an LSTM world model, with a linear controller evolved by CMA-ES. The
  controller is trained in the real game for car racing and inside the model
  for DoomTakeCover.
- [Mastering Diverse Domains through World Models](/papers/2301.04104):
  DreamerV3, a learned recurrent world model with actor and critic trained on
  imagined rollouts and one fixed configuration across benchmarks.
- [Hierarchical World Models as Visual Whole-Body Humanoid Controllers](/papers/2405.18418):
  a frozen TD-MPC2 tracker below a visual TD-MPC2 planner that searches over
  end-effector commands at each step.
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](/papers/2411.14499):
  a survey that frames the world model as transition dynamics and lists the
  planning uses built on it. It reports no experiments.
- [Learning World Models for Unconstrained Goal Navigation](/papers/2411.02446):
  a goal-conditioned Dreamer-style model, trained on episodes that travel
  between farthest-point subgoals from the replay buffer.
- [Vehicle Dynamics Embedded World Models for Autonomous Driving](/papers/2512.02417):
  a Dreamer-style driving agent that separates the car's dynamics from the
  environment, with a goal-output actor and a deployment step for changed cars.
- [RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL](/papers/2512.03556):
  a learned action model and a learned goal model, with a reward built from
  the sparse done signal and the distance to the imagined goal.
- [InDRiVE](/papers/2512.18850): reward-free pretraining with ensemble disagreement, then zero-shot transfer to
  an unseen CARLA town and a short task fine-tune.
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](/papers/1803.00101):
  model-based value expansion, a learned one-step model used to build H-step
  critic targets, with the TD-k trick for the critic's training states.
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](/papers/2203.01387):
  a taxonomy of offline RL methods, with the model-based class and its
  uncertainty penalties. It reports no new experiments.