Model-based reinforcement learning

Overview

A model-based agent predicts what its actions will do and uses those predictions to choose actions, instead of learning only from real trial and error. The agent cards on this page use a model in different ways, and they differ in what the model is and how the policy is found. In Real-Time Bidding by Reinforcement Learning in Display Advertising the model is the estimated distribution of market prices, and the state space is small enough to plan exactly 1701.02490 §2, §4. In Mastering Diverse Domains through World Models the model is a learned recurrent world model, and the actor and critic are trained on rollouts the model imagines 2301.04104 tab. 4. In Recurrent World Models Facilitate Policy Evolution a convolutional VAE and an LSTM form the model, and a single linear controller is searched by evolution against the real reward, which the model never sees 1809.01999 §2, §3. World Models describes the same three-part agent, with the same Car Racing score 1803.10122 tab. 2. Hierarchical World Models as Visual Whole-Body Humanoid Controllers is the one card here that plans at decision time: it runs sampling-based model predictive control over a learned model, with a frozen low-level tracker underneath 2405.18418 §2, §3.

Learning World Models for Unconstrained Goal Navigation keeps a Dreamer-style model and changes the data it learns from. The model is trained on episodes that travel between subgoals taken from the agent's own replay buffer, in both directions and across trajectories, which are the transitions a model otherwise generalises to poorly [arxiv:2411.02446 §1, §3.1].

Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning uses the model in a narrower place. It never trains the policy on imagined rollouts. The actor is updated on real transitions, and the model only builds the target that the critic is trained toward, by looking a fixed number of steps ahead 1803.00101 §3, alg. 1 lines 9–12.

The survey Understanding World or Predicting Future? gives a common frame. It treats the world model as the transition dynamics of a Markov decision process, learned by supervised fitting of one-step predictions, and names planning uses such as model predictive control, which picks the action sequence with the highest predicted return over a horizon 2411.14499 §3.1, eq. 1, eq. 3. The survey reports no experiments of its own 2411.14499 §1–§7.

When the data is a fixed log, the model can be exploited where the log is thin. The offline RL survey A Survey on Offline Reinforcement Learning files model-based methods under one class, which penalises model rollouts by an uncertainty measure 2203.01387 §III-E, §IV-E. See the offline variant below.

The shared structure is a loop: fit or estimate the model, use it to evaluate future outcomes, and update the decision rule. The variants differ in how each step is done. Most learned-model agents on this page do not search at decision time. They train a policy on model rollouts, and the policy then acts directly: 2301.04104 §Critic learning, 1809.01999 §2, 2512.03556 §3.2, 2512.02417 §4.2.1, 2512.18850 §3.3 and 2411.02446 §3.1. Only the humanoid card plans with the model at each step 2405.18418 §2.

Variants

Exact planning over an estimated model

Fill a value table backwards over (steps left, budget left), then bid by a threshold on the value gained from each price. The paper treats one ad campaign as a Markov decision process whose state is the auctions left, the budget left and the features of the current request 1701.02490 §3.1. Market prices are estimated from data by counting 1701.02490 §2. Card: Real-Time Bidding by Reinforcement Learning in Display Advertising.

The table costs O(TB) time and memory, where T is the episode length and B the budget 1701.02490 §4.1. The paper replaces the table with a small network that fits the value differences, with two tanh hidden layers of 30 and 15 units 1701.02490 §4.1, §6.2. The network is only checked inside the range it was trained on 1701.02490 tab. 4. Reported gain: 16.67% more clicks than the Lin baseline at budget ratio 1/8 on iPinYou, averaged over nine campaigns 1701.02490 tab. 2.

Learned world model with imagined rollouts

Train a recurrent world model on replayed experience, then train an actor and a critic on rollouts of 15 steps imagined from the model 2301.04104 tab. 4. The agent acts by sampling from its actor, with no lookahead search at decision time 2301.04104 §Critic learning. Card: Mastering Diverse Domains through World Models.

The paper keeps one configuration for every benchmark. Its main devices are symlog inputs, a categorical critic over 255 exponentially spaced bins trained on twohot targets, a return scale from the 5th to 95th percentile, and free bits on the KL terms 2301.04104 tab. 4 [arxiv:2301.04104 §Robust predictions]. Compute per benchmark is 0.1 to 16.1 A100 GPU days, depending on the suite and the model size 2301.04104 tab. 2. Reported results include a Minecraft Diamond return of 9.1 against 7.1 for IMPALA at 100M steps 2301.04104 tab. 5 and an Atari100K gamer mean of 125 against 105 for IRIS 2301.04104 tab. 9. The paper's own comparison notes that the Atari100K baselines do not all follow one protocol.

Goal-conditioned model trained on subgoal episodes

Keep a goal-conditioned Dreamer-style agent, and change the data it learns from. Besides the usual episodes toward environment goals, each iteration collects episodes that visit two subgoals drawn from a set of key states. The set is picked by farthest-point sampling over the actions in recent episodes 2411.02446 §3.1, §3.2 alg. 1 alg. 2. The model is trained on both kinds of episode, and the policy is trained only in imagination, with reward equal to minus a learned temporal distance 2411.02446 eq. 7 alg. 1. Card: Learning World Models for Unconstrained Goal Navigation.

The paper reports the lowest one-step prediction error among the methods it compares, on all six tasks. Compounding error is lowest on five of the six. On Fetch Slide the ablation without subgoal sampling is slightly lower 2411.02446 tab. 6 2411.02446 tab. 7. On 3-Block navigation across 225 start-goal pairs, MUN reaches 95% against 56% for GC-Dreamer, but the goals there are chosen by hand 2411.02446 tab. 3 [arxiv:2411.02446 §4.5]. Stacking success is above 95% against about 60% for the baselines 2411.02446 §4.3. One run takes 36 to 70 GPU hours on an A100 2411.02446 tab. 1. The released code does not follow the paper's sampler on every task: its 3-Block, Walker and Block Rotation configs pick subgoal states at random instead, which the card records as a discrepancy configs.yaml. Results on those tasks may not come from the method as written.

Learned world model with an evolved linear controller

Train a convolutional VAE to encode each frame into a latent vector, then an LSTM whose mixture-density output predicts the next latent. The controller is one linear layer on the latent and the LSTM state, with 867 parameters for car racing, and it is searched with CMA-ES against the real return 1809.01999 §2, §3 1809.01999 app. A.1–A.5. Card: Recurrent World Models Facilitate Policy Evolution.

The world model is trained on frames and actions from random rollouts, and the reward is used only in the controller search 1809.01999 §3 1803.10122 §3.1. The parameter counts are 867 for the car controller and 1,088 for Take Cover 1803.10122 tab. 1 1803.10122 tab. 3. The card World Models gives the same setup and the same Take Cover temperature table, with τ = 0.10 at 2086 virtual and 193 real 1803.10122 tab. 4. For car racing the controller is trained in the real environment, and the full model reaches 906 ± 21, against 838 ± 11 for the Gym Leader baseline 1809.01999 tab. 1 1809.01999 §3.1. For DoomTakeCover the controller is trained only inside the learned model at sampling temperature 1.15, and it scores 1092 ± 556 in the real game 1809.01999 tab. 2 1809.01999 §4.2. The standard deviation is large. The authors report that the temperature controls how well the controller transfers: at 0.10 the virtual score is 2086 but the real score is 193 1809.01999 tab. 2.

Planning over a learned model at each step

Two TD-MPC2 world models are trained separately. A low-level tracker, pretrained once on 836 motion-capture clips and then frozen, follows end-effector targets on a 56-joint simulated humanoid. A visual high-level planner, trained from scratch for each task on the task reward, outputs those targets from 64×64 camera images 2405.18418 §3, §3.1, §3.2, §4.1. Each action is chosen by sampling candidate action sequences, scoring them with the model's predicted return over a short horizon, and refitting the sampling distribution over a few iterations 2405.18418 §2, tab. 5. Card: Hierarchical World Models as Visual Whole-Body Humanoid Controllers.

On eight tasks the planner's return is comparable to TD-MPC2 except on stairs, where TD-MPC2 scores higher 2405.18418 §4.2 fig. 5. SAC and DreamerV3 did not learn within 3M steps in the same setting 2405.18418 §4.2. In a 51-person study on the five visual tasks, 97.8% of judgements preferred the planner's motion, against 0.4% for TD-MPC2 2405.18418 tab. 6. The cost is inference time: 88.2 ms per step against 50.8 ms for TD-MPC2 2405.18418 tab. 3. The authors report that planning at both levels is critical 2405.18418 §4.3. The benchmark numbers are author-reported.

Learned world model as the RL environment, with a learned reward

Train a world model that predicts the next camera frame from the robot's action, and a second one that imagines the finished scene from an instruction. The policy is then trained with a standard RL algorithm inside these models, with a reward built from the imagined scene 2512.03556 §3.1, §3.2. The reward has two parts: a sparse term that fires when the predicted done signal crosses a threshold, and a dense term that measures the perceptual distance between the current frame and the imagined goal frame 2512.03556 eq. 5, eq. 7, eq. 8. The paper uses PPO 2512.03556 §4.3. Card: RoboScape-R.

The reported gain is on unseen scenes. With an OpenVLA policy, out-of-domain pick-and-place reaches 84.1% with the model reward, against 54.5% for RL with the simulator's reward and 47.5% for supervised fine-tuning [arxiv:2512.03556 tab. 1]. In-domain differences are small, at most 3.0 points by the card's arithmetic on the same table 2512.03556 tab. 1. Training the world model took about 96 hours on an H20 cluster 2512.03556 §4.2. All evaluation is in simulation 2512.03556 §5.

Imagined rollouts with a separate vehicle model

A Dreamer-style agent for driving splits the ego vehicle's dynamics from the rest of the environment. A parameterised vehicle model predicts the car's own state from its control input, and a separate environment model conditions on that state 2512.02417 §4.1.1, eq. 9. The actor outputs a goal rather than a raw action, and a predefined controller turns the goal into an action 2512.02417 §4.2.1. Two mechanisms handle changed car parameters. At deployment, the original policy's action is mapped through the inverse of the new vehicle model, so the new car reaches the states the old car would reach. During training, imagined rollouts are generated under both the nominal and the shifted parameters 2512.02417 §4.3, §4.4. The first mechanism cannot help when the new car cannot reach those states, for example with a smaller maximum steering angle 2512.02417 §4.3. Card: Vehicle Dynamics Embedded World Models for Autonomous Driving.

On MetaDrive's roundabout the success rate is 0.70 against 0.53 for DreamerV3. On the intersection it is 0.88 against 0.91 for DreamerV3, so the baseline is slightly better there 2512.02417 tab. 1. In CARLA's unprotected left turn the success rates are 0.94 and 0.91 2512.02417 tab. 1. The robustness results under changed mass and steering are shown only as figures 2512.02417 fig. 9. The paper states that real-world performance is open 2512.02417 §6.

Reward-free pretraining of the world model

Pretrain a world model and an exploration policy with no task reward, using the disagreement of an ensemble of next-state predictors as an intrinsic reward. Then test the frozen explorer in a town it has not seen, and fine-tune a task policy with 10k steps of task reward [arxiv:2512.18850 §3.2.1, §3.4, §4.3]. The ensemble has eight predictors 2512.18850 tab. 2. Card: InDRiVE.

Zero-shot success on Town01 (seen) is 75.65% for this method, against 62.58% for ICM and 68.58% for RND. On Town02 (unseen) it is 73.20%, against 48.61% and 55.83% 2512.18850 tab. 6. After fine-tuning, collision avoidance reaches 96.43% on Town01 and 94.78% on Town02 2512.18850 tab. 6. For few-shot lane following, RND has the smaller gap between the two towns (0.73 against 3.92) 2512.18850 tab. 6. The setting is discrete steering and throttle on semantic-segmentation input, with transfer tested only from Town01 to Town02 2512.18850 §4.1 2512.18850 §6.

Model rollouts only for critic targets

Fit a one-step dynamics model on replayed transitions. At each update, roll the model forward H steps from a replayed state under the target policy, add up the discounted rewards along the way, and bootstrap from the critic at the end. The actor is trained with its usual loss on the real transition, so imagined states enter only the critic's targets 1803.00101 §3, alg. 1 lines 9–12. The critic is also trained on imagined states, with each imagined step given a target of horizon H minus t, which the paper calls the TD-k trick [arxiv:1803.00101 §3.2]. The imagined rewards come from the true reward function, not from a learned one, so the method needs that function 1803.00101 §3. The horizon is chosen by hand per task: H = 10 for cheetah and swimmer, and H = 3 for walker, where the figure caption says the model class is inadequate at H = 10 1803.00101 fig. 3 caption. Card: Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning.

The paper reports learning curves only, with no numeric returns in the text 1803.00101 §4.1, fig. 3. In the authors' comparison, MVE with the TD-k trick learns fastest on cheetah, swimmer and walker, against DDPG, MVE without TD-k and an imagination-buffer baseline. On walker the gain over DDPG is smaller at H = 3 1803.00101 fig. 3. Without the TD-k trick, large horizons hurt on oracle dynamics 1803.00101 fig. 6. The learned-dynamics runs use 4 seeds 1803.00101 app. A. Unlike the Dreamer-style variant above, this method does not learn its policy from imagined rollouts.

Offline model-based penalties (survey view)

With a fixed dataset, a learned model can be right on the logged states and wrong elsewhere, and a policy that exploits those errors does badly. The survey groups the methods that guard against this under the model-based class. MOReL penalises state-action pairs where an ensemble of models disagrees beyond a threshold. MOPO uses the Frobenius norm of the model covariance as its uncertainty. COMBO adds a conservative value penalty over model rollouts and dataset points, mixed by a fraction f. BREMEN trains a policy inside an ensemble of models under a trust-region constraint 2203.01387 §III-E, §IV-E eq. 31–38. The survey gives no numeric results for these methods, and it states no value for the MOReL threshold, the COMBO α or the BREMEN trust-region ε 2203.01387 §IV-E eq. 32 eq. 35 eq. 36. It notes that the COMBO data fraction changes how conservative the Q-function is [arxiv:2203.01387 §IV-E] eq. 38. Card: A Survey on Offline Reinforcement Learning.

For off-policy evaluation rather than control, the survey reports that model-based and fitted Q evaluation performed best across most metrics on the DOPE benchmark, but that no method was best in every setting 2203.01387 §VI-B1. This is a statement about scoring a policy, not about the learned policy.

When to use which

The headline numbers are author-reported, and the papers use different baselines and metrics. Choose on the constraints above, not on the headline numbers alone.

Implementation sketch

Six sketches, one per variant that has a clear core loop. None is a full program; the comments give the source of each step.

# Exact planning over an estimated model (sketch)  [alg. 1] [arxiv:1701.02490 §4]
# m[d]: probability that the market price equals d  [arxiv:1701.02490 §2]

def value_table(m, T, B, expected_value):
    # With no steps left, no further value  [arxiv:1701.02490 §4]
    V = zeros(T + 1, B + 1)
    for t in range(1, T + 1):
        for b in range(B + 1):
            # Best expected value over the bids this budget allows  [alg. 1]
            V[t][b] = max(
                expected_value(m, V, t, b, a) for a in range(b + 1))
    return V

# Learned world model with imagined rollouts (sketch)  [arxiv:2301.04104 tab. 4]

def train_step(model, actor, critic, replay):
    # Sample stored real experience  [arxiv:2301.04104 §Replay]
    batch = replay.sample()
    # Fit the world model to real transitions  [arxiv:2301.04104 eq. 2]
    model.update(batch)
    # Imagine 15 steps from each real start state  [arxiv:2301.04104 tab. 4]
    imagined = rollout(model, actor, batch.states, horizon=15)
    # Bootstrapped returns over the imagined rewards and critic values  [eq. 5]
    returns = lambda_returns(imagined, critic, gamma=0.997, lam=0.95)
    # Train the critic on those returns, then the actor on the advantage  [eq. 6]
    critic.update(returns)
    actor.update(returns - critic(imagined.states))

# Learned world model with an evolved linear controller (sketch)
# [arxiv:1809.01999 §2] [arxiv:1809.01999 app. A.5]
# env: the real game (car racing) or the learned model (DoomTakeCover)

def train_world_model(random_frames, random_actions):
    # Fit the frame encoder on frames from random rollouts, no reward  [app. A.2]
    vae = train_vae(random_frames)
    # Fit the recurrent model to predict the next latent  [app. A.3]
    mdn_rnn = train_mdn_rnn(vae.encode(random_frames), random_actions)
    return vae, mdn_rnn

def evolve_controller(env, vae, mdn_rnn, generations):
    # Search 867 linear weights with CMA-ES  [app. A.5]
    search = cma_es(num_params=867, population=64)
    for generation in range(generations):
        candidates = search.ask()
        # Score each candidate by its mean return over 16 rollouts  [app. A.5]
        scores = [mean_return(env, vae, mdn_rnn, w, rollouts=16)
                  for w in candidates]
        search.tell(scores)
    return search.best()

# Sampling-based planning over a learned model (sketch)  [arxiv:2405.18418 §2]
# [tab. 5]: 512 candidates, horizon 3, 64 elites, clip to [-1, 1]

def plan(latent, previous_mean):
    # Start from the previous plan, shifted by one step  [§2]
    mean = shift(previous_mean)
    std = full((horizon, action_dim), 2.0)
    for iteration in range(iterations):
        # Sample candidate action sequences and score them with the model  [§2]
        actions = clip(mean + std * noise, -1, 1)
        values = estimate_value(latent, actions)
        # Keep the best candidates and refit the distribution to them  [tab. 5]
        elites = top_k(values, 64)
        weights = softmax(0.5 * elites.values)
        mean = weighted_mean(weights, elites.actions)
        std = clip(weighted_std(weights, elites.actions), 0.05, 2.0)
    # Execute the first step of the refitted plan  [§2]
    return clip(mean[0], -1, 1)

# Goal-conditioned model with subgoal episodes (sketch)  [alg. 1] [§3.1]
# subgoals: key states from farthest-point sampling over actions  [alg. 2]

def subgoal_episode(policy, env, subgoals, n_s, t_s):
    # Visit N_s random subgoals in turn, each with its own step budget  [alg. 1]
    trajectory = []
    for _ in range(n_s):
        target = random_choice(subgoals)
        # Act toward the target until it is reached or t_s steps pass  [alg. 1]
        trajectory += reach(policy, env, target, max_steps=t_s)
    # Stored with the environment-goal episodes for world-model training  [§3.1]
    return trajectory

# Model rollouts only for critic targets (sketch)  [§3, eq. 2] [alg. 1]
# The model changes the critic's target; the actor is trained on real states

def mve_target(s0, actor_target, critic_target, model, reward, H, gamma):
    # Roll the learned model H steps under the target policy  [§3]
    state = s0
    imagined_return = 0.0
    for t in range(H):
        action = actor_target(state)
        # Imagined reward is the true reward function, not a learned one  [§3]
        imagined_return = imagined_return + gamma**t * reward(state, action)
        state = model(state, action)
    # Bootstrap from the target critic at the imagined end state  [eq. 2]
    last_action = actor_target(state)
    return imagined_return + gamma**H * critic_target(state, last_action)

Where the variants differ: the first sketch fills the whole table in one pass and needs no sampling. The second is a loop over replayed data, and every update depends on what the model imagined in the same step. The third trains the model once, then searches a small policy, and the choice of env decides whether the reported score is real or virtual. The fourth searches over action sequences at every control step, so its cost is paid at decision time, not at training time. The fifth reuses the learned-model loop of the second sketch, and changes only which episodes it collects. The sixth also uses the model, but only to form one critic target per update, so the policy never trains on imagined states. The sampling details here follow the paper's description, and the warm start is a simplification of the card's notes on the official code.

Contradictions

• Imagination horizon of DreamerV3. The text gives the critic's prediction horizon as T = 16, and the table gives H = 15 [arxiv:2301.04104 §Critic learning] 2301.04104 tab. 4. This page uses 15, the table's value.
• Reward head of DreamerV3. One section trains the reward predictor with the symlog squared loss, and a later section uses symexp twohot for reward and critic outputs 2301.04104 §Learning algorithm [arxiv:2301.04104 §Robust predictions].
• DMLab comparison of DreamerV3. The text says the method exceeds IMPALA and R2D2+ at 1B steps. The table lists R2D2+ only at 10B steps, where it scores 85.4 against 71.4 for the method 2301.04104 §Results [arxiv:2301.04104 tab. 8].
• MUN paper and its released code. The paper applies farthest-point subgoal selection to every task 2411.02446 alg. 2. The released 3-Block, Walker and Block Rotation configs sample subgoals at random instead configs.yaml. This is a paper-to-code gap, not a paper-to-paper one, and the reported results on those tasks may not come from the method as written.
• Between the two evolved-controller cards. None known. World Models and 1809.01999 report the same Car Racing score of 906 ± 21 1803.10122 tab. 2 1809.01999 tab. 1, and the same Take Cover temperature results 1803.10122 tab. 4 [arxiv:1809.01999 tab. 2].
• DreamerV3 as a reference across cards. DreamerV3 is the baseline in 2405.18418 §4.1, 2512.02417 tab. 1 and [arxiv:2512.18850 §4.6]. Each uses its own tasks, so these numbers do not compare with one another. The humanoid card reports DreamerV3 not learning within 3M steps, and the driving card reports 0.53 success on the roundabout and 0.91 on the intersection 2512.02417 tab. 1. None known to conflict on a shared benchmark.
• Between the other papers. None known. The remaining cards use different domains and metrics and do not compare with each other on any shared benchmark. The survey reports no results to compare.

Papers

• Real-Time Bidding by Reinforcement Learning in Display Advertising: exact value iteration over (steps left, budget left) for bidding, plus a small network for larger campaigns.
• World Models: a VAE, an MDN-RNN and a linear controller evolved with CMA-ES, with the Car Racing and Take Cover results this page cites.
• Recurrent World Models Facilitate Policy Evolution: a VAE and an LSTM world model, with a linear controller evolved by CMA-ES. The controller is trained in the real game for car racing and inside the model for DoomTakeCover.
• Mastering Diverse Domains through World Models: DreamerV3, a learned recurrent world model with actor and critic trained on imagined rollouts and one fixed configuration across benchmarks.
• Hierarchical World Models as Visual Whole-Body Humanoid Controllers: a frozen TD-MPC2 tracker below a visual TD-MPC2 planner that searches over end-effector commands at each step.
• Understanding World or Predicting Future? A Comprehensive Survey of World Models: a survey that frames the world model as transition dynamics and lists the planning uses built on it. It reports no experiments.
• Learning World Models for Unconstrained Goal Navigation: a goal-conditioned Dreamer-style model, trained on episodes that travel between farthest-point subgoals from the replay buffer.
• Vehicle Dynamics Embedded World Models for Autonomous Driving: a Dreamer-style driving agent that separates the car's dynamics from the environment, with a goal-output actor and a deployment step for changed cars.
• RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL: a learned action model and a learned goal model, with a reward built from the sparse done signal and the distance to the imagined goal.
• InDRiVE: reward-free pretraining with ensemble disagreement, then zero-shot transfer to an unseen CARLA town and a short task fine-tune.
• Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning: model-based value expansion, a learned one-step model used to build H-step critic targets, with the TD-k trick for the critic's training states.
• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems: a taxonomy of offline RL methods, with the model-based class and its uncertainty penalties. It reports no new experiments.