Intrinsic motivation

Overview

Intrinsic motivation replaces the task reward, at least during an early phase, with a signal the agent computes from its own experience. The usual signal is how surprising or uncertain the agent's predictions are, so the agent is drawn toward states it does not yet understand. The one card so far uses this as a reward-free pretraining phase: the agent explores and learns a world model, and only afterwards receives a task reward 2512.18850 §1 2512.18850 §3.4.

Variants

Ensemble disagreement

Several predictors each guess the distribution of the next latent state. The reward is how much the guesses spread, averaged over the latent dimensions. The paper uses eight predictors and 1024 dimensions 2512.18850 §3.2.1 2512.18850 tab. 2. Card: InDRiVE.

Forward-model error with an inverse model

A forward model predicts the next latent state from the current state and the action. An inverse model predicts the action from the two states, which shapes the features the forward model sees. The reward is the normalised forward prediction error, scaled by a weight 2512.18850 §3.2.2. The paper calls this variant ICM and uses it as a baseline. Card: InDRiVE.

Error against a fixed random target

A predictor is trained to match the output of a fixed, randomly initialised target over the current state. The reward is the normalised prediction error, which shrinks for states the predictor has seen often 2512.18850 §3.2.3. The paper calls this variant RND and uses it as a baseline. Card: InDRiVE.

When to use which

The only evidence is one controlled comparison with a shared backbone, replay budget and training steps 2512.18850 §3.2 2512.18850 §4.6.

Reported success rates are means over seeds with no spread in the tables. Gaps of one or two points are within unknown noise 2512.18850 §5 2512.18850 tab. 6.

Implementation sketch

The sketch follows the disagreement variant, since it is the paper's proposed signal. The paper states the statistic as a variance across predictors; the authors' released code takes a standard deviation before averaging, so pick one and record which 2512.18850 §3.2.1. See the card's code comparison for the details.

# Disagreement reward for one batch of latent states  [arxiv:2512.18850 §3.2.1]
# Input:  latent u (B, Dz), action a (B, A), K predictors
# Output: intrinsic reward (B,)

def disagreement_reward(u, a, predictors):
    # Each predictor outputs a distribution over the next latent, shape (B, d)
    preds = stack([p(u, a) for p in predictors])
    # Spread across the K predictors, per dimension (variance as stated in the paper)
    spread = variance(preds, axis=0)              # (B, d)
    # Average over the d dimensions to get one scalar per sample
    return mean(spread, axis=-1)                  # (B,)

# Phase 1 uses only the intrinsic reward; Phase 2 uses only the task reward  [arxiv:2512.18850 §3.4]
def pretrain_then_finetune(env, wm, explorer, task_policy):
    for step in range(pretrain_steps):
        collect(env, explorer)
        train_world_model(replay)
        train_predictors(replay)                  # the ensemble is trained on replay
        train_actor_critic(imagined, r_int)       # intrinsic return only
    # zero-shot: freeze everything, run the explorer on an unseen map
    for step in range(finetune_steps):
        train_task_head(replay)                   # extrinsic reward only
        train_actor_critic(imagined, r_ext)

Keep the replay and the imagined rollouts shared across the two phases so the baselines see the same data budget. The paper's numbers depend on that choice 2512.18850 §3.2 2512.18850 tab. 2.

Contradictions

None known across papers, since one card cites this concept so far.

Inside the one source, the paper's own text gives two accounts of the fine-tuning phase. One section says only the reward-head parameters are optimised 2512.18850 §3.1. Another says the task actor and critic are also trained 2512.18850 §3.4 2512.18850 §4.3. The sketch above follows the second.

Papers

• InDRiVE (arXiv:2512.18850): reward-free pretraining with ensemble disagreement, compared with ICM and RND in a CARLA driving study.