InDRiVE: Reward-Free World-Model Pretraining for Autonomous Driving via Latent Disagreement

In plain words

The authors want a self-driving learner that can practise without being told what a good driver does. They let a simulated car wander around a town, guided only by curiosity. The curiosity signal is "where do my several guesses about the future disagree?" Places that are hard to predict get explored more.

The car learns an internal model of the world while it explores, and a policy that acts inside that model. After this reward-free phase, the frozen policy is dropped into a town it has never seen and runs there. The authors then teach it two specific jobs (staying in the lane and avoiding collisions) with a short amount of task feedback.

The point is to check whether a model learned through pure curiosity transfers to new roads and needs little task-specific training. The authors compare it with two other curiosity signals under the same budget. This is a controlled study in one simulator with a simplified action space, not a finished driving system.

TL;DR

• Reward-free pretraining in CARLA on Town01: a DreamerV3-style world model and an exploration policy trained only on an ensemble-disagreement reward.
• Zero-shot: the frozen exploration policy is run on Town01 (seen) and Town02 (unseen). Success rate (SR) averages 75.65% and 73.20%, against 62.58% and 48.61% for ICM and 68.58% and 55.83% for RND tab. 6.
• Few-shot: 10k steps of task-reward fine-tuning. Collision avoidance reaches 96.43% (Town01) and 94.78% (Town02) on average tab. 5, tab. 6.
• Author-reported, one simulator, one town pair, discrete controls, semantic segmentation input. Treat as evidence for the protocol, not for a driving stack.

Problem

Reward design for driving is labour-intensive, and rewards tuned in one town can fail in another §1. Model-based RL can be sample-efficient, but most driving agents still train on task reward throughout §1. The paper asks whether purely intrinsic pretraining produces a reusable driving world model that transfers zero-shot and adapts few-shot to unseen towns and traffic densities §1 §2.

Assumptions: CARLA 0.9.15 synchronous mode at 10 Hz; front semantic-segmentation input (128x128); discrete steer and throttle; brake fixed to 0 §4.1.

Method

The base agent is DreamerV3 with a recurrent state-space model. Its deterministic state is h_t and its stochastic state is s_t. Policies and intrinsic modules read u_t = [h_t; s_t] §3.1 eq. 1. World-model parameters are denoted phi_wm, and the extrinsic reward head is phi_r §3.1.

Phases. Pretraining is reward-free: no task reward is computed, stored or used §3.4 §3.1. Phase 1 is zero-shot: all parameters are frozen and the exploration policy is run on an unseen town §3.4. Phase 2 is few-shot: a task policy starts from the exploration policy and is trained on extrinsic reward, with the world model and intrinsic module frozen §3.4 §4.3.

Intrinsic reward (ensemble disagreement, the proposed signal). K predictors w_k take (u_t, a_t) and output categorical probabilities for the next stochastic latent. Each is trained with cross-entropy on replay, with gradients stopped into the world model. The reward is the mean over dimensions of the ensemble variance §3.2.1 eq. 4. The paper sets d = 32 x 32 = 1024 §3.2.1.

Baselines (same backbone, replay and budget).

• ICM: an inverse model predicts a_t from (u_t, u_{t+1}) with cross-entropy; a forward model predicts u_{t+1}; the reward is a normalised forward error scaled by eta §3.2.2 eq. 5 eq. 6 eq. 7.
• RND: a predictor is trained to match a fixed random target over u_t; the reward is the normalised prediction error §3.2.3 eq. 8 eq. 9.
• All intrinsic signals use running-statistics normalisation with moving-average rate rho §3.2 (paper formula, unnumbered).

Actor-critic. The exploration actor and value are trained on imagined rollouts to maximise discounted intrinsic return §3.3 eq. 10. In Phase 2 a task actor and value are initialised from the exploration policy and trained on discounted extrinsic return §3.3 eq. 11. The extrinsic reward head is trained only in Phase 2 §3.3.

World-model loss. Pretraining minimises reconstruction, KL and continuation terms §3.1 eq. 2. Phase 2 optimises only the reward-head loss, with gradients stopped through (h_t, s_t) §3.1 eq. 3. The KL weight beta_KL and the loss weights lambda_c and lambda_r are symbols in these equations whose values are not stated in the active text §3.1.

# Ensemble-disagreement intrinsic reward  [§3.2.1] [eq. 4]
# Input:  latent u (B, Dz), action a (B, A), predictors w_1..w_K
# Output: intrinsic reward (B,)

def disagreement_reward(latent, action, predictors):
    # Each predictor outputs next-latent probabilities, shape (B, d)  [§3.2.1]
    predictions = stack([w(latent, action) for w in predictors])
    # Variance over the K ensemble members, per dimension  [eq. 4]
    spread = variance(predictions, axis=0)        # (B, d)
    # Average over the d = 1024 dimensions  [eq. 4]
    return mean(spread, axis=-1)                  # (B,)

# Two-phase training order  [§3.4]
# Phase 1 uses only the intrinsic reward; Phase 2 uses only extrinsic reward.

def pretrain_then_finetune(env, world_model, explorer, task_policy):
    # Phase 1: reward-free exploration in Town01  [§3.4] [§4.2]
    for step in range(pretrain_steps):           # 5e5 steps  [§4.2]
        collect(env, explorer)
        train_world_model(replay)                # eq. 2  [§3.1]
        train_intrinsic_module(replay)           # disagreement, ICM or RND
        train_actor_critic(imagined, r_int)      # eq. 10  [§3.3]
    evaluate(explorer, unseen_town)              # zero-shot, all frozen  [§5.1]
    # Phase 2: few-shot task learning  [§3.4] [§4.3]
    for step in range(finetune_steps):           # 1e4 steps  [§4.3]
        train_reward_head(replay)                # eq. 3  [§3.1]
        train_actor_critic(imagined, r_ext)      # eq. 11  [§3.3]
    evaluate(task_policy, towns)                 # few-shot  [§5.2]

Figures: fig. 1 shows how InDRiVE, ICM and RND plug into DreamerV3 §3. fig. 2 shows the actor-critic with the latent-disagreement module §3.4. fig. 3 shows the four evaluation routes in Town01 and Town02 §4.5.

Hyperparameters

Results

Author-reported. Setup: CARLA, trained on Town01, evaluated on Town01 (seen) and Town02 (unseen), four routes, traffic densities {5, 10, 20}, 5 training seeds, N = 50 episodes per configuration §4 §5. Baselines: ICM, RND, and task-trained DreamerV3 (510k steps on the task reward, for the few-shot tables) §4.6. SR is success rate in percent §4.7.

Averages across the four routes tab. 6:

Per-route examples, with the source tables:

• Zero-shot, Town02 Right Turn: InDRiVE 82.10, ICM 44.90, RND 60.22 tab. 3.
• Zero-shot, Town02 Two Turn: InDRiVE 64.78, ICM 49.23, RND 47.10 tab. 3.
• Few-shot lane following, Town02 Two Turn: InDRiVE 88.41, DreamerV3 83.40 tab. 4.
• Few-shot collision avoidance, Town02 Two Turn: InDRiVE 85.55, RND 80.29, ICM 78.21, DreamerV3 78.02 tab. 5.

Few-shot figures are after 10k fine-tuning steps §5.2. The paper reports means over seeds with no spread or confidence intervals in the tables.

Paper vs code

The paper does not link code. The repository feezakk/InDRiVE is treated as official because its owner handle matches the first author's address (feezakk@umich.edu) and the repo name matches the method name. Its root LICENSE is the UC Regents CarDreamer non-commercial licence, and it contains a CarDreamer environment (car_dreamer/). dreamerv3/LICENSE is MIT (Hafner, 2023). Code is cited by path and line only, with no quotes. Searched the GitHub repository search for "InDRiVE" and for CARLA world-model terms (no InDRiVE hit), and the owner list for feezakk and jrkwon (the second author's handle was not confirmed). The repository has no description and an empty README, so the match rests on the repository name, the owner handle, and training scripts that name the disagreement configuration. The reward weights and the environment code (car_dreamer/) were not compared in this pass.

Discrepancies (paper vs code at the pinned commit):

• Disagreement statistic. Paper: variance across the ensemble, averaged over dimensions §3.2.1 eq. 4. Code: standard deviation across the ensemble, then the mean over the last axis expl.py. The code takes the square root of the variance before averaging.
• Disagreement loss. Paper: cross-entropy on categorical next-latent distributions §3.2.1. Code: the disagreement heads use the mse distribution and minimise its negative log-probability, which is a squared-error loss expl.py dreamerv3.yaml.
• Normalisation of the disagreement reward. Paper: all intrinsic signals use the running-statistics normalisation §3.2. Code: the disagreement reward is returned raw; the Disag class has no running-statistics object expl.py. The ICM class does normalise expl.py.
• ICM error. Paper: the squared error, normalised §3.2.2 eq. 7. Code: one half of the squared error, then running normalisation, scaled by eta expl.py. eta defaults to 1.0 when absent expl.py. The code uses an inverse-loss weight of 0.8 dreamerv3.yaml; the paper does not give this weight.
• Batch size. Paper table: 64 tab. 2. Code default: 16 dreamerv3.yaml. No override was found in the pretraining script run_latent_disagreement_experiments.sh.
• Replay size. Paper table: 1e5 tab. 2. Code default: 1e8, with a comment that 1e5 may suffice dreamerv3.yaml.
• Run length. Code default run steps: 510000 dreamerv3.yaml. The pretraining script passes its own step count run_latent_disagreement_experiments.sh.
• Expl. weights. The default config sets extrinsic 1.0 and intrinsic 0.0 dreamerv3.yaml. Pretraining overrides this to intrinsic 1.0 and extrinsic 0.0 run_latent_disagreement_experiments.sh. This matches the paper.
• Optimiser. Exploration optimiser: Adam, lr 1e-4, eps 1e-5, grad clip 100 dreamerv3.yaml. The paper does not state these values.
• Imagination horizon. Code 15 dreamerv3.yaml; paper 15 tab. 2. Match.
• Ensemble size. Code 8 dreamerv3.yaml; paper 8 tab. 2. Match.
• Safety shield. The code has a safe-training shield with modes "one_step" and "long", off by default dreamerv3.yaml dreamerv3.yaml. The fine-tuning script can enable it run_latent_disagreement_experiments.sh. The paper does not describe any shield. Check which runs the reported numbers come from.

Pitfalls

• Equation and table numbers are counted from the LaTeX source, not the compiled PDF. The source's own captions disagree on table numbering (one caption says "Table IV" for the hyperparameters). Check the compiled PDF before citing a number to a reader.
• Disagreement statistic. Implementing the paper's equation as written gives a different reward from the code's std-then-mean statistic. The paper's equation has the variance, and the code takes a square root first. Choose one and say which in your implementation.
• Reward scale. The paper says all intrinsic signals are normalised §3.2, but the code's disagreement reward is not (see Paper vs code). Its absolute scale therefore depends on the ensemble's output, and the paper's numbers do not tell you what scale they used.
• Normalisation epsilon = 1.00 in the hyperparameter table tab. 2 is unusually large for a stabiliser. Confirm against the code's running-moments settings before copying it, since the code's epsilon was not checked here.
• Symbol overload. s_t is the stochastic latent in the world model, but the disagreement formula uses s_{t+1,k} for each predictor's categorical probabilities over the same latent space §3.2.1 eq. 4.
• Few-shot scope is inconsistent. §3.1 says Phase 2 optimises only the reward-head parameters §3.1 eq. 3. §3.4 and §4.3 say the task actor and critic are also trained, with the world model and intrinsic module frozen §3.4 §4.3. Decide which set you train; the paper does not settle it.
• Not goal-conditioned. The policy gets only the image; route waypoints are used only for reward and metrics §4.1 §4.5. The reported success is corridor driving, not navigation to a goal. A downstream agent cannot be told "drive to X" with this policy.
• Rewards and metrics. Success requires no collision, lane violation, wrong-direction or off-road event within the step budget §4.7. Lane violation ends an episode at the first invasion §4.4. The code has a collision calibration script (tools/calibrate_collision.py) that the paper does not mention; its effect on the reported numbers is unknown.
• Gap column. On the few-shot lane-following task, RND has a smaller town gap (0.73) than InDRiVE (3.92) tab. 6. The text's claim that "lane-following transfer is competitive with RND" is accurate; do not read the gap column as InDRiVE leading there.
• Variance is missing. Five training seeds were run §5, but the tables show only means. Differences of 1-2 points between InDRiVE and RND are within unknown noise.
• Compute and cost are not reported: wall-clock, GPU, memory, or the cost of the disagreement ensemble (eight 5-layer, 1024-unit heads in the code dreamerv3.yaml).
• Scope the authors state: Town01 to Town02 transfer only, semantic segmentation input, discrete controls §6.

Lineage

• Builds on: DreamerV3 (world model and actor-critic backbone) §3.
• Uses the same idea as Plan2Explore (latent ensemble disagreement) §3 §2.
• Compared against: ICM and RND, with the same backbone §3.2 §4.6; and task-trained DreamerV3 §4.6.
• Superseded by: not known from the wiki.