Vehicle Dynamics Embedded World Models for Autonomous Driving

In plain words

A self-driving car has to plan by imagining what will happen next. A world model is a learned simulator that does this inside the computer: the agent practises on imagined drives instead of only real ones. Most such models learn the car's own behaviour and the rest of the traffic together, from camera-like images. When the car changes, for example a heavier mass or a smaller maximum steering angle, the learned simulator no longer matches the car and the policy fails.

This paper separates the two. One part models how the ego vehicle moves given its controls. The other models how the surrounding scene changes, and that part only sees the car's motion state, not its raw physical parameters. A policy proposes a target state, and a fixed feedback controller turns that target into steering and throttle, so the car's own response is handled by feedback. Two extra steps help with changed vehicles: one adjusts the trained policy when the car is deployed, and the other trains the policy on imagined drives with several vehicle settings. The results reported are from simulated traffic only.

TL;DR

VDD is a Dreamer-style model-based RL agent for autonomous driving that decouples ego-vehicle dynamics from environment dynamics §4.1. It learns goals rather than raw actions and tracks them with a PID controller §4.2. On MetaDrive it reports higher success on roundabouts than DreamerV3 but slightly lower on intersections, and it is more robust to changed mass and steering limits §5.2. All numbers are author-reported in simulation; the paper itself says real-world performance remains open §6.

Problem

Learned world models for driving are usually trained on images and actions together, so the model mixes the car's own dynamics with the environment's. When the ego vehicle's mass or steering limit changes, a model trained on the nominal car predicts wrong futures and the policy degrades §1 §3.2. The paper assumes the environment transition depends on the ego state, not directly on the action, and that changing the ego dynamics leaves the environment dynamics unchanged §3.2, eq. 3. The method is evaluated in MetaDrive and CARLA simulators, not on real vehicles §5.1.1 §6.

Method

Model structure. VDD builds on Dreamer §4.1. It uses a hierarchical context-aware RSSM (hcRSSM): the lower level is a vehicle dynamics model driven by the ego-vehicle dynamics context, and the upper level is an environmental dynamics model (EDM) that sees the ego state §4.1.1 fig. 2.

World model learning. Each step, the encoder maps the observation to a stochastic latent, a GRU updates the deterministic state using the previous latent and previous ego state, and heads predict the latent prior, reward, episode continuation and the decoded observation §4.1.1 eq. 9 fig. 3. The ego state used as input is s^i = [Δx, Δy, Δψ, v, ψ̇, v̇], which is the relative pose to the end waypoint of the reference lane, speed, yaw rate and acceleration §4.1.1. The paper reports that feeding only acceleration and yaw rate to the sequence model works better than the full state §4.1.1; the implementation section instead says acceleration and yaw rate are the inputs to the EDM §5.1.3. Section 5.2.2 compares the full state with the two-value version, and the card does not resolve which module each choice applies to.

Vehicle dynamics model. The ego state evolves by a parameterised physical model with parameters θ, s^i_t = f_θ(s^i_{t-1}, a_t) eq. 9. It is fitted by mean squared error on the observed next ego state eq. 10. The model is cited to Althoff et al. eq. 8 §3.4; the paper also notes a kinematic bicycle model is suitable only when lateral acceleration stays below 0.5 μg §3.4. The acceleration and steering maps are described only as mappings from the control input, without their forms §3.4.

Loss. The vehicle model is trained separately from the EDM §4.1.2. The EDM loss is L_pred + β_pri · L_pri + β_rep · L_rep eq. 11. The prediction term is the negative log-likelihood of observation, reward and continuation eq. 12; the decoder and reward head use symlog §4.1.2. The prior loss trains the transition with stop-gradient on the posterior, and the representation loss trains the posterior with stop-gradient on the prior, each clipped below at 1 (free bits) eq. 13.

Behaviour learning. The actor outputs a goal g_t ~ π(g | s^e_t) from the environment state s^e = {h, z}. A predefined controller a_t = u(g_t, s^i_t) produces the action, which makes the ego dynamics enter through feedback. The critic estimates values under the actor and controller §4.2.1 eq. 14. Imagined rollouts use the dynamics predictor, actor and controller from replayed latent starts, and λ-returns are computed from predicted rewards and values §4.2.1 eq. 15.

# Imagined behaviour rollout  [§4.2.1] [eq. 14] [eq. 15]
# Input:  latent starts from replayed inputs; actor, critic, controller
# Output: imagined states, actions, rewards and values for the actor loss

def imagine_rollout(starts, horizon, actor, critic, controller, model):
    # Start from latent states of replayed inputs  [§4.2.1]
    states = starts                              # {h, z, s^i}
    for step in range(horizon):                  # horizon not stated
        # Actor picks a goal from the environment state  [eq. 14]
        goals = actor(env_part(states))          # (B, dim_goal)
        # Controller turns the goal into an action  [eq. 14]
        actions = controller(goals, states.ego)  # (B, dim_act)
        # Dynamics predictor and vehicle model advance the state  [eq. 9]
        states = model.step(states, actions)
        # Predicted reward and value feed the lambda-return  [eq. 15]
        rewards = model.reward(states)           # (B,)
        values = critic(env_part(states))        # (B,)
    return states, rewards, values

# Critic: two-hot targets on symlog-transformed lambda-returns  [§4.2.2]
# Actor: return term / max(1, S) + entropy term + reachability term  [eq. 19]

Critic. Returns are symlog-transformed, two-hot encoded on K equally spaced bins, and the critic is trained with categorical cross-entropy against a stop-gradient target §4.2.2 eq. 16 eq. 17. The value is recovered with symexp of the bin-weighted sum eq. 18.

Actor. The actor is trained by backpropagating value gradients through imagined sequences §4.2.3. Its loss has three parts: the return term divided by max(1, S), an entropy bonus, and a reachability penalty that pulls the goal toward the ego state predicted h steps ahead eq. 19. S is the 5th-to-95th percentile range of returns, averaged with exponential decay §4.2.3.

PAD (policy adjustment during deployment). On a vehicle with new parameters θ', the original policy's action a_t is replaced by a'_t = f^{-1}_{θ'}(s^i_{t-1}, f_θ(s^i_{t-1}, a_t)), so the new car reaches the state the original car would reach §4.3 eq. 20 fig. 4. The paper notes PAD cannot help when the new car cannot physically reach those states, for example a smaller maximum steering angle §4.3.

PAT (policy augmentation during training). Imagined trajectories are generated under both the true parameters θ and a shifted set θ'. The world model state is extended with the context ctx = θ'/θ (all ones for the original vehicle). The critic loss is a weighted sum over the two contexts with weight ω_cont, and the actor is trained to maximise return under both §4.4 eq. 21 fig. 4.

Environment reward. R = c1 · R_prog + c2 · R_speed + R_term eq. 22, with progress R_prog = d_t − d_{t−1} in Frenet coordinates and speed term v / v_max §5.1.1. Setting: c1 = 1, c2 = 0.1, v_max = 50 km/h; terminal reward +10 on reaching the destination, −10 off-road, −20 on collision §5.1.1.

Hyperparameters

Results

Author-reported. Setup: MetaDrive simulator, traffic density 0.35, 100 episodes per seed set, three seed sets, final checkpoint §5.1 §5.2.1 tab. 1. Baselines: IDM (rule-based), SAC, PPO, DreamerV3 §5.1.2.

MetaDrive, T-Intersection and Roundabout tab. 1

nuPlan and CARLA unprotected left turn tab. 1

Metric keys: RW reward, RC route completion, SR success rate, CR collision rate tab. 1. The source table leaves the SAC and PPO nuPlan and CARLA cells empty; this card marks them "not reported".

Ablations, roundabout map (figure only, no numbers in text) fig. 7: full-state input vs two-value input; no controller modelling; no reachability loss; with and without PAT §5.2.2.

Robustness to shifted mass and steering (figure only) fig. 9: DreamerV3, DreamerV3-PAD, VDD and VDD-PAT over the 5 × 5 grid. Text says VDD is stable across mass changes and improves with larger steering scale; VDD-PAT significantly outperforms VDD at steering scale 0.5 §5.2.4.

Case study (figure only) fig. 10 fig. 11 fig. 12: same traffic seed, nominal and (0.75, 0.75) scale. DreamerV3 leaves the road in the shifted environment; VDD reaches the destination §5.2.5.

Paper vs code

No official code. Searched: the paper source and the arXiv abstract page for links (none). GitHub repository searches for "vehicle dynamics dreamer", "VDD Dreamer driving", "vehicle dynamics world model", "VDD metadrive" and "Huiqian Li world model" returned no repository that belongs to the authors. Nothing to compare against.

Pitfalls

• No code, so every low-level detail is open. Controller gains, the PID input and output definitions, the goal dimensionality, the reachability horizon h, the entropy scale, the critic bin count K, the λ and γ values, and the PAT weight are all not stated §4.2 eq. 14 eq. 19 eq. 21. Expect to tune them; §5.2.3 says the reachability penalty decides between collapse and slow convergence.
• Inputs to the vehicle and environment models disagree in wording. §4.1.1 says the sequence model uses only acceleration and yaw rate; §5.1.3 says those two feed the EDM. Pick one and test both.
• Steering model in eq. 8 is ambiguous. The text defines a_sw as the front steering angle, but the equation uses f_sw(a_lat), and a_lat is never defined. f_acc and f_sw are described only as mappings §3.4.
• Physical vehicle model is not identified. The paper uses a parameterised physical model with parameters θ but does not say how θ is initialised or estimated in the main experiments. The appendix reports online identification of θ after deployment, with only figures app. fig. 14.
• PAD needs an inverse. f^{-1}_{θ'} in eq. 20 is not given in closed form, and the paper does not say what happens when no inverse exists (for instance when the new steering limit is too small) §4.3.
• Scale of the "17% improvement". The text says VDD gains 17% success rate on the roundabout §5.2.1. In the table this is 0.70 vs 0.53, which is 17 percentage points, a 32% relative gain.
• Mixed or unclear claims. VDD is slightly worse than DreamerV3 on intersections (0.88 vs 0.91 SR), on nuPlan (0.77 vs 0.79) and in reward on the intersection map. The text says VDD converges more slowly and stays at a local optimum for about 1M steps on the roundabout §5.2.1.
• Parameter scope is narrow. The robustness study scales only mass and maximum steering angle §5.1.4. Other dynamics parameters, and combined shifts, are untested.
• Real-world data is weaker. On nuPlan, IDM beats both learned agents §5.2.1. The authors attribute this to IDM using vector input. The CARLA run changes the encoder and decoder inputs; the modified architecture is described only in words §5.1.1.
• Same hyperparameters across simulators. The paper states identical values for MetaDrive and CARLA §5.1.3. It does not say whether the CARLA runs were tuned separately.
• Cost. Imagined rollouts run the vehicle model and controller on every step, and PAT generates a trajectory under each of two contexts (θ and θ') from every replayed start §4.4. The paper gives no wall-clock or memory figures.

Lineage

• Builds on: Dreamer, DreamerV2 and DreamerV3 (Hafner et al.) for the RSSM, symlog, two-hot critic and return normalisation §4.1 §4.2. Also the vehicle model cited to Althoff et al. §3.4, and the ego-controllable / environment split discussed for ISODreamer-style models by Pan et al. §2.
• Compared against: IDM, SAC, PPO, DreamerV3 §5.1.2; DreamerV3-PAD is the DreamerV3 policy with PAD §5.2.4.
• Superseded by: not known from the content of this wiki.