Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving

In plain words

A self-driving car's planner usually learns from recorded human driving, so it is good at staying in the middle of the lane and unsure about situations it never saw, such as drifting toward the edge. This paper builds a video model of the road, a "world model", that takes the current camera views and a planned path and imagines what all the cameras would show if the car drove that way. The planner can then compare several imagined futures and pick the one that looks safest.

The model is a video generator that produces the six surround cameras together and keeps them consistent with each other. It is built on an image generator and extended with layers for time and for the camera views. To keep the views matching where they overlap, the generator first makes the cameras that face in fixed directions and then fills in the gaps between them. The imagined futures are scored with a map check (stay on the road, stay near the lane centre) and an object check (keep away from other road users), both measured by running standard perception models on the generated images. The authors also use the model to create unusual scenes, such as a car that is off-centre, night or rain, and a turn at an intersection, to test the planner.

TL;DR

• Drive-WM is a latent video diffusion model for six-camera nuScenes driving video, conditioned on 3D boxes, HD map and BEV layouts, text and ego motion.
• Multiview consistency comes from a factorized generator: reference views first, then stitched views conditioned on their neighbours §3.2.
• For planning, it scores candidate trajectories from VAD with image-based rewards in a tree rollout §4.
• Author-reported gains: video FID 15.8 and FVD 122.7 on nuScenes tab. 1; open-loop tree planning L2 0.80 m and collision 0.26% versus 1.02 m and 0.93% for VAD with random commands tab. 3. The paper claims to be the first to explore world models for end-to-end planning §1.

Problem

Earlier driving generators produced single-view video or multiview images, not multiview video §1. An end-to-end planner trained only on expert trajectories struggles when the ego vehicle starts off the lane centre, and the paper argues a world model could reveal such failures before the car acts §1 fig. 2 app. A.4. Building that model requires high-resolution pixel generation, consistency across overlapping cameras, and conditioning on several heterogeneous inputs at once §1.

Method

The method has two parts: a multiview video generator §3 and a planning loop that uses it §4.

Notation. A clip is a tensor of T frames by K views with height H and width W §3.1. A latent is encoded per frame and per view, and diffused with a noise schedule indexed by a diffusion time τ. The paper's denoising objective predicts the injected noise §3.1:

\mathbb{E}_{\mathbf{z}\sim p_\text{data},\,\tau\sim p_\tau,\,\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})}\left[\left\|\boldsymbol{\epsilon}-f_{\theta,\phi,\psi}(\mathbf{z}_\tau;\mathbf{c},\tau)\right\|_2^2\right] eq. 1

Here z_τ = α_τ z + σ_τ ε, the condition is c, p_τ is uniform, and θ, φ, ψ are the spatial, temporal and multiview parameters §3.1.

Step 1: lift an image diffusion model to multiview video §3.1.

1. Train a single-view, conditional image latent diffusion model first. This fixes the spatial parameters θ §3.1 app. B.1.
2. Add temporal layers after each spatial block, following VideoLDM. The temporal layers apply a 3D convolution over time and a self-attention over frames §3.1.
3. Add multiview layers: self-attention across the K views, so that views share style and layout §3.1.
4. Freeze θ and fine-tune only φ and ψ on video §3.1 app. B.1.

# Joint multiview video training step  [§3.1] [eq. 1]
# Input:  clip (T, K, 3, H, W); condition c; frozen spatial weights theta
# Output: scalar denoising loss

def training_step(clip, condition):
    # Encode every frame and view into a latent  [§3.1]
    latents = encode(clip)                  # (T*K, C, H', W')
    # Sample a diffusion time uniformly  [§3.1]
    tau = sample_uniform_time()
    # Diffuse the latents with Gaussian noise  [§3.1]
    noise = sample_gaussian(latents.shape)
    noisy = alpha(tau) * latents + sigma(tau) * noise
    # Predict the noise from the noisy latents and the condition  [eq. 1]
    predicted = denoiser(noisy, condition, tau)
    return squared_error(noise, predicted)

Step 2: factorize the joint views §3.2. Modelling all K views jointly is inexpensive but gives only loose consistency in overlapping regions. The paper splits the views into reference views x_r (front, back-left, back-right in nuScenes) and stitched views x_s (front-left, back, front-right), which overlap their neighbours §3.2 fig. 4. The joint distribution is then written as eq. 2 to eq. 4:

p(\mathbf{x}_{1,\dots,K}) = p(\mathbf{x}_1)\,p(\mathbf{x}_2\mid\mathbf{x}_1)\cdots p(\mathbf{x}_K\mid\mathbf{x}_1,\dots,\mathbf{x}_{K-1}) eq. 2

p(\mathbf{x}) = p(\mathbf{x}_s,\mathbf{x}_r) = p(\mathbf{x}_r)\,p(\mathbf{x}_s\mid\mathbf{x}_r) eq. 3

p(\mathbf{x}) = p(\mathbf{x}_s,\mathbf{x}_r\mid\mathbf{x}_{pre}) = p(\mathbf{x}_r\mid\mathbf{x}_{pre})\,p(\mathbf{x}_s\mid\mathbf{x}_r,\mathbf{x}_{pre}) eq. 4

eq. 2 is the full autoregressive factorization, which the paper calls too slow. eq. 3 uses two groups. eq. 4 adds context frames x_pre, for example the last two frames of the previous clip §3.2.

1. Generate the reference views with the joint model §3.2.
2. Generate each stitched view with the same pipeline, plus its two neighbouring reference views and its own previous frame as extra image conditions §3.2 app. B.2.
3. In training, sort the six views clockwise as x_0 to x_5. Each sample is a stitched view x_i, its two neighbours (i-1) and (i+1) mod 6 as references, and its previously generated frame x_i' app. B.2. One stitched view is trained per iteration app. B.2.

# Factorized stitched-view generation  [§3.2] [app. B.2]
# Input:  reference views (F, BL, BR); previous frame per stitched view
# Output: stitched views (FL, B, FR)

def generate_stitched(references, previous, actions):
    stitched = {}
    for view in ["FL", "B", "FR"]:
        # Use the two neighbouring reference views and the previous
        # frame of this view as conditions  [app. B.2]
        neighbours = reference_neighbours(references, view)
        conditions = neighbours + [previous[view]]
        stitched[view] = sample_view(conditions, actions)
    return stitched

reference_neighbours and sample_view are placeholders. The paper names only the FL pair (F and BL) app. B.2; the B and FR pairs follow from the same clockwise-neighbour rule app. B.2 and are not written out.

Step 3: unified conditioning §3.3. Each condition is mapped to a d-dimensional token sequence and all tokens are concatenated eq. 5:

\mathbf{c}_t = [\mathbf{i}_0,\,\mathbf{l}_0,\,\mathbf{e}_0,\,\mathbf{a}_t] \in \mathbb{R}^{(n+k+m+2)\times d} eq. 5

• Image tokens i: initial context frames and reference views, encoded with a ConvNeXt encoder §3.3.
• Layout tokens l: 3D boxes and HD map projected to a 2D perspective view, plus BEV segmentation §3.3 app. C.1.
• Text tokens e: a CLIP text encoder over a sentence built from view, weather and light §3.3.
• Action tokens a: the ego movement (Δx, Δy) per step, mapped by an MLP to two tokens §3.3. Appendix C.1 instead says the action-based data uses speed and steering; see Pitfalls.

Subscript 0 is the current real frame and subscript t the t-th generated frame §3.3. The tokens are attached by cross-attention in each frame §3.3 fig. 3.

Step 4: planning with the world model §4. At each step the pre-trained planner VAD samples trajectory candidates from the real multiview images. Each candidate is turned into per-step displacements (x_{t+1} − x_t, y_{t+1} − y_t). The world model renders the future for each candidate, a reward scores it, and the best candidate extends a planning tree §4.1 fig. 5.

# Tree-based rollout with image-based rewards  [§4.1] [§4.2]
# Input:  current multiview frames; planner; world model; perception models
# Output: the best-scoring trajectory at this step

def plan_next_step(frames, planner, world_model, perception):
    # Sample candidates per command: straight, left, right  [§5.4]
    candidates = planner.sample(frames)
    futures = []
    for trajectory in candidates:
        # Convert positions into per-step displacements  [§4.1]
        actions = displacements(trajectory)
        # Render the multiview future for these actions  [§3.3]
        video = world_model.generate(frames, actions)
        # Detect objects and map elements in the generated frames  [§4.2]
        map_part = map_reward(perception, video)
        object_part = object_reward(perception, video)
        # Total reward is the product of the two parts  [§4.2]
        reward = map_part * object_part
        futures.append((reward, trajectory))
    best = max(futures)
    return best.trajectory

The map reward has two factors: distance from the curb and centreline consistency. The object reward uses longitudinal and lateral distance to other road users. Perception is BEVFormer for 3D objects and MapTR for the online HD map §4.2. The paper also suggests a GPT-4V judgement of the generated videos as a non-vectorized reward, shown qualitatively only §4.2 app. A.5.

Hyperparameters

Results

Author-reported throughout. Setup: nuScenes val for generation and planning (Waymo appears only qualitatively). Planning uses VAD as the base planner, open loop.

Table 1 (a): generation quality tab. 1 Multi-view rows test multi-view image generation, which removes the temporal layers §5.2. Video rows test video generation.

Table 1 (b): generation controllability tab. 1 Layout-conditioned generation, evaluated with pretrained perception on generated views.

Table 2 (a): unified condition ablation tab. 2 Layout-based video generation on nuScenes val §5.3.

Table 2 (b): temporal and multiview tuning tab. 2

Table 2 (c): factorized generation tab. 2

Table 3: planning on nuScenes, open loop tab. 3 Command selection: the GT command for VAD, a random command for VAD, and Drive-WM's tree planner choosing among three commands §5.4.

Table 4: reward ablation for the tree planner tab. 4

Table 5: out-of-domain ego deviation tab. 5 Ego shifted 0.5 m laterally §5.4 app. A.4.

Qualitative only: multiple futures app. A.1, rare night and rain scenes app. A.2, weather and lighting edits app. A.3, counterfactual turns and off-road driving §5.5 fig. 7, Waymo high-resolution images and video app. A.6, and a single GPT-4V reward example app. A.5. None of these has a number.

Paper vs code

The official repository is https://github.com/BraveGroup/Drive-WM at commit eabe3c63bf362a2edf0fb6cbcb908424d2e18980, linked in the paper's title block. It is Apache-2.0 licensed LICENSE. It is a copy of the diffusers library (version 0.24.0.dev0 [code: src/diffusers/init.py#L1]) and the README says the code is "based on the open-source project diffusers" README.md.

The repository does not contain the method. Its to-do list has four unchecked items, including "Training codes for all models", and the released weights for the conditional image, conditional video and action-conditioned models are also unchecked README.md. A search of the tree finds no Drive-WM model, layout encoder, factorization, planner, reward or data-curation code. So none of the hyperparameters, the factorization procedure, the reward or the tree search can be checked against code. The paper's values stand as author statements only.

Pitfalls

• Action definition differs across the paper. Sections §3.3 and §4.1 define the action as the ego displacement (Δx, Δy) per step. Appendix C.1 says the action-based data uses vehicle speed and steering per frame, and the action examples use steering angle in degrees app. C.1 app. A.3. The paper does not say which signal the action-based model used, and no weights or code are released to settle it.
• Single-view or multi-view image result. The text calls the 12.99 FID "single-view image generation" §5.2. Table 1 marks the same row as multi-view tab. 1, and §5.2 describes multi-view image generation as removing the temporal layers. The table reading is more likely. Implementers should not assume single-view.
• "Combined reward outperforms each sub-reward". Table 4 shows the object reward alone gives the same average L2 (0.80 m) as the combined reward; the combined reward is only better on collision (0.26% against 0.27%) §5.4 tab. 4.
• "Near normal levels" after OOD fine-tuning. Average L2 goes from 1.02 to 0.82 m against 0.72 m on normal data, but collision goes only from 1.59% to 0.91%, against 0.22% on normal data §5.4 tab. 5.
• Training and inference planning conditions differ. Training uses ground truth ego motion for the planning condition and inference uses VAD output app. C.1. The paper says this makes predictions consistent with the planned trajectory, but it does not measure the gap.
• Reference pair for B and FR is not written. Appendix B.2 gives the FL pair (F and BL). The B and FR pairs follow from the clockwise neighbour rule in training app. B.2, not from a stated list.
• Unstated diffusion details. The noise schedule, how dropped conditions are replaced, the sampling seed, and the DDIM setting beyond η = 1.0 are not stated §3.1 app. B.1. Reproducing the sampler exactly needs these.
• Chained generation. Training uses clips of T = 8 frames app. B.1, while evaluation segments are about 40 frames app. D.1, built by feeding generated frames back in app. B.1. Drift over long chains is not measured.
• Planner details are missing. The reward formulas and thresholds, the tree depth, candidates per command, the horizon, and whether the whole trajectory or only its first step is executed are not stated §4.2 §5.4. The paper says the best candidate extends the tree and the process repeats §4.1.
• Rewards inherit perception error. The map and object rewards run BEVFormer and MapTR on generated images §4.2. The paper gives no measure of how often these detectors are wrong on generated frames.
• Baseline numbers are copied or re-run, not stated. The source of the BEVGen, BEVControl, MagicDrive, DriveGAN and DriveDreamer numbers is not stated. Single-view and multi-view rows use different protocols tab. 1.
• KPM has no variance and no bounds. KPM is an average ratio of matched points on 8 frames per scene using LoFTR §5.1 app. D.2. No standard deviation or per-scene detail is reported, and the paper does not say whether values above 100% can occur.
• Planning is open loop only. L2 and collision are computed on logged scenes §5.1. No closed-loop driving result is given.
• Qualitative claims without numbers. The GPT-4V reward, the Waymo results, the pedestrian generation and the counterfactual events have no quantitative evaluation app. A.5 app. A.6 §5.5.
• Factorization costs a little FID. Factorized FID is 16.4 against 15.8 for joint modeling, with KPM rising from 45.8% to 94.4% tab. 2.
• No released model. Weights and training code are listed as unreleased README.md. Everything here comes from the paper alone.

Lineage

• Builds on: Stable Diffusion §5.1 and VideoLDM for temporal layers §3.1 (not carded); VAD as the planner §5.4 (not carded); BEVFormer and MapTR as perception §4.2 (not carded); ConvNeXt §3.3 and CLIP §3.3 for encoders.
• Compared against: BEVGen, BEVControl, MagicDrive, DriveGAN, DriveDreamer, LayoutDiffusion and GLIGEN tab. 1.
• Superseded by: not known from content in this wiki.