Vision-Centric 4D Occupancy Forecasting and Planning via Implicit Residual World Models

In plain words

Self-driving cars have to guess what the road will look like a few seconds ahead so they can pick a safe path. Many recent systems try to imagine the whole future scene from scratch, including the static background that does not change. IR-WM takes a different route. It first builds a top-down map of the current scene from the camera images. For the next moment it predicts only what changes on that map: which objects move, and how the scene shifts because the car itself moves. Those changes are added to the previous map, and an alignment step corrects drift so that errors do not pile up.

The model also predicts which parts of space will be occupied by objects in the future, and a planner uses those future features to propose the car's path. The authors found that the predicted future features improve planning, at the cost of extra computation time. They evaluate on the nuScenes driving dataset and report better forecasting and lower trajectory error than the earlier methods they compare against, notably Drive-OccWorld.

TL;DR

• Vision-centric driving world model: a BEVFormer-based encoder builds a bird's-eye-view (BEV) state from camera images, and a future predictor outputs the residual change of that state per step §3.2, §4.1.3.
• Feature alignment with predicted semantics and ego motion gives small per-class IoU gains; the authors call the effect slight §3.2.2, §4.4, tab. 8.
• Author-reported on nuScenes: 4D occupancy forecasting beats Drive-OccWorld on every reported metric group tab. 1–3, and open-loop planning reaches an average L2 of 0.53 m vs 0.85 m for Drive-OccWorld tab. 4.
• The released code is on the ir-wm branch, not the default branch. Several details differ from the paper (losses, collision term, training schedule); see Paper vs code.

Problem

Vision-centric world models for end-to-end planning often regenerate the whole scene for both the current and the future state. This spends capacity on mostly static background and limits how much scene context is encoded from past and current observations §1. The authors move away from full-scene regeneration toward modelling the current state and its future changes §1.

All experiments use nuScenes and its occupancy labels §4.1.1. Training targets are voxel-level occupancy labels plus ego trajectories §4.1.1.

Method

Overview

IR-WM has three parts §3.2, fig. 2:

1. A scene encoder E (BEVFormer) turns past and current camera images into BEV features §3.2.1.
2. An autoregressive future predictor W with a streaming memory of BEV features predicts the residual change of the BEV state at each future step, conditioned on the ego trajectory and a high-level command §3.2.2.
3. Two task heads read the BEV state: an occupancy head D_o for 4D occupancy and a planning head D_p for trajectories §3.2.3.

The planning problem and the decomposition are written as eq. 1–eq. 5:

\tau_{1:f} = \mathcal{F}(I_{-h:0}, \tau_{-h:0}) eq. 1

S_{-h:0} = \mathcal{E}(I_{-h:0}, \tau_{-h:0}) eq. 2

S_t = \mathcal{W}(S_{-h:t-1}, \tau_{-h:t}) eq. 3

\tau_{t+1} = \mathcal{D}_p(S_t) eq. 4

O_t = \mathcal{D}_o(S_t) eq. 5

Here h is the number of past frames, f the number of future steps, S the BEV state and τ the ego trajectory §3.1. The occupancy head is an auxiliary output: it is used for supervision and interpretability, not as an input to planning in the default variant §3.1, §3.2.4.

Residual state update

The predictor does not output the next BEV state directly. It outputs a change ΔS_t, which is added to the previous BEV state §3.2.2, eq. 6:

S^{bev}_t = \Delta S^{bev}_t + S^{bev}_{t-1} eq. 6

The predictor is a stack of transformer layers that takes learnable BEV queries. Each layer runs deformable self-attention, deformable temporal cross-attention to the m memory frames, conditioning on the ego-trajectory embedding, and a feed-forward network that outputs ΔS_t §3.2.2. Reference points for the temporal cross-attention come from the ego-motion transform δτ_t, which is derived from the last two trajectory points, so that memory features are pre-aligned in time before aggregation §3.2.2.

Feature alignment

After the residual update, an alignment module modulates the BEV features with scale and shift parameters from layer normalization §3.2.2. Two separate MLPs produce the parameters: one from the predicted semantic occupancy map O^align_t and one from the ego-motion transform δτ_t §3.2.2. The modulated result is the feature that is pushed into the memory for the next step §3.2.2:

\hat S^{bev}_t = \gamma_s(O^{align}_t)\,\mathrm{LN}(S^{bev}_t) + \beta_s(O^{align}_t) + \gamma_e(\delta\tau_t)\,\mathrm{LN}(S^{bev}_t) + \beta_e(\delta\tau_t) §3.2.2

LN is layer normalization. A light head of MLPs on S^bev_t produces O^align_t, which is supervised with a cross-entropy loss L_align §3.2.2. The authors say the alignment slightly reduces error accumulation over time §4.4, tab. 8.

Occupancy and planning heads

The occupancy head applies an MLP and a channel-to-height mapping to lift each BEV feature into a vertical volume of dense semantic occupancy O_t. Rolling out several steps gives O_{0:f-1} for the current and future frames §3.2.3.

The planning head uses a learnable plan query that is concatenated with a command embedding §3.2.3, fig. 3. A transformer decoder cross-attends from this query to S_t + Adapter(S_t), where the adapter is a stack of convolutions that enhances features §3.2.3. An MLP then outputs the ego translation δτ_{t+1}, and the pose is updated as τ_{t+1} = τ_t + δτ_{t+1} §3.2.3.

Forecasting and planning coupling

The paper compares three ways the planner can use the forecast §3.2.4, fig. 4:

• Tightly coupled: the planner uses predicted occupancy to filter sampled candidate trajectories with a safety cost, then refines the chosen one from the BEV features §3.2.4, eq. 9, eq. 10.
• Semi-coupled (the default): the planner uses only the generated BEV features. Occupancy is auxiliary supervision and optional output §3.2.4.
• Fully decoupled: the planner reads only the current BEV features S_0, and no future features are needed at inference §3.2.4, eq. 11.

The authors adopt semi-coupled by default, based on their ablation [§3.2.4, tab. 5].

Training objective

The total loss is L = L_align + L_occ + λ_plan L_plan §3.3, eq. 14.

Occupancy uses a sum of three terms per future step: cross-entropy, a Lovász loss on IoU, and a binary free/occupied loss §3.3, eq. 12. The planning loss is an imitation term plus a collision term with weight λ_coll §3.3, eq. 13. Both λ_plan and λ_coll are set to 1 §4.1.3.

# Implicit residual rollout  [§3.2]  [fig. 2]
# Input:  images (B, T, N_cam, 3, H, W); ego trajectory (B, T, 2)
# Output: planned trajectory (B, F + 1, 2); aligned BEV (B, F, HW, C)

def rollout(images, ego_traj, command, num_future):
    # Encode history and current frame into BEV features  [§3.2.1]
    memory = scene_encoder(images, ego_traj)     # (B, M, HW, C)
    # Start from the newest BEV state  [§3.2.2]
    bev = memory[:, -1]                          # (B, HW, C)
    traj = zeros(B, 1, 2)                        # (B, t, 2)
    for step in range(num_future):
        # Predict only the change from the previous BEV  [§3.2.2] [eq. 6]
        delta = future_predictor(memory, traj, command, step)
        # Add the residual to the previous BEV  [§3.2.2] [eq. 6]
        bev = bev + delta                        # (B, HW, C)
        # Plan the next displacement from the un-aligned BEV  [§3.2.3]
        step_delta = planner(bev, command[:, step])
        traj = append(traj, traj[:, -1] + step_delta)
        # Predict semantics for the alignment step  [§3.2.2]
        semantics = semantic_head(bev)           # (B, HW, Z, K)
        # Align BEV with semantics and the ego transform  [§3.2.2]
        bev = feature_alignment(bev, semantics, traj)
        # Push the aligned BEV into the memory queue  [§3.2.2]
        memory = push(memory, bev)
    return traj, memory

# Planning loss  [§3.3] [eq. 13]
def planning_loss(pred_traj, gt_traj, valid, gt_boxes):
    # Distance per waypoint, averaged over valid steps  [§3.3]
    # Code uses an unsquared L2 norm  [code: planning_loss.py#L21-L27]
    error = norm(pred_traj - gt_traj, axis=-1)   # (B, F)
    imitation = masked_mean(error, valid)
    # Overlap of the ego box with GT boxes  [code: planning_loss.py#L36-L69]
    collision = box_overlap(ego_boxes(pred_traj), gt_boxes)
    return imitation + lambda_coll * collision

Training details (as stated in the paper)

The scene encoder is BEVFormer-based §4.1.3. Training uses AdamW with an initial learning rate of 2e-4, a cosine annealing schedule, and 8 NVIDIA A100 GPUs §4.1.3. The paper does not state the backbone, weight decay, warmup, gradient clipping or number of epochs. The released code sets these, as listed in the hyperparameter table.

Hyperparameters

Values marked "not stated" are absent from the paper. Code values are from the ir-wm branch at code_commit; configs are in projects/configs/.

Results

Author-reported. Setup: nuScenes, with nuScenes-Occupancy for fine-grained labels §4.1, tab. 1–4. The proposed model and most planning baselines use camera input; four planning baselines use LiDAR tab. 4. The model is a BEVFormer-based encoder with a three-layer future predictor, trained on 8 A100 GPUs §4.1.3. The baseline numbers come from Drive-OccWorld; whether they were re-run or copied is not stated. Hardware for the latency column is not stated.

Headline claims from the paper:

• Inflated GMO forecasting (nuScenes): IoU_c 40.8, IoU_f 37.2 vs 39.8 and 36.3 for Drive-OccWorld tab. 1.
• Fine-grained GMO (nuScenes-Occupancy): IoU_c 16.2, IoU_f 14.5 vs 13.6 and 12.0 tab. 1.
• Open-loop planning on nuScenes validation: average L2 0.53 m vs 0.85 m for Drive-OccWorld; average collision rate 0.17% vs 0.29% tab. 4. The abstract's "top performance" does not hold for average collision, where SSR is lower at 0.15% tab. 4.

Table 1 tab. 1: inflated GMO (nuScenes) and fine-grained GMO (nuScenes-Occupancy).

† SPC is SurroundDepth + PCPNet + Cylinder3D tab. 1.

Table 2 tab. 2: fine-grained GMO and GSO (nuScenes-Occupancy).

† SPC is SurroundDepth + PCPNet + Cylinder3D tab. 2.

Table 3 tab. 3: fine-grained MMO and MSO (nuScenes-Occupancy), per-class IoU~_f.

Table 4 tab. 4: open-loop planning on nuScenes validation. L2 in metres, collision in percent; lower is better. Aux. Sup. is the auxiliary supervision the method uses.

Table 5 tab. 5: coupling variants, fine-grained MMO and MSO on nuScenes-Occupancy validation, with planning L2 and latency. The latency device is not stated.

Table 6 tab. 6: temporal self-supervision on BEV features (fine-grained GMO and GSO).

Table 7 tab. 7: action conditioning by addition vs cross-attention (fine-grained MMO and MSO).

Table 8 tab. 8: feature alignment, per-class IoU_f on nuScenes-Occupancy validation.

Table 9 tab. 9: BEV resolution, fine-grained GMO on nuScenes-Occupancy validation.

The paper also includes qualitative visualisations of forecasts at different times of day fig. 5, described in words only here.

Paper vs code

Code: the repository yuyang-cloud/Drive-OccWorld is linked in the abstract. Owner yuyang-cloud matches co-author Yu Yang; the paper's author list includes Yu Yang. The default branch of that repository is the Drive-OccWorld baseline. The README links the IR-WM code on the ir-wm branch [README, News], and that branch is what we recorded: commit a83e4a2 ("update readme"). The ir-wm branch is Apache-2.0 (LICENSE). All paths below are relative to the repository root at that commit. The paper's default semi-coupled variant is the config at projects/configs/e2e/MMO_MSO_with_plan_semi_coupled.py.

Searched for an official repository: the abstract link, and the README. No other repository was used.

Differences found on reading the code (not run):

• Training path of the default config appears broken. The semi-coupled config uses detector type Drive_OccWorld MMO_MSO_with_plan_semi_coupled.py. Its forward_train reads ret_dict['next_bev_preds'] drive_occworld.py, but future_pred returns a dict in which that key is commented out drive_occworld.py. The Drive_OccWorld_V2 class does return it drive_occworldV2.py, and is used by the fully decoupled config only. As read, the default training path raises a KeyError. We did not run it.
• Planning loss is not squared. The paper writes the imitation term as a squared L2 norm §3.3, eq. 13. The code takes the square root of the summed squares, i.e. a Euclidean distance planning_loss.py.
• Collision term uses ground-truth boxes, not occupancy voxels. The paper says the collision term penalises waypoints that overlap occupied voxels of the target occupancy §3.3, eq. 13. The code computes overlap of the ego box with ground-truth future bounding boxes planning_loss.py, passed in from compute_plan_loss drive_occworld.py.
• Collision weights differ from λ_coll = 1. The planning head has three collision terms at margins 0.0, 0.5 and 1.0 with weights 2.5, 1.0 and 0.25 MMO_MSO_with_plan_semi_coupled.py. The paper states λ_coll = 1 §4.1.3. The imitation term has no explicit weight plan_head.py.
• Occupancy loss terms differ. The paper uses cross-entropy, Lovász and a binary occupancy term §3.3, eq. 12. The code's voxel loss uses cross-entropy, a semantic scene-class term, Lovász and a geometric scene-class term. There is no BCE term world_head_v1.py. Cross-entropy also uses class weight 5 for all non-empty classes world_head_v1.py.
• Averaging over future steps is not visible. The paper averages the occupancy loss over the f future steps §3.3, eq. 12. loss_occ returns one dictionary per term per decoder layer, with no 1/f factor visible world_head_v1.py.
• Occupancy loss is at 256 x 256 x 20. The config's grid is 512 x 512 x 40 MMO_MSO_with_plan_semi_coupled.py. The loss head defaults to output scale 2, so targets are downsampled by majority vote to 256 x 256 x 20 world_head_v1.py world_head_v1.py.
• Teacher forcing for the first 12 epochs. During training, the first 12 of 24 epochs roll out with ground-truth trajectories rather than predicted ones drive_occworld.py MMO_MSO_with_plan_semi_coupled.py. The paper does not mention this.
• Semantic conditioning is an argmax. The code takes the argmax of the predicted semantics, detached from the graph, because sem_gt_train is False conditionalnorm.py MMO_MSO_with_plan_semi_coupled.py. The paper describes the conditioning as coming from the predicted map O^align §3.2.2.
• Ego conditioning uses a transform, not δτ. The ego branch takes the 3 x 4 future-to-history transform, positional-encoded to a 144-dimensional input conditionalnorm.py conditionalnorm.py. The paper writes this term as a function of δτ_t §3.2.2.
• Sum of two modulations. The code sums the semantic and ego modulated outputs conditionalnorm.py. The paper's formula applies layer normalization to the same input in both terms §3.2.2. Both are the same idea, but the normalization placement is not identical.
• Temporal self-supervision is off in released configs. use_lwm defaults to False drive_occworld.py and no released config sets it. The TSS ablation tab. 6 cannot be reproduced from the released configs.
• Pretrained initialisation is not in the paper. The fine-grained configs start from an R101 FCOS3D pretrain checkpoint action_condition_MMO_MSO.py. The e2e config starts from the fine-grained MMO/MSO epoch-24 checkpoint MMO_MSO_with_plan_semi_coupled.py.
• Planner inputs. The planner's forward takes BEV features and a command only plan_head.py. That matches the caption that ego status is not used by planning tab. 4. The BEVFormer perception head uses CAN bus MMO_MSO_with_plan_semi_coupled.py; the paper does not say whether the encoder uses it.
• Horizon differs by config. Fine-grained configs use 4 future frames action_condition_MMO_MSO.py. The e2e config uses 5 MMO_MSO_with_plan_semi_coupled.py. The paper's dataset section says four timestamps §4.1.1, and its tables use both 2 s and 2.5 s tab. 2, tab. 3.

Pitfalls

• The default training path does not run as released (as read). Use the V2 class or fix the missing next_bev_preds before reproducing the semi-coupled result. See Paper vs code.
• Use the ir-wm branch. The default branch is the Drive-OccWorld baseline, not IR-WM.
• Compare numbers only at the same horizon. Fine-grained tables use 2 s and the MMO/MSO and e2e settings use 2.5 s tab. 3, tab. 5. The paper's text says four timestamps §4.1.1.
• Ground-truth trajectories are used for 12 of 24 epochs. If you reproduce from scratch, the early rollout uses GT ego trajectories, and the trained model may behave differently when it switches to predicted ones at epoch 12 drive_occworld.py.
• Pretraining is required for the e2e result. The e2e config starts from a fine-grained checkpoint MMO_MSO_with_plan_semi_coupled.py. The paper does not say so, so a from-scratch run will differ.
• Collision signal differs from the paper's description. The code's collision term checks the predicted ego box against ground-truth boxes of other objects planning_loss.py. An implementer who follows the paper's voxel-overlap description will get a different planning signal.
• Planning error feeds back into the memory. The memory alignment uses the planned trajectory §3.2.2, so a bad plan also distorts the aligned BEV used by the next step. The paper does not discuss this.
• Unstated values. Weight decay, warmup, gradient clipping, epochs, backbone, head counts and loss class weights are not in the paper. See the hyperparameter table for the code values.
• Latency hardware is not stated. The latency column tab. 5 cannot be compared with other papers without the device.
• Baseline provenance is not stated. The paper does not say whether the Drive-OccWorld and other baseline numbers were re-run or copied from their papers.
• Input frame count is ambiguous. The paper says "two past frames and the current frame" §4.1.1; the config's queue_length = 2 is described as the history frame count MMO_MSO_with_plan_semi_coupled.py. Whether the current frame is additional is not clear from either.
• No ground-truth semantic conditioning in released configs. The early-epoch ground-truth path in the detector needs sem_gt_train True, and every released config sets it to False MMO_MSO_with_plan_semi_coupled.py. Only the trajectory teacher forcing above applies.
• Average collision is not best in the paper's own table. SSR reports 0.15% vs 0.17% for IR-WM tab. 4. Treat the abstract's "top performance" as about L2 and forecasting, not collision.

Lineage

• Builds on: BEVFormer as the scene encoder §3.2.1; Drive-OccWorld for the overall setup and the fine-grained occupancy comparison §2, tab. 1–4. The Drive-OccWorld arXiv id is 2408.14197, from the code README.
• Compared against: OpenOccupancy-C, PowerBEV-3D, BEVDet4D, Cam4DOcc (OCFNet), OccProphet, Drive-OccWorld for forecasting tab. 1–3; IL, NMP, FF, EO, ST-P3, UniAD, VAD, OccNet, GenAD, UAD, RenderWorld, SSR, PARA-Drive and GaussianAD for planning tab. 4.
• Superseded by: not known from content already in this wiki.