Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving

In plain words

A self-driving car needs a rough idea of what will happen next: where other vehicles and people will be, and which parts of the road will stay free. This paper builds a "world model" for that. It looks at the camera images from the last few moments, builds a 3D picture of the space around the car, and predicts how that picture changes over the next two seconds. The picture marks which parts of space are occupied and how the occupied parts move.

The prediction can be steered. You can tell the model what the car is doing, such as its speed, its turn or a high-level command like "turn left", and it imagines the future that matches. The authors also use the prediction to drive. A planner tries many possible paths, scores each one by whether it runs into predicted occupied space or leaves the road, and keeps the cheapest. The chosen path is fed back into the model, so the forecast and the plan keep going step by step.

TL;DR

• A camera-only world model that forecasts future 4D occupancy and flow over two seconds §3.2 app. C.1.
• It adds ego-motion normalisation to its memory and takes actions (velocity, steering, trajectory, command) through a cross-attention layer §3.2 §3.3.
• It plans by sampling trajectories and scoring them on the forecast. Author-reported, the planner's average L2 error is 0.85 m against 1.03 m for UniAD under the NoAvg protocol tab. 5.
• The released code covers the occupancy-forecasting configs only. No planning config and no Lyft config is released, and the released configs use CAN-bus ego motion as the action, not the predicted trajectory. See Paper vs code.

Problem

Planners that map sensors straight to trajectories lack world knowledge of how a dynamic scene evolves, which the authors argue weakens generalisation and safety §1. Most world models for driving either generate video, to create training data, or pretrain encoders. Few use the forecast in planning §1 §2.1. Earlier 3D forecasting methods predict future occupancy from history alone, and they do not condition on ego actions or plan app. A.2. The method assumes multi-view camera input with two history frames plus the current frame, and a future horizon of four steps at 0.5 s, i.e. two seconds on nuScenes app. C.1 app. C.3.

Method

Preliminary

An end-to-end driver maps past sensor observations and past ego trajectories to future ego trajectories eq. 1 §3.1. A driving world model maps past observations and ego actions to plausible future environment states eq. 2 §3.1. Drive-OccWorld combines the two in an autoregressive loop. The world model \mathcal{W} predicts the next state from observations, past states and actions eq. 3. An occupancy-based planner \mathcal{P} then picks the trajectory with the lowest occupancy cost f_o among sampled proposals eq. 4 §3.1. For action-controllable generation the planner is removed, which avoids leaking the ego status. For planning, the chosen trajectory becomes the action for the next forecast step §3.1.

4D forecasting with the world model

The model has three parts §3.2.

• History encoder. It takes multi-view camera images and produces BEV embeddings with a BEVFormer-style visual encoder §3.2 app. C.3.
• Memory queue with conditional normalisation. Each historical BEV embedding is layer-normalised without learned affine parameters, then rescaled and shifted by parameters (\gamma^*, \beta^*) derived from a condition eq. 5 §3.2. Three conditions are described:
  • Semantic: a light head with an argmax predicts voxel-wise semantic labels, which are one-hot encoded and convolved into (\gamma^s, \beta^s) §3.2.
  • Ego motion: the 3×4 ego transform between timestamps -t and +t is flattened, encoded and passed through an MLP to give (\gamma^e, \beta^e) §3.2.
  • Agent motion: voxel-wise 3D backward centripetal flow, which points from each voxel at time t to its instance centre at t-1, is encoded into (\gamma^f, \beta^f) §3.2.
• World decoder. An autoregressive transformer with learnable BEV queries. Each layer runs deformable self-attention, temporal deformable cross-attention to the memory, cross-attention to the action embedding, and a feed-forward network §3.2 app. B.1. Prediction heads convert the BEV features to 3D occupancy and flow with a channel-to-height operation §3.2 fig. 6.

\tilde{F}^{bev} = \gamma^* \cdot \mathrm{LayerNorm}(F^{bev}) + \beta^* eq. 5

The loop below shows how the parts fit at test time. It is written from the paper's description and the code's control flow, not copied from either. The code's rollout is at drive_occworld.py.

# Forecast-then-plan rollout  [§3.4] [eq. 3] [eq. 4]
# Input:  h + 1 camera frames; command per future step
# Output: occupancy per future step; one trajectory per future step

def rollout(frames, command, candidates, steps):
    # Encode each history frame into BEV features  [§3.2]
    bev_history = [history_encoder(frame) for frame in frames]
    # Keep the newest memory frames, normalised by ego motion  [§3.2] [eq. 5]
    memory = normalise_with_ego_motion(bev_history[-memory_len:])
    trajectories = []
    occupancies = []
    for step in range(steps):
        # Choose the action condition for this step  [§3.3] [§3.4]
        action = action_for(step, trajectories)
        # Predict the next BEV features and decode occupancy  [§3.2]
        bev_next = world_decoder(memory, action)
        occupancy = occupancy_head(bev_next)
        # Candidates are grouped by command: left, forward, right  [§3.4]
        group = candidates[command[step]]
        # Score every candidate in the group on the forecast  [§3.4] [eq. 4]
        costs = occupancy_cost(group, occupancy)
        # Keep the cheapest candidate as this step's trajectory  [§3.4]
        trajectories.append(group[argmin(costs)])
        occupancies.append(occupancy)
        # Push the forecast into memory for the next step  [§3.1]
        memory = push(memory, bev_next)
    return occupancies, trajectories

Action-controllable generation

Four action types are defined §3.3:

• Velocity (v_x, v_y) in m/s, decomposed along the ego axes.
• Steering angle, read from the steering sensor and converted to curvature in m⁻¹ following VAD.
• Trajectory (\Delta x, \Delta y) in metres to the next timestamp.
• Command: forward, turn left or turn right.

Each action is Fourier-embedded, the embeddings are concatenated, and learned projections fuse them to the width of the decoder's conditional cross-attention §3.3. The paper reports that this cross-attention interface controls the output better than adding the embedding to the BEV queries §4.4 tab. 7.

End-to-end planning with the world model

• Occupancy cost. The total cost is a sum of three factors §3.4. The agent-safety term penalises overlap with occupied cells of other road users, and also penalises trajectories that come too close laterally or longitudinally. The road-safety term penalises leaving the drivable area. The learned-volume term is a 2D cost map predicted from the decoder's BEV features, inspired by ST-P3.
• Candidates. A trajectory sampler follows ST-P3 and places candidates on a 2D grid around the ego vehicle, guided by the high-level command §3.4. The number of candidates is not stated in the paper.
• Selection. The planner returns the candidate with the minimum total cost §3.4.
• BEV refinement. The selected trajectory is encoded, concatenated with a command embedding to form an ego query, and cross-attends to the forecast BEV features. An MLP then regresses the final trajectory §3.4.
• Training. The planning loss has a max-margin term that makes the expert trajectory cheaper than sampled alternatives app. B.2 eq. 8, an L2 imitation term, and a collision term over obstacle grids §3.4 eq. 8.

Training objective

The total loss is the sum of a normalisation loss, a forecasting loss and a planning loss, with no weights shown eq. 6 app. B.2. The normalisation loss is cross-entropy on the historical semantic predictions and L1 on the historical flow app. B.2. The forecasting loss has L1 on flow and, on occupancy, an average over future frames of cross-entropy, Lovász and binary occupancy terms eq. 7 app. B.2. The planning loss is in eq. 8.

Hyperparameters

Results

Author-reported. Setup: nuScenes and Lyft-Level5 for inflated (movable-object) occupancy and flow, nuScenes-Occupancy for fine-grained occupancy, and open-loop planning on nuScenes §4.1. The paper reports a 2 s horizon for nuScenes and 0.8 s for Lyft-Level5, citing annotation rates of 2 Hz and 5 Hz app. C.1. Baselines are Cam4DOcc, PowerBEV-3D, OpenOccupancy-C, SPC and planners UniAD, VAD, ST-P3, OccNet, BEV-Planner and Drive-WM tab. 1 tab. 2 tab. 5. The "A" variant uses ground-truth actions and "P" uses the predicted trajectory §4.2.

Table 1 — Inflated GMO and flow, and fine-grained GMO tab. 1

Table 2 — Fine-grained GMO and GSO on nuScenes-Occupancy tab. 2

Table 3 — Controllability with action conditions (ablation setting, 1 s horizon) tab. 3

Table 4 — Planning upper bound with ground-truth versus predicted trajectory as action tab. 4

Table 5 — End-to-end planning on nuScenes, open loop tab. 5

Protocols: † is NoAvg, ‡ is TemAvg, and ‡* adds ego status to the planner tab. 5.

Table 6 — Conditional normalisation ablation (no action conditions) tab. 6

Table 7 — Action-conditioning interface ablation tab. 7

Table 8 — Occupancy cost factors and BEV refinement tab. 8

Table 9 — Input frames, memory length, and latency on an A6000 GPU tab. 9

Table 10 — Semantic occupancy loss ablation tab. 10

Headline claims, as the authors state them §4.2 §4.3:

• Inflated GMO, nuScenes: Drive-OccWorld^P gets mIoU_f 36.3 versus 26.8 for Cam4DOcc, and VPQ_f 25.1 versus 20.0 for PowerBEV-3D tab. 1.
• Lyft-Level5: mIoU_f 39.7 versus 33.6 for Cam4DOcc, and VPQ_f 33.4 versus 28.2 for Cam4DOcc tab. 1.
• Planning, NoAvg: the average L2 is 0.85 m against 1.03 m for UniAD and the average collision rate is 0.29 % against 0.31 % tab. 5.
• Planning, TemAvg: the average L2 is 0.47 m against 0.69 m for UniAD tab. 5. The text's relative gains of 33 %, 22 % and 9.7 % at 1 s, 2 s and 3 s are against UniAD† §4.3.

Paper vs code

Code: yuyang-cloud/Drive-OccWorld at commit 3bd90b39fc83c944e047b35e26c46b3807b203af (Apache-2.0), linked from the project page. Citations below use paths relative to the repository root. Paper table, equation and figure numbers are the ones derived from source order; see Pitfalls.

1. No planning config is released. Every released config sets turn_on_plan = False action_condition_GMO.py, and none of them passes a plan_head to the model. The planner code exists drive_occworld.py but cannot be switched on from the released configs. The planning results tab. 5 and the predicted-trajectory rows tab. 1 tab. 3 cannot be reproduced from these configs.
2. Predicted trajectories are not the action in the released configs. The paper says the predicted trajectory serves as the action in training and testing §3.4. In code, the action comes from the CAN bus of the future frame whenever use_can_bus is set world_head_base.py, and every released config sets use_can_bus=True and use_plan_traj=False action_condition_GMO.py. The predicted-trajectory branch is skipped.
3. Ground-truth warm-up. When planning is on, the code uses the ground-truth trajectory as the action for the first 12 epochs of training and the predicted trajectory afterwards drive_occworld.py. The paper says predicted trajectories are used in training from the start §3.4.
4. Normalisation is partial. The paper's full model uses semantic, ego and agent-motion normalisation §3.2. The released configs set sem_norm=False, ego_motion_ln=True and obj_motion_ln=False action_condition_GMO.py. Only the ego-motion normalisation is active. The semantic and motion normalisation losses are commented out in the forward pass drive_occworld.py.
5. Fourier embedding is off. The paper Fourier-embeds the actions §3.3 tab. 7. Every released config sets use_fourier=False action_condition_GMO.py. Steering and trajectory actions are also disabled action_condition_GMO.py action_condition_GMO.py. Velocity is the x and y components only world_head_base.py.
6. Flow uses a separate head. The paper describes one world decoder that predicts occupancy and flow §3.2. The code builds a second copy of the head with three output classes for flow, with its own parameters drive_occworld.py.
7. Occupancy loss terms differ. The paper's Eq. 7 averages cross-entropy, Lovász and binary occupancy over future frames eq. 7. The code uses cross-entropy with class weight 5 for non-free classes, Lovász, a semantic scene-completion term, and a geometric term built from precision, recall and specificity world_head_v1.py semkitti_loss.py. The semantic term is not in the paper. The loss is applied to the output of every decoder layer and summed world_head_v1.py.
8. Loss resolution differs from the grid. The paper states the 512×512×40 voxel grid app. C.3. The occupancy losses are computed at 256×256×20: predictions are trilinearly resized, and ground truth is block-reduced by two along each axis world_head_v1.py.
9. Flow loss is Smooth-L1. The paper says L1 §3.2 app. B.2. The code uses 0.5 × Smooth-L1 world_head_v1.py semkitti_loss.py.
10. The planner cost has extra terms. The paper lists agent-safety, road-safety and learned-volume costs §3.4. The code adds a headway cost, which penalises occupied cells 10 m ahead cost.py. Comfort, progress and lane-divider costs are commented out cost.py.
11. Planner class IDs are hard-coded. Instance classes are 2–10 and the drivable area is class 11 plan_head.py. These follow the 17-class fine-grained label space that the detector also uses for vehicle IDs drive_occworld.py. The paper does not state the label mapping.
12. Planning loss weighting. The code multiplies the cost-based planning terms by 0.1 on the v1 path drive_occworld.py. The paper's Eq. 8 has no such weight eq. 8.
13. Training uses ground-truth occupancy for the planner. When planning is on, the training-time cost is computed on ground-truth occupancy drive_occworld.py. At test time it uses the predicted semantic occupancy drive_occworld.py. The paper does not describe this.
14. Backbone and schedule are in code only. The paper names the BEVFormer encoder but not the image backbone §3.2. The code uses ResNet-101 with DCNv2, initialised from an FCOS3D checkpoint, with backbone learning rate 0.1× and 24 epochs action_condition_GMO.py action_condition_GMO.py action_condition_GMO.py action_condition_GMO.py.
15. Lyft is not reproducible from the repo. The loader has a use_lyft flag loading_instance.py, but no released config sets it. The Lyft results tab. 1 cannot be reproduced from the released configs.
16. CAN-bus fields are rewritten at load time. The loader overwrites the translation and the patch-angle entries of each sample's CAN-bus vector nuscenes_dataset.py. The released configs read entries 0–2 and 17 of that vector as action input world_head_base.py. Any other ego-pose source must provide the same fields.

Pitfalls

• Numbering. Table, figure and equation numbers in this card follow the order they appear in the source, not the LaTeX labels. For example, the source label tab8 is the planning upper bound, which is Table 4 in this card. The paper's compiled PDF should be checked against these numbers before quoting them.
• "Percent" in the text. The text reports gains as percentages, e.g. "9.4 %" over Cam4DOcc on timestamp-weighted mIoU. These are absolute differences in mIoU or VPQ points: 37.4 − 28.0 = 9.4 tab. 1. The 5.1 % VPQ claim on nuScenes is measured against PowerBEV-3D (25.1 − 20.0), not Cam4DOcc (18.6) §4.2 tab. 1. Use the table values when quoting.
• Timestamp-weighted metric. The weighted mIoU averages, over horizons, the running mean of per-step IoU eq. 11. A single bad early step affects every later term, so compare the per-step numbers as well.
• Ground-truth actions in forecasting. The "A" rows condition on ground-truth ego actions, and the released configs condition on CAN-bus motion tab. 1 world_head_base.py. Forecast numbers from these rows are not a planner-free benchmark.
• Ego-motion normalisation depends on the transform chain. The normalisation uses the transform from each future frame back to each history frame. Errors in alignment appear as blur in memory. The code aligns with the ego transforms drive_occworld.py conditionalnorm.py.
• Loss grid versus evaluation grid. Training losses use 256×256×20 and the block-reduced ground truth world_head_v1.py. Evaluation uses the 200×200×16 prediction grid. Numbers from different grids are not directly comparable.
• Label-space mismatch for the planner. The planner's instance and drivable IDs assume the 17-class fine-grained label space plan_head.py. If you train a two-class GMO model, the instance mask will be empty unless you override these IDs. This was not run.
• Planner candidates are fixed. The candidate set is 1800 trajectories in three groups by command plan_head.py plan_head.py. Sampling is generated from speed and steering at 0.5 s intervals nuscenes_world_dataset_template.py. Candidate-set choices change planning cost, and the paper does not report them.
• Latency is dominated by the encoder. Table 9 reports encoder latency of 140–406 ms, decoder 188–220 ms, and memory 10–26 ms on an A6000 tab. 9. The paper calls the lowest configuration real-time, but its own table shows the encoder dominates.
• Data preparation is heavy. Training needs the Cam4DOcc derived dataset and nuScenes-Occupancy annotations DATASET.MD. The configs expect data/ paths relative to the working directory action_condition_GMO.py.
• Hardware for training. The paper reports 8 A100 GPUs app. C.3. The configs set one sample per GPU, so batch size scales with GPU count action_condition_GMO.py.
• Stated but unverified hyperparameters. The paper does not state the learning-rate warm-up, weight decay, gradient clipping, or epochs. Those values are in the released config only action_condition_GMO.py.

Lineage

• Builds on: the BEVFormer-style visual encoder §3.2; ST-P3 for the planner cost and sampler §3.4; the Cam4DOcc benchmark protocol and metrics §4.1 app. C.2; the channel-to-height head from FlashOcc §3.2; action conditioning inspired by Vista and Drive-WM §3.3.
• Compared against: Cam4DOcc, PowerBEV-3D, OpenOccupancy-C, SPC and CONet-C for forecasting tab. 1 tab. 2; UniAD, VAD, ST-P3, OccNet, BEV-Planner and Drive-WM for planning tab. 5.
• Related world-model work named by the paper: OccWorld, OccSora, DriveWorld, ViDAR, UniWorld, Copilot4D and GAIA-1 §2.1.