Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications

In plain words

A self-driving car needs to know not only where things are now, but where they will be in the next couple of seconds. This paper builds a test bench for that problem using only cameras, with no lidar. The world around the car is cut into a 3D grid of small cubes. Each cube is marked as empty, as belonging to a moving object such as a car or a pedestrian, as belonging to a static thing such as a building, or as free road. The task is to predict how these marks change over the next few moments.

The authors build the data from existing driving datasets. They also define how to score the predictions, and they try four ways to get a prediction from camera images. Three of the ways are adapted from existing methods. The fourth is a new network that takes a short run of past camera frames and predicts the grid for the present and the near future in one pass. It also predicts, for each cube, which way the matter in it is moving, so that the same object can be followed from one moment to the next.

The point of the work is to give other researchers a common benchmark and a reference model, so that results on this task can be compared.

TL;DR

• A benchmark for camera-only 4D occupancy forecasting, built from nuScenes, nuScenes-Occupancy and Lyft-Level5, with a fixed split, metrics and four task levels §3.2, §3.3.
• Four baselines: a static-world model, voxelised point-cloud prediction, 2D-to-3D instance prediction, and the authors' end-to-end network OCFNet §3.4.
• OCFNet fuses camera frames over time with ego-motion, predicts future occupancy and 3D backward flow, and is trained with a weighted sum of occupancy and flow losses §4.
• Author-reported: OCFNet beats the other baselines on the future-IoU metrics; for example IoU_f 23.89 vs 21.25 for PowerBEV-3D on nuScenes at 2 s §5.2, tab. 1.
• Check before use: the code's training config (24 epochs, depth loss off) and its evaluation differ from the paper in several places (see Paper vs code).

Problem

Camera-based occupancy methods estimate the occupied space around a vehicle now, but not its future state §1. Forecasting methods that predict future instances are mostly limited to bird's-eye-view (BEV) maps and to vehicle classes §1, §2. Occupancy forecasters exist, but they need lidar point clouds as input §1, §2. The paper frames camera-only 4D occupancy forecasting as the missing piece and proposes a benchmark for it §1.

Assumptions: input is six surround-view cameras and a few past frames, with ego-poses available §4.1, app. B. Labels for movable objects are bounding-box inflated, which the authors recommend for this task §3.3, app. A.

Method

Benchmark data §3.2, fig. 2.

1. Each original nuScenes sequence is split into windows of N = N_p + N_f + 1 frames, with N_p = 2 past frames, the present frame and N_f = 4 future frames §3.2, §5.1.
2. Instances of the eight general movable object (GMO) classes are taken from the bounding-box annotations and moved into the present coordinate frame. The classes are bicycle, bus, car, construction, motorcycle, trailer, truck and pedestrian §3.2.
3. An instance is dropped if its visibility is under 40% when it first appears in the history window, if it first appears in the future window, or if it leaves the predefined range §3.2.
4. Gaps in an instance's track are filled by assuming constant velocity §3.2.
5. General static objects (GSO) and free space come from the nuScenes-Occupancy voxel labels, which are combined with the instance annotations §3.2.
6. The 3D backward centripetal flow of a voxel at time t points to the centre of its instance at t − 1 §3.2. Its use is described in §5.3, app. D.

Evaluation tasks and metrics §3.3.

1. Forecasting inflated GMO: GMO are the bounding-box voxels, and everything else is one class §3.3.
2. Forecasting fine-grained GMO: GMO are the voxel labels from nuScenes-Occupancy §3.3. Lyft-Level5 has no occupancy labels, so it is evaluated only on task 1 §3.3.
3. Forecasting inflated GMO, fine-grained GSO and free space together §3.3.
4. Forecasting fine-grained GMO, fine-grained GSO and free space together §3.3.
5. The metric is intersection over union (IoU). IoU_c scores the present frame. IoU_f averages the per-frame IoU over the future frames eq. 1, eq. 2. A third score, IoU~_f, weights the near future more heavily eq. 3.
6. For 3D instance prediction the paper uses video panoptic quality (VPQ), where a predicted instance counts as a true positive at IoU above 0.2 and with a correct tracked ID app. E, eq. 5.

Baselines §3.4, fig. 3.

1. Static-world occupancy: a present-frame camera occupancy model (OpenOccupancy-C) is copied to all future steps §3.4, §5.1.
2. Voxelised point-cloud prediction (SPC): surround depth is ray-cast into pseudo point clouds, a point-cloud predictor forecasts future points, a point segmenter labels them as movable or static, and the result is voxelised §3.4, §5.1.
3. 2D-3D instance prediction (PowerBEV-3D): a BEV instance forecaster predicts 2D occupancy, which is replicated along the height axis §3.4, §5.1. The paper notes that this assumes a flat road and equal object heights §3.4.
4. End-to-end OCFNet, described next §3.4, §4.

OCFNet §4, fig. 4, app. B.

1. Multi-frame feature aggregation: each camera image is encoded, lifted into a 3D voxel volume by an LSS-style lifting module app. B, and all volumes are warped into the present frame with 6-DoF ego-poses §4.1, app. B. Each frame's relative 6-DoF pose is then concatenated as extra channels, and the frame axis is merged into the channel axis §4.1.
2. Future state prediction: a voxel encoder produces features at four scales §4.2. A predictor expands each scale's channel count so that it covers the future frames, using stacked 3D residual blocks §4.2, app. B.
3. A 3D decoder upsamples the features and fuses them with each scale §4.2.
4. Two heads read the fused features: an occupancy head that outputs class scores per future frame, and a flow head that outputs a 3D vector per voxel with a 1x1 convolution and no softmax §4.2.
5. The coarse outputs are interpolated to full resolution, and the occupancy class is the argmax §4.2.
6. Training loss, eq. 4: an occupancy cross-entropy term, a smooth-L1 flow term, and a depth term that is applied only to the present frame. The paper gives weights 0.5, 0.05 and 0.5 §4.3, app. B, eq. 4.

3D instance prediction (not in the training loss) app. E. Instance centres are found by non-maximum suppression on the present-frame occupancy. Instance IDs are then propagated through the future frames using the predicted flow app. E.

# OCFNet, one training step  [§4] [fig. 4] [code: ocfnet.py#L478-L544]
# Input:  images (B, 3, 6, 3, H, W): 2 past + present frame, 6 cameras
# Input:  ego-poses, occupancy labels, flow labels
# Output: scalar loss

def train_step(images, egomotions, occ_labels, flow_labels):
    # 1. Encode each camera image with ResNet50 and an FPN neck  [app. B]
    image_feats = image_encoder(images)
    # 2. Lift image features into a 3D voxel volume per frame  [§4.1] [app. B]
    voxels = lift_to_voxels(image_feats)           # (B, T_obs, C, X, Y, Z)
    # 3. Warp past volumes into the present frame with ego-motion  [§4.1]
    aligned = warp_to_present(voxels, egomotions)
    # 4. Append each frame's relative 6-DoF pose as channels  [§4.1]
    #    The present frame gets a zero pose  [code: ocfnet.py#L364-L371]
    motion_aware = concat_poses(aligned, egomotions)
    # 5. Encode the stacked frames with a 3D ResNet18 at four scales  [§4.2]
    occ_scales = occ_encoder(motion_aware)
    flow_scales = flow_encoder(motion_aware)
    # 6. Expand each scale to the future frames, with blocks 2, 1, 2  [app. B]
    occ_future = occ_predictor(occ_scales)
    flow_future = flow_predictor(flow_scales)
    # 7. Fuse each scale with the upsampled decoder feature  [§4.2]
    occ_fused = occ_decoder(occ_future)
    flow_fused = flow_decoder(flow_future)
    # 8. Occupancy head gives class scores; flow head gives 3 values  [§4.2]
    occ_logits = occ_head(occ_fused)               # per scale, coarse grid
    flow_pred = flow_head(flow_fused)              # per scale, coarse grid
    # 9. Occupancy loss: class-weighted cross-entropy, weight 0.5 per scale
    #    The paper's lambda_1 = 0.5 is applied here  [eq. 4] [app. B]
    loss_occ = sum(0.5 * weighted_ce(z, occ_labels) for z in occ_logits)
    # 10. Flow loss: smooth L1 on voxels with valid labels, weight 0.05
    #     The paper's lambda_2 = 0.05 is applied here  [eq. 4] [app. B]
    loss_flow = sum(0.05 * smooth_l1(f, flow_labels) for f in flow_pred)
    # 11. Depth loss is in the paper's objective, but the code turns it off
    #     [code: OCFNet_in_Cam4DOcc_V1.2.py#L185]
    return loss_occ + loss_flow

# Evaluation: IoU from one pooled confusion matrix  [code: ocfnet.py#L662-L711]
# Input:  predicted classes and ground-truth labels per scene
# Output: per-class IoU

def evaluate(predictions, labels):
    # 1. Score the future frames t = 1..4 by default  [code: ocfnet.py#L648]
    scored = select_future_frames(predictions, labels)
    # 2. Upsample and take argmax  [code: ocfnet.py#L664-L673]
    classes = argmax(trilinear_upsample(scored.pred), dim=class)
    # 3. Pool one confusion matrix over all frames  [code: ocfnet.py#L685]
    confusion = zeros(num_classes, num_classes)
    for pred, gt in zip(classes, scored.gt):
        keep = gt != 255                           # ignore noise label
        confusion += histogram(pred[keep], gt[keep])
    # 4. Per-class IoU  [code: cam4docc_dataset.py#L462-L463]
    return iou_from_confusion(confusion)

Hyperparameters

Results

Author-reported. Setup: nuScenes (with nuScenes-Occupancy) and Lyft-Level5, trained on 8 A100 GPUs §5.1.

Headline text claims: OCFNet and OCFNet† beat the other baselines on IoU_f and IoU~_f §5.2. The paper states gains of 12.4% and 13.3% over PowerBEV-3D on nuScenes (IoU_f and IoU~_f) and of 20.8% and 21.8% on Lyft-Level5 §5.2. Recomputed from tab. 1, the nuScenes figures match OCFNet; the Lyft figures also match OCFNet, not OCFNet†.

Table 1 tab. 1: forecasting inflated GMO (task 1). Percentages are IoU.

OCFNet (ours) is trained on one sixth of the training sequences; OCFNet† on all of them §5.1, tab. 1.

Table 2 tab. 2: forecasting fine-grained GMO on nuScenes-Occupancy (task 2).

Table 3 tab. 3: inflated GMO, fine-grained GSO and free space (task 3).

Table 4 tab. 4: fine-grained GMO, fine-grained GSO and free space (task 4).

Table 5 tab. 5: ablation on the flow head (nuScenes, task 1).

The paper reads this as about 4% from the multi-task flow head §5.3.

Table 6 tab. 6: forecasting by time horizon, appendix C. Columns are the horizons of each dataset.

Table 7 tab. 7: 3D instance prediction (VPQ, appendix E).

OCFNet* takes its occupancy from OCFNet† and takes the flow of the grids it shares with PowerBEV-3D from PowerBEV-3D; the other grids keep OCFNet†'s flow app. E. The paper says PowerBEV-3D beats OCFNet and OCFNet† on nuScenes, and that OCFNet* is best on both datasets app. E.

Paper vs code

Official code found through the abstract link (github.com/haomo-ai/Cam4DOcc, MIT, commit 542f14a9d9e142d9faf9044df47c8488a7c3166a). The repository is published by the authors' organisation. The code reproduces the architecture and most of the data pipeline. Differences:

• Epochs. Paper: 15 epochs for OCFNet §5.1. Shipped configs: 24 epochs in both the nuScenes and the Lyft variants OCFNet_in_Cam4DOcc_V1.2.py.
• Depth loss. Paper: the explicit depth loss is used, with weight 0.5 and applied only to the present frame §4.3, app. B, eq. 4. Code: disable_loss_depth=True in the shipped configs OCFNet_in_Cam4DOcc_V1.2.py, and the depth term is skipped in the training forward ocfnet.py. The code comment says a version with depth fine-tuning is planned.
• Loss form. Paper: a sum over the frames, normalised by N_f + 1, with the three weights eq. 4. Code: a sum of head losses over four pyramid scales, with the weights built into each term (0.5 for occupancy, 0.05 for flow); there is no N_f + 1 factor occ_head.py flow_head.py ocfnet.py.
• Predicted frames. Paper: 5 frames, t in [0, N_f] §4.2, app. B. Code: n_future_frames_plus = 6, and the losses use the last 6 ground-truth frames OCFNet_in_Cam4DOcc_V1.2.py ocfnet.py.
• Present-frame IoU. Paper: IoU_c is reported in tables 1 to 4 tab. 1–4. Code: test_present = False in the shipped configs, so the test path scores only frames t = 1..4 OCFNet_in_Cam4DOcc_V1.2.py ocfnet.py. The present frame is scored only when test_present is set. The frame indices assume the 7-frame window, which is inferred from the code, not run.
• IoU definition. Paper: IoU_f is the mean of per-frame IoUs eq. 2. Code: IoU comes from one confusion matrix pooled over all scenes and frames ocfnet.py cam4docc_dataset.py. These two can differ.
• IoU~_f. Paper: defined in eq. 3. No code computing this weighting was found in the files read.
• VPQ. Paper: VPQ is reported in tab. 7 app. E, eq. 5. Code: the test path sets the VPQ value to a constant 0.1, and the instance evaluation call is commented out ocfnet.py. The shipped test path does not produce the tab. 7 numbers.
• Flow resolution and units. Paper: 3D backward flow per voxel §3.2. Code: flow labels are built on a grid reduced by 4 in each axis, that is 0.8 m cells rather than 0.2 m voxels loading_instance.py loading_instance.py. The flow head is compared against labels at that size flow_head.py.
• Occupancy label resolution. Paper: losses over the 512 x 512 x 40 grid §5.1. Code: occupancy labels are reduced to the head grid by a mode over 4 x 4 x 4 blocks before the loss occ_head.py.
• Class weights. Paper: not stated. Code: cross-entropy weight 5 for each non-empty class occ_head.py.
• Image neck. Paper: "FPN" app. B. Code: SECONDFPN for the image features OCFNet_in_Cam4DOcc_V1.2.py. The 3D decoder is an FPN3D OCFNet_in_Cam4DOcc_V1.2.py.
• Input size. Paper: 900 x 1600 app. B. Code: 896 x 1600 input from a 900 x 1600 source, with augmentation OCFNet_in_Cam4DOcc_V1.2.py.
• Optimiser details. The paper gives AdamW, learning rate 3e-4 and weight decay 0.01, which match the code §5.1 OCFNet_in_Cam4DOcc_V1.2.py. The code also uses a backbone LR multiplier, a cosine schedule with warm-up and gradient clipping, none of which the paper gives.
• Predictor blocks. Paper: 2, 1, 2 residual blocks with kernel (3, 3, 1) app. B. Code matches pred_block.py.
• Ego-motion. Paper: 6-DoF relative pose concatenated per frame §4.1. Code matches, with a zero pose for the present frame ocfnet.py.
• Lyft variant. The Lyft config drops the vehicle prefix from the class names and uses the Lyft dataset class OCFNet_in_Cam4DOcc_V1.2_lyft.py OCFNet_in_Cam4DOcc_V1.2_lyft.py. The paper's Lyft setup has GMO and free space only §3.3.
• Binary variant. The shipped binary config (V1.1) uses two class names, vehicle and human, without separate classes OCFNet_in_Cam4DOcc_V1.1.py. The eight-class config matches the paper's eight GMO classes OCFNet_in_Cam4DOcc_V1.2.py.
• Dataset split for 1/6 training. The paper trains OCFNet on one sixth of the training sequences §5.1. The shipped configs do not include that setting; the default is all sequences OCFNet_in_Cam4DOcc_V1.2.py.

Pitfalls

• Evaluation window. Frames are scored from t = 1, and the present frame is skipped unless test_present is set ocfnet.py. Reproducing IoU_c needs that flag. Check the frame count against the paper before you compare numbers.
• Pooled versus per-frame IoU. The code pools confusion counts over all frames and scenes ocfnet.py. A per-frame average, as eq. 2 writes it, gives different numbers. Choose one and state it.
• Instance metric. The VPQ path in the shipped test code is a constant placeholder ocfnet.py. Do not cite tab. 7 as reproduced from this code.
• Coarse training grid. Heads and losses work at the coarser grid, while evaluation upsamples the logits trilinearly before the argmax ocfnet.py. Predicted surfaces at 0.2 m are interpolations, not direct outputs.
• Flow units and rounding. Flow is in units of the 0.8 m grid, and each target is the rounded mean of the previous frame's instance mask minus the voxel index loading_instance.py. Voxels whose instance mask is empty in the previous frame keep the ignore value loading_instance.py.
• Ignore label. Label 255 is ignored in cross-entropy, smooth L1 and evaluation. Label 255 marks noise in the labels ocfnet.py.
• Class weighting is not in the paper. The factor of 5 on non-empty classes changes how much the object classes weigh against the empty class in the occupancy loss occ_head.py. Runs that omit it will not match the reported numbers.
• Dead fine-grained loss path. The point-level losses (cross-entropy, scene-class losses, geometric loss and Lovász) are defined, but the main loss path calls only the voxel loss occ_head.py. Do not assume they are active.
• Visibility rule. The code filters on visibility_token == 1. The paper states 40%. The code does not show which visibility bin that token corresponds to cam4docc_dataset.py.
• Instance coverage. The appendix reports that over 30% of instances first appear in the present frame, so the model must learn their motion from their current position and surroundings app. A. It also argues that long instance durations make training more effective app. A.
• Cost. The paper reports 370 M parameters, 6434 GFLOPs and 57 GB of training memory app. B. These were not checked against the code. Training was on 8 A100 GPUs §5.1. The repository pins torch 1.10.1, mmcv-full 1.4.0, mmdet 2.14.0 and mmdetection3d v0.17.1, and needs a custom CUDA occupancy-pooling op README.md.
• Data terms. The benchmark depends on nuScenes, nuScenes-Occupancy and Lyft-Level5, each with its own terms. Those terms were not checked for this card. Derived files may need separate checks before redistribution README.md.
• Table numbers. Table and figure numbers were counted in the LaTeX source. The compiled PDF was not checked.

Lineage

• Builds on: OpenOccupancy's camera baseline and the nuScenes-Occupancy labels §3.2, §5.1; PowerBEV for 2D instance prediction and the flow idea §2, §3.2; LSS-style lifting and ResNet/FPN image encoders app. B. The source gives no arXiv ids for these, so none are linked here.
• Compared against: the four baselines in §3.4 (static-world, voxelised point-cloud prediction, 2D-3D instance prediction).
• Superseded by: not known.