BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework

In plain words

Self-driving cars find other objects by combining a camera, which sees colour and texture, with a LiDAR, which measures distances with laser light. Most earlier systems used the LiDAR points to decide where to look in the camera image. If the LiDAR misses points, for example on objects that reflect little light, or has a limited field of view, those systems have nothing to look with and stop producing useful output.

BEVFusion keeps the two sensors apart until late. The camera side takes each picture and guesses how far away each pixel is, then places the picture's content onto a top-down map of the road. The LiDAR side builds its own top-down map from its points. The two maps are stacked, mixed with a small learned layer, and sent to a standard detector. Because the camera map does not depend on LiDAR points, the system can still produce a result when the LiDAR is weak, and it does better still when both sensors are working.

TL;DR

• Two independent streams build features in the same bird's-eye-view (BEV) grid: a camera stream using Lift-Splat-Shoot-style depth lifting, and a LiDAR stream reused from PointPillars, CenterPoint or TransFusion-L §3 §3.1 §3.2.
• Fusion is one 3x3 convolution over the concatenated maps, followed by a channel gate §3.3 eq. 1 eq. 2.
• On nuScenes validation with TransFusion-L as the LiDAR stream, fusion gives 67.9 mAP against 64.9 for LiDAR alone tab. 1.
• Under simulated LiDAR loss (field-of-view limits or dropped object points), the abstract reports margins of 15.7 to 28.9 mAP points over the earlier methods, measured under those training settings; the tables give the per-setting numbers against LiDAR-only rows §4.4.1 tab. 3 tab. 4.
• Author-reported. The code is official, and several training choices differ from the paper (see Paper vs code).

Problem

Current LiDAR-camera fusion methods pick image features by projecting LiDAR points, or proposals made from them, into the image. Without a usable LiDAR point cloud these methods cannot produce a meaningful prediction §1 §2 §4.4.1. The paper lists causes such as low reflectance from object surfaces, an internal data-transfer glitch, and a LiDAR field of view below 360 degrees because of hardware limits §1. The target is 3D object detection on nuScenes, with six cameras, one LiDAR point cloud per frame and 10 classes §4.1.

Method

BEVFusion has two streams that map raw inputs to BEV features of the same spatial size, one fusion module, and a standard detection head §3 fig. 2. The camera stream has three parts: an image encoder, a view projector that lifts image features into 3D ego coordinates with a predicted depth distribution, and a BEV encoder §3.1 fig. 2. The paper uses Dual-Swin-Tiny as the image backbone, a standard FPN, and a small adaptive module (ADP) that resamples and concatenates the multi-scale FPN outputs; the ADP architecture is in app. A and fig. 5 §3.1.

1. Image encoder. Multi-view images go through the backbone and FPN. The ADP refines each upsampled level before concatenation §3.1 app. A.
2. View projection (LSS-style lift). A convolution predicts a depth distribution over bins and a set of context features per pixel. Their product gives a frustum of features, which is projected into the ego frame using the camera extrinsics and summed into a voxel grid §3.1.
3. Spatial-to-channel. Height is folded into the channel axis instead of pooled or compressed with strided 3D convolutions, which keeps full resolution §3.1 cam_stream_lss.py.
4. BEV encoder. Four 3x3 conv layers reduce the folded channels to the camera BEV width §3.1.
5. LiDAR stream. Any LiDAR BEV network can be used. The paper tests PointPillars, CenterPoint and TransFusion-L, which voxelise points and apply sparse 3D convolution to reach a BEV map §3.2.
6. Fusion. The two BEV maps are concatenated on channels. A 3x3 conv reduces the channels to the LiDAR width, and a channel gate from global average pooling, a linear map and a sigmoid rescales them §3.3 eq. 1 eq. 2.
7. Head. The fused map goes to one of three detection heads: anchor-based (PointPillars), anchor-free (CenterPoint) or transformer-based (TransFusion-L) §3.4.

At inference the camera stream takes images and the camera-to-LiDAR matrices only; the LiDAR points enter through the LiDAR stream and the fusion module §3 §3.1. In the PointPillars training config, projected LiDAR depth also supervises the camera depth during training (see Paper vs code).

# BEVFusion forward pass, TransFusion-L variant  [§3] [eq. 1] [eq. 2]
# Input:  images (B, N, 3, 448, 800), N = 6; points: list of (P_b, 5)
#         lidar2img: (B, N, 4, 4), matrices for the original 1600x900 frame
# Output: per-object class scores and 3D boxes; no NMS at test time

def forward(images, points, lidar2img):
    # Image backbone (Dual-Swin-T), FPN and ADP  [§3.1] [app. A]
    image_feats = fpn_adp(dual_swin_t(images))  # (B*N, 256, 112, 200)

    # Depth: softmax over 41 bins, 4 m to 44 m in 1 m steps  [§3.1]
    # [code: mmdet3d/models/detectors/cam_stream_lss.py#L130-L141]
    # [code: mmdet3d/models/detectors/bevf_faster_rcnn.py#L31]
    depth_probs = softmax(depth_conv(image_feats), dim=depth_bins)

    # Context: 64 channels, each weighted by the depth probabilities  [§3.1]
    # [code: mmdet3d/models/detectors/cam_stream_lss.py#L140]
    context = context_conv(image_feats)  # (B*N, 64, 112, 200)
    frustum = depth_probs[:, None] * context[:, :, None]

    # Splat: move each cell to ego coordinates and sum into a grid  [§3.1]
    # Grid: 0.6 m cells, x, y in [-54, 54] m, z in [-5, 3] m
    # [code: mmdet3d/models/detectors/cam_stream_lss.py#L229-L258]
    voxels = splat_sum(frustum, lidar2img)  # (B, 64, 13, 180, 180)

    # Spatial-to-channel: fold the 13 height slices into channels  [§3.1]
    # [code: mmdet3d/models/detectors/cam_stream_lss.py#L318-L322]
    bev = reshape(voxels, (B, 64 * 13, 180, 180))

    # Four 3x3 conv layers, 832 -> 512 -> 512 -> 256 channels  [§3.1]
    # [code: mmdet3d/models/detectors/cam_stream_lss.py#L194-L207]
    cam_bev = bev_encoder(bev)  # (B, 256, 180, 180)

    # LiDAR stream: voxels 0.075 x 0.075 x 0.2 m  [§3.2]
    # [code: configs/bevfusion/bevf_tf_4x8_6e_nusc.py#L62-L79]
    lidar_bev = lidar_stream(points)  # (B, 512, 180, 180)

    # Fusion f_static: 3x3 conv on the channel concatenation  [§3.3] [eq. 1]
    # The code also applies BatchNorm and ReLU here
    # [code: mmdet3d/models/detectors/bevf_faster_rcnn.py#L59-L67]
    stacked = concat([cam_bev, lidar_bev], dim=channels)  # (B, 768, ...)
    fused = conv3x3_bn_relu(stacked)  # (B, 512, 180, 180)

    # Fusion f_adaptive: channel gate from global average pooling  [eq. 2]
    # [code: mmdet3d/models/detectors/bevf_faster_rcnn.py#L15-L24]
    gate = sigmoid(conv1x1(global_avg_pool(fused)))
    fused = gate * fused  # (B, 512, 180, 180)

    # Head, TransFusion-L: 200 queries seeded from a heatmap  [§3.4]
    # [code: configs/bevfusion/bevf_tf_4x8_6e_nusc.py#L102-L113]
    queries = heatmap_queries(fused, num_proposals=200)

    # Training matches queries to ground truth with the Hungarian method
    # [code: configs/bevfusion/bevf_tf_4x8_6e_nusc.py#L133-L139]
    outputs = transfusion_head(queries, fused)
    return outputs

Key equations, as written in the paper:

\mathbf{F}_\text{fused} = f_\text{adaptive}\big(f_\text{static}\big(\left[\mathbf{F}_\text{Camera}, \mathbf{F}_\text{LiDAR}\right]\big)\big) eq. 1

f_\text{adaptive}(\mathbf{F}) = \sigma\big(\mathbf{W} f_\text{avg}(\mathbf{F})\big) \cdot \mathbf{F} eq. 2

Robustness augmentations, as the code runs them. They are training data transforms, and the object-failure one is also applied in the halfbox test pipeline [code: configs/base/datasets/nusc_halfbox_tf.py#L38] [code: configs/base/datasets/nusc_halfbox_tf.py#L68].

# Object-failure augmentation, as implemented  [§4.4.1]
# [code: mmdet3d/datasets/pipelines/transforms_3d.py#L985-L994]
# Paper: each frame 0.5, and each object within it 0.5 [§4.4.1]
def drop_object_points(points, gt_boxes):
    # Frame-level coin: applied with probability 0.5 per call
    if random() < 0.5:
        # Remove every point inside any ground-truth box, with no
        # per-object coin
        inside = points_in_any_box(points, gt_boxes)
        points = points[~inside]
    return points

Hyperparameters

Values marked "not stated" are absent from the paper. Values from the code are the official config for that variant.

Results

Author-reported throughout. Setup: nuScenes detection, mAP and NDS §4.1 app. B.1. The generalisation and robustness results are on the validation set according to the text §4.2 §4.4.1; Table 2 also gives test results. Test results use no test-time augmentation or ensemble §4.3. The paper does not say how many runs were averaged; its checklist says results were observed with a fixed seed.

Generalisation over LiDAR streams (nuScenes validation) tab. 1

Comparison with state of the art tab. 2. Subset of the table: mAP and NDS only. * uses BEV-space augmentation during training; † uses double-flip at test time (as labelled in the source). Per-class APs are in tab. 2.

FOV-limited LiDAR tab. 3. Gains in parentheses are absolute mAP or NDS points over the LiDAR row. "LC" is the TransFusion LiDAR-camera baseline §4.4.1.

Object-point dropping tab. 4. Parenthesised gains in the augmented rows are measured against the LiDAR-only row without augmentation, not against an augmented LiDAR baseline (checked: for example 41.6 - 12.7 = 28.9).

Camera failures (nuScenes validation) tab. 5. "Missing F" removes the front camera; "Preserve F" keeps only the front camera; "Stuck" freezes 50 % of the camera frames §4.4.2.

Camera-stream ablation (PointPillars head, validation) tab. 6 §4.5.

Fusion-module ablation (validation) tab. 7 §4.5. CSF is channel and spatial fusion (the 3x3 conv); AFS is the adaptive channel gate.

Appendix results (author-reported; figure and table numbers follow source order).

• Fusion module under malfunction, TransFusion-L tab. 8: clean, object failure, and the camera failures. CSF only: clean 67.3 mAP; object failure 50.1 mAP; missing F 65.4; preserve F 63.5; stuck 65.6. CSF + AFS: clean 67.9; object failure 50.3; missing F 65.9; preserve F 65.1; stuck 66.2. LiDAR-only TransFusion-L: clean 64.9; object failure 34.6.
• Both malfunctions, BEVFusion vs TransFusion tab. 9: BEVFusion mAP is 67.9 clean, 47.8 object failure plus missing F, 39.3 plus preserve F, 43.6 plus stuck. TransFusion LC is 66.9 clean, 34.2, 33.6 and 33.9.
• Lighting tab. 10: mAP (metric not labelled in the table) by day / night: CenterPoint (L) 62.8 / 35.4; TransFusion-L (L) 64.8 / 36.2; TransFusion (LC) 67.0 / 41.8; BEVFusion (LC) 68.0 / 42.4.
• Distance from ego vehicle tab. 11 tab. 12: BEVFusion with TransFusion-L, mAP for <15 m / 15-30 m / >30 m: 77.3 / 69.4 / 49.2, against 76.3 / 66.1 / 43.2 for LiDAR only. In tab. 12 the authors' TransFusion re-implementation gets 77.6 / 68.3 / 47.7 and their reported TransFusion gets 75.5 / 66.9 / 43.7.
• Latency and memory tab. 13: batch size 1, Intel Xeon Gold 6126 CPU and one V100 GPU. Memory in MB / time in ms. LiDAR only: PointPillars 7190 / 189.38; CenterPoint 12468 / 199.87; TransFusion-L 12536 / 263.61. Camera only: 7948 / 1264.55; 7956 / 1278.94; 7950 / 1264.45. Both: 7968 / 1513.65; 18078 / 1464.21; 18086 / 1529.66. The text says the camera-to-BEV projector alone costs more than 957 ms app. E and can be sped up by running streams concurrently.

Paper vs code

Code: github.com/ADLab-AutoDrive/BEVFusion at fb4384ca05cae05be548dd96a0f940da70263cdb (cloned with --depth 1). The paper's abstract links this repository, so it is the authors' own code. The repository is a fork of MMDetection3D (Apache-2.0, LICENSE). The BEVFusion code lives in mmdet3d/models/detectors/, and configs/bevfusion/ holds the per-variant configs.

Consistent with the paper: nuScenes input size 448 x 800, Dual-Swin-T backbone, TransFusion-L fusion learning rate 1e-3 with the 6-epoch and 4/5 schedule app. B.1 bevf_tf_4x8_6e_nusc.py, 12-epoch schedule for PointPillars [code: configs/base/schedules/schedule_1x.py#L11-L14], LiDAR frozen for CenterPoint and TransFusion-L but not for PointPillars train.py, FOV limits of 60 and 90 degrees [code: configs/base/datasets/nusc_fov60_tf.py#L30], and no NMS at test time bevf_tf_4x8_6e_nusc.py.

Discrepancies found:

1. Depth supervision is not in the paper. The code adds a KL loss between predicted depth and depth projected from LiDAR, with weight 1.0 bevf_transfusion.py. The code comments say this is "not mentioned in paper" bevf_faster_rcnn.py. Only the PointPillars data pipeline loads projected depth [code: configs/base/datasets/nusc_pp.py#L32]. The TransFusion and CenterPoint data configs do not [code: configs/base/datasets/nusc_tf.py#L19-L42]. The PointPillars fusion config also freezes the camera stream bevf_pp_2x8_1x_nusc.py, so the effect of this loss on that model is not clear from the fusion config alone.
2. Camera stream frozen in fusion training. The paper describes the camera stream as trained in stage one and inherited in stage two. It says nothing about freezing it. The configs set freeze_img=True bevf_tf_4x8_6e_nusc.py bevf_cp_4x8_6e_nusc.py bevf_pp_2x8_1x_nusc.py, and the freeze also covers the lift module bevf_faster_rcnn.py.
3. Object-failure augmentation is per frame in code, per object in paper. The paper says each frame has a 0.5 chance and each object a further 0.5 chance §4.4.1. The code makes one 0.5 draw per call and removes points in every ground-truth box transforms_3d.py.
4. Object-failure config also limits the field of view in training. The halfbox training pipeline restricts LiDAR points to ±60 degrees [code: configs/base/datasets/nusc_halfbox_tf.py#L30]. The paper does not describe this for the object-failure experiment.
5. Object-failure evaluation is random. The halfbox test pipeline applies the same drop [code: configs/base/datasets/nusc_halfbox_tf.py#L68]. The drop calls np.random.rand() with no seed set in that function transforms_3d.py. The paper's checklist says results were observed with a fixed seed, which does not obviously cover this draw.
6. Batch size. The paper states 32 on 8 V100 GPUs app. B.1. The TransFusion and CenterPoint configs give 4 per GPU, which is 32 on 8 GPUs bevf_tf_4x8_6e_nusc.py. The PointPillars config gives 2 per GPU bevf_pp_2x8_1x_nusc.py, and its name ("2x8") suggests 16 in total. The GPU count is not in that file. The robust fine-tune configs also give 2 per GPU halfbox_bevf_tf_2x8_1x_nusc.py.
7. Fusion schedule length. §4.1 says the fusion stage runs "another 9 epochs" §4.1. Appendix B.1 gives 12 epochs for PointPillars and 6 for CenterPoint and TransFusion-L, and the TransFusion config matches the 6 bevf_tf_4x8_6e_nusc.py. The paper is internally inconsistent here.
8. Fusion conv. Eq. 1 shows a convolution only. The code uses convolution, BatchNorm and ReLU bevf_faster_rcnn.py.
9. Undocumented choices in code. Depth bins, BEV grid, voxel limits, sweep count, optimizer type and gradient clipping are set in the configs and not in the paper (see Hyperparameters). The paper gives the optimizer learning rate and batch size only.
10. BEV-space augmentation has a different source. The BEVFusion* rows use BEV-space augmentation, which the paper's footnote credits to another repository (mit-han-lab/bevfusion). The code README gives the same credit [README.md]. This card does not use that repository's code.

No code was found that conflicts with the paper on input size, backbone, learning rate, FOV limits or the fusion equations.

Pitfalls

• Frustum canvas differs from the input size. Images are resized to 448 x 800 [code: configs/base/datasets/nusc_tf.py#L16], but the frustum is built on the original 900 x 1600 frame cam_stream_lss.py, and lidar2img is not rescaled by the resize transform. Downsampling by 8 gives 112 x 200, which matches the stride-4 features of the 448 x 800 input app. A. Changing img_scale without also changing final_dim and downsample breaks this match.
• Depth is a soft distribution over 41 bins from 4 m to 44 m. The bin range is a code default, not a paper value, and the paper does not say what happens beyond it cam_stream_lss.py.
• The spatial-to-channel reshape swaps the BEV axes. The camera BEV is permuted to swap its two horizontal axes cam_stream_lss.py. The fusion code only interpolates on a shape mismatch, so it does not check the axis order against the LiDAR BEV bevf_faster_rcnn.py.
• Robustness results are for removed or cropped points, not for an empty point cloud. The code path for a frame with zero LiDAR points was not checked, so do not assume the model handles total LiDAR loss without a test.
• "%" in the paper means percentage points. For example, the 18.4% gain for PointPillars is 53.5 - 35.1 mAP tab. 1 §1.
• The table gains in tab. 4 are against the non-augmented LiDAR baseline. The augmented LiDAR-only rows are not reported.
• Vanilla fusion can get worse after augmentation. TransFusion LC drops from 38.1 to 37.2 mAP with object-failure augmentation tab. 4; the paper hypothesises that missing foreground points give wrong supervision §4.4.1.
• Camera-only stream training details are in code, not the paper. The config uses 20 epochs, AdamW and lr 1e-4 bevf_tf_4x8_20e_nusc_cam.py bevf_tf_4x8_20e_nusc_cam.py. The paper only says "official" LiDAR settings app. B.1.
• The camera-stream BEV grid (0.6 m) is coarser than the voxel grid (0.075 m) but matches the LiDAR BEV resolution bevf_tf_4x8_6e_nusc.py bevf_tf_4x8_6e_nusc.py. Re-tuning one stream without the other changes the BEV size.
• Table and figure numbers follow caption order in the source. LaTeX floats may appear in a different order in the PDF, so check the PDF before quoting a number.
• Parts of the paper were added in review. Appendix A to F and several sentences in the main text are wrapped in a \rebuttal macro in the source [raw source: latex/appendix.tex].

Lineage

• Builds on: Lift-Splat-Shoot for the depth-based view projection §3.1. Dual-Swin-T (CBNetV2) for the image backbone §3.1. Spatial-to-channel from a prior work the paper cites §3.1. PointPillars, CenterPoint and TransFusion-L as LiDAR streams and heads §3.2 §3.4. Squeeze-and- Excitation for the channel gate §3.3. MMDetection3D for the code base §4.1.
• Compared against: FUTR3D, PointAugmenting, MVX-Net, PointPainting, 3D-CVF, MVP, FusionPainting, TransFusion and DETR3D tab. 2 tab. 5.