wikillm

Paper · 2023-04-27 · cs.CV

SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection

Fuses LiDAR and camera detections as sparse object candidates, not dense BEV or point maps. Each sensor has its own 3D detector with 200 queries; camera boxes move into the LiDAR frame, and one self-attention layer fuses the 400 candidates. Depth and semantics are transferred between sensors before detection. Author-reported nuScenes val NDS 72.8, mAP 70.4 (ResNet-50).

Multimodal fusionObject queries (learned query slots in transformer decoders)Bird's-eye-view (BEV) representationBipartite matching for set-prediction losses3d-object-detectionlidar-camera-fusionmulti-sensor-fusionsparse-representationbird-eye-viewnuscenesautonomous-driving

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A self-driving car finds other road users in two ways. A laser scanner (LiDAR) gives an accurate 3D picture, and cameras give colour and appearance. Most earlier fusion methods first merge the two sensors into one dense picture of the whole scene, such as a top-down grid or a cloud of points, and then search that picture for objects. Most of that picture is empty road and background, so the work is wasted and slow.

This paper keeps only the objects. Each sensor gets its own detector, and each detector produces a short list of candidate objects. The camera's list is first re-expressed in the laser's 3D coordinates, and then the two lists are mixed by a single attention step that lets each candidate look at the others. Before the detectors run, each sensor also gets a hint from the other one. The laser provides depth to the camera, and the camera provides labels to the laser. On the standard nuScenes driving benchmark, the authors report the best scores among the methods they compare against, and the fastest inference of those methods. All numbers are the authors' own.

TL;DR

SparseFusion runs a LiDAR detector and a camera detector in parallel, each producing 200 object candidates. The camera candidates are converted into LiDAR coordinates, and all 400 candidates are fused with a lightweight self-attention layer and a shared prediction head. Two transfer modules, one for depth from LiDAR to camera and one for semantics from camera to LiDAR, are applied before the detectors. Author-reported nuScenes validation NDS 72.8 and mAP 70.4 with a ResNet-50 camera backbone, and test NDS 73.8 and mAP 72.0 tab. 2. The camera branch needs LiDAR points at inference, so it is not a stand-alone camera detector.

Problem

Existing LiDAR-camera 3D detectors either fuse dense representations (point features, BEV grids or dense image features) or query dense features for every object §1 tab. 1. The authors argue that objects cover only a small part of a driving scene, so dense candidates and dense scene representations are noisy and slow §1. Cross-modal attention between dense features is also costly, and the two sensors have a large domain gap §1 §2. The paper asks whether fusion can work using only sparse, object-level candidates from both sensors §1.

Method

The pipeline has two parallel 3D detectors, one per modality §3.1 fig. 2.

LiDAR branch. The LiDAR backbone (VoxelNet) produces a BEV feature map. The branch follows TransFusion-L and uses one decoder layer. Its N_L = 200 object queries are placed on the BEV plane, attend to the BEV features with cross-attention, and each produce one candidate instance feature and one 3D box in LiDAR coordinates §3.1 §4.2.

Camera branch. Deformable DETR is extended with 3D box regression heads. Its N_C = 200 queries sit on the image of each of the six views. Each query attends only to features of its own view with deformable attention, and the outputs of all views form the camera candidates §3.1 §7.1.

Transformation of camera candidates. The camera head predicts a box for each camera candidate. The box is projected into the LiDAR frame using the camera's extrinsic and intrinsic matrices. A box embedding is built from the projected box, a camera embedding is built from the camera parameters, and the two are combined with the candidate feature as in eq. 1. A self-attention layer then aggregates candidates that come from different cameras §3.1 §7.1. The paper treats the class as view-invariant and only transforms the box part of the representation §3.1.

Sparse candidate fusion. Both candidate sets are passed through separate learned projections f_L and f_C (a linear layer followed by layer normalisation §7.1) and concatenated into one set of N_L + N_C candidates eq. 2. A self-attention layer and feed-forward network mix them, and a LiDAR-view prediction head outputs the final class scores and boxes §3.1 §7.1.

Geometric transfer (LiDAR to camera). LiDAR points are projected into each view to form a sparse depth map of size 200 x 112 for nuScenes §7.1. A small shared encoder turns it into four feature levels, following alg. 1. Each level is concatenated with the matching image FPN level and passed through a 3x3 convolution. The result is a depth-aware image feature that the camera branch uses §3.2 fig. 3.

Semantic transfer (camera to LiDAR). LiDAR points in occupied BEV cells are projected into the images. Image features at those points are max-pooled over scales and added to the BEV features. The result queries the multi-scale image features with deformable attention. The updated queries replace the BEV features at those cells and form a semantics-aware BEV map §3.2 §7.1. The paper says this map is used to initialise the LiDAR queries §7.1 §7.3.

Query initialisation. Both branches initialise queries from a heatmap, following CenterFormer and TransFusion §7.3. For LiDAR, object centres are splatted onto a Gaussian heatmap per class, combined by maximum, and the highest-scoring positions become reference points. A learned category embedding is added to the BEV feature at each position §7.3. For the camera, 3D box centres are projected into every view and assigned to one of four feature levels by the larger side of their box, using the thresholds 0, 48, 96, 192 and infinity §7.3.

Objective. Each detector's predictions are matched one-to-one to ground truth with the Hungarian algorithm. Classification uses focal loss, and 3D box regression uses L1 loss. The camera detector is matched in camera coordinates and the others in LiDAR coordinates §3.3. The total detection loss is eq. 4. The heatmap term is eq. 3, and the full loss is alpha times the heatmap term plus beta times the detection term, with gamma = 1, alpha = 0.1 and beta = 1 §3.3.

Key equations:

\mathbf{q}^{L}_{C,i} = g(\mathbf{b}^{L}_{i}) + h\big(\mathbf{q}^{P}_{C,i} \cdot m(\mathbf{R}_v, \mathbf{t}_v, \mathbf{K}_v)\big) eq. 1

where g and h are MLPs, m is an MLP over the flattened camera parameters, and b^L is the projected box §3.1.

\mathbf{Q}_{LC} = \{f_L(\mathbf{q}_{L,i})\}_{i=1}^{N_L} \cup \{f_C(\mathbf{q}_{C,i})\}_{i=1}^{N_C} eq. 2

\mathcal{L}_{init} = \mathcal{L}_{GFocal}(\hat{\mathbf{Y}}_L, \mathbf{Y}_L) + \mathcal{L}_{GFocal}(\hat{\mathbf{Y}}_C, \mathbf{Y}_C) eq. 3

\mathcal{L}_{detect} = \gamma\,\mathcal{L}^{camera}_{detect} + \mathcal{L}^{trans}_{detect} + \mathcal{L}^{LiDAR}_{detect} + \mathcal{L}^{fusion}_{detect} eq. 4

The pseudocode below follows the paper's description. Where the released code differs, see Paper vs code.

Pseudocode
# SparseFusion training forward pass  [fig. 2] [alg. 1]
# Input:  points (N, 5); images (V, 3, 448, 800); camera params per view
# Output: classes and boxes in the LiDAR frame, one set per object query

def sparsefusion_forward(points, images, cameras):
    # LiDAR backbone gives a BEV feature map  [§3.1]
    bev_feat = lidar_backbone(points)                  # (B, C, H, W)

    # Image backbone and FPN give four feature levels per view  [§7.1]
    img_feats = fpn(resnet50(images))                  # 4 x (B*V, C, h, w)

    # Project LiDAR points into each view as a sparse depth map  [§7.1]
    sparse_depth = project_depth(points, cameras)      # (B, V, 2, 112, 200)

    # Encode depth and fuse it into each image level  [§3.2] [alg. 1]
    img_feats = depth_encoder(sparse_depth, img_feats)

    # Add image semantics to BEV cells that contain LiDAR points  [§3.2]
    bev_sem = semantic_transfer(bev_feat, img_feats, points, cameras)

    # Initialise LiDAR queries from the top heatmap peaks  [§7.3]
    lidar_queries, lidar_refs = init_from_heatmap(bev_sem, n=200)

    # One LiDAR decoder layer with cross-attention to BEV  [§3.1]
    lidar_cands = lidar_decoder(lidar_queries, lidar_refs, bev_feat)

    # Initialise camera queries from projected box centres  [§7.3]
    cam_queries, cam_refs = init_camera_queries(img_feats, cameras, n=200)

    # One camera decoder layer, each query uses only its own view  [§3.1]
    cam_cands, cam_boxes = camera_decoder(cam_queries, cam_refs, img_feats)

    # Move camera boxes into the LiDAR frame and embed them  [§3.1] [eq. 1]
    cam_cands = to_lidar_frame(cam_cands, cam_boxes, cameras)

    # Project both candidate sets and concatenate them  [eq. 2]
    fused_in = concat(f_lidar(lidar_cands), f_cam(cam_cands))

    # Self-attention over all 400 candidates, then prediction  [§3.1]
    fused = self_attention_ffn(fused_in)
    return lidar_view_head(fused)                      # classes, boxes


def sparsefusion_loss(outputs, targets):
    # Gaussian focal loss on the two heatmaps, weight alpha = 0.1  [eq. 3]
    init_loss = gaussian_focal(outputs.bev_heat, targets.bev_heat) \
        + gaussian_focal(outputs.img_heat, targets.img_heat)

    # Match each detector's predictions to ground truth, one-to-one  [§3.3]
    matches = hungarian_match(outputs, targets)

    # Focal loss for classes and L1 for boxes, summed over detectors  [§3.3] [eq. 4]
    det_loss = detection_loss(outputs, targets, matches)

    # Combine with alpha = 0.1 and beta = 1  [§3.3]
    return 0.1 * init_loss + 1.0 * det_loss

Hyperparameters

Values are as stated in the paper unless the "Where" column says they come from code. Paper settings are in §4.2. Code citations are to the nuScenes config configs/sparsefusion_nusc_voxel_LC_r50.py in the repository at code_commit.

NameValueWhereNotes
point cloud rangex, y in [-54, 54] m; z in [-5, 3] m§4.2 sparsefusion_nusc_voxel_LC_r50.pymatches
voxel size0.075 x 0.075 x 0.2 m§4.2 sparsefusion_nusc_voxel_LC_r50.pymatches
image resolution800 x 448§4.2 sparsefusion_nusc_voxel_LC_r50.pymatches
camera views6§4.1 sparsefusion_nusc_voxel_LC_r50.py
queries per branchN_L = N_C = 200§4.2 sparsefusion_nusc_voxel_LC_r50.pyat most 400 objects per scene §4.2
decoder layers per branch1 (LiDAR, camera, fusion)§4.2 sparsefusion_nusc_voxel_LC_r50.py
hidden dimension128, except backbones§4.2 sparsefusion_nusc_voxel_LC_r50.py
attention heads / FFN width8 / 256code only sparsefusion_nusc_voxel_LC_r50.pynot stated in paper
dropout0.1code only sparsefusion_nusc_voxel_LC_r50.pynot stated in paper
image backboneResNet-50 with FPN, 4 levels used§4.2 §7.1 sparsefusion_nusc_voxel_LC_r50.pyinitialisation: see Paper vs code
LiDAR backboneVoxelNet (sparse encoder, SECOND backbone and neck in code)§4.2 sparsefusion_nusc_voxel_LC_r50.py
optimiserAdamW§4.2 sparsefusion_nusc_voxel_LC_r50.py
learning rate1e-4 initial§4.2 sparsefusion_nusc_voxel_LC_r50.pyimage backbone and LiDAR layers use 0.1x sparsefusion_nusc_voxel_LC_r50.py
learning-rate schedule"one-cycle"§4.2 sparsefusion_nusc_voxel_LC_r50.pycode uses mmcv "cyclic" with target_ratio (8, 0.0001) and step_ratio_up 0.4; exact curve not stated in paper
weight decay1e-2§4.2 sparsefusion_nusc_voxel_LC_r50.py
gradient clippingmax norm 0.1, L2code only sparsefusion_nusc_voxel_LC_r50.pynot stated in paper
LiDAR-only pretraining20 epochs (TransFusion-L)§4.2 transfusion_nusc_voxel_L.py
fusion training6 epochs, LiDAR components and detector frozen§4.2 sparsefusion_nusc_voxel_LC_r50.py sparsefusion_nusc_voxel_LC_r50.py
batch size16 (4 GPUs x 4 samples)§4.2 sparsefusion_nusc_voxel_LC_r50.pythe config's gpu_ids lists 8 GPUs sparsefusion_nusc_voxel_LC_r50.py
class balancingCBGS§4.2 sparsefusion_nusc_voxel_LC_r50.py
LiDAR augmentationrotation +/-0.785 rad; scale 0.9 to 1.1; translation std 0.5; random flipstypes in §4.2; values code only sparsefusion_nusc_voxel_LC_r50.pyvalues not stated in paper
camera augmentationscale 0.9 to 1.1; horizontal flip p = 0.5types in §4.2; values code only sparsefusion_nusc_voxel_LC_r50.py
LiDAR sweeps10code only sparsefusion_nusc_voxel_LC_r50.pysweeps not mentioned in paper
sparse depth mapscale 4, so 200 x 112; 2 channels in code§7.1 sparsefusion_nusc_voxel_LC_r50.py
depth normalisationmean 14.41, variance 156.89code only loading.pynot stated in paper
geometric transfer levelsL = 4alg. 1 §7.1 sparsefusion_head_deform.py
camera level thresholds0, 48, 96, 192, infinity (box side)§7.3 sparsefusion_head_deform.py
loss weightsgamma = 1, alpha = 0.1, beta = 1§3.3code weights differ, see Paper vs code
heatmap loss weight0.1 (both heatmaps)§3.3 sparsefusion_nusc_voxel_LC_r50.py
classification lossfocal, weight 1.0§3.3 sparsefusion_nusc_voxel_LC_r50.pygamma 2, alpha 0.25 in code
box L1 loss weightnot stated in papercode: 0.25 sparsefusion_nusc_voxel_LC_r50.py
camera-centre L1 weightnot stated in papercode: 5.0 sparsefusion_nusc_voxel_LC_r50.py
matching costsfocal 0.15; BEV L1 0.25; 3D IoU 0.25code only sparsefusion_nusc_voxel_LC_r50.pypaper says only "Hungarian" §3.3
velocity loss weight0.2code only sparsefusion_nusc_voxel_LC_r50.py
heatmap Gaussian overlap / min radius0.1 / 2code only sparsefusion_nusc_voxel_LC_r50.py
query NMS kernel3code only sparsefusion_nusc_voxel_LC_r50.py
test NMS"circle"code only sparsefusion_nusc_voxel_LC_r50.py

Results

Author-reported. Setup: nuScenes, no test-time augmentation and no model ensembles §4.3 tab. 2. Validation and test scores are from the paper's own table. The paper reports 5.6 FPS against 3.1 FPS for DeepInteraction on one A6000 GPU, which the authors call 1.8x §4.3, and claims the fastest inference among the compared methods fig. 1.

Comparison with existing methods tab. 2:

MethodModalityLiDAR backboneCamera backboneval NDSval mAPtest NDStest mAP
FCOS3DCamera-ResNet-10141.534.342.835.8
PETRCamera-ResNet-10144.237.045.539.1
CenterPointLiDARVoxelNet-66.859.667.360.3
TransFusion-LLiDARVoxelNet-70.165.170.265.5
PointAugmentingLiDAR + cameraVoxelNetDLA34--71.066.8
FUTR3DLiDAR + cameraVoxelNetResNet-10168.364.5--
UVTRLiDAR + cameraVoxelNetResNet-10170.265.471.167.1
TransFusionLiDAR + cameraVoxelNetResNet-5071.367.571.668.9
AutoAlignV2LiDAR + cameraVoxelNetCSPNet71.267.172.468.4
BEVFusionLiDAR + cameraVoxelNetDual-Swin-T72.169.673.371.3
BEVFusionLiDAR + cameraVoxelNetSwin-T71.468.572.970.2
DeepInteractionLiDAR + cameraVoxelNetResNet-5072.669.973.470.8
CMTLiDAR + cameraVoxelNetResNet-5070.867.9--
SparseFusion (proposed)LiDAR + cameraVoxelNetResNet-5072.870.473.872.0

Performance breakdown on the validation set tab. 3:

MethodsModalityNDSmAP
TransFusion-LL70.165.1
LiDAR branch (with semantic transfer)L + ST71.868.4
PETR (ResNet-50)C38.131.3
Camera branch, before view transformationC + GT43.540.6
Camera branch, after view transformationC + GT44.341.5
SparseFusionL + C72.870.4

Image backbone comparison on the validation set, no test-time augmentation tab. 4. The FPS column is measured on one A6000 GPU:

MethodsImage backboneInput resolutionNDSmAPFPS
BEVFusionDual-Swin-T1600 x 90072.169.60.8
CMTVoVNet-991600 x 64072.970.33.8 (flash-attention)
SparseFusionResNet-50800 x 44872.870.45.6
SparseFusionSwin-T800 x 44873.171.05.3

Ablations tab. 5. The paper trains these on a 1/5 split of the nuScenes training set and evaluates on the full validation set, so they are not directly comparable with the main table §4.5:

Sparse fusion strategy:

Fusion strategyNDSmAP
MLP, no cross-instance mixing67.864.5
Cross-attention (LiDAR queries, camera keys/values)68.665.8
Optimal transport68.065.6
Self-attention (used)68.866.4

Information transfer:

Geometric transferSemantic transferNDSmAP
yesno67.764.2
noyes68.465.7
yesyes68.866.4

Parallel versus sequential detectors:

Sequential variantNDSmAP
camera queries take LiDAR positions; features come from image features68.165.3
camera queries take both LiDAR positions and instance features67.964.2
parallel detectors (used)68.866.4

The text states that removing the view transformation lowers the score from 68.8 NDS and 66.4 mAP to 68.3 NDS and 65.6 mAP §4.5.

Category-wise results on the validation set tab. 6:

MethodsNDSmAPcartruckbustrailerconst. veh.ped.moto.bicycletraffic conebarrier
TransFusion-L70.265.186.559.674.442.225.486.672.156.074.174.1
SparseFusion72.870.488.564.477.144.330.389.881.571.080.676.6

The paper also reports, in words, that camera detections recall objects that LiDAR misses, mainly at long range and for some classes §4.4 fig. 5 and that many fused objects draw attention mostly from neighbouring instances §4.4 fig. 6.

Paper vs code

Code: official repository https://github.com/yichen928/SparseFusion, linked from the paper's abstract. Commit 22537781e033b0269249a894158db6dc8933997f, Apache-2.0. The code is built on MMDetection3D. The model code is in mmdet3d/models/dense_heads/sparsefusion_head_deform.py and mmdet3d/models/utils/sparsefusion_models.py.

The config matches the paper for the input resolution, voxel size, point range, query counts, number of decoder layers, hidden width, the 20 and 6 epoch schedule, the batch size of 16, the 200 x 112 depth map and the four-level camera-size thresholds.

Pitfalls

  • The camera branch needs LiDAR at inference. The test pipeline builds sparse depth from the same LiDAR points sparsefusion_nusc_voxel_LC_r50.py. The paper does not describe a camera-only mode.
  • Two-stage training with frozen components. Training needs a LiDAR-only TransFusion-L trained for 20 epochs, then 6 epochs of fusion with the LiDAR components frozen §4.2. The code freezes both the LiDAR backbone and the LiDAR detector sparsefusion_nusc_voxel_LC_r50.py. Reproduction also needs the external "initial weights" README.md. The config also sets find_unused_parameters = True sparsefusion_nusc_voxel_LC_r50.py.
  • Python loops in the camera branch. The camera decoder loops over samples and views in Python sparsefusion_models.py. The heatmap deformable attention also loops over views sparsefusion_head_deform.py. Speed figures are for the authors' implementation on one A6000 GPU tab. 4.
  • Speed comparison. The 1.8x speed-up compares against DeepInteraction, timed by the authors on the same hardware §4.3. The main table gives no timings.
  • Paper-internal inconsistency. TransFusion-L has validation NDS 70.1 in tab. 2 and tab. 3, but 70.2 in tab. 6.
  • Ablation numbers are not comparable. Ablations use a 1/5 training split §4.5, so their absolute values are lower than the main table.
  • Loss balance. The paper says gamma, alpha and beta were set empirically §3.3 and does not give the L1 weight. The code's effective L1 weights differ from the paper's unit weighting (see Paper vs code), so implementing the equations literally will not match the code's loss balance.
  • Dense transfer work remains. The semantic transfer visits every occupied BEV cell and projects it into the images §7.1, and the depth encoder runs on every view §3.2 alg. 1.

Lineage

  • Builds on: TransFusion-L for the LiDAR branch and its heatmap query initialisation §3.1 §7.3; Deformable DETR for the camera branch §3.1 §7.1; CenterFormer for heatmap initialisation §7.3; the camera-embedding multiplication of BEVDepth §3.1; FCOS-style scale assignment §7.3. These are cited by name, as in the paper.
  • Compared against: the methods in tab. 2, including BEVFusion, DeepInteraction, CMT, UVTR, FUTR3D, PointAugmenting and AutoAlignV2.
  • Superseded by: not stated in the paper, and no such paper is known from this wiki.

Source and license

Adapted from “SparseFusion: Fusing Multi-Modal Sparse Representations for Multi-Sensor 3D Object Detection” by Yichen Xie, Chenfeng Xu, Marie-Julie Rakotosaona, Patrick Rim, Federico Tombari, Kurt Keutzer, Masayoshi Tomizuka, Wei Zhan (arXiv:2304.14340), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yichen928/SparseFusion at commit 22537781e033 (Apache-2.0). Any quoted code is shown under that license.