DeepInteraction: 3D Object Detection via Modality Interaction

In plain words

Self-driving cars often carry a laser scanner (LiDAR) and several cameras, and a detector has to find cars, pedestrians and other objects in that data. The usual approach squeezes the laser and camera information into one combined picture and searches that picture.

This paper argues that the squeeze throws away what each sensor does well. The laser gives geometry at low resolution, and the cameras give rich appearance at high resolution. So the method keeps one picture per sensor all the way through. The two pictures pass information to each other in a few steps, using the known geometry that links each laser point to the camera pixels it lands on. A set of candidate objects is then refined in turns, some turns reading the camera picture and the others the laser picture.

The authors test this on nuScenes, a public driving dataset with ten object classes, and report top results on its leaderboard.

TL;DR

DeepInteraction detects 3D objects from one LiDAR and six cameras on nuScenes. Instead of merging both sensors into one map, it keeps a representation per sensor and lets them exchange information in an encoder and in a decoder. The author-reported test scores are 70.8 mAP and 73.4 NDS for the base model with a ResNet-50 image backbone, and first place on the leaderboard for the ensemble. The official code is MIT-licensed, but several paper configurations are not in the repository, and paper and code disagree on details that change results.

Problem

Point clouds give precise localisation and geometry at low resolution, and images give rich appearance at high resolution, so combining them should help 3D detection §1. Existing multi-modal detectors combine the two sensors into one hybrid representation, for example by painting image semantics onto points or by merging both into a joint bird's-eye-view (BEV) map §1 fig. 1. The authors argue that this fusion discards modality-specific strengths that a single unified map cannot keep §1.

Cost is also a problem. Feature extraction for high-resolution multi-view camera images accounts for most of the latency of a multi-modal detector, as reported by a prior work the authors cite §4.3.

The method depends on calibration: it builds 2D-3D correspondences from sensor calibration, so calibration errors propagate into the interaction. The authors name this dependency and efficiency as limitations app. A.3.

Method

The network has two backbones that run in parallel: a LiDAR backbone that outputs a BEV feature map, and an image backbone that processes the six camera views §3 §4.2. An encoder of two layers lets the two feature sets interact §3.1 §4.2. A decoder then refines 200 object queries in a DETR-style set prediction, alternating between image and LiDAR §3.2 §4.2.

Encoder: multi-modal representational interaction §3.1

Each encoder layer has three parts §3.1:

1. Cross-modal interaction (MMRI). Every cell of one modality attends to the cells of the other modality that its geometry links it to. The two correspondence maps are:
   • Image to LiDAR BEV (the paper's M_c→p): project LiDAR points into the cameras to get a sparse depth map, complete it to a dense depth map, back-project each pixel to 3D and read off its BEV cell. The neighbourhood is a (2k+1) x (2k+1) sample around each pixel §3.1 (i).
   • LiDAR BEV to image (M_p→c): for each BEV cell, take the LiDAR points inside its pillar and project them into the cameras §3.1 (i).
2. Intra-modal learning (IML). Each modality attends to a k x k grid of its own features, with the same local attention §3.1 (II).
3. Integration. Each modality's two outputs are merged with its input through two feed-forward steps eq. 2 §3.1 (III).

The attention is local attention over neighbour sets:

f_{\phi}(h_c, h_p)^{[i,j]} = \sum_{k, v \in \mathcal{N}_q} \mathrm{softmax}\!\left(\frac{q\,k}{\sqrt{d}}\right) v eq. 1

The integration step is

h_p' = \mathrm{FFN}\big(\mathrm{Concat}\big(\mathrm{FFN}(\mathrm{Concat}(h_p^{p \to p},\, h_p^{c \to p})),\, h_p\big)\big) eq. 2

with the symmetric expression for the image branch §3.1.

# One encoder layer, applied twice  [§3.1] [fig. 2]
# lidar: (B, C, H_p, W_p) BEV map;  image: (B*N, C, H_c, W_c) camera maps
def encoder_layer(lidar, image, calib):
    # Image to LiDAR: each BEV cell attends to the camera pixels that its
    # pillar's points project to  [§3.1 (i)] [eq. 1]
    # [code: encoder_utils.py#L216-L320]
    image_to_lidar = cross_attend(lidar, image, calib)
    # Inside the BEV map: attend over a 9x9 window  [§3.1 (II)]
    # [code: deepinteraction_encoder.py#L12]
    lidar_to_lidar = window_attend(lidar, lidar, size=9)
    # Integration: 1x1 conv + BN twice, no activation  [eq. 2]
    # [code: deepinteraction_encoder.py#L26-L27]
    lidar_mix = conv1x1_bn(concat(image_to_lidar, lidar_to_lidar))
    lidar_out = conv1x1_bn(concat(lidar_mix, lidar))
    # LiDAR to image: the BEV map is warped into each camera using depth
    # completion, then attended to  [§3.1 (i)]
    # [code: encoder_utils.py#L137-L213] [code: deepinteraction_encoder.py#L29]
    lidar_to_image = cross_attend(image, warp_bev_to_camera(lidar, calib))
    # Inside the camera maps: the same 9x9 window  [§3.1 (II)]
    # [code: deepinteraction_encoder.py#L17]
    image_to_image = window_attend(image, image, size=9)
    # Image integration  [code: deepinteraction_encoder.py#L31-L32]
    image_mix = conv1x1_bn(concat(lidar_to_image, image_to_image))
    image_out = conv1x1_bn(concat(image_mix, image))
    return image_out, lidar_out

Decoder: multi-modal predictive interaction §3.2

The decoder starts from object queries and refines them over several layers §3.2 fig. 3:

1. Query start. Queries are initialised from the strongest heatmap peaks, as in TransFusion §4.2. A heatmap is built from the BEV features, and the top 200 peaks become queries deepinteraction_decoder.py.
2. Alternating interaction. The first and every other layer read the image. Each 3D box is projected into every camera, its projection is replaced by the smallest axis-aligned rectangle that encloses it, and a 7 x 7 RoI is pooled from that rectangle §3.2 (I). The other layers read the LiDAR BEV map with the same procedure, with each box's size doubled before pooling §3.2 (II).
3. Query update. Each query's parameters are generated from the query itself and applied to the RoI features §3.2. The updated queries feed prediction heads for centre, height, size, rotation and velocity §3.2.
4. Auxiliary supervision. Every layer's predictions are supervised with the same matching and loss as TransFusion §3.2 deepinteraction_decoder.py deepinteraction_decoder.py.

# Decoder: start from heatmap peaks, then refine per layer  [§3.2] [fig. 3]
# heatmap: (C, H, W) class scores on the BEV grid
def decode(heatmap, lidar, image, calib):
    # 200 queries at the strongest peaks  [§4.2]
    # [code: deepinteraction_decoder.py#L225-L253]
    queries, boxes = top_peaks(heatmap, count=200)
    for step in range(4):
        # Layers 1 and 3 read the cameras, layers 2 and 4 the BEV map
        # [§3.2] [code: deepinteraction_decoder.py#L281-L289]
        if step % 2 == 0:
            # Project each box into every camera and pool a 7x7 RoI
            # [§3.2 (I)] [code: decoder_utils.py#L641-L645]
            rois = image_rois(boxes, image, calib)
        else:
            # Pool a 7x7 RoI from the BEV box, doubled in size first
            # [§3.2 (II)] [code: decoder_utils.py#L807]
            rois = bev_rois(boxes, lidar)
        # Query-conditioned dynamic conv on the RoI, then FFN  [§3.2]
        # [code: decoder_utils.py#L584-L631]
        queries = refine(queries, rois)
        # Heads over the concatenated new and previous query features
        # [code: deepinteraction_decoder.py#L289-L290]
        boxes, scores = predict(queries, previous=boxes)
    return boxes, scores

Training recipe §4.2

Training has two stages. The LiDAR-only baseline (TransFusion-L) is trained for 20 epochs. The fusion model is then trained for 6 epochs, with the image branch frozen §4.2. The fusion config loads a checkpoint named r50_fpn_voxel_0075.pth Fusion_0075_refactor.py, which the README calls pretrained backbone weights README.md. The LiDAR-only stage has no config in the repository. The paper does not say which LiDAR parts are frozen in the second stage (see Paper vs code).

Hyperparameters

Unless stated otherwise, values are from the base model. Paper citations are section or appendix numbers; code citations use the official repository at 97ae7c8.

Results

Author-reported. The evaluation metrics are mAP (averaged over 0.5, 1, 2 and 4 m thresholds and over 10 classes) and NDS §4.1. The paper reports test-set numbers and states that its ensemble ranks first on the nuScenes leaderboard §4.3 tab. 1. Entries marked with a dagger use test-time augmentation; entries marked with a section sign use test-time augmentation and ensembling tab. 1.

Main comparison, nuScenes test and val tab. 1

Efficiency tab. 2 (mAP and NDS on val unless marked; FPS on A100, A6000 and V100 as listed)

Decoder ablations, val tab. 3

(a) Layer type, LiDAR branch / image branch

(b) Single vs multiple representations

(c) Number of decoder layers

(d) Queries, training / inference

Encoder ablations, val tab. 4

(a) Encoder design

(b) Number of encoder layers

(c) Interaction vs fusion, with the same encoder and decoder depth as TransFusion §4.5

LiDAR backbones, val tab. 5

Per-class, val tab. 6

Per-class, test app. A.1 tab. 7

Key findings, as reported by the authors:

• The base model with a ResNet-50 image backbone reaches 70.8 mAP and 73.4 NDS on test and beats BEVFusion, which uses Swin-Tiny, at 70.2 and 72.9 §4.3 tab. 1.
• The large model reaches 74.1 mAP and 75.5 NDS on test with test-time augmentation tab. 1. The ensemble reaches 75.6 mAP and 76.3 NDS and is reported as first on the leaderboard among all solutions §4.3 tab. 1.
• Efficiency: DeepInteraction runs at 2.6 FPS on V100 and 4.9 FPS on A100 tab. 2. The authors say it runs faster than PointAugmenting and FUTR3D and slower than TransFusion §4.3 tab. 2.
• Decoder: replacing the DETR decoder layer with the proposed interaction layer adds 1.3 mAP and 1.0 NDS on val §4.4 tab. 3(a). Alternating image and LiDAR layers beats LiDAR-only layers by 0.7 mAP tab. 3(b).
• Encoder: interaction and intra-modal learning are complementary; two encoder layers give the best val result §4.5 tab. 4(a)(b).
• Backbones: the gain over the LiDAR-only baseline is 5.5 mAP with PointPillars and 4.8 mAP with VoxelNet, from the table values §4.6 tab. 5.
• Per class, the largest gains over TransFusion-L on val are bicycle, motorcycle and traffic cone §4.7 tab. 6.

Paper vs code

The official repository is fudan-zvg/DeepInteraction (MIT, Copyright 2022 Fudan Zhang Vision Group), at commit 97ae7c87706f5a73179b9e64797fc08842cbb5a4. Its owner matches the authors' group: the paper links this URL in its author block, and the corresponding author Li Zhang is at Fudan University [paper author block] LICENSE. The repository has one base configuration (Fusion_0075_refactor.py) and one Swin variant it names DeepInteraction++ (Fusion_0075_plusplus.py), which is not the paper's large model. The README reports val 69.85 / 72.63 for the base configuration README.md. Its test numbers are 70.78 / 73.43 (base), 74.12 / 75.52 (large) and 75.74 / 76.34 (ensemble) README.md.

Matches, after checking:

• Point cloud range, voxel size 0.075 m, and the rotation, scaling and translation augmentation ranges match §4.2 Fusion_0075_refactor.py.
• Batch size 16 (8 GPUs x 2) and 6 fusion epochs match §4.2 Fusion_0075_refactor.py.
• Encoder depth 2 and 200 queries match §3.1 §4.2 Fusion_0075_refactor.py.
• The first-layer interaction is on the image and the second on LiDAR, as stated §3.2 deepinteraction_decoder.py.

Differences:

• Optimizer. The paper says Adam §4.2; the code uses AdamW with weight decay 0.01 Fusion_0075_refactor.py.
• Gradient clipping. The code clips the gradient norm at 0.1 Fusion_0075_refactor.py. The paper does not mention clipping.
• Frozen modules. The paper freezes only the image branch §4.2. The code sets freeze_img=True and freeze_pts=True Fusion_0075_refactor.py. The freeze rule sets requires_grad=False for every parameter whose name contains "pts", except the BEV head and the interaction encoder. It then freezes the encoder's LiDAR projection convolution, the first decoder layer, the heatmap head and the first prediction head deepinteraction.py. The interaction encoder's fusion blocks and the rest of the decoder stay trainable.
• Flip augmentation. The paper uses horizontal flips §4.2. The code uses horizontal and vertical flips, each with probability 0.5 Fusion_0075_refactor.py.
• Image size. The paper rescales images to half of 1600 x 900 §4.2. The code uses 800 x 448 Fusion_0075_refactor.py.
• Decoder depth. The paper says 5 cascaded decoder layers §4.2. The code has one transformer decoder layer (the class default) and num_mmpi=4 interaction layers deepinteraction_decoder.py Fusion_0075_refactor.py. Five is consistent only if the transformer layer is counted. The code asserts that num_mmpi is even deepinteraction_decoder.py. The configuration that produced the layer ablation in Table 3(c) is not in the repository.
• Query count at test. The paper uses 200 queries for training and testing §4.2, but its best setting is 200 train and 300 test tab. 3(d). The code uses 200 Fusion_0075_refactor.py.
• Image feature level. The code takes the first FPN level only deepinteraction.py. The paper does not say which level is used.
• Encoder details the paper leaves open. The encoder width is 128 Fusion_0075_refactor.py. The window is 9 x 9 deepinteraction_encoder.py. The "FFN" of eq. 2 is a 1 x 1 convolution with batch norm and no activation deepinteraction_encoder.py. Its attention normalises the scores over the neighbours, not as the sum outside the softmax written in eq. 1 encoder_utils.py.
• Direction of the image-to-LiDAR attention. The paper's text gives the image feature as query and the LiDAR cells as neighbours for the c-to-p interaction §3.1 (ii). Fig. 2's caption describes the same c-to-p arrow as spreading image signal into the LiDAR representation fig. 2. The code follows the figure: the LiDAR cell is the query and the image pixels are keys and values encoder_utils.py encoder_utils.py.
• Pillar grid. The interaction uses 0.6 m pillars and at most 20 points per pillar Fusion_0075_refactor.py. The paper does not state the grid. Its 0.2 m voxel setting for PointPillars §4.6 is for the LiDAR backbone, not this grid.
• Heatmap. The LiDAR heatmap is computed and detached, so it receives no gradient. Only the image-side heatmap is supervised deepinteraction_decoder.py deepinteraction_decoder.py deepinteraction_decoder.py.
• Interaction masking. A query that projects into no camera keeps the first layer's predictions and is excluded from the image-stage loss deepinteraction_decoder.py deepinteraction_decoder.py. A camera with one or fewer queries is skipped decoder_utils.py.
• Activations. The RoI blocks use GELU decoder_utils.py decoder_utils.py. The transformer decoder uses ReLU Fusion_0075_refactor.py.
• Test-time post-processing. No NMS is applied at test time Fusion_0075_refactor.py deepinteraction_decoder.py. The paper does not mention NMS.
• LiDAR sweeps. The code aggregates 10 sweeps Fusion_0075_refactor.py. The paper does not mention sweeps.
• Missing configs. The large model (Swin-Tiny, voxel 0.5 m, channel doubling, TTA, ensemble) and the LiDAR-only baseline have no configuration in the repository. The Swin config in the repository is the DeepInteraction++ variant, with 9 epochs and multi_scale=True Fusion_0075_plusplus.py Fusion_0075_plusplus.py. Paper numbers for these settings cannot be checked against the code.
• README test score. The README's ensemble test score is 75.74 / 76.34 README.md. Table 1 gives 75.6 / 76.3 tab. 1, and the appendix gives 75.7 / 76.3 tab. 7.

Pitfalls

• Loss and query-direction ambiguity. Eq. 1 writes the sum outside the softmax, but the code normalises over neighbours encoder_utils.py. The caption and the text disagree on which modality is the query for the image-to-LiDAR path. Follow the code, which uses LiDAR cells as queries for that path.
• Calibration and projection order. Both projection paths work on point clouds that have been transformed back from the augmented frame to the original LiDAR frame before projection encoder_utils.py encoder_utils.py decoder_utils.py. Projecting augmented points directly misaligns the image and LiDAR. The paper does not mention this step.
• Depth completion cost. The image-side warp runs classical multi-scale depth completion on the CPU for every camera and sample, with a device-to-host copy encoder_utils.py. This is a synchronisation point in each training step and is probably the costliest non-learned step. The authors state that efficiency was not considered app. A.3.
• Hard-coded range. The warp hard-codes the nuScenes range of plus or minus 54 m in x and y, with a TODO noting other datasets are not supported encoder_utils.py. Other ranges need code changes.
• Hard-coded group sizes. The image-to-LiDAR attention groups neighbours into sizes 20, 40, 80 and 120 encoder_utils.py. These match 20 points per pillar times 6 cameras. Changing the pillar point cap requires changing the groups.
• Frozen BatchNorm at test time. Frozen LiDAR modules are set to track_running_stats=False deepinteraction.py. In PyTorch, that makes BatchNorm use batch statistics even in evaluation, so test outputs depend on batch composition. The image ResNet uses norm_eval=True instead Fusion_0075_refactor.py.
• Two-stage recipe. The LiDAR initialisation and the frozen set in the second stage decide the outcome. The paper does not list which parts are frozen, so a reproduction will differ unless it follows the code.
• Margin claims. The abstract says the method "surpasses all prior arts often by a large margin". Derived from Table 1 against the strongest rival in the same setting: base +0.6 mAP and +0.5 NDS over BEVFusion (Liu et al.); large +3.0 mAP and +1.3 NDS over LargeKernel3D-F, both with TTA; ensemble +0.6 mAP and +0.2 NDS over BEVFusion-e.
• Reduced model in Table 4(c). The "ours" entry in Table 4(c) (68.7 mAP) uses fewer encoder and decoder layers than the full model (69.9 mAP) §4.5 tab. 4(c). Do not compare it with the full model.
• Mixed bases in Table 2. Table 2 mixes test (PointAugmenting) and val numbers, and the GPU behind the FPS column of Table 3(c) is not stated tab. 2 tab. 3(c).
• Decoder-layer latency. Table 3(c) says extra decoder layers add negligible latency. Between 2 and 5 layers, FPS goes from 2.8 to 2.6. The drop from 8.7 to 2.8 comes from adding the image branch, not from adding a layer tab. 3(c).
• LiDAR backbone gains. Section 4.6 says 5.5 mAP for voxel and 4.4 mAP for pillar. Table 5 gives 4.8 for voxel (69.9 minus 65.1) and 5.5 for pillar (60.0 minus 54.5) §4.6 tab. 5.
• Table 6 and Table 1 disagree. Table 6 gives TransFusion-L val NDS 70.2 and DeepInteraction val NDS 72.7. Table 1 gives 70.1 and 72.6 tab. 1 tab. 6.
• Table 7 errors. The FUTR3D row duplicates AutoAlign's numbers and cites AutoAlign's reference tab. 7. Table 7 also gives TransFusion test NDS 71.7 (Table 1: 71.6) and DeepInteraction-e test mAP 75.7 (Table 1: 75.6) tab. 7. Treat Table 7 as unreliable for the FUTR3D row.
• Percent and points. The paper writes "1.3%" and "11.8%" for absolute differences in mAP points, not relative changes §4.4 §4.7.
• Test-time augmentation and ensembling. The large and ensemble results need TTA and ensembling, and their configurations are missing from the repository. Compare them only with other entries marked the same way.

Lineage

• Builds on: TransFusion (Bai et al., 2022, not yet a card) for the LiDAR backbone, query initialisation, matching and loss §3.2 §4.2. DETR decoder layers, as the comparison baseline §4.4 tab. 3(a); see DETR.
• Compared against: BEVFusion (Liang et al.) [/papers/2205.13790], BEVFusion (Liu et al.), FUTR3D, PointAugmenting, MVP, AutoAlignV2, CenterPoint, Focals Conv, LargeKernel3D, PolarFormer and others listed in tab. 1.
• Superseded by: not known from content in this wiki.