CenterFormer: Center-based Transformer for 3D Object Detection

In plain words

Self-driving cars often use a LiDAR, which returns a cloud of points. One common way to find cars, pedestrians and cyclists in such a cloud is to flatten it into a top-down map and look for the centre of each object. This paper keeps that idea and adds a transformer, the attention mechanism behind many modern models. Each candidate centre becomes a query, and the query looks at the features around its own spot to pick up context. The same idea is extended to earlier scans of the scene, so the detector can follow objects that move. The authors argue that this trains more easily than standard transformer detectors, which learn their queries from scratch.

TL;DR

• A centre heatmap on a bird's-eye-view (BEV) map picks candidate object centres. Their features become the queries of a transformer decoder §3.2 §3.2.1.
• Cross-attention reads a 3x3 window at three scales, or 15 learned sampling offsets per head and scale in the deformable variant §3.2.2 §3.2.3 eq. 3 eq. 4.
• Multi-frame fusion adds the previous frame's BEV feature to the heatmap and to the cross-attention §3.3 app. A.
• Author-reported: 73.7 L2 mAPH on Waymo validation with 8 frames and 75.6 on test with a single model tab. 1 tab. 2.
• Official code (MIT) exists. Several details differ between the paper and the code, listed under Paper vs code.

Problem

Two-stage LiDAR detectors refine each proposal using only features inside its box, so they miss context outside it §1 app. C. DETR-style transformer decoders can model that context, but on LiDAR they are hard to apply: the BEV map is too large for global attention, and learned queries converge slowly §1 §4.5. Multi-frame methods that concatenate points assume that objects do not move between frames; the authors note that independently moving objects break this assumption §2 app. C.

Method

The network has four parts: a voxel encoder that produces a BEV feature map, a multi-scale center proposal network (CPN), a center transformer decoder and regression heads fig. 2. Training uses the center-based assignment of CenterPoint and no Hungarian matching app. A.

1. Backbone. Points are averaged per voxel, then a sparse-convolution VoxelNet-style backbone produces a BEV map. The down-sampling is by 8 in x and y and by 16 in z, so the BEV cell is 0.8 m app. A. The Waymo grid is 0.1 m x 0.1 m x 0.15 m over x,y in [-75.2, 75.2] m and z in [-2, 4] m §4.1.
2. Multi-scale CPN. The BEV map is up-sampled and down-sampled into three scales through one up-sampling and one down-sampling layer. Each scale ends with a CBAM block (channel and spatial attention) §3.2.1 app. A rpn_transformer.py.
3. Centre proposals. A centre head on the finest scale gives one heatmap channel per class. The top N peaks become proposals, with N = 500 in training and N = 1000 at evaluation §3.2.1 §4.2 waymo_centerformer.py dist_test.py.
4. Queries and position. The feature at each proposal is its query. A linear layer embeds the proposal's location as a position term §3.2.2 rpn_transformer.py.
5. Decoder layer. A layer is self-attention among the proposals, then cross-attention to the multi-scale BEV features, then a feed-forward block. Each sub-layer has a pre-LayerNorm and a residual connection §3.2.2 transformer.py transformer.py. The base model uses 3 layers and 4 heads; the deformable model uses 2 layers and 6 heads §4.2 waymo_centerformer.py.
6. Base cross-attention. Keys and values come from a 3x3 window around the proposal at each of the three scales, so 27 neighbours per query. The paper states the cost as O(9SN) for S scales and N proposals §3.2.2 rpn_transformer.py.
7. Deformable cross-attention. A linear layer predicts a 2-D offset for each head, scale and sampling point (K = 15 in the experiments). Features are read at p + Δp by bilinear sampling. Attention weights come from a linear layer on the query. Outputs are summed over scales §3.2.3 eq. 4 §4.2 waymo_centerformer_deformable.py.
8. Regression heads. Each enhanced query predicts the box offset (2), height (1), size (3), yaw as two channels (2) and an IoU score (1) §3.1 waymo_centerformer.py center_head_iou_1d.py.
9. Training assignment. Ground-truth centres are added to the proposal ranking so that they are always selected. Only proposals at ground-truth centres are trained as positives. The paper says this avoids costly matching of oriented 3-D boxes app. A rpn_transformer.py.
10. Corner auxiliary head. The centre and the four edge midpoints of each box are drawn as Gaussians with half the centre radius. The corner MSE is applied only where the target is above 0 §3.4 preprocess.py center_head_iou_1d.py.
11. IoU head and rectification. The IoU head is trained against the IoU of each positive proposal's box. At test time each score is multiplied by the predicted IoU raised to a class exponent beta §3.4 §4.2 center_head_iou_1d.py.
12. Loss. Total loss is the heatmap focal loss plus 2 times the box L1 loss, plus the IoU loss, plus the corner MSE §3.4 center_head_iou_1d.py center_head_iou_1d.py.
13. Multi-frame. Each frame passes through the same backbone. The previous frame's last BEV feature is moved into the current coordinates §3.3. Spatial-aware fusion uses the current BEV feature as the reference for a pixel-wise attention, concatenates it with the weighted previous feature and applies a convolution. A time embedding is added by relative time §3.3 app. A fig. 4. In cross-attention, previous-frame features at the proposal are added to the keys (base model) or sampled by the deformable layer (deformable model) §3.3 fig. 3. The 8-frame configuration combines 8 sweeps into 2 chunks of 4 sweeps, so only two BEV maps are fused §4.2 loading.py waymo_centerformer_multiframe_deformable_8frames.py.

Sketch of the single-frame path. It is reconstructed from the paper and the code, not an algorithm in the paper.

# Sketch of the single-frame path  [§3.2] [code: rpn_transformer.py#L415-L489]
# Input:  points (N, 4) for one frame, gt_centres only during training
# Output: per proposal: class, box parameters and IoU score

def centerformer_forward(points, gt_centres=None):
    # Voxelise the points and encode them into a BEV map  [app. A]
    bev = voxel_backbone(points)                       # (B, C, H, W)

    # Build fine, base and coarse scales  [§3.2.1] [app. A]
    fine, base, coarse = multi_scale_fpn(bev)          # H, H/2, H/4

    # Score every BEV cell by its best class  [§3.2.1]
    heatmap = sigmoid(center_head(fine))               # (B, K, H, W)
    scores = max_over_classes(heatmap)                 # (B, H*W)

    # Force ground-truth centres to the front of the ranking  [app. A]
    if gt_centres is not None:
        scores = boost_scores(scores, gt_centres)

    # Keep the top-N peaks: N = 500 in training, 1000 at evaluation  [§4.2]
    proposal_index = top_k(scores, num_proposals)      # (B, N)
    queries = gather(fine, proposal_index)             # (B, N, C)
    centre_xy = grid_xy(proposal_index)                # (B, N, 2)
    position = linear_embed(centre_xy)                 # (B, N, C)

    # Gather 3x3 neighbours at each scale for cross-attention  [§3.2.2]
    neighbours = window_features(centre_xy, [fine, base, coarse], 3)

    # Run the decoder layers; each sub-layer is pre-norm and residual  [§3.2.2]
    for layer in decoder_layers:
        queries = queries + self_attention(queries + position)
        queries = queries + cross_attention(queries + position, neighbours)
        queries = queries + feed_forward(queries)

    # Regression, IoU and corner outputs from the enhanced queries  [§3.4]
    return regression_heads(queries)

Inference rescoring, with the paper's notation for the score α and β.

# Inference rescoring  [§3.4] [§4.2]
# Code: det3d/models/bbox_heads/center_head_iou_1d.py#L361-L363 and #L484-L487

def rescore(score, iou_logit, label, beta):
    # IoU output mapped to [0, 1]  [code: center_head_iou_1d.py#L362]
    iou = clamp((iou_logit + 1) * 0.5, 0.0, 1.0)
    # Raise IoU to the class exponent, then multiply the score  [§3.4]
    return score * iou ** beta[label]

Hyperparameters

Values in "Where" are from the paper unless marked code. "not stated" means the paper does not give the value. Config paths are for the Waymo single-frame configs in the official repository.

Results

Author-reported. Setup: Waymo Open Dataset (WOD) for the main results, with the standard L2 difficulty; a single model; baselines as the authors list them. The test set was scored by the online server §4.3. nuScenes results are in app. B.

Validation set, L2 metrics (mAP / mAPH). Source: tab. 1.

Test set, single model, L2 metrics. Source: tab. 2. Rows are the baselines the paper lists with L2 values.

Vehicle-only training, L2 on validation tab. 3: CenterFormer 70.7 mAP / 70.2 mAPH. The next best listed method is CT3D with 69.0 mAP, and BtcDet has 70.1 mAP / 69.6 mAPH. The paper trains CenterFormer on vehicles only for this comparison.

Ablation, single frame, validation mean L2 mAPH tab. 4. Columns are the components: CPN (multi-scale proposal network), SA (self-attention), CA (cross-attention), Corner (corner head), DCA (deformable cross-attention), IoU (IoU rectification), Fade (copy-and-paste off late).

Multi-frame, LEVEL_2 mAPH on validation, trained without IoU rectification app. C. Star marks point concatenation, the usual baseline.

nuScenes validation app. B. The base model with no IoU rectification and no multi-frame fusion is marked by the paper as ‡. The CenterPoint row is the authors' re-run with the same configuration.

Smaller studies on 20% of the training data app. C. These are the author-reported mean L2 mAPH values from the appendix tables.

• Layers / heads, base: 3/4 gives 64.0, the best; 2/4 gives 63.7; 2/6 gives 63.7; 3/2 gives 63.2; 3/6 gives 63.7; 4/2 gives 63.0; 4/4 gives 63.5.
• Layers / heads, deformable: 2/6 gives 63.9, the best; 1/3 gives 63.3; 2/3 gives 63.4; 3/3 gives 63.7; 3/6 gives 63.1.
• Window sizes, base: 3,3,3 gives 64.0; 5,3,3 gives 64.0; 5,5,3 gives 63.8; 5,5,5 gives 62.7; 7,3,3 gives 63.6.
• Sampling offsets, deformable: 5 gives 62.8; 9 gives 63.2; 15 gives 63.9; 20 gives 63.2.
• Position encoding: none 59.5; sinusoidal 61.7; learned linear 64.0.
• DETR-style set matching in the same setup gives 46.3 mAPH, more than 20 points below the authors' training recipe app. C §4.5.
• Initial queries from object centres beat learned queries by 1.5 mAPH §4.5.

Paper vs code

Code: official repository TuSimple/centerformer at commit 5a949b88ed7bb15aafb39bf78c95f1452063ebea. Licence MIT (Copyright 2022 Zixiang Zhou). The repository's NOTICE says parts are derived from det3d and second, both MIT LICENSE NOTICE. I searched the paper's abstract link and the repository it names; no other repositories were used.

• Position in values. Eq. 2 gives V without the position term eq. 2. In the code, the position embedding is added to the input before the key and value projection in both self-attention and cross-attention, so V includes it transformer.py transformer.py transformer.py.
• Coordinate normalisation for the position embedding. The base model and the multi-frame model use raw grid indices rpn_transformer.py rpn_transformer.py. The deformable models divide by the map height and width rpn_transformer.py rpn_transformer.py. The paper states no normalisation §3.2.2.
• IoU target. The paper's IoU target is the best overlap of each predicted box with any ground-truth box §3.4. The code computes the IoU against the box assigned to each positive centre center_head_iou_1d.py. It also rescales the target to 2·IoU − 1 before smooth L1 center_head_iou_1d.py. The paper mentions neither point.
• IoU output at test time. The head output is mapped from [-1, 1] to [0, 1] before rescoring center_head_iou_1d.py. The paper gives the formula α' = α · iou^β, not this mapping §3.4.
• Copy-and-paste augmentation ("fade"). The config's enable=False key is not read anywhere in the code. The augmentation is active from the start and is removed at epoch 15 by a hook waymo_centerformer.py preprocess.py disable_dbsampler.py. The paper does not give the epoch §4.2.
• Evaluation proposals. N = 1000 is applied only when running tools/dist_test.py, which overwrites the neck's obj_num dist_test.py. The config file alone keeps 500.
• Multi-frame chunking. The paper says 4 and 8 frames are split into 2-frame and 4-frame mini-batches §4.2. The code combines 4 sweeps per chunk, so 8 sweeps give 2 chunks loading.py waymo_centerformer_multiframe_deformable_8frames.py. The two descriptions agree, but the paper's wording is ambiguous.
• Optimiser. The paper says AdamW. The code uses PyTorch Adam with betas (0.9, 0.99) and weight decay applied outside the step, which is AdamW-like train.py. The optimiser is created with a hard-coded placeholder rate of 3e-3 marked "TODO", and the one-cycle schedule sets the real rate from the config train.py learning_schedules_fastai.py.
• Corner head at test time. Corner outputs are computed only in training rpn_transformer.py. The paper's claim that the corner head is auxiliary supervision is consistent with this.
• CBAM. The paper says each scale ends with CBAM §3.2.1. The code adds channel and spatial attention to each block rpn_transformer.py. This matches.
• Ablation numbers. The text says the 2-, 4- and 8-frame gains over point concatenation are 1.0, 0.5 and 1.1 app. C. Those numbers match the deformable rows of the multi-frame table, not the base rows (the base model gains 0.5 at 2 frames).

Not checked: the nuScenes configuration and the vehicle-only training config.

Pitfalls

• Training depends on the ground-truth centre rule. Ground-truth centres are forced into the proposal set, and only proposals at those centres are trained as positives rpn_transformer.py app. A. The paper presents this rule as the reason training converges quickly app. A. The paper gives no ablation of this rule alone.
• Position embedding normalisation changes across variants. Porting a variant to another model needs its own normalisation rule (see Paper vs code).
• The evaluation proposal count is set in one script. N = 1000 applies only under tools/dist_test.py (see Paper vs code).
• Cyclist exponent. The default beta for cyclists is 4 and the 8-frame model uses 1 §4.2 center_head_iou_1d.py. Changing beta changes the cyclist ranking, so cyclist numbers depend on it.
• Dropout and head sizes are not in the paper. The code uses dropout 0.3, head dimension 64 and MLP dimension 256 waymo_centerformer.py. An implementer who follows only the paper has to guess these.
• Ablation tables use 20% of the data. The appendix studies (layers, windows, offsets, position encoding) use 20% of the training set app. C. The full-data optimum may differ.
• Fade is part of the ablation gain. The +0.4 and +0.7 gains from Fade in tab. 4 come with a switch-off at an epoch the paper does not state.
• Memory. Multi-frame runs use 2 samples per GPU, against 4 for single-frame, and the paper splits 4- and 8-frame inputs to fit memory §4.2 waymo_centerformer_multiframe_2frames.py.
• Custom CUDA operators. The deformable attention and IoU-NMS kernels are compiled from source in the repository setup.py setup.py. They were not built or tested here.
• Two-stage comparison. The paper argues that the decoder is one-stage, so it avoids the second RoI stage app. C. This is a design claim, not a measured speed comparison in the paper.

Lineage

• Builds on: CenterPoint (yin2021center in the paper's bibliography) for the VoxelNet backbone, the centre heatmap and the training scheme; the paper states this in §3.2 and app. A. DETR (carion2020end) for the decoder and object queries, with the encoder removed and centre-based queries §1. Deformable DETR (zhu2020deformable) for the deformable attention layer §3.2.3. CIA-SSD (zheng2020ciassd) for the IoU rectification §3.4. CBAM (Woo_2018_ECCV) for the attention blocks §3.2.1. arXiv ids for these papers are not in the paper's bibliography.
• Compared against: PV-RCNN++, SST, AFDetV2, CenterPoint, LiDAR R-CNN, Part-A2, 3D-MAN, M3DETR, RSN, PPBA, StarNet, Voxel Transformer, CT3D, BoxeR-3D and Pyramid R-CNN, on the Waymo tables tab. 1 tab. 2 tab. 3. nuScenes is compared with CenterPoint only app. B.
• Superseded by: not known from content already in this wiki.